I am trying to populate input for my fzf sript from multiple parallel processes.

fzf = subprocess.Popen(
    [
        "fzf",
    ],
    stdin=subprocess.PIPE,
    text=True,
)

so I want to be able write to directly to this PIPE from other proccesses.

As multiprocessing does not support pickling IO objects I used file descriptor id. Like this:

python_process = multiprocessing.Process(
    target=generate_input,
    args=(fzf.stdin.fileno(),),
)

Inside my process I did write like this:

os.dup(write_fd) # I thought it could help to resolve error with fd.
os.write(write_fd, "Any string".encode("utf-8"))

This was leading to the OSError: [Errno 9] Bad file descriptor. I dont understand why I am gettings this, because all file descriptors are inherited from parent process. To ensure this, I tried to call this code inside my child process.

_fd_types = (
    ("REG", stat.S_ISREG),
    ("FIFO", stat.S_ISFIFO),
    ("DIR", stat.S_ISDIR),
    ("CHR", stat.S_ISCHR),
    ("BLK", stat.S_ISBLK),
    ("LNK", stat.S_ISLNK),
    ("SOCK", stat.S_ISSOCK),
)

for fd in range(100):
    try:
        s = os.fstat(fd)
    except:
        continue
    for fd_type, func in _fd_types:
        if func(s.st_mode):
            break
    else:
        fd_type = str(s.st_mode)
    print(f"fd: {fd}, type: {fd_type}")

This snippet printed that my file descriptor was available via os.fstat function.

I googled a lot a have not found something similar. Or at least some projects that utilize python in this way. I understand that I can approach this in other ways.
For example:

    child = subprocess.Popen(
        [
            "my_python_program --child",
        ],
        stdout=subprocess.PIPE,
    )
    fzf = subprocess.Popen(
        [
            "fzf",
        ],
        stdin=child.stdout,
    )

MacM1, Python3.12

But I really want to learn how to it in a more "basic" way. I will be very grateful for any info related, thank you.

I am trying to populate input for my fzf sript from multiple parallel processes.

fzf = subprocess.Popen(
    [
        "fzf",
    ],
    stdin=subprocess.PIPE,
    text=True,
)

so I want to be able write to directly to this PIPE from other proccesses.

As multiprocessing does not support pickling IO objects I used file descriptor id. Like this:

python_process = multiprocessing.Process(
    target=generate_input,
    args=(fzf.stdin.fileno(),),
)

Inside my process I did write like this:

os.dup(write_fd) # I thought it could help to resolve error with fd.
os.write(write_fd, "Any string".encode("utf-8"))

This was leading to the OSError: [Errno 9] Bad file descriptor. I dont understand why I am gettings this, because all file descriptors are inherited from parent process. To ensure this, I tried to call this code inside my child process.

_fd_types = (
    ("REG", stat.S_ISREG),
    ("FIFO", stat.S_ISFIFO),
    ("DIR", stat.S_ISDIR),
    ("CHR", stat.S_ISCHR),
    ("BLK", stat.S_ISBLK),
    ("LNK", stat.S_ISLNK),
    ("SOCK", stat.S_ISSOCK),
)

for fd in range(100):
    try:
        s = os.fstat(fd)
    except:
        continue
    for fd_type, func in _fd_types:
        if func(s.st_mode):
            break
    else:
        fd_type = str(s.st_mode)
    print(f"fd: {fd}, type: {fd_type}")

This snippet printed that my file descriptor was available via os.fstat function.

I googled a lot a have not found something similar. Or at least some projects that utilize python in this way. I understand that I can approach this in other ways.
For example:

    child = subprocess.Popen(
        [
            "my_python_program --child",
        ],
        stdout=subprocess.PIPE,
    )
    fzf = subprocess.Popen(
        [
            "fzf",
        ],
        stdin=child.stdout,
    )

MacM1, Python3.12

But I really want to learn how to it in a more "basic" way. I will be very grateful for any info related, thank you.

• python
• multiprocessing
• subprocess
• file-descriptor

• 1 The file descriptor table is process-local. You can pass a fd from a parent to a child when the child is started but without fancy tricks (involving UNIX domain sockets), not after that point. – Charles Duffy Commented 20 hours ago
• (Separately, Linux lets you get a handle on another process's descriptors from procfs, but as that's an OS-specific nonstandard extension, I'm filling it under "fancy tricks" too). – Charles Duffy Commented 20 hours ago
• Sorry, for missleading. I can not get this. "You can pass a fd from a parent to a child when the child is started but without fancy tricks (involving UNIX domain sockets), not after that point" So that I do on my snippets. As I know from docs. They all are inherited by default from parent process. docs.python.org/3/library/subprocess.html#subprocess.Popen Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors. – Vladislav Zikunov Commented 20 hours ago
• So I wonder why my code is having this exception "OSError: [Errno 9] Bad file descriptor". When trying to open (os.fdopen() ) or write to this fd. – Vladislav Zikunov Commented 20 hours ago
• 1 Look at procfs to see what was actually inherited. I suspect that multiprocessing is effectively using close_fds=True. – Charles Duffy Commented 20 hours ago

1 Answer 1

Still learning, but I found solution that satisfies me at least.

1. Create fifo descriptor in parent

if not os.path.exists(fifo_path):
        os.mkfifo(fifo_path)

2. Launch process that will write to this fifo

    python_process = multiprocessing.Process(
        target=child_func,
    )
    python_process.start()
    
   
    write_fd = open(fifo_path, "w")
    write_fd.close()

3. Open fifo and pass to subprocess

      fifo = os.open(fifo_path, os.O_RDONLY)
      fzf = subprocess.Popen(
           [
               "fzf",
           ],
           stdin=fifo,
           text=True,
       )