Bug #10583
closedProcess.spawn stalls forever opening named pipes (fifo)
Description
Ruby's implementation of Process.spawn seems to attempt to send error/success information from the spawned process back to the parent, and the parent won't continue until it gets this information. However, a named pipe (fifo) is mapped to the spawned process' IO, it will stall opening the IO stream, and never be able to send the error/success status back to the parent.
While stalled, the parent process is sitting here: https://github.com/ruby/ruby/blob/ruby_2_1/process.c#L3403 This prevents spawning multiple commands that communicate through a named pipe.
Example testcase is attached.
Files
Updated by nobu (Nobuyoshi Nakada) over 9 years ago
- Status changed from Open to Rejected
open(2)
either blocks until the file gets ready or fails with ENXIO
.
Open in a thread.
Updated by ezran (Justin Greer) over 9 years ago
While open(2) blocks, the spawn should NOT block. Only the spawned process itself should block waiting on the open() - the parent process should continue, and should not require threads to make this happen.
Using the system calls directly this would not happen; it's an artifact of the way Ruby is trying to send status information between the spawned process and the parent process. Please reopen this ticket so the bug can be fixed.
Updated by nobu (Nobuyoshi Nakada) over 9 years ago
- Status changed from Rejected to Feedback
The current behavior is necessary to handle exec failures properly.
Also, as it uses vfork(2)
for huge memory apps if available, and it doesn't allow the parent and child processes to run in parallel.
I think this issue is a rare case, and not worth to drop the feature by default.
So what about a new async
option?
Updated by akr (Akira Tanaka) over 9 years ago
I feel this is rare usage too.
nobu's proposal (async option) is a possible way to use spawn method in this usage.
The option name should be descriptive and longer, such as ignore_error, though.
Updated by nobu (Nobuyoshi Nakada) over 9 years ago
OK, https://github.com/nobu/ruby/compare/spawn-ignore_error-option
But I'm afraid that ignore_error
's concern and async
's (or nonblock
's, etc) might be different.
Updated by ezran (Justin Greer) over 9 years ago
While the reduced testcase I attached previously sounds like a rare usage, it's actually pretty common any time you're tying together multiple programs and one of them only knows how to work with a file/fifo instead of stdin/stdout. Our actual usage is more along the lines of this pattern:
fifo_path = '/path/to/data_fifo'
`mkfifo '#{fifo_path}'`
decoder_pid = Process.spawn("decode_data", "/path/to/input_file", fifo_path, :close_others => true)
filter_pid = Process.spawn("filter_data", :STDIN => fifo_path, :STDOUT => "/path/to/filtered_file", :close_others => true)
result = Process.wait2(decoder_pid)
# Handle result...
result = Process.waitpid(filter_pid)
# Handle result...
The above seems to be expected/common usage when you want to be able to monitor process status better than if it's being run by a shell. (Possibly also reading its stderr as it runs.)
I'm not sure that ignore_error
really conveys the usage correctly here. Generally speaking, I would expect that the default behavior of Process.spawn
matches the description of posix_spawn
which it's based on. Having a version that checks for specific issues with exec seems like it should be an alternate form, maybe Process.safe_spawn
or Process.spawn(..., :synchronous => true)
.
Either that, or it should use a non-blocking open when it tries to open files, so that the stall doesn't happen. (Then it would need to switch back to non-blocking, so it doesn't change the standard behavior for the exec'ed process.)
Updated by nobu (Nobuyoshi Nakada) over 9 years ago
Justin Greer wrote:
While the reduced testcase I attached previously sounds like a rare usage, it's actually pretty common any time you're tying together multiple programs and one of them only knows how to work with a file/fifo instead of stdin/stdout. Our actual usage is more along the lines of this pattern:
I have never used a fifo for such purpose, rather would use a normal pipe directly or open3
.
Either that, or it should use a non-blocking open when it tries to open files, so that the stall doesn't happen.
And just fails with ENXIO
.
Updated by akr (Akira Tanaka) about 9 years ago
- Status changed from Feedback to Rejected
After thinking while, I decided to reject this issue.
Invoke spawn() in a thread for this usage: Thread.new { spawn(command, :in => fifo_path) }
Accepting this issue needs spawn() use fork() system call instead of vfork() system call.
So spawn() will be slower.
fork() is slower on bigger parent process.
So Thread.new { spawn_using_vfork() } will be faster than spawn_using_fork() for big parent process.
I'm not sure the threshold, though.
Also, opening a named pipe should not block other threads.
This problem is fixed at the latest trunk.
But the fix needs opening files in the parent process.
So it contradict this issue.