Bug #22456
Updated by nobu (Nobuyoshi Nakada) 1 day ago
## Description If a thread is blocked writing to an IO **inside a trap handler**, and another thread closes that IO, `IO#close` never returns. The writer gets `IOError: stream closed in another thread` and moves on. The closing thread stays in `IO#close` forever (status `sleep`). This is already fixed on master by ba68343d3a "Allow wakeup mutex to be used in trap context" ([GitHub (GitHub PR #13684], #13684, released in 4.0.0). It has no Redmine ticket, and it was never backported. Both 3.4.11 and 3.3.12 still have the old code: `thread_io_wake_pending_closer` locks `busy->wakeup_mutex` with `rb_mutex_lock`, and `rb_notify_fd_close` creates that mutex without `rb_mutex_allow_trap`. The waiting on close came to 3.3 with the backport of #20723, so the hang is a regression in 3.3 and 3.4. ## Reproduction No gems needed: ```ruby # IO#close hangs forever when the thread writing to that IO is running a trap handler. # Usage: ruby repro.rb [trap|thread] mode = ARGV[0] || "trap" r, w = IO.pipe begin loop { w.write_nonblock("x" * 4096) } # fill the pipe so the next write blocks rescue IO::WaitWritable end closer = nil write = lambda do closer = Thread.new { sleep 0.3; w.close; :closed } # close while the other thread is blocked writing begin w.syswrite("y") puts "write: returned" rescue Exception => e puts "write: #{e.class}: #{e.message}" end end if mode == "trap" Signal.trap("USR1") { write.call } Process.kill("USR1", Process.pid) sleep 0.1 until closer else write.call end result = closer.join(3) puts "#{RUBY_DESCRIPTION}, #{mode}: " + (result ? "IO#close returned" : "IO#close HANGS (#{closer.status}) at #{closer.backtrace&.first}") ``` Output (Debian 13, x86_64): ``` $ ruby repro.rb trap write: IOError: stream closed in another thread ruby 3.3.10 (2025-10-23 revision 343ea05002) [x86_64-linux], trap: IO#close HANGS (sleep) at repro.rb:13:in `close' $ ruby repro.rb thread write: IOError: stream closed in another thread ruby 3.3.10 (2025-10-23 revision 343ea05002) [x86_64-linux], thread: IO#close returned $ ruby repro.rb trap write: IOError: stream closed in another thread ruby 3.4.10 (2026-06-30 revision 2b0b7728dc) +PRISM [x86_64-linux], trap: IO#close HANGS (sleep) at repro.rb:13:in 'IO#close' $ ruby repro.rb thread write: IOError: stream closed in another thread ruby 3.4.10 (2026-06-30 revision 2b0b7728dc) +PRISM [x86_64-linux], thread: IO#close returned ``` Same with the latest releases of each branch (`trap` mode): ``` ruby 3.3.12 (2026-07-16 revision 0581089df9) [x86_64-linux], trap: IO#close HANGS (sleep) at repro.rb:13:in `close' ruby 3.4.11 (2026-09-23 revision 592f1ffdb3) +PRISM [x86_64-linux], trap: IO#close HANGS (sleep) at repro.rb:13:in 'IO#close' ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [x86_64-linux], trap: IO#close returned ``` `IO#write` / `<<` behave the same as `syswrite`. ## Why it hangs (3.4) `rb_notify_fd_close` moves the writer's `waiting_fd` to `busy->pending_fd_users`, and `rb_notify_fd_close_wait` sleeps on `busy->wakeup_mutex` until that list is empty. The writer removes itself in `thread_io_wake_pending_closer`, which first calls `rb_mutex_lock(wfd->busy->wakeup_mutex)`. In trap context, `do_mutex_lock` refuses the lock ("can't be called from trap context"), so `rb_mutex_lock` raises before `ccan_list_del()` and `rb_thread_wakeup()`. The entry is never removed, and the closing thread is never woken. Because the `waiting_fd` lives on the writer's C stack, the list is also left pointing at a dead frame. The exception that surfaces is the pending `IOError`, not a `ThreadError`. With a 3.4.10 build instrumented around that call, `do_mutex_lock` takes the trap branch, and `rb_mutex_lock` returns with `TAG_RAISE` and `IOError` in `errinfo`. `TracePoint(:raise)` sees no `ThreadError` at all. ## Real-world impact This is the hang that ba68343d3a was written for: Puma stopping on SIGTERM. In single mode, Puma runs the whole graceful stop inside its SIGTERM trap. `Server#stop` writes to its notification pipe (`notify_safely`) while the server thread closes that pipe in `handle_servers`. The server thread hangs in `IO#close` (puma/server.rb:432), the main thread hangs in `join` (server.rb:668) inside the trap, and systemd ends up sending SIGKILL. We hit it in production with Puma 7.2.1 and Solid Queue running inside Puma, on Ruby 3.3.10. Five of seven stops hung. A thread dump taken during the hang showed exactly those two frames. The puma/puma#3940 change (`syswrite` instead of `<<`) does not avoid it, because the writer is still in a trap. As a workaround, we changed Puma's TERM/INT traps to `Thread.new { stop }`, so that the stop runs outside trap context. With that change, five of five stops were clean. ## Suggested backport (3.4 / 3.3) The fix on master is in `rb_thread_io_close_interrupt`, which does not exist on these branches: master moved to per-IO blocking operations in 425fa0aeb5 (GH-13127) before the fix, so ba68343d3a does not cherry-pick. The equivalent change is in `rb_notify_fd_close` (thread.c). It mirrors what `io.c` already does for `fptr->write_lock`. Patch against 3.4.10 (attached as `allow-trap-3.4.patch`): ```diff if (has_any) { wakeup_mutex = rb_mutex_new(); RBASIC_CLEAR_CLASS(wakeup_mutex); /* hide from ObjectSpace */ + rb_mutex_allow_trap(wakeup_mutex, 1); } ``` Tested on Debian 13, x86_64, with the script above (`trap` mode, 20 runs each): | Ruby | Result | |---|---| | 3.4.10, unpatched | IO#close hangs 20/20 | | 3.4.10 + this patch (built from the release tarball) | IO#close returns 20/20 | | 4.0.7 (already contains ba68343d3a) | IO#close returns 20/20 | `thread` mode returns in all three. With the patch, `make test-all TESTS="ruby/test_io.rb ruby/test_thread.rb ruby/test_signal.rb ruby/test_thread_queue.rb"` passes: 406 tests, 8899 assertions, 0 failures, 0 errors, 1 skip. The same hunk should apply to 3.3, where `rb_notify_fd_close` is identical. Only the line numbers change. I have not built it on 3.3. [GitHub PR #13684]: https://github.com/ruby/ruby/pull/13684