Closing a TCPServer while another thread is blocked in accept can discard a just-accepted connection's file descriptor, leaving the peer connected to nothing
Bug #22286:Closing a TCPServer while another thread is blocked in accept can discard a just-accepted connection's file descriptor, leaving the peer connected to nothing
Closing a TCPServer while another thread is blocked in accept can discard a just-accepted connection's file descriptor, leaving the peer connected to nothing.
The script to reproduce the issue in each iteration binds a listener, starts a thread blocked in accept, connects a client, then tears the listener down in one of three orders. It then closes whatever the accept thread actually received and checks whether the client sees the peer go away
within half a second. After a GC it reports how many descriptors the process still holds beyond its starting count. Results on Ruby 4.0.6, Linux 6.18.
Every stuck client is one the accept thread never handed back, and the leaked descriptor count matches. So accept(2) returned a socket, but Ruby raised the "closed in another thread" interrupt in that thread before wrapping the descriptor, and the descriptor was dropped. Joining the accept thread before calling close avoids the issue.
Run the script with:
# The number of issue occurrences will be reported in hang/not-accepted/fd-delta value.
ruby accept_race.rb close # issue visible
ruby accept_race.rb shutdown # confirms issue coming from close-interrupt
ruby accept_race.rb shutdown-join-close # workaround to fix the issue
I can confirm this on macOS too, and it isn't new. I ran into it with a small local TCP service that closes its listener from another thread during shutdown. Every so often a client would just sit there waiting, and it turned out to be exactly what's described here.
Thread#kill on the accepting thread loses the descriptor the same way, not just IO#close.
IO.select followed by accept_nonblock does not leak, which helps narrow down where the descriptor gets dropped.
My repro script is at the bottom and runs on both 2.6 and 4.0. Each iteration binds a listener on an ephemeral port and starts one accept thread that closes whatever it accepts. Four client threads connect in a loop, and after a random 0-5 ms delay the listener gets torn down in one of three ways. At the end the script runs GC, then uses lsof to count server-side descriptors that are still open for any of the listener ports. It also counts open BasicSocket objects, to show that nothing on the Ruby side still owns them.
(*) On 2.6.10, kill combined with nonblock occasionally left a single descriptor behind (1/300 in 2 of 4 runs). I haven't tracked that one down and it may be something separate. I never saw it on 4.0.7.
On the server side the leaked descriptors sit in CLOSE_WAIT because the client eventually closed its end. They survive GC and no Ruby object refers to them, so nothing will ever close them. A client without a read timeout just hangs.
If the "stream closed in another thread" interrupt (or a kill) is pending at that point, it raises. The tag jump unwinds past rsock_s_accept, and val, which by now is a perfectly good file descriptor, is simply dropped. Nothing closes it.
I haven't built master. Reading it, though, rsock_s_accept is unchanged there, and rb_thread_io_blocking_call still runs RUBY_VM_CHECK_INTS_BLOCKING(ec) after func returns, so I'd expect the same behavior.
I'm not sure what the cleanest fix is. Since accept returns a resource rather than just data, maybe the socket extension could handle it locally: notice that accept succeeded and close (or wrap) the fd before letting the pending interrupt propagate. I don't know the interrupt machinery well enough to say whether a general fix in rb_thread_io_blocking_call would be better.
Workarounds, for anyone who lands here:
Wait with IO.select([server]), then call server.accept_nonblock(exception: false), treating :wait_readable as "try again". Closing the server from another thread then just makes the accept thread exit with IOError, and no descriptor is lost.
Or, as in the original report, make sure the accept thread has stopped before you close the listener.
Script (usage is in the header comment; it needs lsof):
# ruby accept_leak.rb MODE [ITERS] [nonblock]# MODE: close | kill | close_then_kill# pass "nonblock" as the third arg to use IO.select + accept_nonblockrequire'socket'mode=ARGV[0]||'close'iters=Integer(ARGV[1]||'300')nonblock=ARGV[2]=='nonblock'ports=[]iters.timesdosrv=TCPServer.new('127.0.0.1',0)port=srv.addr[1]ports<<portacceptor=Thread.newdobeginloopdoifnonblockIO.select([srv])c=srv.accept_nonblock(exception: false)nextifc==:wait_readableelsec=srv.acceptendc.closeendrescueIOError,Errno::EBADFnilendendrunning=trueclients=Array.new(4)doThread.newdowhilerunningbeginSocket.tcp('127.0.0.1',port,connect_timeout: 0.2).closerescueStandardErrornilendendendendsleep(rand*0.005)casemodewhen'close'srv.closewhen'kill'acceptor.killwhen'close_then_kill'srv.closeacceptor.join(0.1)acceptor.killendacceptor.join(1)running=falseclients.each(&:join)srv.closeunlesssrv.closed?endGC.startsleep0.2# Count server-side TCP descriptors (local port is one of our listener ports)# that are still open in this process.lsof=IO.popen(['lsof','-nP','-a','-p',Process.pid.to_s,'-iTCP'],&:read)leaked=lsof.lines.selectdo|line|ports.any?{|p|line.include?("127.0.0.1:#{p}->")}endopen_objects=ObjectSpace.each_object(BasicSocket).count{|s|!s.closed?}putsformat('%-8s %-16s %-9s leaked=%d/%d open socket objects=%d',RUBY_VERSION,mode,nonblock?'nonblock':'blocking',leaked.size,iters,open_objects)