Bug #11600: indeterministic failures on socket operations when running out of file descriptors

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.2.4p180 (2015-10-06 revision 52046) [x86_64-linux]
• Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN

[erkkie (Erkki Eilonen)]

We've been getting indeterministic failures on socket operations when running out of file descriptors, instead of the correct Errno::EMFILE being raised we've seen various failure modes, on both linux and osx:

1. This is what we're seeing in production the most, masquerading as a dns failure

ruby test.rb
1017 tempfiles
freeing 4
test.rb:24:in getaddrinfo': getaddrinfo: Name or service not known (SocketError) from test.rb:24:in block (3 levels) in '

ruby test.rb 

1017 tempfiles
freeing 4
test.rb:24: [BUG] † - errno == 0
ruby 2.2.4p180 (2015-10-06 revision 52046) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0009 e:000008 CFUNC :getaddrinfo
c:0002 p:0018 s:0004 e:000003 BLOCK test.rb:24 [FINISH]
c:0001 p:---- s:0002 e:000001 TOP [FINISH]

-- Ruby level backtrace information ----------------------------------------
test.rb:24:in block (3 levels) in <main>' test.rb:24:in getaddrinfo'

ruby test.rb 

1017 tempfiles
freeing 4
test.rb:24: [BUG] rb_sys_fail(getaddrinfo) - errno == 0
ruby 2.2.4p180 (2015-10-06 revision 52046) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0009 e:000008 CFUNC :getaddrinfo
c:0002 p:0018 s:0004 e:000003 BLOCK test.rb:24 [FINISH]
c:0001 p:---- s:0002 e:000001 TOP [FINISH]

-- Ruby level backtrace information ----------------------------------------
test.rb:24:in block (3 levels) in <main>' test.rb:24:in getaddrinfo'

Similar errors also happen doing other operations (eg TCPSocket.open)

Test case attached

[tarmo_t (Tarmo Tänav)]

• File testcase.rb testcase.rb added

Managed to get the testcase slightly simpler with a guaranteed ruby crash.

[kosaki (Motohiro KOSAKI)]

In my system, getaddrinfo() return -11(EAI_SYSTEM) and errno=0. That seems glibc's bug.
I guess this is related to https://sourceware.org/bugzilla/show_bug.cgi?id=15339.

[erkkie (Erkki Eilonen)]

Motohiro KOSAKI wrote:

In my system, getaddrinfo() return -11(EAI_SYSTEM) and errno=0. That seems glibc's bug.
I guess this is related to https://sourceware.org/bugzilla/show_bug.cgi?id=15339.

Happens on OSX (10.10.5) as well (on 2.2.3 and earlier, presumably on latest 2_2 as well)

➜ ruby test.rb
1017 tempfiles
freeing 1
[BUG] Illegal instruction at 0x007fff94258e4e
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:
     * ~/Library/Logs/CrashReporter
     * /Library/Logs/CrashReporter
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.

-- Control frame information -----------------------------------------------


-- Machine register context ------------------------------------------------
 rax: 0x0000000000000018 rbx: 0x00007fe91255a790 rcx: 0x00007fff9427adf9
 rdx: 0xffffffffffffffff rdi: 0x0000000000000018 rsi: 0x0000000000000003
 rbp: 0x00000001012fa550 rsp: 0x00000001012fa540  r8: 0x0000000101a01080
  r9: 0x0000000101a00000 r10: 0x010000000000fff6 r11: 0x0000000000000203
 r12: 0x00007fff785e6158 r13: 0x00007fff94258d1f r14: 0x00007fff785e6158
 r15: 0x00000001012fa580 rip: 0x00007fff94258e4e rfl: 0x0000000000010246

-- C level backtrace information -------------------------------------------
0   ruby                                0x0000000100e2ff45 rb_vm_bugreport + 149
1   ruby                                0x0000000100cd7c82 rb_bug_context + 466
2   ruby                                0x0000000100dac7f3 sigill + 83
3   libsystem_platform.dylib            0x00007fff86b1ef1a _sigtramp + 26
4   libdispatch.dylib                   0x00007fff94258e4e _dispatch_kq_init + 303
5   libsystem_c.dylib                   0x00007fff77d3e6b8 __global_locale + 0

-- Other runtime information -----------------------------------------------

* Loaded script: test.rb

* Loaded features:

    0 enumerator.so
    1 rational.so
    2 complex.so
    3 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/enc/encdb.bundle
    4 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/enc/trans/transdb.bundle
    5 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/unicode_normalize.rb
    6 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/rbconfig.rb
    7 thread.rb
    8 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/thread.bundle
    9 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/compatibility.rb
   10 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/defaults.rb
   11 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/deprecate.rb
   12 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/errors.rb
   13 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/version.rb
   14 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/requirement.rb
   15 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/platform.rb
   16 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/basic_specification.rb
   17 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/stub_specification.rb
   18 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/util/stringio.rb
   19 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/specification.rb
   20 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/exceptions.rb
   21 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb
   22 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb
   23 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb
   24 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems.rb
   25 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/socket.bundle
   26 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/socket.rb
   27 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/delegate.rb
   28 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/etc.bundle
   29 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/fileutils.rb
   30 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/tmpdir.rb
   31 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/tempfile.rb

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
Don't forget to include the above Crash Report log file.
For details: http://www.ruby-lang.org/bugreport.html

[1]    69774 abort      ruby test.rb

[nobu (Nobuyoshi Nakada)]

Erkki Eilonen wrote:

Happens on OSX (10.10.5) as well (on 2.2.3 and earlier, presumably on latest 2_2 as well)

It's a bug of a system library.
We can do nothing for it.

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

Accurately, it is not a bug but an intentional crash.

Citing from libdispatch-442.1.4/src/source.c:

		int err = errno;
		switch (err) {
		case EMFILE:
			DISPATCH_CLIENT_CRASH("kqueue() failure: "
					"process is out of file descriptors");
			break;
		case ENFILE:
			DISPATCH_CLIENT_CRASH("kqueue() failure: "
					"system is out of file descriptors");
			break;
		case ENOMEM:
			DISPATCH_CLIENT_CRASH("kqueue() failure: "
					"kernel is out of memory");
			break;
		default:
			(void)dispatch_assume_zero(err);
			DISPATCH_CRASH("kqueue() failure");
			break;
		}

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset r52190.

---

init.c: glibc bug

• ext/socket/init.c (rsock_raise_socket_error): get rid of a glibc
  bug. [ruby-core:71100] [Bug #11600]