Bug #11600
closedindeterministic failures on socket operations when running out of file descriptors
Description
We've been getting indeterministic failures on socket operations when running out of file descriptors, instead of the correct Errno::EMFILE
being raised we've seen various failure modes, on both linux and osx:
- This is what we're seeing in production the most, masquerading as a dns failure
ruby test.rb
1017 tempfiles
freeing 4
test.rb:24:in getaddrinfo': getaddrinfo: Name or service not known (SocketError) from test.rb:24:in
block (3 levels) in '
ruby test.rb
1017 tempfiles
freeing 4
test.rb:24: [BUG] † - errno == 0
ruby 2.2.4p180 (2015-10-06 revision 52046) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0003 p:---- s:0009 e:000008 CFUNC :getaddrinfo
c:0002 p:0018 s:0004 e:000003 BLOCK test.rb:24 [FINISH]
c:0001 p:---- s:0002 e:000001 TOP [FINISH]
-- Ruby level backtrace information ----------------------------------------
test.rb:24:in block (3 levels) in <main>' test.rb:24:in
getaddrinfo'
ruby test.rb
1017 tempfiles
freeing 4
test.rb:24: [BUG] rb_sys_fail(getaddrinfo) - errno == 0
ruby 2.2.4p180 (2015-10-06 revision 52046) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0003 p:---- s:0009 e:000008 CFUNC :getaddrinfo
c:0002 p:0018 s:0004 e:000003 BLOCK test.rb:24 [FINISH]
c:0001 p:---- s:0002 e:000001 TOP [FINISH]
-- Ruby level backtrace information ----------------------------------------
test.rb:24:in block (3 levels) in <main>' test.rb:24:in
getaddrinfo'
Similar errors also happen doing other operations (eg TCPSocket.open
)
Test case attached
Files
Updated by tarmo_t (Tarmo Tänav) over 8 years ago
- File testcase.rb testcase.rb added
Managed to get the testcase slightly simpler with a guaranteed ruby crash.
Updated by kosaki (Motohiro KOSAKI) over 8 years ago
In my system, getaddrinfo() return -11(EAI_SYSTEM) and errno=0. That seems glibc's bug.
I guess this is related to https://sourceware.org/bugzilla/show_bug.cgi?id=15339.
Updated by erkkie (Erkki Eilonen) over 8 years ago
Motohiro KOSAKI wrote:
In my system, getaddrinfo() return -11(EAI_SYSTEM) and errno=0. That seems glibc's bug.
I guess this is related to https://sourceware.org/bugzilla/show_bug.cgi?id=15339.
Happens on OSX (10.10.5) as well (on 2.2.3 and earlier, presumably on latest 2_2 as well)
➜ ruby test.rb
1017 tempfiles
freeing 1
[BUG] Illegal instruction at 0x007fff94258e4e
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
-- Crash Report log information --------------------------------------------
See Crash Report log file under the one of following:
* ~/Library/Logs/CrashReporter
* /Library/Logs/CrashReporter
* ~/Library/Logs/DiagnosticReports
* /Library/Logs/DiagnosticReports
for more details.
-- Control frame information -----------------------------------------------
-- Machine register context ------------------------------------------------
rax: 0x0000000000000018 rbx: 0x00007fe91255a790 rcx: 0x00007fff9427adf9
rdx: 0xffffffffffffffff rdi: 0x0000000000000018 rsi: 0x0000000000000003
rbp: 0x00000001012fa550 rsp: 0x00000001012fa540 r8: 0x0000000101a01080
r9: 0x0000000101a00000 r10: 0x010000000000fff6 r11: 0x0000000000000203
r12: 0x00007fff785e6158 r13: 0x00007fff94258d1f r14: 0x00007fff785e6158
r15: 0x00000001012fa580 rip: 0x00007fff94258e4e rfl: 0x0000000000010246
-- C level backtrace information -------------------------------------------
0 ruby 0x0000000100e2ff45 rb_vm_bugreport + 149
1 ruby 0x0000000100cd7c82 rb_bug_context + 466
2 ruby 0x0000000100dac7f3 sigill + 83
3 libsystem_platform.dylib 0x00007fff86b1ef1a _sigtramp + 26
4 libdispatch.dylib 0x00007fff94258e4e _dispatch_kq_init + 303
5 libsystem_c.dylib 0x00007fff77d3e6b8 __global_locale + 0
-- Other runtime information -----------------------------------------------
* Loaded script: test.rb
* Loaded features:
0 enumerator.so
1 rational.so
2 complex.so
3 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/enc/encdb.bundle
4 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/enc/trans/transdb.bundle
5 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/unicode_normalize.rb
6 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/rbconfig.rb
7 thread.rb
8 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/thread.bundle
9 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/compatibility.rb
10 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/defaults.rb
11 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/deprecate.rb
12 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/errors.rb
13 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/version.rb
14 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/requirement.rb
15 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/platform.rb
16 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/basic_specification.rb
17 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/stub_specification.rb
18 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/util/stringio.rb
19 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/specification.rb
20 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/exceptions.rb
21 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb
22 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb
23 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb
24 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/rubygems.rb
25 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/socket.bundle
26 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/socket.rb
27 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/delegate.rb
28 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/x86_64-darwin14/etc.bundle
29 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/fileutils.rb
30 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/tmpdir.rb
31 /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/2.2.0/tempfile.rb
[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
Don't forget to include the above Crash Report log file.
For details: http://www.ruby-lang.org/bugreport.html
[1] 69774 abort ruby test.rb
Updated by nobu (Nobuyoshi Nakada) over 8 years ago
Erkki Eilonen wrote:
Happens on OSX (10.10.5) as well (on 2.2.3 and earlier, presumably on latest 2_2 as well)
It's a bug of a system library.
We can do nothing for it.
Updated by nobu (Nobuyoshi Nakada) over 8 years ago
- Description updated (diff)
Accurately, it is not a bug but an intentional crash.
Citing from libdispatch-442.1.4/src/source.c
:
int err = errno;
switch (err) {
case EMFILE:
DISPATCH_CLIENT_CRASH("kqueue() failure: "
"process is out of file descriptors");
break;
case ENFILE:
DISPATCH_CLIENT_CRASH("kqueue() failure: "
"system is out of file descriptors");
break;
case ENOMEM:
DISPATCH_CLIENT_CRASH("kqueue() failure: "
"kernel is out of memory");
break;
default:
(void)dispatch_assume_zero(err);
DISPATCH_CRASH("kqueue() failure");
break;
}
Updated by nobu (Nobuyoshi Nakada) over 8 years ago
- Status changed from Open to Closed
Applied in changeset r52190.
init.c: glibc bug
- ext/socket/init.c (rsock_raise_socket_error): get rid of a glibc
bug. [ruby-core:71100] [Bug #11600]