Backport #7291: split gives core dump on large file

• Status: Rejected
• Assignee: usa (Usaku NAKAMURA)

[niels_tolstrup (Niels Tolstrup)]

When applying split on a large file, we get a segmentation fault.

We have tried on ubuntu 12.04LTS with ruby 1.9.2p180 and ruby 1.9.3p194.

The bug can be reproduced with this code:

data = ">ENSG00000010072|ENST00000008440|1|1|231474002|231488267|SprT-like N-terminal domain [Source:HGNC Symbol;Acc:25356]
GACGGGCCGTCTCGAGAGCCGGCATCTCCTAGGAGCTAGTCCTGGTCCTCGGCTAGGCGG
CTTGGGGTCGCGGCGTAACTGGGGAGCCAGCCTGACGCCGGCGGACCCCGCCTGTGATCC
TGGCAACGATGGATGATGACTTGATGTTGGCACTGCGGCTTCAGGAGGAGTGGAACTTGC
AGGAGGCGGAGCGCGATCATGCCCAGGAGTCCCTGTCGCTAGTGGACGCGTCGTGGGAGT
TGGTGGACCCCACACCGGACTTGCAGGCACTGTTTGTTCAGTTTAACGACCAATTCTTCT
GGGGCCAGCTGGAGGCCGTCGAGGTGAAGTGGAGCGTGCGAATGACCCTGTGAGTTCCGA
GCCCCGCTGGGGAAAGAGGCGGGACTGGCAGCTTTCCTGCAGCCCCCGGCCCTGGTTTTC
TCTCCTTTCTCTAGTCCGACGGTCCCAGGGGGCGTTAAATGAGGGGAGTCTGGTTTTGGA
CCTGGCAATTCCTGCCTCGGCGTGTTTCTGTCTTCCTTACCTTTTCTCCCACTCGAACAA"

fh = File.open('test.txt','w')
(1..15000000).each {|i| fh.puts(data)}
fh.close

str = File.read("test.txt")

str.split(/>ENS/).each do |match|

if match.split(">ENS").length > 1
puts match
exit
end

end

We get this error:

segfault.rb:22: [BUG] Segmentation fault
ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]

-- control frame ----------
c:0007 p:---- s:0021 b:0021 l:000020 d:000020 CFUNC :split
c:0006 p:0013 s:0017 b:0017 l:0012d8 d:000016 BLOCK segfault.rb:22
c:0005 p:---- s:0014 b:0014 l:000013 d:000013 FINISH
c:0004 p:---- s:0012 b:0012 l:000011 d:000011 CFUNC :each
c:0003 p:0093 s:0009 b:0009 l:0012d8 d:0002c0 EVAL segfault.rb:20
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0012d8 d:0012d8 TOP

-- Ruby level backtrace information ----------------------------------------
segfault.rb:20:in <main>' segfault.rb:20:in each'
segfault.rb:22:in block in <main>' segfault.rb:22:in split'

-- C level backtrace information -------------------------------------------
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_vm_bugreport+0x61) [0x7ffa89323101]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x5f24e) [0x7ffa8920d24e]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_bug+0xa5) [0x7ffa8920e075]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x10b874) [0x7ffa892b9874]
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7ffa88e254c0]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_memsearch+0x68) [0x7ffa89295758]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x1298f5) [0x7ffa892d78f5]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x16e5a6) [0x7ffa8931c5a6]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x164978) [0x7ffa89312978]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x16a80b) [0x7ffa8931880b]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_yield+0x47) [0x7ffa8931dcf7]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_ary_each+0x46) [0x7ffa891def16]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x16e5a6) [0x7ffa8931c5a6]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x164978) [0x7ffa89312978]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x16a80b) [0x7ffa8931880b]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(rb_iseq_eval_main+0xb1) [0x7ffa8931e631]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(+0x65292) [0x7ffa89213292]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(ruby_exec_node+0x1d) [0x7ffa8921414d]
/usr/share/ruby-rvm/rubies/ruby-1.9.2-p180/lib/libruby.so.1.9(ruby_run_node+0x1e) [0x7ffa8921640e]
ruby(main+0x4b) [0x40082b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7ffa88e1076d]
ruby() [0x400859]

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted (core dumped)

best regards

Niels and Jesper

[mrkn (Kenta Murata)]

• Category set to core

In OS X Mountain Lion, I couldn't reproduce segmentation fault, but found to occur EINVAL on 1.9.3-p286 and r37494.

$ ruby-1.9.3-p286 -v test.rb
ruby 2.0.0dev (2012-11-06 trunk 37494) [x86_64-darwin12.2.0]
test.rb:18:in read': Invalid argument - test.txt (Errno::EINVAL) from test.rb:18:in '

$ ruby-trunk -v test.rb
ruby 2.0.0dev (2012-11-06 trunk 37494) [x86_64-darwin12.2.0]
test.rb:18:in read': Invalid argument - test.txt (Errno::EINVAL) from test.rb:18:in '

[sorah (Sorah Fukumori)]

"ruby -v" wrote on this ticket: ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
but in your crash log: ruby 1.9.2p180 (2011-02-18 revision 30909) [x86_64-linux]

which one is right?

[niels_tolstrup (Niels Tolstrup)]

The bug appears in both 1.9.3 and 1.9.2. I have attached a crash log for 1.9.3 below:

best regards

Niels

nt@vega:~/tmp$ time ruby segfault.rb
segfault.rb:22: [BUG] Segmentation fault
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0007 p:---- s:0021 b:0021 l:000020 d:000020 CFUNC :split
c:0006 p:0013 s:0017 b:0017 l:000028 d:000016 BLOCK segfault.rb:22
c:0005 p:---- s:0014 b:0014 l:000013 d:000013 FINISH
c:0004 p:---- s:0012 b:0012 l:000011 d:000011 CFUNC :each
c:0003 p:0093 s:0009 b:0009 l:000028 d:0002c0 EVAL segfault.rb:20
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:000028 d:000028 TOP

-- Ruby level backtrace information ----------------------------------------
segfault.rb:20:in <main>' segfault.rb:20:in each'
segfault.rb:22:in block in <main>' segfault.rb:22:in split'

-- C level backtrace information -------------------------------------------
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x17bff5) [0x7f9b8fd91ff5] vm_dump.c:796
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x5d13b) [0x7f9b8fc7313b] isoc99_vscanf.c:31
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_bug+0xb7) [0x7f9b8fc73fe7] vfprintf.c:1310
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x11062f) [0x7f9b8fd2662f] getnetgrent_r.c:297
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7f9b8f88f4c0] ../sysdeps/posix/killpg.c:38
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_memsearch+0x68) [0x7f9b8fd02468] ../string/bits/string2.h:1247
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x12f265) [0x7f9b8fd45265] dl-sym.c:158
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x175751) [0x7f9b8fd8b751] vm_insnhelper.c:404
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x16b419) [0x7f9b8fd81419] insns.def:1015
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x1718aa) [0x7f9b8fd878aa] vm.c:1220
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_yield+0x47) [0x7f9b8fd8ceb7] vm.c:654
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_ary_each+0x52) [0x7f9b8fc41e42] gconv_cache.c:460
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x175751) [0x7f9b8fd8b751] vm_insnhelper.c:404
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x16b419) [0x7f9b8fd81419] insns.def:1015
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x1718aa) [0x7f9b8fd878aa] vm.c:1220
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(rb_iseq_eval_main+0xae) [0x7f9b8fd8da5e] vm.c:1461
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(+0x62e62) [0x7f9b8fc78e62] vfprintf.c:1623
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(ruby_exec_node+0x1d) [0x7f9b8fc79c0d] vfprintf.c:1986
/usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9(ruby_run_node+0x1e) [0x7f9b8fc7bcae] vfscanf.c:2450
ruby() [0x4007fb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f9b8f87a76d] libc-start.c:226
ruby() [0x400829]

-- Other runtime information -----------------------------------------------

• Loaded script: segfault.rb

• Loaded features:

  0 enumerator.so
  1 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
  2 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
  3 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
  4 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb
  5 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
  6 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
  7 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb
  8 /usr/share/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems.rb

• Process memory map:

00400000-00401000 r-xp 00000000 08:02 229377 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/bin/ruby
00600000-00601000 r--p 00000000 08:02 229377 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/bin/ruby
00601000-00602000 rw-p 00001000 08:02 229377 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/bin/ruby
018a6000-9d45d000 rw-p 00000000 00:00 0 [heap]
7f96e945d000-7f993bd44000 rw-p 00000000 00:00 0
7f9b8bdfa000-7f9b8df90000 rw-p 00000000 00:00 0
7f9b8e377000-7f9b8e38c000 r-xp 00000000 08:01 1466741 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9b8e38c000-7f9b8e58b000 ---p 00015000 08:01 1466741 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9b8e58b000-7f9b8e58c000 r--p 00014000 08:01 1466741 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9b8e58c000-7f9b8e58d000 rw-p 00015000 08:01 1466741 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9b8e58d000-7f9b8e62d000 rw-p 00000000 00:00 0
7f9b8e62d000-7f9b8e62f000 r-xp 00000000 08:02 352275 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f9b8e62f000-7f9b8e82f000 ---p 00002000 08:02 352275 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f9b8e82f000-7f9b8e830000 r--p 00002000 08:02 352275 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f9b8e830000-7f9b8e831000 rw-p 00003000 08:02 352275 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7f9b8e831000-7f9b8e833000 r-xp 00000000 08:02 344076 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f9b8e833000-7f9b8ea32000 ---p 00002000 08:02 344076 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f9b8ea32000-7f9b8ea33000 r--p 00001000 08:02 344076 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f9b8ea33000-7f9b8ea34000 rw-p 00002000 08:02 344076 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7f9b8ea34000-7f9b8ecfd000 r--p 00000000 08:01 1461034 /usr/lib/locale/locale-archive
7f9b8ecfd000-7f9b8edf6000 r-xp 00000000 08:01 1470638 /lib/x86_64-linux-gnu/libm-2.15.so
7f9b8edf6000-7f9b8eff5000 ---p 000f9000 08:01 1470638 /lib/x86_64-linux-gnu/libm-2.15.so
7f9b8eff5000-7f9b8eff6000 r--p 000f8000 08:01 1470638 /lib/x86_64-linux-gnu/libm-2.15.so
7f9b8eff6000-7f9b8eff7000 rw-p 000f9000 08:01 1470638 /lib/x86_64-linux-gnu/libm-2.15.so
7f9b8eff7000-7f9b8f000000 r-xp 00000000 08:01 1470650 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f9b8f000000-7f9b8f200000 ---p 00009000 08:01 1470650 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f9b8f200000-7f9b8f201000 r--p 00009000 08:01 1470650 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f9b8f201000-7f9b8f202000 rw-p 0000a000 08:01 1470650 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f9b8f202000-7f9b8f230000 rw-p 00000000 00:00 0
7f9b8f230000-7f9b8f232000 r-xp 00000000 08:01 1470655 /lib/x86_64-linux-gnu/libdl-2.15.so
7f9b8f232000-7f9b8f432000 ---p 00002000 08:01 1470655 /lib/x86_64-linux-gnu/libdl-2.15.so
7f9b8f432000-7f9b8f433000 r--p 00002000 08:01 1470655 /lib/x86_64-linux-gnu/libdl-2.15.so
7f9b8f433000-7f9b8f434000 rw-p 00003000 08:01 1470655 /lib/x86_64-linux-gnu/libdl-2.15.so
7f9b8f434000-7f9b8f43b000 r-xp 00000000 08:01 1470645 /lib/x86_64-linux-gnu/librt-2.15.so
7f9b8f43b000-7f9b8f63a000 ---p 00007000 08:01 1470645 /lib/x86_64-linux-gnu/librt-2.15.so
7f9b8f63a000-7f9b8f63b000 r--p 00006000 08:01 1470645 /lib/x86_64-linux-gnu/librt-2.15.so
7f9b8f63b000-7f9b8f63c000 rw-p 00007000 08:01 1470645 /lib/x86_64-linux-gnu/librt-2.15.so
7f9b8f63c000-7f9b8f654000 r-xp 00000000 08:01 1470659 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9b8f654000-7f9b8f853000 ---p 00018000 08:01 1470659 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9b8f853000-7f9b8f854000 r--p 00017000 08:01 1470659 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9b8f854000-7f9b8f855000 rw-p 00018000 08:01 1470659 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9b8f855000-7f9b8f859000 rw-p 00000000 00:00 0
7f9b8f859000-7f9b8fa0c000 r-xp 00000000 08:01 1470639 /lib/x86_64-linux-gnu/libc-2.15.so
7f9b8fa0c000-7f9b8fc0b000 ---p 001b3000 08:01 1470639 /lib/x86_64-linux-gnu/libc-2.15.so
7f9b8fc0b000-7f9b8fc0f000 r--p 001b2000 08:01 1470639 /lib/x86_64-linux-gnu/libc-2.15.so
7f9b8fc0f000-7f9b8fc11000 rw-p 001b6000 08:01 1470639 /lib/x86_64-linux-gnu/libc-2.15.so
7f9b8fc11000-7f9b8fc16000 rw-p 00000000 00:00 0
7f9b8fc16000-7f9b8fe2e000 r-xp 00000000 08:02 2056214 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7f9b8fe2e000-7f9b9002d000 ---p 00218000 08:02 2056214 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7f9b9002d000-7f9b90032000 r--p 00217000 08:02 2056214 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7f9b90032000-7f9b90036000 rw-p 0021c000 08:02 2056214 /var/cache/ruby-rvm/rubies/ruby-1.9.3-p194/lib/libruby.so.1.9.1
7f9b90036000-7f9b90052000 rw-p 00000000 00:00 0
7f9b90052000-7f9b90074000 r-xp 00000000 08:01 1470641 /lib/x86_64-linux-gnu/ld-2.15.so
7f9b90144000-7f9b9024a000 rw-p 00000000 00:00 0
7f9b9026d000-7f9b9026e000 rw-p 00000000 00:00 0
7f9b9026e000-7f9b9026f000 ---p 00000000 00:00 0
7f9b9026f000-7f9b90274000 rw-p 00000000 00:00 0
7f9b90274000-7f9b90275000 r--p 00022000 08:01 1470641 /lib/x86_64-linux-gnu/ld-2.15.so
7f9b90275000-7f9b90277000 rw-p 00023000 08:01 1470641 /lib/x86_64-linux-gnu/ld-2.15.so
7fff47e1b000-7fff47e30000 rw-p 00000000 00:00 0 [stack]
7fff47fc9000-7fff47fca000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted (core dumped)

real 0m56.451s
user 0m21.230s
sys 0m32.870s

[mame (Yusuke Endoh)]

• Tracker changed from Bug to Backport
• Project changed from Ruby master to Backport193
• Category deleted (core)
• Status changed from Open to Assigned
• Assignee set to usa (Usaku NAKAMURA)

I succeeded to reproduce this issue in 1.9.3, but failed in 2.0.0.

As I look at the trace, the segfault occurred in rb_memsearch which was rewritten in 2.0.0, AFAIK.
So I think that this issue will occur in 1.9 (Sorry in advance if I am wrong). Moved to 1.9.3 tracker.

--
Yusuke Endoh [e-mail]

[usa (Usaku NAKAMURA)]

I don't have any reproducible environment, so I can't debug it.

If mame-san's expectation is right, current rb_memsearch_ss() implementaion
may still have this bug, because the difference between 1.9.3 and 2.0.0 is
only 2 point.
(1) r37793 (performance implovement)
(2) using memmem(3), if it exists
So, if (1) does not fix this, all platforms without memmem(3) should have same
bug on 2.0.0.

Mame-san, can you help me to debug?

[usa (Usaku NAKAMURA)]

• Status changed from Assigned to Rejected

1.9.3 is out of date