Bug #8543
closednew rb_iseq_load crash
Added by alvoskov (Alexey Voskov) over 11 years ago. Updated almost 9 years ago.
Description
I noticed an unusual behaviour of undocumented rb_iseq_load function.
Its work differs in different Ruby versions. I'm trying to protect some Ruby
source code by its conversion to YARV p-code and using the next strategy:
-
Convert code to array
data = RubyVM::InstructionSequence.compile_file('hello.rb').to_a
-
Pass a compiled source to the rb_iseq_load function and evaluate it
iseq = iseq_load.(data) iseq.eval
Sample programs are supplied in the attachments.
"hello.rb"
puts "tralivali"
def funct(a,b)
a**b
end
3.times { |i|
puts "Hello, world#{funct(2,i)}!"
}
The differences
Ruby 1.9.3 (ruby 1.9.3p194 (2012-04-20) [i386-mingw32])
Correct work. Output:
tralivali
Hello, world1!
Hello, world2!
Hello, world4!
Ruby 2.0.0 (ruby 2.0.0p193 (2013-05-14) [x64-mingw32])
Incorrect work (omits the code inside code blocks). Output
tralivali
Attempts of loading bigger programs by means of rb_iseq_load in Ruby 2.0.0 usually ends with a segmentation fault.
Such behaviour also can be reproduced by means of iseq Ruby extension ("for iseq freaks")
https://github.com/wanabe/iseq
P.S. I understand that it is an undocumented feature.
Files
hello.rb (102 Bytes) hello.rb | Codes for conversion to YARV PCODE | alvoskov (Alexey Voskov), 06/19/2013 09:39 PM | |
rb_pack.rb (931 Bytes) rb_pack.rb | "Compiler" | alvoskov (Alexey Voskov), 06/19/2013 09:39 PM | |
iseq-load-test3.rb (210 Bytes) iseq-load-test3.rb | Program which calls RubyVM::InstructionSequence.load | spatulasnout (B Kelly), 04/16/2014 10:03 AM | |
iseq-load-test3-file.rb (369 Bytes) iseq-load-test3-file.rb | Program fragment which triggers SEGV when loaded by rb_iseq_load | spatulasnout (B Kelly), 04/16/2014 10:03 AM | |
please-fix-rb_iseq_load-thank-you.pdf (444 KB) please-fix-rb_iseq_load-thank-you.pdf | PDF slide describing our use case for rb_iseq_load, per: [ruby-core:63332] | spatulasnout (B Kelly), 06/26/2014 11:36 PM | |
iseq-load-test5.rb (4.7 KB) iseq-load-test5.rb | Small ruby program reproducing iseq_load crash | spatulasnout (B Kelly), 09/09/2015 11:25 PM |
Updated by nagachika (Tomoyuki Chikanaga) over 11 years ago
- Tracker changed from Misc to Bug
Updated by nagachika (Tomoyuki Chikanaga) over 11 years ago
- Priority changed from Normal to 3
- Backport set to 1.9.3: DONTNEED, 2.0.0: REQUIRED
- ruby -v set to ruby 2.0.0p234 (2013-06-19 revision 41434) [x86_64-darwin11.4.2]
Updated by hsbt (Hiroshi SHIBATA) almost 11 years ago
- Target version changed from 2.1.0 to 2.2.0
Updated by spatulasnout (B Kelly) over 10 years ago
Hi,
Just a data point:
With ruby 2.2.0dev (2014-04-16 trunk 45576) [i386-mswin32_100],
the rb_iseq_load example in https://bugs.ruby-lang.org/issues/8543
is producing the correct output.
(I will attempt to test with more complicated programs soon.)
Regards,
Bill
Updated by spatulasnout (B Kelly) over 10 years ago
- File iseq-load-test3.rb iseq-load-test3.rb added
- File iseq-load-test3-file.rb iseq-load-test3-file.rb added
Hello again,
The attached files comprise a small example which appears to consistently
reproduce a segmentation fault with rb_iseq_load.
Interpreter is: ruby 2.2.0dev (2014-04-16 trunk 45597) [i386-mswin32_100]
The crash appears related to the case statement.
Call stack:
The st_table pointer at st_foreach_check is bad:
msvcr100-ruby220.dll!st_foreach_check(st_table * table=0x00000008, int (void)* func=0x0f3f3580, unsigned long arg=4512588, unsigned long never=6) Line 891 + 0x3 bytes C
msvcr100-ruby220.dll!hash_foreach_call(unsigned long arg=4512588) Line 258 + 0x17 bytes C
msvcr100-ruby220.dll!rb_ensure(unsigned long (void)* b_proc=0x0f3f3530, unsigned long data1=4512588, unsigned long (void)* e_proc=0x0f3f3480, unsigned long data2=10318080) Line 888 + 0x7 bytes C
msvcr100-ruby220.dll!rb_hash_foreach(unsigned long hash=10318080, int (void)* func=0x0f45ac20, unsigned long farg=4512696) Line 275 + 0x17 bytes C
> msvcr100-ruby220.dll!iseq_set_sequence(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58) Line 1507 + 0x12 bytes C
msvcr100-ruby220.dll!iseq_setup(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58) Line 1026 + 0xd bytes C
msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58, unsigned long body=10318900, st_table * labels_table=0x00d27d10) Line 5855 + 0xd bytes C
msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d279b0, unsigned long locals=10320520, unsigned long args=5, unsigned long exception=10320460, unsigned long body=10318900) Line 5932 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4) Line 561 + 0x19 bytes C
msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4) Line 582 + 0x17 bytes C
msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27500, unsigned long op=10320540) Line 5701 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27500, iseq_link_anchor * anchor=0x0044dfd4, unsigned long body=10318680, st_table * labels_table=0x00d27860) Line 5816 + 0x13 bytes C
msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27500, unsigned long locals=10320900, unsigned long args=3, unsigned long exception=10320840, unsigned long body=10318680) Line 5932 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4) Line 561 + 0x19 bytes C
msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4) Line 582 + 0x17 bytes C
msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27110, unsigned long op=10320920) Line 5701 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27110, iseq_link_anchor * anchor=0x0044e250, unsigned long body=10318560, st_table * labels_table=0x00d27470) Line 5783 + 0xd bytes C
msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27110, unsigned long locals=10321160, unsigned long args=1, unsigned long exception=10321100, unsigned long body=10318560) Line 5932 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4) Line 561 + 0x19 bytes C
msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4) Line 582 + 0x17 bytes C
msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d25c08, unsigned long op=10321180) Line 5701 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d25c08, iseq_link_anchor * anchor=0x0044e4cc, unsigned long body=10318500, st_table * labels_table=0x00d25f68) Line 5783 + 0xd bytes C
msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d25c08, unsigned long locals=10339520, unsigned long args=1, unsigned long exception=10339460, unsigned long body=10318500) Line 5932 + 0x15 bytes C
msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10339540, unsigned long parent=0, unsigned long opt=4) Line 561 + 0x19 bytes C
msvcr100-ruby220.dll!iseq_s_load(int argc=1, unsigned long * argv=0x0057005c, unsigned long self=10542580) Line 576 + 0x13 bytes C
msvcr100-ruby220.dll!call_cfunc_m1(unsigned long (void)* func=0x0f443b50, unsigned long recv=10542580, int argc=1, const unsigned long * argv=0x0057005c) Line 1330 + 0xf bytes C
msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0) Line 1502 + 0x20 bytes C
msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0) Line 1592 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0) Line 1786 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0) Line 1941 + 0x11 bytes C
msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0) Line 1028 + 0x1a bytes C
msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588) Line 1328 + 0xd bytes C
msvcr100-ruby220.dll!invoke_block_from_c(rb_thread_struct * th=0x00c15588, const rb_block_struct * block=0x005effe0, unsigned long self=10187260, int argc=1, const unsigned long * argv=0x0044f0ec, const rb_block_struct * blockptr=0x00000000, const RNode * cref=0x00000000, unsigned long defined_class=4, int splattable=1) Line 752 + 0x9 bytes C
msvcr100-ruby220.dll!vm_yield(rb_thread_struct * th=0x00c15588, int argc=1, const unsigned long * argv=0x0044f0ec) Line 784 + 0x28 bytes C
msvcr100-ruby220.dll!rb_yield_0(int argc=1, const unsigned long * argv=0x0044f0ec) Line 936 + 0x13 bytes C
msvcr100-ruby220.dll!rb_yield(unsigned long val=10340180) Line 946 + 0xb bytes C
msvcr100-ruby220.dll!rb_ary_each(unsigned long array=10342440) Line 1806 + 0x2f bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x0f3cd6b0, unsigned long recv=10342440, int argc=0, const unsigned long * argv=0x00570040) Line 1336 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8) Line 1502 + 0x20 bytes C
msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8) Line 1592 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8) Line 1786 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8) Line 1941 + 0x11 bytes C
msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0) Line 999 + 0x1a bytes C
msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588) Line 1328 + 0xd bytes C
msvcr100-ruby220.dll!rb_iseq_eval_main(unsigned long iseqval=10505960) Line 1586 + 0x9 bytes C
msvcr100-ruby220.dll!ruby_exec_internal(void * n=0x00a04ee8) Line 254 + 0x46 bytes C
msvcr100-ruby220.dll!ruby_exec_node(void * n=0x00a04ee8) Line 316 + 0x9 bytes C
msvcr100-ruby220.dll!ruby_run_node(void * n=0x00a04ee8) Line 308 + 0x9 bytes C
ruby_t.exe!main(int argc=3, char * * argv=0x00c115b8) Line 36 + 0x16 bytes C
ruby_t.exe!__tmainCRTStartup() Line 555 + 0x17 bytes C
In the iseq_set_sequence above marked by ">", the bad hash value was
fetched from operands[j]:
case TS_CDHASH:
{
VALUE map = operands[j];
struct cdhash_set_label_struct data;
data.hash = map;
data.pos = pos;
data.len = len;
rb_hash_foreach(map, cdhash_set_label_i, (VALUE)&data);
hide_obj(map);
generated_iseq[pos + 1 + j] = map;
break;
}
And the iobj->insn_id is consistently YARVINSN_opt_case_dispatch:
iobj 0x00d27c6c {link={...} insn_id=YARVINSN_opt_case_dispatch line_no=5 ...} iseq_insn_data *
link {type=ISEQ_ELEMENT_INSN next=0x00d27c8c prev=0x00d27ba4 } iseq_link_element
insn_id YARVINSN_opt_case_dispatch ruby_vminsn_type
line_no 5 unsigned int
operand_size 2 int
sc_state 0 int
I had invoked the program with --disable-gems for simplicity, however
the crash occurs either way.
If there's any further information I could provide please let me know.
Thanks for your help!
Regards,
Bill
Updated by spatulasnout (B Kelly) over 10 years ago
Salutations,
Attached is a one-page PDF slide for the Ruby 2.2 feature proposal
developer meeting. It describes our use case for rb_iseq_load, per:
[ruby-core:63332] [ANN] Request for "slide-show" of your feature proposal
Thank you!
Bill
Updated by naruse (Yui NARUSE) over 10 years ago
received, thanks!
Updated by ko1 (Koichi Sasada) over 10 years ago
Thank you for your report.
I'll check it (best effort. I ask my boss about priority).
Updated by nobu (Nobuyoshi Nakada) over 10 years ago
- Tracker changed from Bug to Feature
- Description updated (diff)
Updated by spatulasnout (B Kelly) about 10 years ago
Hi,
Koichi Sasada wrote:
Thank you for your report.
I'll check it (best effort. I ask my boss about priority).
Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?
I attempted a git bisect
this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:
commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000
* parse.y: fix build error with bison-3.0.
And unfortunately the iseq.load problem already existed by that
point.
(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)
Very much hoping iseq.load might be fixable before 2.2 code
freeze.
Thanks & Regards,
Bill
Updated by normalperson (Eric Wong) about 10 years ago
billk@cts.com wrote:
Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?
Not speaking for the rest of ruby-core, but I welcome occasional
reminders like these :)
I attempted a
git bisect
this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000* parse.y: fix build error with bison-3.0.
And unfortunately the iseq.load problem already existed by that
point.
Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?
(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)
Yes. "git bisect run" is awesome for scripting these things.
Updated by spatulasnout (B Kelly) about 10 years ago
Eric Wong wrote:
Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?
Good call -- cygwin indeed allowed me to roll back to bison 2.7.x
I wasn't able to fully automate git bisect
as there was various build
breakage on some of the commits (usually to do with changes in enc and
enc/trans.)
But ultimately, the result of the manual bisect was:
66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000
* compile.c: fix to output warning when the same literals
are available as a condition of same case clause.
And remove infomation ('#n') because we can find duplicated
condition with explicit line numbers.
[ruby-core:38343] [Ruby 1.9 - Bug #5068]
* test/ruby/test_syntax.rb: add a test for above.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
:100644 100644 bcde6499fd43af4fc7eae9496d7eb529e52d5465 5f48bda3d2c5787acfc93c7d209964b45b4405bf M ChangeLog
:100644 100644 508d599d081ddd3676efc513c25f76f00216116b 74982db138d5432f0077c49265e7b177a906ec97 M compile.c
:040000 040000 b0608f2c1a2b0eaab543fb26ac4c2a78cb9b0c57 4f78f071cacaf4c8da9d8ccfecb027092f94bc54 M test
My test script was running both Alexey Voskov's "tralivali" test as
well as my segfault test relating to the case statement.
I suspect the above commit is what introduced the case statement-
related segfault. I'm not sure if it will also relate to Alexey
Voskov's "tralivali" test well, or whether that might be a separate
issue. (But if the above can be fixed, I'm happy to try another
bisect if other issues remain.)
Thanks & Regards,
Bill
Updated by normalperson (Eric Wong) almost 10 years ago
Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/rb_iseq_load_fix-wip@v0.txt
Thanks.
Updated by normalperson (Eric Wong) almost 10 years ago
Eric Wong normalperson@yhbt.net wrote:
Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/rb_iseq_load_fix-wip@v0.txt
Don't bother with that, I forgot to test iseq-load-test3-file.rb
hello.rb works fine, at least :x Fixing...
Updated by normalperson (Eric Wong) almost 10 years ago
billk@cts.com wrote:
But ultimately, the result of the manual bisect was:
66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000* compile.c: fix to output warning when the same literals are available as a condition of same case clause. And remove infomation ('#n') because we can find duplicated condition with explicit line numbers. [ruby-core:38343] [Ruby 1.9 - Bug #5068] * test/ruby/test_syntax.rb: add a test for above.
That was only one of the breakages :)
Things have bitrotted a lot over the years.
The following patch might be ready to commit to trunk:
http://80x24.org/spew/m/rb_iseq_load_fix@v1.txt
It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.
There's no public API change in this version of my patch.
The test cases are probably not complete, though. I am mainly
interested in the feature because I'm working on another (in-core)
optimization which may utilize this.
I can definitely use some help thinking of better test cases since I
don't believe I remember all of Ruby syntax :x
Updated by normalperson (Eric Wong) almost 10 years ago
Eric Wong normalperson@yhbt.net wrote:
The following patch might be ready to commit to trunk:
Still some bugs related to keyword args I'm working on, but am out of
time for the moment:
http://80x24.org/spew/m/20141123055537.GA1002%40dcvr.yhbt.net.txt
Updated by spatulasnout (B Kelly) almost 10 years ago
Howdy,
Eric Wong wrote:
That was only one of the breakages :)
Things have bitrotted a lot over the years.The following patch might be ready to commit to trunk:
http://80x24.org/spew/m/rb_iseq_load_fix@v1.txt
It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.
Thanks so much for working on this.
Just some initial feedback -- after applying both patches to trunk,
I'm still seeing a segfault in the iseq.eval call on iseq-load-test3.rb,
though in a different location than before.
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
omg: {:mid=>:each_char, :flag=>256, :orig_argc=>0, :blockptr=>nil}
omg: {:mid=>:inject, :flag=>0, :orig_argc=>1, :blockptr=>["YARVInstructionSequence/SimpleDataFormat", 2, 2, 1, {:arg_size=>2, :local_size=>3, :stack_max=>4}, "block in user_mask", "./iseq-load-test3-file.rb", "P:/code/ruby/bkelly/test/iseq-load-test3-file.rb", 4, :block, [:mask, :chr], {:lead_num=>2}, [[:redo, nil, :label_2, :label_102, :label_2, 0], [:next, nil, :label_2, :label_102, :label_102, 0]], [4, [:trace, 256], :label_2, 5, [:trace, 1], [:getlocal_OP__WC__0, 2], [:dup], [:opt_case_dispatch, ["u", :label_60, "g", :label_71, "o", :label_82, "a", :label_93], :label_38], 6, [:dup], [:putobject, "u"], [:checkmatch, 2], [:branchif, :label_60], 8, [:dup], [:putobject, "g"], [:checkmatch, 2], [:branchif, :label_71], 10, [:dup], [:putobject, "o"], [:checkmatch, 2], [:branchif, :label_82], 12, [:dup], [:putobject, "a"], [:checkmatch, 2], [:branchif, :label_93], :label_38, 15, [:pop], [:trace, 1], [:putself], [:getinlinecache, :label_49, 0], [:getconstant, :ArgumentError], [:setinlinecache, 0], :label_49, [:putobject, "invalid who' symbol in file mode: "], [:getlocal_OP__WC__0, 2], [:tostring], [:concatstrings, 2], [:opt_send_without_block, {:mid=>:raise, :flag=>264, :orig_argc=>2, :blockptr=>nil}], [:jump, :label_102], :label_60, 16, [:pop], 7, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 2496], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_71, [:pop], 9, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 1080], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_82, [:pop], 11, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 519], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_93, [:pop], 13, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 4095], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], :label_102, 17, [:trace, 512], 15, [:leave]]]} omg: {:mid=>:"core#define_method", :flag=>256, :orig_argc=>3, :blockptr=>nil} ./iseq-load-test3-file.rb:3:in
module:FileUtils'iseq-load-test3.rb: [BUG] Segmentation fault
ruby 2.2.0dev (2014-11-24 trunk 48553) [i386-mswin32_100]
-- Control frame information -----------------------------------------------
c:0005 p:---- s:0010 e:000009 CFUNC :to_s
c:0004 p:---- s:0008 e:000007 CFUNC :to_str
c:0003 p:---- s:0006 e:000005 CFUNC :to_s
c:0002 p:---- s:0004 e:000003 CFUNC :message
c:0001 p:0000 s:0002 E:000f5c TOP [FINISH]
-- Ruby level backtrace information ----------------------------------------
iseq-load-test3.rb:0:in message' iseq-load-test3.rb:0:in
to_s'
iseq-load-test3.rb:0:in to_str' iseq-load-test3.rb:0:in
to_s'
-- C level backtrace information -------------------------------------------
C:\Windows\SysWOW64\ntdll.dll(ZwWaitForSingleObject+0x15) [0x76EAF8D1]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObjectEx+0x43) [0x76511194]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObject+0x12) [0x76511148]
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_print_backtrace+0x38) [0x52AD8468] p:\code\ruby-git\ruby-trunk\vm_dump.c:712
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_vm_bugreport+0x65) [0x52AD8915] p:\code\ruby-git\ruby-trunk\vm_dump.c:974
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_bug_context+0x74) [0x529D24F4] p:\code\ruby-git\ruby-trunk\error.c:389
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(sigsegv+0x28) [0x52A5D638] p:\code\ruby-git\ruby-trunk\signal.c:850
C:\Windows\system32\MSVCR100.dll(XcptFilter+0x13e) [0x71B5B9DF]
M:\dev\ruby-build\trunk\bin\ruby_t.exe(__tmainCRTStartup+0x14a) [0x00DD11F5] f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c:572
C:\Windows\system32\MSVCR100.dll(seh_longjmp_unwind4+0x2e) [0x71AD2F54]
C:\Windows\syswow64\kernel32.dll(BaseThreadInitThunk+0x12) [0x7651338A]
C:\Windows\SysWOW64\ntdll.dll(RtlInitializeExceptionChain+0x63) [0x76EC9F72]
-- Other runtime information -----------------------------------------------
- Loaded script: iseq-load-test3.rb
The debugger backtrace was:
msvcr100-ruby220.dll!str_replace_shared_without_enc(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 874 + 0x3 bytes C
msvcr100-ruby220.dll!str_replace_shared(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 892 + 0xd bytes C
msvcr100-ruby220.dll!str_new_shared(unsigned long klass=0x0024e76c, unsigned long str=0x00000000) Line 900 + 0x16 bytes C
msvcr100-ruby220.dll!rb_sym_to_s(unsigned long sym=0x0000000e) Line 8599 + 0x19 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52254cd0, unsigned long recv=0x0000000e, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x0000000e, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027b118, unsigned long defined_class=0x0024e0a0) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1abc) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!rb_obj_as_string(unsigned long obj=0x0000000e) Line 1111 + 0x12 bytes C
msvcr100-ruby220.dll!rb_str_format(int argc=0x00000003, const unsigned long * argv=0x004cef90, unsigned long fmt=0x02ea19b8) Line 711 + 0xc bytes C
msvcr100-ruby220.dll!rb_f_sprintf(int argc=0x00000003, const unsigned long * argv=0x004cef90) Line 449 + 0x3e bytes C
msvcr100-ruby220.dll!name_err_mesg_to_str(unsigned long obj=0x02ea3330) Line 1204 + 0xb bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x522142d0, unsigned long recv=0x02ea1abc, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1abc, unsigned long id=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027ef38, unsigned long defined_class=0x0024dbb4) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1abc, unsigned long mid=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!convert_type(unsigned long val=0x02ea1abc, const char * tname=0x523fe264, const char * method=0x523fe25c, int raise=0x00000000) Line 2623 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_convert_type(unsigned long val=0x02ea1abc, int type=0x00000005, const char * tname=0x523fe264, const char * method=0x523fe25c) Line 2673 + 0x13 bytes C
msvcr100-ruby220.dll!rb_check_string_type(unsigned long str=0x02ea1abc) Line 1744 + 0x15 bytes C
msvcr100-ruby220.dll!rb_String(unsigned long val=0x02ea1abc) Line 3022 + 0x9 bytes C
msvcr100-ruby220.dll!exc_to_s(unsigned long exc=0x02ea1a94) Line 693 + 0x9 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213b10, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bbb8, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1a94) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!exc_message(unsigned long exc=0x02ea1a94) Line 709 + 0x1b bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213be0, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bc38, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!error_print() Line 133 + 0x25 bytes C
msvcr100-ruby220.dll!error_handle(int ex=0x00000006) Line 312 C
msvcr100-ruby220.dll!ruby_cleanup(volatile int ex=0x00000006) Line 193 + 0x9 bytes C
msvcr100-ruby220.dll!ruby_run_node(void * n=0x02e55234) Line 309 + 0x12 bytes C
ruby_t.exe!main(int argc=0x00000002, char * * argv=0x002a1678) Line 36 + 0x16 bytes C
ruby_t.exe!__tmainCRTStartup() Line 555 + 0x17 bytes C
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes
It seemed to be dying in an rb_f_sprintf() call. The 'desc' variable in
name_err_mesg_to_str(), which was invoking sprintf, evaluated to:
desc = 0x02e11420 "RubyVM::InstructionSequence:user_mask@./iseq-load-test3-file.rb"
If there's anything I could try that might help narrow it down further,
please let me know.
Thanks again for your help,
Bill
Updated by normalperson (Eric Wong) almost 10 years ago
Sorry, the inline patch was an extremely hacky work-in-progress,
but I think rb_iseq_load_fix@v1.txt should've been OK with your
(non keyword) use cases.
Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
This is on top of r48554, which was purely a cleanup commit in
preparation for this.
Notable changes since v1 (and my WIP v2):
- iseq#to_ary dumps :kwbits => iseq->params.keyword->bits_start
Current hacks:
-
peephole optimization seems to be not idempotent wrt jump elimination
A second pass (optimize => to_ary => load(optimize) eliminates
extra useless jumps such as:jump LABEL
LABEL:
I currently disable peephole optimization to on load to avoid
running the optimizer twice, but ideally, the first pass should
eliminate the above jump insn. -
iseq->stack_size seems not calculated correctly upon load,
so I'm currently blindly loading it from the misc hash :x
Updated by spatulasnout (B Kelly) almost 10 years ago
Eric Wong wrote:
Sorry, the inline patch was an extremely hacky work-in-progress,
but I think rb_iseq_load_fix@v1.txt should've been OK with your
(non keyword) use cases.Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
Ah, I see.
Thanks! v3 is indeed no longer segfaulting on any of the iseq tests
I have.
Awesome!
Thanks again,
Bill
Updated by ko1 (Koichi Sasada) almost 10 years ago
Seems nice! Thank you!
(maybe we need fix rdoc)
Updated by normalperson (Eric Wong) almost 10 years ago
ko1: thank you for checking, any comment on the extra jumps insns which
I hack around by disabling peephole optimization?
It seems caused by the following in iseq_set_sequence:
else if (orig_sp - sp == 0) {
/* jump to next insn */
if (last_line != (unsigned int)adjust->line_no) {
line_info_table[k].line_no = last_line = adjust->line_no;
line_info_table[k].position = pos;
k++;
}
generated_iseq[pos++] = BIN(jump);
generated_iseq[pos++] = 0;
My roundtrip test fails because:
1. iseq_peephole_optimize called in original
2. iseq_set_sequence generates useless jump (above)
3. ISeq#to_a called (showing useless jump)
4. ISeq#load on 3. array
5. iseq_peephole_optimize now deletes jump emitted in 2.
6. ISeq#to_a now has no useless jumps
So now I'm trying to figure out how to eliminate the useless jump in 2
so roundtrip test always succeeds (and my normal Ruby code has
no useless jump insns)
Updated by ko1 (Koichi Sasada) almost 10 years ago
I use nop insns instead of "jump to next insn" (2 words) in r48579.
How about it?
Updated by normalperson (Eric Wong) almost 10 years ago
nop; nop works, but I don't think it's an improvement (or regression).
I prefer to save 2 words entirely.
I planned to workaround the issue by doing ISeq#load twice:
first = iseq.to_a # useless jump|nop here
junk = ISeq.load(first) # second pass optimization removes jump|nop
a = ISeq.load(junk.to_a)
b = ISeq.load(a.to_a)
assert_equal a, b
But I encountered other bugs with to_a + load on test/ruby/test_io.rb
and test_settracefunc.rb which I have not investigated, yet...
Works-in-progress:
[PATCH 1/2] http://80x24.org/spew/m/iseq_calc_param_size%40r48578.txt
[PATCH 2/2] http://80x24.org/spew/m/rb_iseq_load_fix-v4%40r48578.txt
Updated by normalperson (Eric Wong) almost 10 years ago
Work-in-progress -v5 based on r48597:
http://80x24.org/spew/m/rb_iseq_load-fix-v5@r48597.txt
Still some failures, particularly around:
def x(a, (b, *c), d: false); end
I'm not sure if we handle ISeq#to_a correctly for "(b, *c)"
right now...
I might be busy with other stuff for a few days.
Updated by wanabe (_ wanabe) almost 10 years ago
https://github.com/ruby/ruby/blob/33ea2646b98adb49ae2e1781753bf22d33729ac0/iseq.c#L1720
iseq_data_to_ary() ignores hidden variables named by id_internal().
It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?
Updated by normalperson (Eric Wong) almost 10 years ago
s.wanabe@gmail.com wrote:
iseq_data_to_ary() ignores hidden variables named by id_internal().
It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?
Thank you! Committed as r48678. That seems to fix the
def x(a, (b, *c), d: false); end
case for me, now #to_a seems to to not be idempotent and segfaults,
investigating.
Updated by Anonymous almost 10 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Applied in changeset r48705.
mostly fix rb_iseq_load
This allows reporters commenters of [Feature #8543] to load
instruction sequences directly. Some test cases are still failing
but documented in test/-ext-/iseq_load/test_iseq_load.rb.
- compile.c (rb_iseq_build_from_exception): entry->sp is unsigned
(iseq_build_callinfo_from_hash): account for kw_arg
(iseq_build_from_ary_body): update for r35459
(CHECK_STRING, CHECK_INTEGER): remove unused checks
(int_param): new function for checking newparams' hash (iseq_build_kw): new function for loading rb_iseq_param_keyword (rb_iseq_build_from_ary): account for
misc' entry and general
structure changes
[Feature #8543] - iseq.c (CHECK_HASH): new macro (for
misc' and
param' entries)
(iseq_load): account formisc' and
params' hashes
(iseq_data_to_ary): add final opt to arg_opt_labels,
fix kw support, account for unsigned entry->sp - ext/-test-/iseq_load/iseq_load.c: new ext for test
- ext/-test-/iseq_load/extconf.rb: ditto
- test/-ext-/iseq_load/test_iseq_load.rb: new test
Updated by normalperson (Eric Wong) almost 10 years ago
I think the stack mismatches in current trunk are hard to avoid, but the
loaded bytecode is still valid and runnable for the currently-skipped
cases. I propose the following to test more thoroughly.
diff --git a/test/-ext-/iseq_load/test_iseq_load.rb b/test/-ext-/iseq_load/test_iseq_load.rb
index 5bbd49e..7251603 100644
--- a/test/-ext-/iseq_load/test_iseq_load.rb
+++ b/test/-ext-/iseq_load/test_iseq_load.rb
@@ -52,16 +52,21 @@ class TestIseqLoad < Test::Unit::TestCase
end
def test_next_in_block_in_block
- skip "failing due to stack_max mismatch"
- assert_iseq_roundtrip <<-'end;'
- 3.times { 3.times { next } }
+ @next_broke = false
+ src = <<-'end;'
+ 3.times { 3.times { next; @next_broke = true } }
end;
+ a = ISeq.compile(src).to_a
+ iseq = ISeq.iseq_load(a)
+ iseq.eval
+ assert_equal false, @next_broke
+ skip "failing due to stack_max mismatch"
+ assert_iseq_roundtrip(src)
end
def test_break_ensure
- skip "failing due to exception entry sp mismatch"
- assert_iseq_roundtrip <<-'end;'
- def m
+ src = <<-'end;'
+ def test_break_ensure_def_method
bad = true
while true
begin
@@ -70,8 +75,15 @@ class TestIseqLoad < Test::Unit::TestCase
bad = false
end
end
+ bad
end
end;
+ a = ISeq.compile(src).to_a
+ iseq = ISeq.iseq_load(a)
+ iseq.eval
+ assert_equal false, test_break_ensure_def_method
+ skip "failing due to exception entry sp mismatch"
+ assert_iseq_roundtrip(src)
end
# FIXME: still failing
Updated by spatulasnout (B Kelly) about 9 years ago
- File iseq-load-test5.rb iseq-load-test5.rb added
- Subject changed from rb_iseq_load to new rb_iseq_load crash
Hi,
We've encountered a new crash related to RubyVM::InstructionSequence.load.
So far it has been reproducible on multiple platforms:
ruby 2.2.3p158 (2015-08-12 revision 48555) [i386-mswin32_100]
ruby 2.2.3p158 (2015-08-12 revision 51190) [i386-darwin13]
Attached is a small program which reproduces the crash on our systems.
(The attached file contains further details about the crash.)
Updated by spatulasnout (B Kelly) about 9 years ago
Addendum: Attached file in previous post also reproduces the crash
in the latest ruby trunk:
:3: [BUG] Segmentation fault
ruby 2.3.0dev (2015-09-09 trunk 48555) [i386-mswin32_100]
Updated by nagachika (Tomoyuki Chikanaga) about 9 years ago
- Tracker changed from Feature to Bug
- ruby -v set to -
- Backport set to 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: REQUIRED
At least r51816 should be backported into 2.1/2.2.
Updated by nagachika (Tomoyuki Chikanaga) almost 9 years ago
- Backport changed from 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: DONE
Backported into ruby_2_2
branch at r52740.
Updated by usa (Usaku NAKAMURA) almost 9 years ago
- Backport changed from 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: DONE to 2.0.0: UNKNOWN, 2.1: DONTNEED, 2.2: DONE
2.0/2.1 use different code about r51816.
So they don't have this problem, I think.