Bug #8543: new rb_iseq_load crash

• Status: Closed
• Assignee: ko1 (Koichi Sasada)
• Target version: -
• ruby -v: -
• Backport: 2.0.0: UNKNOWN, 2.1: DONTNEED, 2.2: DONE

[alvoskov (Alexey Voskov)]

I noticed an unusual behaviour of undocumented rb_iseq_load function.
Its work differs in different Ruby versions. I'm trying to protect some Ruby
source code by its conversion to YARV p-code and using the next strategy:

1. Convert code to array

   data = RubyVM::InstructionSequence.compile_file('hello.rb').to_a
   

2. Pass a compiled source to the rb_iseq_load function and evaluate it

   iseq = iseq_load.(data)
   iseq.eval
   

Sample programs are supplied in the attachments.
"hello.rb"

puts "tralivali"
def funct(a,b)
  a**b
end

3.times { |i|
  puts "Hello, world#{funct(2,i)}!"
}

The differences
Ruby 1.9.3 (ruby 1.9.3p194 (2012-04-20) [i386-mingw32])
Correct work. Output:

tralivali
Hello, world1!
Hello, world2!
Hello, world4!

Ruby 2.0.0 (ruby 2.0.0p193 (2013-05-14) [x64-mingw32])
Incorrect work (omits the code inside code blocks). Output

tralivali

Attempts of loading bigger programs by means of rb_iseq_load in Ruby 2.0.0 usually ends with a segmentation fault.

Such behaviour also can be reproduced by means of iseq Ruby extension ("for iseq freaks")
https://github.com/wanabe/iseq

P.S. I understand that it is an undocumented feature.

[nagachika (Tomoyuki Chikanaga)]

• Tracker changed from Misc to Bug

[nagachika (Tomoyuki Chikanaga)]

• Priority changed from Normal to 3
• Backport set to 1.9.3: DONTNEED, 2.0.0: REQUIRED
• ruby -v set to ruby 2.0.0p234 (2013-06-19 revision 41434) [x86_64-darwin11.4.2]

[hsbt (Hiroshi SHIBATA)]

• Target version changed from 2.1.0 to 2.2.0

[spatulasnout (B Kelly)]

Hi,

Just a data point:

With ruby 2.2.0dev (2014-04-16 trunk 45576) [i386-mswin32_100],
the rb_iseq_load example in https://bugs.ruby-lang.org/issues/8543
is producing the correct output.

(I will attempt to test with more complicated programs soon.)

Regards,

Bill

[spatulasnout (B Kelly)]

• File iseq-load-test3.rb iseq-load-test3.rb added
• File iseq-load-test3-file.rb iseq-load-test3-file.rb added

Hello again,

The attached files comprise a small example which appears to consistently
reproduce a segmentation fault with rb_iseq_load.

Interpreter is: ruby 2.2.0dev (2014-04-16 trunk 45597) [i386-mswin32_100]

The crash appears related to the case statement.

Call stack:

The st_table pointer at st_foreach_check is bad:

  msvcr100-ruby220.dll!st_foreach_check(st_table * table=0x00000008, int (void)* func=0x0f3f3580, unsigned long arg=4512588, unsigned long never=6)  Line 891 + 0x3 bytes C
  msvcr100-ruby220.dll!hash_foreach_call(unsigned long arg=4512588)  Line 258 + 0x17 bytes        C
  msvcr100-ruby220.dll!rb_ensure(unsigned long (void)* b_proc=0x0f3f3530, unsigned long data1=4512588, unsigned long (void)* e_proc=0x0f3f3480, unsigned long data2=10318080)  Line 888 + 0x7 bytes       C
  msvcr100-ruby220.dll!rb_hash_foreach(unsigned long hash=10318080, int (void)* func=0x0f45ac20, unsigned long farg=4512696)  Line 275 + 0x17 bytes       C
> msvcr100-ruby220.dll!iseq_set_sequence(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58)  Line 1507 + 0x12 bytes  C
  msvcr100-ruby220.dll!iseq_setup(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58)  Line 1026 + 0xd bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58, unsigned long body=10318900, st_table * labels_table=0x00d27d10)  Line 5855 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d279b0, unsigned long locals=10320520, unsigned long args=5, unsigned long exception=10320460, unsigned long body=10318900)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27500, unsigned long op=10320540)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27500, iseq_link_anchor * anchor=0x0044dfd4, unsigned long body=10318680, st_table * labels_table=0x00d27860)  Line 5816 + 0x13 bytes  C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27500, unsigned long locals=10320900, unsigned long args=3, unsigned long exception=10320840, unsigned long body=10318680)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27110, unsigned long op=10320920)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27110, iseq_link_anchor * anchor=0x0044e250, unsigned long body=10318560, st_table * labels_table=0x00d27470)  Line 5783 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27110, unsigned long locals=10321160, unsigned long args=1, unsigned long exception=10321100, unsigned long body=10318560)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d25c08, unsigned long op=10321180)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d25c08, iseq_link_anchor * anchor=0x0044e4cc, unsigned long body=10318500, st_table * labels_table=0x00d25f68)  Line 5783 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d25c08, unsigned long locals=10339520, unsigned long args=1, unsigned long exception=10339460, unsigned long body=10318500)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10339540, unsigned long parent=0, unsigned long opt=4)  Line 561 + 0x19 bytes    C
  msvcr100-ruby220.dll!iseq_s_load(int argc=1, unsigned long * argv=0x0057005c, unsigned long self=10542580)  Line 576 + 0x13 bytes       C
  msvcr100-ruby220.dll!call_cfunc_m1(unsigned long (void)* func=0x0f443b50, unsigned long recv=10542580, int argc=1, const unsigned long * argv=0x0057005c)  Line 1330 + 0xf bytes        C
  msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1502 + 0x20 bytes      C
  msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1592 + 0x11 bytes C
  msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1786 + 0x11 bytes    C
  msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1941 + 0x11 bytes       C
  msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0)  Line 1028 + 0x1a bytes    C
  msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588)  Line 1328 + 0xd bytes   C
  msvcr100-ruby220.dll!invoke_block_from_c(rb_thread_struct * th=0x00c15588, const rb_block_struct * block=0x005effe0, unsigned long self=10187260, int argc=1, const unsigned long * argv=0x0044f0ec, const rb_block_struct * blockptr=0x00000000, const RNode * cref=0x00000000, unsigned long defined_class=4, int splattable=1)  Line 752 + 0x9 bytes	C
  msvcr100-ruby220.dll!vm_yield(rb_thread_struct * th=0x00c15588, int argc=1, const unsigned long * argv=0x0044f0ec)  Line 784 + 0x28 bytes       C
  msvcr100-ruby220.dll!rb_yield_0(int argc=1, const unsigned long * argv=0x0044f0ec)  Line 936 + 0x13 bytes       C
  msvcr100-ruby220.dll!rb_yield(unsigned long val=10340180)  Line 946 + 0xb bytes C
  msvcr100-ruby220.dll!rb_ary_each(unsigned long array=10342440)  Line 1806 + 0x2f bytes  C
  msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x0f3cd6b0, unsigned long recv=10342440, int argc=0, const unsigned long * argv=0x00570040)  Line 1336 + 0x7 bytes C
  msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1502 + 0x20 bytes      C
  msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1592 + 0x11 bytes C
  msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1786 + 0x11 bytes    C
  msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1941 + 0x11 bytes       C
  msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0)  Line 999 + 0x1a bytes     C
  msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588)  Line 1328 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_eval_main(unsigned long iseqval=10505960)  Line 1586 + 0x9 bytes   C
  msvcr100-ruby220.dll!ruby_exec_internal(void * n=0x00a04ee8)  Line 254 + 0x46 bytes     C
  msvcr100-ruby220.dll!ruby_exec_node(void * n=0x00a04ee8)  Line 316 + 0x9 bytes  C
  msvcr100-ruby220.dll!ruby_run_node(void * n=0x00a04ee8)  Line 308 + 0x9 bytes   C
  ruby_t.exe!main(int argc=3, char * * argv=0x00c115b8)  Line 36 + 0x16 bytes     C
  ruby_t.exe!__tmainCRTStartup()  Line 555 + 0x17 bytes   C


In the iseq_set_sequence above marked by ">", the bad hash value was
fetched from operands[j]:

                case TS_CDHASH:
                  {
                      VALUE map = operands[j];
                      struct cdhash_set_label_struct data;
                      data.hash = map;
                      data.pos = pos;
                      data.len = len;
                      rb_hash_foreach(map, cdhash_set_label_i, (VALUE)&data);

                      hide_obj(map);
                      generated_iseq[pos + 1 + j] = map;
                      break;
                  }

And the iobj->insn_id is consistently YARVINSN_opt_case_dispatch:

    iobj            0x00d27c6c {link={...} insn_id=YARVINSN_opt_case_dispatch line_no=5 ...}        iseq_insn_data *
    link            {type=ISEQ_ELEMENT_INSN next=0x00d27c8c prev=0x00d27ba4 }       iseq_link_element
    insn_id         YARVINSN_opt_case_dispatch      ruby_vminsn_type
    line_no         5       unsigned int
    operand_size    2       int
    sc_state        0       int

I had invoked the program with --disable-gems for simplicity, however
the crash occurs either way.

If there's any further information I could provide please let me know.

Thanks for your help!

Regards,

Bill

[spatulasnout (B Kelly)]

• File please-fix-rb_iseq_load-thank-you.pdf please-fix-rb_iseq_load-thank-you.pdf added

Salutations,

Attached is a one-page PDF slide for the Ruby 2.2 feature proposal
developer meeting. It describes our use case for rb_iseq_load, per:
[ruby-core:63332] [ANN] Request for "slide-show" of your feature proposal

Thank you!

Bill

[naruse (Yui NARUSE)]

received, thanks!

[ko1 (Koichi Sasada)]

Thank you for your report.

I'll check it (best effort. I ask my boss about priority).

[nobu (Nobuyoshi Nakada)]

• Tracker changed from Bug to Feature
• Description updated (diff)

[spatulasnout (B Kelly)]

Hi,

Koichi Sasada wrote:

Thank you for your report.

I'll check it (best effort. I ask my boss about priority).

Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?

I attempted a git bisect this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:

commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000

* parse.y: fix build error with bison-3.0.

And unfortunately the iseq.load problem already existed by that
point.

(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)

Very much hoping iseq.load might be fixable before 2.2 code
freeze.

Thanks & Regards,

Bill

[normalperson (Eric Wong)]

[e-mail] wrote:

Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?

Not speaking for the rest of ruby-core, but I welcome occasional
reminders like these :)

I attempted a git bisect this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:

commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000

* parse.y: fix build error with bison-3.0.

And unfortunately the iseq.load problem already existed by that
point.

Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?

(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)

Yes. "git bisect run" is awesome for scripting these things.

[spatulasnout (B Kelly)]

Eric Wong wrote:

Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?

Good call -- cygwin indeed allowed me to roll back to bison 2.7.x

I wasn't able to fully automate git bisect as there was various build
breakage on some of the commits (usually to do with changes in enc and
enc/trans.)

But ultimately, the result of the manual bisect was:

66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000

* compile.c: fix to output warning when the same literals
  are available as a condition of same case clause.
  And remove infomation ('#n') because we can find duplicated
  condition with explicit line numbers.
  [ruby-core:38343] [Ruby 1.9 - Bug #5068]
* test/ruby/test_syntax.rb: add a test for above.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

:100644 100644 bcde6499fd43af4fc7eae9496d7eb529e52d5465 5f48bda3d2c5787acfc93c7d209964b45b4405bf M ChangeLog
:100644 100644 508d599d081ddd3676efc513c25f76f00216116b 74982db138d5432f0077c49265e7b177a906ec97 M compile.c
:040000 040000 b0608f2c1a2b0eaab543fb26ac4c2a78cb9b0c57 4f78f071cacaf4c8da9d8ccfecb027092f94bc54 M test

My test script was running both Alexey Voskov's "tralivali" test as
well as my segfault test relating to the case statement.

I suspect the above commit is what introduced the case statement-
related segfault. I'm not sure if it will also relate to Alexey
Voskov's "tralivali" test well, or whether that might be a separate
issue. (But if the above can be fixed, I'm happy to try another
bisect if other issues remain.)

Thanks & Regards,

Bill

[normalperson (Eric Wong)]

Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/[e-mail]
Thanks.

[normalperson (Eric Wong)]

Eric Wong [e-mail] wrote:

Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/[e-mail]

Don't bother with that, I forgot to test iseq-load-test3-file.rb
hello.rb works fine, at least :x Fixing...

[normalperson (Eric Wong)]

[e-mail] wrote:

But ultimately, the result of the manual bisect was:

66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000

* compile.c: fix to output warning when the same literals
  are available as a condition of same case clause.
  And remove infomation ('#n') because we can find duplicated
  condition with explicit line numbers.
  [ruby-core:38343] [Ruby 1.9 - Bug #5068]
* test/ruby/test_syntax.rb: add a test for above.

That was only one of the breakages :)
Things have bitrotted a lot over the years.

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/[e-mail]

It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.

There's no public API change in this version of my patch.

The test cases are probably not complete, though. I am mainly
interested in the feature because I'm working on another (in-core)
optimization which may utilize this.

I can definitely use some help thinking of better test cases since I
don't believe I remember all of Ruby syntax :x

[normalperson (Eric Wong)]

Eric Wong [e-mail] wrote:

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/[e-mail]

Still some bugs related to keyword args I'm working on, but am out of
time for the moment:

http://80x24.org/spew/m/20141123055537.GA1002%40dcvr.yhbt.net.txt

[spatulasnout (B Kelly)]

Howdy,

Eric Wong wrote:

That was only one of the breakages :)
Things have bitrotted a lot over the years.

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/[e-mail]

It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.

Thanks so much for working on this.

Just some initial feedback -- after applying both patches to trunk,
I'm still seeing a segfault in the iseq.eval call on iseq-load-test3.rb,
though in a different location than before.

ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
omg: {:mid=>:each_char, :flag=>256, :orig_argc=>0, :blockptr=>nil}
omg: {:mid=>:inject, :flag=>0, :orig_argc=>1, :blockptr=>["YARVInstructionSequence/SimpleDataFormat", 2, 2, 1, {:arg_size=>2, :local_size=>3, :stack_max=>4}, "block in user_mask", "./iseq-load-test3-file.rb", "P:/code/ruby/bkelly/test/iseq-load-test3-file.rb", 4, :block, [:mask, :chr], {:lead_num=>2}, [[:redo, nil, :label_2, :label_102, :label_2, 0], [:next, nil, :label_2, :label_102, :label_102, 0]], [4, [:trace, 256], :label_2, 5, [:trace, 1], [:getlocal_OP__WC__0, 2], [:dup], [:opt_case_dispatch, ["u", :label_60, "g", :label_71, "o", :label_82, "a", :label_93], :label_38], 6, [:dup], [:putobject, "u"], [:checkmatch, 2], [:branchif, :label_60], 8, [:dup], [:putobject, "g"], [:checkmatch, 2], [:branchif, :label_71], 10, [:dup], [:putobject, "o"], [:checkmatch, 2], [:branchif, :label_82], 12, [:dup], [:putobject, "a"], [:checkmatch, 2], [:branchif, :label_93], :label_38, 15, [:pop], [:trace, 1], [:putself], [:getinlinecache, :label_49, 0], [:getconstant, :ArgumentError], [:setinlinecache, 0], :label_49, [:putobject, "invalid who' symbol in file mode: "], [:getlocal_OP__WC__0, 2], [:tostring], [:concatstrings, 2], [:opt_send_without_block, {:mid=>:raise, :flag=>264, :orig_argc=>2, :blockptr=>nil}], [:jump, :label_102], :label_60, 16, [:pop], 7, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 2496], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_71, [:pop], 9, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 1080], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_82, [:pop], 11, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 519], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_93, [:pop], 13, [:trace, 1], [:getlocal_OP__WC__0, 3], [:putobject, 4095], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], :label_102, 17, [:trace, 512], 15, [:leave]]]} omg: {:mid=>:"core#define_method", :flag=>256, :orig_argc=>3, :blockptr=>nil} ./iseq-load-test3-file.rb:3:in module:FileUtils'iseq-load-test3.rb: [BUG] Segmentation fault
ruby 2.2.0dev (2014-11-24 trunk 48553) [i386-mswin32_100]

-- Control frame information -----------------------------------------------
c:0005 p:---- s:0010 e:000009 CFUNC :to_s
c:0004 p:---- s:0008 e:000007 CFUNC :to_str
c:0003 p:---- s:0006 e:000005 CFUNC :to_s
c:0002 p:---- s:0004 e:000003 CFUNC :message
c:0001 p:0000 s:0002 E:000f5c TOP [FINISH]

-- Ruby level backtrace information ----------------------------------------
iseq-load-test3.rb:0:in message' iseq-load-test3.rb:0:in to_s'
iseq-load-test3.rb:0:in to_str' iseq-load-test3.rb:0:in to_s'

-- C level backtrace information -------------------------------------------
C:\Windows\SysWOW64\ntdll.dll(ZwWaitForSingleObject+0x15) [0x76EAF8D1]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObjectEx+0x43) [0x76511194]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObject+0x12) [0x76511148]
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_print_backtrace+0x38) [0x52AD8468] p:\code\ruby-git\ruby-trunk\vm_dump.c:712
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_vm_bugreport+0x65) [0x52AD8915] p:\code\ruby-git\ruby-trunk\vm_dump.c:974
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_bug_context+0x74) [0x529D24F4] p:\code\ruby-git\ruby-trunk\error.c:389
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(sigsegv+0x28) [0x52A5D638] p:\code\ruby-git\ruby-trunk\signal.c:850
C:\Windows\system32\MSVCR100.dll(XcptFilter+0x13e) [0x71B5B9DF]
M:\dev\ruby-build\trunk\bin\ruby_t.exe(__tmainCRTStartup+0x14a) [0x00DD11F5] f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c:572
C:\Windows\system32\MSVCR100.dll(seh_longjmp_unwind4+0x2e) [0x71AD2F54]
C:\Windows\syswow64\kernel32.dll(BaseThreadInitThunk+0x12) [0x7651338A]
C:\Windows\SysWOW64\ntdll.dll(RtlInitializeExceptionChain+0x63) [0x76EC9F72]

-- Other runtime information -----------------------------------------------

• Loaded script: iseq-load-test3.rb

The debugger backtrace was:

msvcr100-ruby220.dll!str_replace_shared_without_enc(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 874 + 0x3 bytes C
msvcr100-ruby220.dll!str_replace_shared(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 892 + 0xd bytes C
msvcr100-ruby220.dll!str_new_shared(unsigned long klass=0x0024e76c, unsigned long str=0x00000000) Line 900 + 0x16 bytes C
msvcr100-ruby220.dll!rb_sym_to_s(unsigned long sym=0x0000000e) Line 8599 + 0x19 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52254cd0, unsigned long recv=0x0000000e, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x0000000e, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027b118, unsigned long defined_class=0x0024e0a0) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1abc) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!rb_obj_as_string(unsigned long obj=0x0000000e) Line 1111 + 0x12 bytes C
msvcr100-ruby220.dll!rb_str_format(int argc=0x00000003, const unsigned long * argv=0x004cef90, unsigned long fmt=0x02ea19b8) Line 711 + 0xc bytes C
msvcr100-ruby220.dll!rb_f_sprintf(int argc=0x00000003, const unsigned long * argv=0x004cef90) Line 449 + 0x3e bytes C
msvcr100-ruby220.dll!name_err_mesg_to_str(unsigned long obj=0x02ea3330) Line 1204 + 0xb bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x522142d0, unsigned long recv=0x02ea1abc, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1abc, unsigned long id=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027ef38, unsigned long defined_class=0x0024dbb4) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1abc, unsigned long mid=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!convert_type(unsigned long val=0x02ea1abc, const char * tname=0x523fe264, const char * method=0x523fe25c, int raise=0x00000000) Line 2623 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_convert_type(unsigned long val=0x02ea1abc, int type=0x00000005, const char * tname=0x523fe264, const char * method=0x523fe25c) Line 2673 + 0x13 bytes C
msvcr100-ruby220.dll!rb_check_string_type(unsigned long str=0x02ea1abc) Line 1744 + 0x15 bytes C
msvcr100-ruby220.dll!rb_String(unsigned long val=0x02ea1abc) Line 3022 + 0x9 bytes C
msvcr100-ruby220.dll!exc_to_s(unsigned long exc=0x02ea1a94) Line 693 + 0x9 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213b10, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bbb8, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1a94) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!exc_message(unsigned long exc=0x02ea1a94) Line 709 + 0x1b bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213be0, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bc38, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!error_print() Line 133 + 0x25 bytes C
msvcr100-ruby220.dll!error_handle(int ex=0x00000006) Line 312 C
msvcr100-ruby220.dll!ruby_cleanup(volatile int ex=0x00000006) Line 193 + 0x9 bytes C
msvcr100-ruby220.dll!ruby_run_node(void * n=0x02e55234) Line 309 + 0x12 bytes C
ruby_t.exe!main(int argc=0x00000002, char * * argv=0x002a1678) Line 36 + 0x16 bytes C
ruby_t.exe!__tmainCRTStartup() Line 555 + 0x17 bytes C
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes

It seemed to be dying in an rb_f_sprintf() call. The 'desc' variable in
name_err_mesg_to_str(), which was invoking sprintf, evaluated to:

desc = 0x02e11420 "RubyVM::InstructionSequence:user_mask@./iseq-load-test3-file.rb"

If there's anything I could try that might help narrow it down further,
please let me know.

Thanks again for your help,

Bill

[normalperson (Eric Wong)]

Sorry, the inline patch was an extremely hacky work-in-progress,
but I think [e-mail] should've been OK with your
(non keyword) use cases.

Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/[e-mail]
http://80x24.org/spew/m/[e-mail]

This is on top of r48554, which was purely a cleanup commit in
preparation for this.

Notable changes since v1 (and my WIP v2):

• iseq#to_ary dumps :kwbits => iseq->params.keyword->bits_start

Current hacks:

• peephole optimization seems to be not idempotent wrt jump elimination
  A second pass (optimize => to_ary => load(optimize) eliminates
  extra useless jumps such as:

  jump LABEL
  

  LABEL:

  I currently disable peephole optimization to on load to avoid
  running the optimizer twice, but ideally, the first pass should
  eliminate the above jump insn.

• iseq->stack_size seems not calculated correctly upon load,
  so I'm currently blindly loading it from the misc hash :x

[spatulasnout (B Kelly)]

Eric Wong wrote:

Sorry, the inline patch was an extremely hacky work-in-progress,
but I think [e-mail] should've been OK with your
(non keyword) use cases.

Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/[e-mail]
http://80x24.org/spew/m/[e-mail]

Ah, I see.

Thanks! v3 is indeed no longer segfaulting on any of the iseq tests
I have.

Awesome!

Thanks again,

Bill

[ko1 (Koichi Sasada)]

Seems nice! Thank you!
(maybe we need fix rdoc)

[normalperson (Eric Wong)]

ko1: thank you for checking, any comment on the extra jumps insns which
I hack around by disabling peephole optimization?

It seems caused by the following in iseq_set_sequence:

	    else if (orig_sp - sp == 0) {
		/* jump to next insn */
		if (last_line != (unsigned int)adjust->line_no) {
		    line_info_table[k].line_no = last_line = adjust->line_no;
		    line_info_table[k].position = pos;
		    k++;
		}
		generated_iseq[pos++] = BIN(jump);
		generated_iseq[pos++] = 0;

My roundtrip test fails because:

1. iseq_peephole_optimize called in original
2. iseq_set_sequence generates useless jump (above)
3. ISeq#to_a called (showing useless jump)
4. ISeq#load on 3. array
5. iseq_peephole_optimize now deletes jump emitted in 2.
6. ISeq#to_a now has no useless jumps

So now I'm trying to figure out how to eliminate the useless jump in 2
so roundtrip test always succeeds (and my normal Ruby code has
no useless jump insns)

[ko1 (Koichi Sasada)]

I use nop insns instead of "jump to next insn" (2 words) in r48579.

How about it?

[normalperson (Eric Wong)]

nop; nop works, but I don't think it's an improvement (or regression).
I prefer to save 2 words entirely.

I planned to workaround the issue by doing ISeq#load twice:

first = iseq.to_a       # useless jump|nop here
junk = ISeq.load(first) # second pass optimization removes jump|nop
a = ISeq.load(junk.to_a)
b = ISeq.load(a.to_a)
assert_equal a, b

But I encountered other bugs with to_a + load on test/ruby/test_io.rb
and test_settracefunc.rb which I have not investigated, yet...

Works-in-progress:

[PATCH 1/2] http://80x24.org/spew/m/iseq_calc_param_size%40r48578.txt
[PATCH 2/2] http://80x24.org/spew/m/rb_iseq_load_fix-v4%40r48578.txt

[normalperson (Eric Wong)]

Work-in-progress -v5 based on r48597:

http://80x24.org/spew/m/[e-mail]

Still some failures, particularly around:

   def x(a, (b, *c), d: false); end

I'm not sure if we handle ISeq#to_a correctly for "(b, *c)"
right now...
I might be busy with other stuff for a few days.

[wanabe (_ wanabe)]

https://github.com/ruby/ruby/blob/33ea2646b98adb49ae2e1781753bf22d33729ac0/iseq.c#L1720
iseq_data_to_ary() ignores hidden variables named by id_internal().

It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?

[normalperson (Eric Wong)]

[e-mail] wrote:

iseq_data_to_ary() ignores hidden variables named by id_internal().

It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?

Thank you! Committed as r48678. That seems to fix the

     def x(a, (b, *c), d: false); end

case for me, now #to_a seems to to not be idempotent and segfaults,
investigating.

• Status changed from Open to Closed
• % Done changed from 0 to 100

Applied in changeset r48705.

---

mostly fix rb_iseq_load

This allows reporters commenters of [Feature #8543] to load
instruction sequences directly. Some test cases are still failing
but documented in test/-ext-/iseq_load/test_iseq_load.rb.

• compile.c (rb_iseq_build_from_exception): entry->sp is unsigned
  (iseq_build_callinfo_from_hash): account for kw_arg
  (iseq_build_from_ary_body): update for r35459
  (CHECK_STRING, CHECK_INTEGER): remove unused checks
  (int_param): new function for checking new params' hash (iseq_build_kw): new function for loading rb_iseq_param_keyword (rb_iseq_build_from_ary): account for misc' entry and general
  structure changes
  [Feature #8543]
• iseq.c (CHECK_HASH): new macro (for misc' and param' entries)
  (iseq_load): account for misc' and params' hashes
  (iseq_data_to_ary): add final opt to arg_opt_labels,
  fix kw support, account for unsigned entry->sp
• ext/-test-/iseq_load/iseq_load.c: new ext for test
• ext/-test-/iseq_load/extconf.rb: ditto
• test/-ext-/iseq_load/test_iseq_load.rb: new test

[normalperson (Eric Wong)]

I think the stack mismatches in current trunk are hard to avoid, but the
loaded bytecode is still valid and runnable for the currently-skipped
cases. I propose the following to test more thoroughly.

 diff --git a/test/-ext-/iseq_load/test_iseq_load.rb b/test/-ext-/iseq_load/test_iseq_load.rb
 index 5bbd49e..7251603 100644
 --- a/test/-ext-/iseq_load/test_iseq_load.rb
 +++ b/test/-ext-/iseq_load/test_iseq_load.rb
 @@ -52,16 +52,21 @@ class TestIseqLoad < Test::Unit::TestCase
    end
  
    def test_next_in_block_in_block
 -    skip "failing due to stack_max mismatch"
 -    assert_iseq_roundtrip <<-'end;'
 -      3.times { 3.times { next } }
 +    @next_broke = false
 +    src = <<-'end;'
 +      3.times { 3.times { next; @next_broke = true } }
      end;
 +    a = ISeq.compile(src).to_a
 +    iseq = ISeq.iseq_load(a)
 +    iseq.eval
 +    assert_equal false, @next_broke
 +    skip "failing due to stack_max mismatch"
 +    assert_iseq_roundtrip(src)
    end
  
    def test_break_ensure
 -    skip "failing due to exception entry sp mismatch"
 -    assert_iseq_roundtrip <<-'end;'
 -      def m
 +    src = <<-'end;'
 +      def test_break_ensure_def_method
          bad = true
          while true
            begin
 @@ -70,8 +75,15 @@ class TestIseqLoad < Test::Unit::TestCase
              bad = false
            end
          end
 +        bad
        end
      end;
 +    a = ISeq.compile(src).to_a
 +    iseq = ISeq.iseq_load(a)
 +    iseq.eval
 +    assert_equal false, test_break_ensure_def_method
 +    skip "failing due to exception entry sp mismatch"
 +    assert_iseq_roundtrip(src)
    end
  
    # FIXME: still failing

[spatulasnout (B Kelly)]

• File iseq-load-test5.rb iseq-load-test5.rb added
• Subject changed from rb_iseq_load to new rb_iseq_load crash

Hi,

We've encountered a new crash related to RubyVM::InstructionSequence.load.

So far it has been reproducible on multiple platforms:

ruby 2.2.3p158 (2015-08-12 revision 48555) [i386-mswin32_100]
ruby 2.2.3p158 (2015-08-12 revision 51190) [i386-darwin13]

Attached is a small program which reproduces the crash on our systems.
(The attached file contains further details about the crash.)

[spatulasnout (B Kelly)]

Addendum: Attached file in previous post also reproduces the crash
in the latest ruby trunk:

:3: [BUG] Segmentation fault
ruby 2.3.0dev (2015-09-09 trunk 48555) [i386-mswin32_100]

[nagachika (Tomoyuki Chikanaga)]

• Tracker changed from Feature to Bug
• ruby -v set to -
• Backport set to 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: REQUIRED

At least r51816 should be backported into 2.1/2.2.

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_2 branch at r52740.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: UNKNOWN, 2.1: REQUIRED, 2.2: DONE to 2.0.0: UNKNOWN, 2.1: DONTNEED, 2.2: DONE

2.0/2.1 use different code about r51816.
So they don't have this problem, I think.