Feature #8543

rb_iseq_load

Added by Alexey Voskov almost 2 years ago. Updated 5 months ago.

[ruby-core:55557]
Status:Closed
Priority:Low
Assignee:Koichi Sasada

Description

I noticed an unusual behaviour of undocumented rb_iseq_load function.
Its work differs in different Ruby versions. I'm trying to protect some Ruby
source code by its conversion to YARV p-code and using the next strategy:

  1. Convert code to array
   data = RubyVM::InstructionSequence.compile_file('hello.rb').to_a
  1. Pass a compiled source to the rb_iseq_load function and evaluate it
   iseq = iseq_load.(data)
   iseq.eval

Sample programs are supplied in the attachments.
"hello.rb"

puts "tralivali"
def funct(a,b)
  a**b
end

3.times { |i|
  puts "Hello, world#{funct(2,i)}!"
}

The differences
Ruby 1.9.3 (ruby 1.9.3p194 (2012-04-20) [i386-mingw32])
Correct work. Output:

tralivali
Hello, world1!
Hello, world2!
Hello, world4!

Ruby 2.0.0 (ruby 2.0.0p193 (2013-05-14) [x64-mingw32])
Incorrect work (omits the code inside code blocks). Output

tralivali

Attempts of loading bigger programs by means of rb_iseq_load in Ruby 2.0.0 usually ends with a segmentation fault.

Such behaviour also can be reproduced by means of iseq Ruby extension ("for iseq freaks")
https://github.com/wanabe/iseq

P.S. I understand that it is an undocumented feature.

hello.rb Magnifier - Codes for conversion to YARV PCODE (102 Bytes) Alexey Voskov, 06/19/2013 09:39 PM

rb_pack.rb Magnifier - "Compiler" (931 Bytes) Alexey Voskov, 06/19/2013 09:39 PM

iseq-load-test3.rb Magnifier - Program which calls RubyVM::InstructionSequence.load (210 Bytes) B Kelly, 04/16/2014 10:03 AM

iseq-load-test3-file.rb Magnifier - Program fragment which triggers SEGV when loaded by rb_iseq_load (369 Bytes) B Kelly, 04/16/2014 10:03 AM

please-fix-rb_iseq_load-thank-you.pdf - PDF slide describing our use case for rb_iseq_load, per: [ruby-core:63332] (444 KB) B Kelly, 06/26/2014 11:36 PM

Associated revisions

Revision 48705
Added by normal 6 months ago

mostly fix rb_iseq_load

This allows reporters commenters of [Feature #8543] to load
instruction sequences directly. Some test cases are still failing
but documented in test/-ext-/iseq_load/test_iseq_load.rb.

  • compile.c (rb_iseq_build_from_exception): entry->sp is unsigned (iseq_build_callinfo_from_hash): account for kw_arg (iseq_build_from_ary_body): update for r35459 (CHECK_STRING, CHECK_INTEGER): remove unused checks (int_param): new function for checking new params' hash (iseq_build_kw): new function for loading rb_iseq_param_keyword (rb_iseq_build_from_ary): account formisc' entry and general structure changes [Feature #8543]
  • iseq.c (CHECK_HASH): new macro (for misc' andparam' entries) (iseq_load): account for misc' andparams' hashes (iseq_data_to_ary): add final opt to arg_opt_labels, fix kw support, account for unsigned entry->sp
  • ext/-test-/iseq_load/iseq_load.c: new ext for test
  • ext/-test-/iseq_load/extconf.rb: ditto
  • test/-ext-/iseq_load/test_iseq_load.rb: new test

Revision 48705
Added by normal 6 months ago

mostly fix rb_iseq_load

This allows reporters commenters of [Feature #8543] to load
instruction sequences directly. Some test cases are still failing
but documented in test/-ext-/iseq_load/test_iseq_load.rb.

  • compile.c (rb_iseq_build_from_exception): entry->sp is unsigned (iseq_build_callinfo_from_hash): account for kw_arg (iseq_build_from_ary_body): update for r35459 (CHECK_STRING, CHECK_INTEGER): remove unused checks (int_param): new function for checking new params' hash (iseq_build_kw): new function for loading rb_iseq_param_keyword (rb_iseq_build_from_ary): account formisc' entry and general structure changes [Feature #8543]
  • iseq.c (CHECK_HASH): new macro (for misc' andparam' entries) (iseq_load): account for misc' andparams' hashes (iseq_data_to_ary): add final opt to arg_opt_labels, fix kw support, account for unsigned entry->sp
  • ext/-test-/iseq_load/iseq_load.c: new ext for test
  • ext/-test-/iseq_load/extconf.rb: ditto
  • test/-ext-/iseq_load/test_iseq_load.rb: new test

History

#1 Updated by Tomoyuki Chikanaga almost 2 years ago

  • Tracker changed from Misc to Bug

#2 Updated by Tomoyuki Chikanaga almost 2 years ago

  • Backport set to 1.9.3: DONTNEED, 2.0.0: REQUIRED
  • ruby -v set to ruby 2.0.0p234 (2013-06-19 revision 41434) [x86_64-darwin11.4.2]
  • Priority changed from Normal to Low

#3 Updated by Hiroshi SHIBATA over 1 year ago

  • Target version changed from 2.1.0 to current: 2.2.0

#4 Updated by B Kelly about 1 year ago

Hi,

Just a data point:

With ruby 2.2.0dev (2014-04-16 trunk 45576) [i386-mswin32_100],
the rb_iseq_load example in https://bugs.ruby-lang.org/issues/8543
is producing the correct output.

(I will attempt to test with more complicated programs soon.)

Regards,

Bill

#5 Updated by B Kelly about 1 year ago

Hello again,

The attached files comprise a small example which appears to consistently
reproduce a segmentation fault with rb_iseq_load.

Interpreter is: ruby 2.2.0dev (2014-04-16 trunk 45597) [i386-mswin32_100]

The crash appears related to the case statement.

Call stack:

The st_table pointer at st_foreach_check is bad:

  msvcr100-ruby220.dll!st_foreach_check(st_table * table=0x00000008, int (void)* func=0x0f3f3580, unsigned long arg=4512588, unsigned long never=6)  Line 891 + 0x3 bytes C
  msvcr100-ruby220.dll!hash_foreach_call(unsigned long arg=4512588)  Line 258 + 0x17 bytes        C
  msvcr100-ruby220.dll!rb_ensure(unsigned long (void)* b_proc=0x0f3f3530, unsigned long data1=4512588, unsigned long (void)* e_proc=0x0f3f3480, unsigned long data2=10318080)  Line 888 + 0x7 bytes       C
  msvcr100-ruby220.dll!rb_hash_foreach(unsigned long hash=10318080, int (void)* func=0x0f45ac20, unsigned long farg=4512696)  Line 275 + 0x17 bytes       C
> msvcr100-ruby220.dll!iseq_set_sequence(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58)  Line 1507 + 0x12 bytes  C
  msvcr100-ruby220.dll!iseq_setup(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58)  Line 1026 + 0xd bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d279b0, iseq_link_anchor * anchor=0x0044dd58, unsigned long body=10318900, st_table * labels_table=0x00d27d10)  Line 5855 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d279b0, unsigned long locals=10320520, unsigned long args=5, unsigned long exception=10320460, unsigned long body=10318900)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320540, unsigned long parent=10318300, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27500, unsigned long op=10320540)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27500, iseq_link_anchor * anchor=0x0044dfd4, unsigned long body=10318680, st_table * labels_table=0x00d27860)  Line 5816 + 0x13 bytes  C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27500, unsigned long locals=10320900, unsigned long args=3, unsigned long exception=10320840, unsigned long body=10318680)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10320920, unsigned long parent=10318380, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d27110, unsigned long op=10320920)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d27110, iseq_link_anchor * anchor=0x0044e250, unsigned long body=10318560, st_table * labels_table=0x00d27470)  Line 5783 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d27110, unsigned long locals=10321160, unsigned long args=1, unsigned long exception=10321100, unsigned long body=10318560)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4)  Line 561 + 0x19 bytes     C
  msvcr100-ruby220.dll!rb_iseq_load(unsigned long data=10321180, unsigned long parent=10318460, unsigned long opt=4)  Line 582 + 0x17 bytes       C
  msvcr100-ruby220.dll!iseq_build_load_iseq(rb_iseq_struct * iseq=0x00d25c08, unsigned long op=10321180)  Line 5701 + 0x15 bytes  C
  msvcr100-ruby220.dll!iseq_build_from_ary_body(rb_iseq_struct * iseq=0x00d25c08, iseq_link_anchor * anchor=0x0044e4cc, unsigned long body=10318500, st_table * labels_table=0x00d25f68)  Line 5783 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_build_from_ary(rb_iseq_struct * iseq=0x00d25c08, unsigned long locals=10339520, unsigned long args=1, unsigned long exception=10339460, unsigned long body=10318500)  Line 5932 + 0x15 bytes       C
  msvcr100-ruby220.dll!iseq_load(unsigned long self=10542580, unsigned long data=10339540, unsigned long parent=0, unsigned long opt=4)  Line 561 + 0x19 bytes    C
  msvcr100-ruby220.dll!iseq_s_load(int argc=1, unsigned long * argv=0x0057005c, unsigned long self=10542580)  Line 576 + 0x13 bytes       C
  msvcr100-ruby220.dll!call_cfunc_m1(unsigned long (void)* func=0x0f443b50, unsigned long recv=10542580, int argc=1, const unsigned long * argv=0x0057005c)  Line 1330 + 0xf bytes        C
  msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1502 + 0x20 bytes      C
  msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1592 + 0x11 bytes C
  msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1786 + 0x11 bytes    C
  msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005eff80, rb_call_info_struct * ci=0x00cdcfc0)  Line 1941 + 0x11 bytes       C
  msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0)  Line 1028 + 0x1a bytes    C
  msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588)  Line 1328 + 0xd bytes   C
  msvcr100-ruby220.dll!invoke_block_from_c(rb_thread_struct * th=0x00c15588, const rb_block_struct * block=0x005effe0, unsigned long self=10187260, int argc=1, const unsigned long * argv=0x0044f0ec, const rb_block_struct * blockptr=0x00000000, const RNode * cref=0x00000000, unsigned long defined_class=4, int splattable=1)  Line 752 + 0x9 bytes   C
  msvcr100-ruby220.dll!vm_yield(rb_thread_struct * th=0x00c15588, int argc=1, const unsigned long * argv=0x0044f0ec)  Line 784 + 0x28 bytes       C
  msvcr100-ruby220.dll!rb_yield_0(int argc=1, const unsigned long * argv=0x0044f0ec)  Line 936 + 0x13 bytes       C
  msvcr100-ruby220.dll!rb_yield(unsigned long val=10340180)  Line 946 + 0xb bytes C
  msvcr100-ruby220.dll!rb_ary_each(unsigned long array=10342440)  Line 1806 + 0x2f bytes  C
  msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x0f3cd6b0, unsigned long recv=10342440, int argc=0, const unsigned long * argv=0x00570040)  Line 1336 + 0x7 bytes C
  msvcr100-ruby220.dll!vm_call_cfunc_with_frame(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1502 + 0x20 bytes      C
  msvcr100-ruby220.dll!vm_call_cfunc(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1592 + 0x11 bytes C
  msvcr100-ruby220.dll!vm_call_method(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1786 + 0x11 bytes    C
  msvcr100-ruby220.dll!vm_call_general(rb_thread_struct * th=0x00c15588, rb_control_frame_struct * reg_cfp=0x005effd0, rb_call_info_struct * ci=0x00cdc5a8)  Line 1941 + 0x11 bytes       C
  msvcr100-ruby220.dll!vm_exec_core(rb_thread_struct * th=0x00c15588, unsigned long initial=0)  Line 999 + 0x1a bytes     C
  msvcr100-ruby220.dll!vm_exec(rb_thread_struct * th=0x00c15588)  Line 1328 + 0xd bytes   C
  msvcr100-ruby220.dll!rb_iseq_eval_main(unsigned long iseqval=10505960)  Line 1586 + 0x9 bytes   C
  msvcr100-ruby220.dll!ruby_exec_internal(void * n=0x00a04ee8)  Line 254 + 0x46 bytes     C
  msvcr100-ruby220.dll!ruby_exec_node(void * n=0x00a04ee8)  Line 316 + 0x9 bytes  C
  msvcr100-ruby220.dll!ruby_run_node(void * n=0x00a04ee8)  Line 308 + 0x9 bytes   C
  ruby_t.exe!main(int argc=3, char * * argv=0x00c115b8)  Line 36 + 0x16 bytes     C
  ruby_t.exe!__tmainCRTStartup()  Line 555 + 0x17 bytes   C


In the iseq_set_sequence above marked by ">", the bad hash value was
fetched from operands[j]:

                case TS_CDHASH:
                  {
                      VALUE map = operands[j];
                      struct cdhash_set_label_struct data;
                      data.hash = map;
                      data.pos = pos;
                      data.len = len;
                      rb_hash_foreach(map, cdhash_set_label_i, (VALUE)&data);

                      hide_obj(map);
                      generated_iseq[pos + 1 + j] = map;
                      break;
                  }

And the iobj->insn_id is consistently YARVINSN_opt_case_dispatch:

    iobj            0x00d27c6c {link={...} insn_id=YARVINSN_opt_case_dispatch line_no=5 ...}        iseq_insn_data *
    link            {type=ISEQ_ELEMENT_INSN next=0x00d27c8c prev=0x00d27ba4 }       iseq_link_element
    insn_id         YARVINSN_opt_case_dispatch      ruby_vminsn_type
    line_no         5       unsigned int
    operand_size    2       int
    sc_state        0       int

I had invoked the program with --disable-gems for simplicity, however
the crash occurs either way.

If there's any further information I could provide please let me know.

Thanks for your help!

Regards,

Bill

#6 Updated by B Kelly 11 months ago

Salutations,

Attached is a one-page PDF slide for the Ruby 2.2 feature proposal
developer meeting. It describes our use case for rb_iseq_load, per:
[ANN] Request for "slide-show" of your feature proposal

Thank you!

Bill

#7 Updated by Yui NARUSE 11 months ago

received, thanks!

#8 Updated by Koichi Sasada 10 months ago

Thank you for your report.

I'll check it (best effort. I ask my boss about priority).

#9 Updated by Nobuyoshi Nakada 10 months ago

  • Tracker changed from Bug to Feature
  • Description updated (diff)

#10 Updated by B Kelly 8 months ago

Hi,

Koichi Sasada wrote:

Thank you for your report.

I'll check it (best effort. I ask my boss about priority).

Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?

I attempted a git bisect this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:

commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000

* parse.y: fix build error with bison-3.0.

And unfortunately the iseq.load problem already existed by that
point.

(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)

Very much hoping iseq.load might be fixable before 2.2 code
freeze.

Thanks & Regards,

Bill

#11 Updated by Eric Wong 8 months ago

billk@cts.com wrote:

Sorry to be the squeaky wheel, but I was wondering if there still
might be a chance to look into this before 2.2 is released?

Not speaking for the rest of ruby-core, but I welcome occasional
reminders like these :)

I attempted a git bisect this evening in the hope of narrowing
down where the iseq.load problems began -- however I ran into a
problem of being unable to build ruby-trunk prior to this patch:

commit 434826c0e9d3e3b48d99a39b7ad7626a6f1ae2eb
Author: kazu kazu@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Wed Jul 31 13:01:57 2013 +0000

* parse.y: fix build error with bison-3.0.

And unfortunately the iseq.load problem already existed by that
point.

Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?

(I wonder how best to proceed with the bisect. I suppose one
could write a script to attempt to apply the parse.y patch at
each stage...)

Yes. "git bisect run" is awesome for scripting these things.

#12 Updated by B Kelly 8 months ago

Eric Wong wrote:

Thanks for that data point, it was before I started mucking with iseq.
Can you try installing/running an older bison?

Good call -- cygwin indeed allowed me to roll back to bison 2.7.x

I wasn't able to fully automate git bisect as there was various build
breakage on some of the commits (usually to do with changes in enc and
enc/trans.)

But ultimately, the result of the manual bisect was:

66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000

* compile.c: fix to output warning when the same literals
  are available as a condition of same case clause.
  And remove infomation ('#n') because we can find duplicated
  condition with explicit line numbers.
   [Ruby 1.9 - Bug #5068]
* test/ruby/test_syntax.rb: add a test for above.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

:100644 100644 bcde6499fd43af4fc7eae9496d7eb529e52d5465 5f48bda3d2c5787acfc93c7d209964b45b4405bf M ChangeLog
:100644 100644 508d599d081ddd3676efc513c25f76f00216116b 74982db138d5432f0077c49265e7b177a906ec97 M compile.c
:040000 040000 b0608f2c1a2b0eaab543fb26ac4c2a78cb9b0c57 4f78f071cacaf4c8da9d8ccfecb027092f94bc54 M test

My test script was running both Alexey Voskov's "tralivali" test as
well as my segfault test relating to the case statement.

I suspect the above commit is what introduced the case statement-
related segfault. I'm not sure if it will also relate to Alexey
Voskov's "tralivali" test well, or whether that might be a separate
issue. (But if the above can be fixed, I'm happy to try another
bisect if other issues remain.)

Thanks & Regards,

Bill

#13 Updated by Eric Wong 6 months ago

Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/rb_iseq_load_fix-wip@v0.txt
Thanks.

#14 Updated by Eric Wong 6 months ago

Eric Wong normalperson@yhbt.net wrote:

Work-in-progress fix here, can you please test?
http://80x24.org/spew/m/rb_iseq_load_fix-wip@v0.txt

Don't bother with that, I forgot to test iseq-load-test3-file.rb
hello.rb works fine, at least :x Fixing...

#15 Updated by Eric Wong 6 months ago

billk@cts.com wrote:

But ultimately, the result of the manual bisect was:

66d247bcb50a29769ff940100223544c125521aa is the first bad commit
commit 66d247bcb50a29769ff940100223544c125521aa
Author: ko1 ko1@b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Date: Tue Apr 24 09:20:42 2012 +0000

* compile.c: fix to output warning when the same literals
  are available as a condition of same case clause.
  And remove infomation ('#n') because we can find duplicated
  condition with explicit line numbers.
   [Ruby 1.9 - Bug #5068]
* test/ruby/test_syntax.rb: add a test for above.

That was only one of the breakages :)
Things have bitrotted a lot over the years.

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/rb_iseq_load_fix@v1.txt

It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.

There's no public API change in this version of my patch.

The test cases are probably not complete, though. I am mainly
interested in the feature because I'm working on another (in-core)
optimization which may utilize this.

I can definitely use some help thinking of better test cases since I
don't believe I remember all of Ruby syntax :x

#16 Updated by Eric Wong 6 months ago

Eric Wong normalperson@yhbt.net wrote:

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/rb_iseq_load_fix@v1.txt

Still some bugs related to keyword args I'm working on, but am out of
time for the moment:

http://80x24.org/spew/m/20141123055537.GA1002%40dcvr.yhbt.net.txt

#17 Updated by B Kelly 6 months ago

Howdy,

Eric Wong wrote:

That was only one of the breakages :)
Things have bitrotted a lot over the years.

The following patch might be ready to commit to trunk:

http://80x24.org/spew/m/rb_iseq_load_fix@v1.txt

It's better than the complete breakage we have right now, so I might
commit the above in a few days. The new test case I added should
help (or force) other core committers to maintain iseq loading,
though.

Thanks so much for working on this.

Just some initial feedback -- after applying both patches to trunk,
I'm still seeing a segfault in the iseq.eval call on iseq-load-test3.rb,
though in a different location than before.

ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
ci->kw_arg 00000000
omg: {:mid=>:each_char, :flag=>256, :orig_argc=>0, :blockptr=>nil}
omg: {:mid=>:inject, :flag=>0, :orig_argc=>1, :blockptr=>["YARVInstructionSequence/SimpleDataFormat", 2, 2, 1, {:arg_size=>2, :local_size=>3, :stack_max=>4}, "block in user_mask", "./iseq-load-test3-file.rb", "P:/code/ruby/bkelly/test/iseq-load-test3-file.rb", 4, :block, [:mask, :chr], {:lead_num=>2}, [[:redo, nil, :label_2, :label_102, :label_2, 0], [:next, nil, :label_2, :label_102, :label_102, 0]], [4, [:trace, 256], :label_2, 5, [:trace, 1], [:getlocal_OP_WC0, 2], [:dup], [:opt_case_dispatch, ["u", :label_60, "g", :label_71, "o", :label_82, "a", :label_93], :label_38], 6, [:dup], [:putobject, "u"], [:checkmatch, 2], [:branchif, :label_60], 8, [:dup], [:putobject, "g"], [:checkmatch, 2], [:branchif, :label_71], 10, [:dup], [:putobject, "o"], [:checkmatch, 2], [:branchif, :label_82], 12, [:dup], [:putobject, "a"], [:checkmatch, 2], [:branchif, :label_93], :label_38, 15, [:pop], [:trace, 1], [:putself], [:getinlinecache, :label_49, 0], [:getconstant, :ArgumentError], [:setinlinecache, 0], :label_49, [:putobject, "invalid `who' symbol in file mode: "], [:getlocal_OPWC0, 2], [:tostring], [:concatstrings, 2], [:opt_send_without_block, {:mid=>:raise, :flag=>264, :orig_argc=>2, :blockptr=>nil}], [:jump, :label_102], :label_60, 16, [:pop], 7, [:trace, 1], [:getlocal_OPWC0, 3], [:putobject, 2496], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_71, [:pop], 9, [:trace, 1], [:getlocal_OPWC0, 3], [:putobject, 1080], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_82, [:pop], 11, [:trace, 1], [:getlocal_OPWC0, 3], [:putobject, 519], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], 16, [:jump, :label_102], :label_93, [:pop], 13, [:trace, 1], [:getlocal_OPWC0, 3], [:putobject, 4095], [:opt_send_without_block, {:mid=>:|, :flag=>256, :orig_argc=>1, :blockptr=>nil}], :label_102, 17, [:trace, 512], 15, [:leave]]]}
omg: {:mid=>:"core#define_method", :flag=>256, :orig
argc=>3, :blockptr=>nil}
./iseq-load-test3-file.rb:3:in `module:FileUtils'iseq-load-test3.rb: [BUG] Segmentation fault
ruby 2.2.0dev (2014-11-24 trunk 48553) [i386-mswin32_100]

-- Control frame information -----------------------------------------------
c:0005 p:---- s:0010 e:000009 CFUNC :to_s
c:0004 p:---- s:0008 e:000007 CFUNC :to_str
c:0003 p:---- s:0006 e:000005 CFUNC :to_s
c:0002 p:---- s:0004 e:000003 CFUNC :message
c:0001 p:0000 s:0002 E:000f5c TOP [FINISH]

-- Ruby level backtrace information ----------------------------------------
iseq-load-test3.rb:0:in message'
iseq-load-test3.rb:0:in
to_s'
iseq-load-test3.rb:0:in to_str'
iseq-load-test3.rb:0:in
to_s'

-- C level backtrace information -------------------------------------------
C:\Windows\SysWOW64\ntdll.dll(ZwWaitForSingleObject+0x15) [0x76EAF8D1]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObjectEx+0x43) [0x76511194]
C:\Windows\syswow64\kernel32.dll(WaitForSingleObject+0x12) [0x76511148]
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_print_backtrace+0x38) [0x52AD8468] p:\code\ruby-git\ruby-trunk\vm_dump.c:712
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_vm_bugreport+0x65) [0x52AD8915] p:\code\ruby-git\ruby-trunk\vm_dump.c:974
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(rb_bug_context+0x74) [0x529D24F4] p:\code\ruby-git\ruby-trunk\error.c:389
M:\dev\ruby-build\trunk\bin\msvcr100-ruby220.dll(sigsegv+0x28) [0x52A5D638] p:\code\ruby-git\ruby-trunk\signal.c:850
C:\Windows\system32\MSVCR100.dll(XcptFilter+0x13e) [0x71B5B9DF]
M:\dev\ruby-build\trunk\bin\ruby_t.exe(__tmainCRTStartup+0x14a) [0x00DD11F5] f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c:572
C:\Windows\system32\MSVCR100.dll(seh_longjmp_unwind4+0x2e) [0x71AD2F54]
C:\Windows\syswow64\kernel32.dll(BaseThreadInitThunk+0x12) [0x7651338A]
C:\Windows\SysWOW64\ntdll.dll(RtlInitializeExceptionChain+0x63) [0x76EC9F72]

-- Other runtime information -----------------------------------------------

  • Loaded script: iseq-load-test3.rb

The debugger backtrace was:

msvcr100-ruby220.dll!str_replace_shared_without_enc(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 874 + 0x3 bytes C
msvcr100-ruby220.dll!str_replace_shared(unsigned long str2=0x02ea1990, unsigned long str=0x00000000) Line 892 + 0xd bytes C
msvcr100-ruby220.dll!str_new_shared(unsigned long klass=0x0024e76c, unsigned long str=0x00000000) Line 900 + 0x16 bytes C
msvcr100-ruby220.dll!rb_sym_to_s(unsigned long sym=0x0000000e) Line 8599 + 0x19 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52254cd0, unsigned long recv=0x0000000e, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004ceadc, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x0000000e, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027b118, unsigned long defined_class=0x0024e0a0) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1abc) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x0000000e, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!rb_obj_as_string(unsigned long obj=0x0000000e) Line 1111 + 0x12 bytes C
msvcr100-ruby220.dll!rb_str_format(int argc=0x00000003, const unsigned long * argv=0x004cef90, unsigned long fmt=0x02ea19b8) Line 711 + 0xc bytes C
msvcr100-ruby220.dll!rb_f_sprintf(int argc=0x00000003, const unsigned long * argv=0x004cef90) Line 449 + 0x3e bytes C
msvcr100-ruby220.dll!name_err_mesg_to_str(unsigned long obj=0x02ea3330) Line 1204 + 0xb bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x522142d0, unsigned long recv=0x02ea1abc, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf0f0, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1abc, unsigned long id=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027ef38, unsigned long defined_class=0x0024dbb4) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1abc, unsigned long mid=0x00000c41, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!convert_type(unsigned long val=0x02ea1abc, const char * tname=0x523fe264, const char * method=0x523fe25c, int raise=0x00000000) Line 2623 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_convert_type(unsigned long val=0x02ea1abc, int type=0x00000005, const char * tname=0x523fe264, const char * method=0x523fe25c) Line 2673 + 0x13 bytes C
msvcr100-ruby220.dll!rb_check_string_type(unsigned long str=0x02ea1abc) Line 1744 + 0x15 bytes C
msvcr100-ruby220.dll!rb_String(unsigned long val=0x02ea1abc) Line 3022 + 0x9 bytes C
msvcr100-ruby220.dll!exc_to_s(unsigned long exc=0x02ea1a94) Line 693 + 0x9 bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213b10, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf348, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bbb8, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_call0(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL, unsigned long self=0x02ea1a94) Line 348 + 0x21 bytes C
msvcr100-ruby220.dll!rb_call(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int argc=0x00000000, const unsigned long * argv=0x00000000, call_type scope=CALL_FCALL) Line 610 + 0x23 bytes C
msvcr100-ruby220.dll!rb_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00000ca1, int n=0x00000000, ...) Line 812 + 0x17 bytes C
msvcr100-ruby220.dll!exc_message(unsigned long exc=0x02ea1a94) Line 709 + 0x1b bytes C
msvcr100-ruby220.dll!call_cfunc_0(unsigned long (void)* func=0x52213be0, unsigned long recv=0x02ea1a94, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 1193 + 0x7 bytes C
msvcr100-ruby220.dll!vm_call0_cfunc_with_frame(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 127 + 0x1a bytes C
msvcr100-ruby220.dll!vm_call0_cfunc(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 144 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0_body(rb_thread_struct * th=0x002ab438, rb_call_info_struct * ci=0x004cf580, const unsigned long * argv=0x00000000) Line 184 + 0x11 bytes C
msvcr100-ruby220.dll!vm_call0(rb_thread_struct * th=0x002ab438, unsigned long recv=0x02ea1a94, unsigned long id=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000, const rb_method_entry_struct * me=0x0027bc38, unsigned long defined_class=0x0024e028) Line 59 + 0x11 bytes C
msvcr100-ruby220.dll!rb_check_funcall(unsigned long recv=0x02ea1a94, unsigned long mid=0x00001c61, int argc=0x00000000, const unsigned long * argv=0x00000000) Line 449 + 0x21 bytes C
msvcr100-ruby220.dll!error_print() Line 133 + 0x25 bytes C
msvcr100-ruby220.dll!error_handle(int ex=0x00000006) Line 312 C
msvcr100-ruby220.dll!ruby_cleanup(volatile int ex=0x00000006) Line 193 + 0x9 bytes C
msvcr100-ruby220.dll!ruby_run_node(void * n=0x02e55234) Line 309 + 0x12 bytes C
ruby_t.exe!main(int argc=0x00000002, char * * argv=0x002a1678) Line 36 + 0x16 bytes C
ruby_t.exe!tmainCRTStartup() Line 555 + 0x17 bytes C
kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes

ntdll.dll!
RtlUserThreadStart@8() + 0x27 bytes

ntdll.dll!
_RtlUserThreadStart@8() + 0x1b bytes

It seemed to be dying in an rb_f_sprintf() call. The 'desc' variable in
name_err_mesg_to_str(), which was invoking sprintf, evaluated to:

desc = 0x02e11420 "RubyVM::InstructionSequence:user_mask@./iseq-load-test3-file.rb"

If there's anything I could try that might help narrow it down further,
please let me know.

Thanks again for your help,

Bill

#18 Updated by Eric Wong 6 months ago

Sorry, the inline patch was an extremely hacky work-in-progress,
but I think rb_iseq_load_fix@v1.txt should've been OK with your
(non keyword) use cases.

Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt

This is on top of r48554, which was purely a cleanup commit in
preparation for this.

Notable changes since v1 (and my WIP v2):
- iseq#to_ary dumps :kwbits => iseq->params.keyword->bits_start

Current hacks:

  • peephole optimization seems to be not idempotent wrt jump elimination
    A second pass (optimize => to_ary => load(optimize) eliminates
    extra useless jumps such as:

    jump LABEL
    LABEL:

I currently disable peephole optimization to on load to avoid
running the optimizer twice, but ideally, the first pass should
eliminate the above jump insn.

  • iseq->stack_size seems not calculated correctly upon load, so I'm currently blindly loading it from the misc hash :x

#19 Updated by B Kelly 6 months ago

Eric Wong wrote:

Sorry, the inline patch was an extremely hacky work-in-progress,
but I think rb_iseq_load_fix@v1.txt should've been OK with your
(non keyword) use cases.

Here is a slightly less broken, but still hacky work-in-progress:
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt
http://80x24.org/spew/m/rb_iseq_load_fix@v3.txt

Ah, I see.

Thanks! v3 is indeed no longer segfaulting on any of the iseq tests
I have.

Awesome!

Thanks again,

Bill

#20 Updated by Koichi Sasada 6 months ago

Seems nice! Thank you!
(maybe we need fix rdoc)

#21 Updated by Eric Wong 6 months ago

ko1: thank you for checking, any comment on the extra jumps insns which
I hack around by disabling peephole optimization?

It seems caused by the following in iseq_set_sequence:

        else if (orig_sp - sp == 0) {
        /* jump to next insn */
        if (last_line != (unsigned int)adjust->line_no) {
            line_info_table[k].line_no = last_line = adjust->line_no;
            line_info_table[k].position = pos;
            k++;
        }
        generated_iseq[pos++] = BIN(jump);
        generated_iseq[pos++] = 0;

My roundtrip test fails because:

1. iseq_peephole_optimize called in original
2. iseq_set_sequence generates useless jump (above)
3. ISeq#to_a called (showing useless jump)
4. ISeq#load on 3. array
5. iseq_peephole_optimize now deletes jump emitted in 2.
6. ISeq#to_a now has no useless jumps

So now I'm trying to figure out how to eliminate the useless jump in 2
so roundtrip test always succeeds (and my normal Ruby code has
no useless jump insns)

#22 Updated by Koichi Sasada 6 months ago

I use nop insns instead of "jump to next insn" (2 words) in r48579.

How about it?

#23 Updated by Eric Wong 6 months ago

nop; nop works, but I don't think it's an improvement (or regression).
I prefer to save 2 words entirely.

I planned to workaround the issue by doing ISeq#load twice:

first = iseq.to_a       # useless jump|nop here
junk = ISeq.load(first) # second pass optimization removes jump|nop
a = ISeq.load(junk.to_a)
b = ISeq.load(a.to_a)
assert_equal a, b

But I encountered other bugs with to_a + load on test/ruby/test_io.rb
and test_settracefunc.rb which I have not investigated, yet...

Works-in-progress:

[PATCH 1/2] http://80x24.org/spew/m/iseq_calc_param_size%40r48578.txt
[PATCH 2/2] http://80x24.org/spew/m/rb_iseq_load_fix-v4%40r48578.txt

#24 Updated by Eric Wong 6 months ago

Work-in-progress -v5 based on r48597:

http://80x24.org/spew/m/rb_iseq_load-fix-v5@r48597.txt

Still some failures, particularly around:

   def x(a, (b, *c), d: false); end

I'm not sure if we handle ISeq#to_a correctly for "(b, *c)"
right now...
I might be busy with other stuff for a few days.

#25 Updated by _ wanabe 6 months ago

https://github.com/ruby/ruby/blob/33ea2646b98adb49ae2e1781753bf22d33729ac0/iseq.c#L1720
iseq_data_to_ary() ignores hidden variables named by id_internal().

It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?

#26 Updated by Eric Wong 6 months ago

s.wanabe@gmail.com wrote:

iseq_data_to_ary() ignores hidden variables named by id_internal().

It seems to be mismatched for the variable that recieving keyword arguments, for aforementioned reason.
How about push rb_uint_new(lid) when rb_id2str(lid) is false?

Thank you! Committed as r48678. That seems to fix the

     def x(a, (b, *c), d: false); end

case for me, now #to_a seems to to not be idempotent and segfaults,
investigating.

#27 Updated by Anonymous 6 months ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Applied in changeset r48705.


mostly fix rb_iseq_load

This allows reporters commenters of [Feature #8543] to load
instruction sequences directly. Some test cases are still failing
but documented in test/-ext-/iseq_load/test_iseq_load.rb.

  • compile.c (rb_iseq_build_from_exception): entry->sp is unsigned (iseq_build_callinfo_from_hash): account for kw_arg (iseq_build_from_ary_body): update for r35459 (CHECK_STRING, CHECK_INTEGER): remove unused checks (int_param): new function for checking new params' hash (iseq_build_kw): new function for loading rb_iseq_param_keyword (rb_iseq_build_from_ary): account formisc' entry and general structure changes [Feature #8543]
  • iseq.c (CHECK_HASH): new macro (for misc' andparam' entries) (iseq_load): account for misc' andparams' hashes (iseq_data_to_ary): add final opt to arg_opt_labels, fix kw support, account for unsigned entry->sp
  • ext/-test-/iseq_load/iseq_load.c: new ext for test
  • ext/-test-/iseq_load/extconf.rb: ditto
  • test/-ext-/iseq_load/test_iseq_load.rb: new test

#28 Updated by Eric Wong 5 months ago

I think the stack mismatches in current trunk are hard to avoid, but the
loaded bytecode is still valid and runnable for the currently-skipped
cases. I propose the following to test more thoroughly.

 diff --git a/test/-ext-/iseq_load/test_iseq_load.rb b/test/-ext-/iseq_load/test_iseq_load.rb
 index 5bbd49e..7251603 100644
 --- a/test/-ext-/iseq_load/test_iseq_load.rb
 +++ b/test/-ext-/iseq_load/test_iseq_load.rb
 @@ -52,16 +52,21 @@ class TestIseqLoad < Test::Unit::TestCase
    end

    def test_next_in_block_in_block
 -    skip "failing due to stack_max mismatch"
 -    assert_iseq_roundtrip <<-'end;'
 -      3.times { 3.times { next } }
 +    @next_broke = false
 +    src = <<-'end;'
 +      3.times { 3.times { next; @next_broke = true } }
      end;
 +    a = ISeq.compile(src).to_a
 +    iseq = ISeq.iseq_load(a)
 +    iseq.eval
 +    assert_equal false, @next_broke
 +    skip "failing due to stack_max mismatch"
 +    assert_iseq_roundtrip(src)
    end

    def test_break_ensure
 -    skip "failing due to exception entry sp mismatch"
 -    assert_iseq_roundtrip <<-'end;'
 -      def m
 +    src = <<-'end;'
 +      def test_break_ensure_def_method
          bad = true
          while true
            begin
 @@ -70,8 +75,15 @@ class TestIseqLoad < Test::Unit::TestCase
              bad = false
            end
          end
 +        bad
        end
      end;
 +    a = ISeq.compile(src).to_a
 +    iseq = ISeq.iseq_load(a)
 +    iseq.eval
 +    assert_equal false, test_break_ensure_def_method
 +    skip "failing due to exception entry sp mismatch"
 +    assert_iseq_roundtrip(src)
    end

    # FIXME: still failing

Also available in: Atom PDF