Bug #12192: Wrong nth_ref_max check in parse.y

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 2.3.0
• Backport: 2.1: DONE, 2.2: DONE, 2.3: DONE

[norc (Victor Nawothnig)]

$ ruby -e 'a = $1073741824'
-e: integer 2147483648 too big to convert to `int' (RangeError)

The following is a stack trace from the issue:

* thread #1: tid = 0x91034, 0x00000001000e52d8 ruby`rb_out_of_int(num=2147483648) + 40 at numeric.c:2379, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00000001000e52d8 ruby`rb_out_of_int(num=2147483648) + 40 at numeric.c:2379
    frame #1: 0x00000001000e5358 ruby`check_int(num=2147483648) + 40 at numeric.c:2387
    frame #2: 0x00000001000e53b1 ruby`rb_fix2int(val=4294967297) + 81 at numeric.c:2420
    frame #3: 0x00000001002292e0 ruby`iseq_set_sequence(iseq=0x00000001018b36a0, anchor=0x00007fff5fbfeea0) + 1504 at compile.c:1661
    frame #4: 0x0000000100224741 ruby`iseq_setup(iseq=0x00000001018b36a0, anchor=0x00007fff5fbfeea0) + 161 at compile.c:1122
    frame #5: 0x000000010021544f ruby`rb_iseq_compile_node(iseq=0x00000001018b36a0, node=0x00000001018b3998) + 3071 at compile.c:643

The error is platform-dependent, but is reproduceable on OSX using LLVM/Clang.

The following 2 steps cause this error:

1. iseq_compile_each wraps n inside a VALUE using INT2FIX
2. iseq_set_sequence/vm_getspecial unwraps n using FIX2INT, checking for loss of information during casting

The reason that check fails is because n gets left-shifted before being passed to INT2FIX for the back-ref flag. That means n must not be larger than INT_MAX >> 1.

parse_numvar from parse.y contains an a wrong limit on systems where sizeof(long) == sizeof(int)

The fix is simple, parse.y needs to use the following boundary for checking:
nth_ref_max = ((FIXNUM_MAX < INT_MAX) ? FIXNUM_MAX : INT_MAX) >> 1;

[norc (Victor Nawothnig)]

I created a pull request for the fix:

https://github.com/ruby/ruby/pull/1296

[shyouhei (Shyouhei Urabe)]

• Description updated (diff)

[norc (Victor Nawothnig)]

• Description updated (diff)

Pull requested was accepted, bug can be closed.

[nagachika (Tomoyuki Chikanaga)]

• Status changed from Open to Closed
• Backport changed from 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN to 2.1: REQUIRED, 2.2: REQUIRED, 2.3: REQUIRED

Merged at r54172.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.1: REQUIRED, 2.2: REQUIRED, 2.3: REQUIRED to 2.1: DONE, 2.2: REQUIRED, 2.3: REQUIRED

ruby_2_1 r54275 merged revision(s) 54172.

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.1: DONE, 2.2: REQUIRED, 2.3: REQUIRED to 2.1: DONE, 2.2: DONE, 2.3: REQUIRED

Backported into ruby_2_2 branch at r54335.

[naruse (Yui NARUSE)]

• Backport changed from 2.1: DONE, 2.2: DONE, 2.3: REQUIRED to 2.1: DONE, 2.2: DONE, 2.3: DONE

ruby_2_3 r54382 merged revision(s) 54172.