Bug #20159: Prism assertion failure even if it is executed with --parser=parse.y

• Status: Closed
• Assignee: kddnewton (Kevin Newton)
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-01-07T16:23:58Z master 8b86d6f0c1) [x86_64-linux]
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[tompng (tomoya ishida)]

Prism fails to parse this code with assertion failed. Reported at https://github.com/ruby/prism/issues/1616

<<A+%
A

Ruby exits with assertion failed even if it's run with option --parser=parse.y

# ruby -v
ruby 3.4.0dev (2024-01-07T16:23:58Z master 8b86d6f0c1) [x86_64-linux]
# (echo "<<A+%"; echo "A") > a.rb
# ruby --parser=parse.y a.rb
ruby: prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted

eval will correclty result in SyntaxError, require and load exits with assertion failed.

# irb
irb(main):001> eval File.read('a.rb')
(irb):1:in `eval': (eval at (irb):1):1: unterminated string meets end of file (SyntaxError)
irb(main):002> load 'a.rb'
prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted

[kddnewton (Kevin Newton)]

• Assignee set to kddnewton (Kevin Newton)

[kddnewton (Kevin Newton)]

I'm having difficulty reproducing this.

$ ruby -v
ruby 3.4.0dev (2024-01-07T16:23:58Z master 8b86d6f0c1) [arm64-darwin23]
$ (echo "<<A+%"; echo "A") > a.rb
$ ruby --parser=parse.y a.rb
a.rb: a.rb:1: unterminated string meets end of file (SyntaxError)
<<A+%
     ^

and with IRB:

irb(main):001> RUBY_VERSION
=> "3.4.0"
irb(main):002> eval File.read('a.rb')
(irb):2:in `eval': (eval at (irb):2):1: unterminated string meets end of file (SyntaxError)
<<A+%
     ^

	from (irb):2:in `<main>'
	from <internal:kernel>:187:in `loop'
	from /Users/kddnewton/.rubies/ruby-yjit/lib/ruby/gems/3.4.0+0/gems/irb-1.11.0/exe/irb:9:in `<top (required)>'
	from /Users/kddnewton/.rubies/ruby-yjit/bin/irb:25:in `load'
	from /Users/kddnewton/.rubies/ruby-yjit/bin/irb:25:in `<main>'
irb(main):003> load 'a.rb'
(irb):3:in `load': a.rb:1: unterminated string meets end of file (SyntaxError)
<<A+%
     ^

	from (irb):3:in `<main>'
	from <internal:kernel>:187:in `loop'
	from /Users/kddnewton/.rubies/ruby-yjit/lib/ruby/gems/3.4.0+0/gems/irb-1.11.0/exe/irb:9:in `<top (required)>'
	from /Users/kddnewton/.rubies/ruby-yjit/bin/irb:25:in `load'
	from /Users/kddnewton/.rubies/ruby-yjit/bin/irb:25:in `<main>'

Do you have any RUBYOPT environment variables set?

[tompng (tomoya ishida)]

I found that this is because syntax_suggest is using prism.
It's not caused by iseq loading, I now think this is not a bug.

% (echo "<<A+%"; echo "A") > a.rb
% ruby a.rb
Assertion failure
% ruby --disable-gems a.rb
SyntaxError
% ruby --disable-gems -r syntax_suggest a.rb
Assertion failure

[mame (Yusuke Endoh)]

@kddnewton (Kevin Newton) I could reproduce the issue with no RUBYOPT. The error occurs because SyntaxSuggest uses Prism.lex_compat

$ cat a.rb
<<A+%
A

$ ruby -v a.rb
ruby 3.4.0dev (2024-01-09T08:14:55Z master 149373ce7f) [x86_64-linux]
a.rb:1: warning: possibly useless use of + in void context
ruby: prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed.
Aborted

$ gdb --args ruby a.rb
...
(gdb) r
Starting program: /home/mame/work/ruby/ruby a.rb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffde56f640 (LWP 55846)]
[New Thread 0x7fffdc5ef640 (LWP 55847)]
ruby: prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed.

Thread 1 "ruby" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737349576640) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) p rb_backtrace()
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/core_ext.rb:40:in `detailed_message'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/api.rb:94:in `call'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/timeout.rb:196:in `timeout'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/timeout.rb:42:in `handle_timeout'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/timeout.rb:187:in `block in timeout'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/api.rb:96:in `block in call'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/api.rb:96:in `new'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/code_search.rb:61:in `initialize'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/code_search.rb:61:in `new'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/clean_document.rb:89:in `initialize'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/code_line.rb:31:in `from_source'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/code_line.rb:31:in `new'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/lex_all.rb:22:in `initialize'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/syntax_suggest/lex_all.rb:45:in `lex'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/prism.rb:47:in `lex_compat'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/prism/lex_compat.rb:610:in `result'
        from /home/mame/work/ruby/local/lib/ruby/3.4.0+0/prism/lex_compat.rb:610:in `lex'
$1 = void
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737349576640) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737349576640) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737349576640, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7beb476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7bd17f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7bd171b in __assert_fail_base (fmt=0x7ffff7d86130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x5555559ea548 "list->size == 0 || newline_offset > list->offsets[list->size - 1]",
    file=0x5555559ea500 "prism/util/pm_newline_list.c", line=42, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007ffff7be2e96 in __GI___assert_fail (
    assertion=assertion@entry=0x5555559ea548 "list->size == 0 || newline_offset > list->offsets[list->size - 1]",
    file=file@entry=0x5555559ea500 "prism/util/pm_newline_list.c", line=line@entry=42,
    function=function@entry=0x5555559ea5b0 <__PRETTY_FUNCTION__.1> "pm_newline_list_append") at ./assert/assert.c:101
#7  0x00005555557de7e6 in pm_newline_list_append (list=list@entry=0x7fffffffcdd8, cursor=<optimized out>) at prism/util/pm_newline_list.c:42
#8  0x00005555557eb307 in parser_lex (parser=0x7fffffffcbb0) at prism/prism.c:9057
#9  0x0000555555806d68 in parse_expression_infix (parser=0x7fffffffcbb0, node=0x5555566bdda0,
    previous_binding_power=PM_BINDING_POWER_STATEMENT, binding_power=39, accepts_command_call=<optimized out>) at prism/prism.c:16836
#10 0x00005555557fb94a in parse_expression (parser=parser@entry=0x7fffffffcbb0, binding_power=binding_power@entry=PM_BINDING_POWER_STATEMENT,
    accepts_command_call=accepts_command_call@entry=true, diag_id=diag_id@entry=PM_ERR_CANNOT_PARSE_EXPRESSION) at prism/prism.c:17166
#11 0x00005555557fbb55 in parse_statements (parser=parser@entry=0x7fffffffcbb0, context=context@entry=PM_CONTEXT_MAIN) at prism/prism.c:11045
#12 0x00005555557fbd5c in parse_program (parser=parser@entry=0x7fffffffcbb0) at prism/prism.c:17244
#13 0x000055555580b049 in pm_parse (parser=parser@entry=0x7fffffffcbb0) at prism/prism.c:17478
#14 0x0000555555990e07 in parse_lex_input (input=input@entry=0x7fffffffce80, options=options@entry=0x7fffffffcea0,
    return_nodes=return_nodes@entry=false) at prism/extension.c:525
#15 0x0000555555991b6a in lex (argc=<optimized out>, argv=<optimized out>, self=<optimized out>) at prism/extension.c:571
#16 0x000055555578f7f4 in vm_call_cfunc_with_frame_ (stack_bottom=<optimized out>, argv=<optimized out>, argc=<optimized out>,
    calling=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>) at /home/mame/work/ruby/vm_insnhelper.c:3490
#17 vm_call_cfunc_with_frame (calling=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
    at /home/mame/work/ruby/vm_insnhelper.c:3518
...

[kddnewton (Kevin Newton)]

• Status changed from Open to Closed

This is now fixed by 88d7838445ec84b1cc630ce3bd97bb71cd0aefd4