Bug #20809: Ripper fails to parse/tokenize nested heredocs within embexpr

• Status: Open
• Assignee: -
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-10-22T09:04:30Z master 54065f3b7b) +PRISM [x86_64-linux]
• Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[tompng (tomoya ishida)]

In this example, Ripper wrongly concats "s333\n" and "s4444".

require 'ripper'

code1 = '<<HEREDOC1
s1
#{<<HEREDOC2}s333
s22
HEREDOC2
s4444#{55555}
HEREDOC1'

p Ripper.tokenize(code1)
# => ["<<HEREDOC1", "\n", "s1\n", "\#{", "<<HEREDOC2", "}", "s333\ns4444", "s22\n", "HEREDOC2\n", "\#{", "55555", "}", "\n", "HEREDOC1"]

In this example, result of Ripper.sexp is wrong. "s333" disappears.

code2 = '<<HEREDOC1
s1
#{<<HEREDOC2}s333
s22
HEREDOC2
s4444
HEREDOC1'

p eval(code2) # => "s1\ns22\ns333\ns4444\n"
p Ripper.sexp(code2)
# => 
[:program,
 [[:string_literal,
   [:string_content,
    [:@tstring_content, "s1\n", [2, 0]],
    [:string_embexpr,
     [[:string_literal,
       [:string_content,
        [:@tstring_content, "s22\n", [4, 0]]]]]],
    [:@tstring_content, "s4444\n", [6, 0]]]]]]