Bug #18963: Ripper.tokenize(code).join != code when heredoc and multiline string is on the same line

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [x86_64-darwin20]
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[tompng (tomoya ishida)]

Expect Ripper.tokenize(code).join to match the original code but it doesn't.

code = %[
<<A + "hello
A
world"
]
eval code # => "ruby\nhello\nworld"
code == Ripper.tokenize(code).join # => false
eval Ripper.tokenize(code).join # => Syntax Error

Other codes that does not match.
"<<A;/a\nA\nb/" "<<A;:'a\nA\nb'" "<<A;%[a\nA\nb]"

IRB fails coloring those codes.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|ace2eee544378eb03ea95b95c89434508325e8c8.

---

[Bug #18963] Separate string contents by here document terminator

[nobu (Nobuyoshi Nakada)]

• Related to Bug #20521: Memory leak in Ripper parsing added