Bug #19291: Ripper's on_parse_error token has wrong content

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-darwin20]
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[tompng (tomoya ishida)]

Ruby 3.1

Ripper::Lexer.new("{a:,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nhelloworld").scan.map(&:tok)
# =>
["{", "a:", ",", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "", "\n", "helloworld"]

Ruby 3.2.0

Ripper::Lexer.new("{a:,aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nhelloworld").scan.map(&:tok)
# => 
["{",
 "a:",
 ",",
 "oworld\u0000\u0000\u0000\u0000\u0000\u0000z`\u0001\u0000\u0000\u0000\u0000\u0000\xC0\xF0$\u0010\u0001\u0000\u0000\u0000\x80A\xB8\u0019\xB5\u007F\u0000\u0000\x91\x94",
 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
 "\n",
 "helloworld"]

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|3becc4a105bcf873fdc6e83c2a957f73718c5084.

---

[Bug #19291] Rewind to the previous line

When rewinding looking ahead after newline token, also reset the last
line string, the pointers to it, and the location, not only the line
number.

[nobu (Nobuyoshi Nakada)]

This may be a potential bug since older versions.