Bug #20579: `Ripper.tokenize("1\n ")` does not include trailing spaces

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-06-13T09:49:46Z master 8b843b0dc7) [x86_64-linux]
• Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[tompng (tomoya ishida)]

# trailing :on_sp, :on_ignored_nl are included in the result
Ripper.tokenize("1\n\t \t\n\t \t")
# => ["1", "\n", "\t \t", "\n", "\t \t"]

# trailing :on_sp disappears
Ripper.tokenize("1\n\t \t")
# => ["1", "\n"]

Found with Ripper.tokenize(random_string).join != random_string while investigating https://bugs.ruby-lang.org/issues/20578

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|a1f72a563b74a44198b34add3aa9fdcef54c2405.

---

[Bug #20579] ripper: Dispatch spaces at END-OF-INPUT without newline