Project

General

Profile

Actions

Bug #20579

closed

`Ripper.tokenize("1\n ")` does not include trailing spaces

Added by tompng (tomoya ishida) 14 days ago. Updated 13 days ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 3.4.0dev (2024-06-13T09:49:46Z master 8b843b0dc7) [x86_64-linux]
[ruby-core:118315]

Description

# trailing :on_sp, :on_ignored_nl are included in the result
Ripper.tokenize("1\n\t \t\n\t \t")
# => ["1", "\n", "\t \t", "\n", "\t \t"]

# trailing :on_sp disappears
Ripper.tokenize("1\n\t \t")
# => ["1", "\n"]

Found with Ripper.tokenize(random_string).join != random_string while investigating https://bugs.ruby-lang.org/issues/20578

Actions

Also available in: Atom PDF

Like0
Like0