Bug #20030: `Ripper.tokenize('"\\C-あ"')` separates encoding valid string to encoding invalid string.

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.3.0dev (2023-11-30T16:23:25Z master d048bae96b) [x86_64-linux]
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[tompng (tomoya ishida)]

Ripper.tokenize '"\\C-あ"' # or Ripper.tokenize "\"\\C-\u3042\""
# =>
["\"", "\x81", "\x82", "\""]

I expect all tokens to be valid_encoding if the source string is valid_encoding.

This is causing IRB crash when typing "\C-あ".

[nobu (Nobuyoshi Nakada)]

https://github.com/ruby/ruby/pull/9091
This fixes another ripper scanner event issue at a syntax error too.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|d503e1b95a40e45d7767e0175de60092de4ba54e.

---

[Bug #20030] dispatch invalid escaped character without ignoring it