Bug #20517: `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string.

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux]
• Backport: 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE

[tompng (tomoya ishida)]

Ripper.tokenize '"\\M-あ"'
=> ["\"", "\\M-\xE3", "\x81", "\x82", "\""]

I expect all tokens to be valid_encoding if the source string is valid_encoding.
Similar to https://bugs.ruby-lang.org/issues/20030

[nobu (Nobuyoshi Nakada)]

• Backport changed from 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN to 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED

[nobu (Nobuyoshi Nakada)]

https://github.com/ruby/ruby/pull/10893

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|05553cf22d43dd78b8f30cc4591230b5c000c538.

---

[Bug #20517] Make a multibyte character one token at meta escape

[k0kubun (Takashi Kokubun)]

• Backport changed from 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED to 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE

ruby_3_3 05553cf22d43dd78b8f30cc4591230b5c000c538.