Bug #13321: String#codepoints for one-byte encodings

• Status: Rejected
• Assignee: -
• Target version: -
• Backport: 2.2: UNKNOWN, 2.3: UNKNOWN, 2.4: UNKNOWN

[InfraRuby (InfraRuby Vision)]

On many versions of Ruby, including 2.4.0:

"\x80".force_encoding("WINDOWS-1252").codepoints.first # => 0x80

I expected 0x20AC: https://en.wikipedia.org/wiki/Windows-1252

See:
https://github.com/ruby/ruby/blob/v2_4_0/string.c#L7817-L7818
https://github.com/ruby/ruby/blob/v2_4_0/string.c#L422-L424

[InfraRuby (InfraRuby Vision)]

• Description updated (diff)

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Rejected

0x20AC is euro sign in Unicode, it is 0x80 in Windows-1252.

[InfraRuby (InfraRuby Vision)]

That's surprising to me but I can see that. Thanks!

[duerst (Martin Dürst)]

I tried to improve the documentation with r58000. Please tell me if that helps, or if further explanations are needed.

[InfraRuby (InfraRuby Vision)]

Please update the documentation for String#codepoints too.

String#codepoints does return (Unicode) codepoints for US-ASCII and ISO-8859-1 as those encodings are the basis of Unicode.

Maybe add Encoding#unicode_codepoints? which returns true for these encodings: US-ASCII, ISO-8859-1, UTF-8, UTF-16(BE|LE), UTF-32(BE|LE).

(Also, there's an unrelated change in that revision.)

[stomar (Marcus Stollsteimer)]

@duerst (Martin Dürst), @normal

r58000 accidentally reverts r57997 ("deduplicate static rb_str_format format strings") for string.c.

[duerst (Martin Dürst)]

InfraRuby (InfraRuby Vision) wrote:

Please update the documentation for String#codepoints too.

That says "This is a shorthand for str.each_codepoint.to_a".

String#codepoints does return (Unicode) codepoints for US-ASCII and ISO-8859-1 as those encodings are the basis of Unicode.

Well, yes, and for almost all encodings, the returned values are Unicode code points for the ASCII characters, and for some other encodings, there is a bit more of overlap. I don't think we need to go too much into details.

Maybe add Encoding#unicode_codepoints? which returns true for these encodings: US-ASCII, ISO-8859-1, UTF-8, UTF-16(BE|LE), UTF-32(BE|LE).

There are quite a few other cases where behavior of String methods changes depending on the string's Encoding. I think it would be good to have access to this information, but methods with more general names may be needed.

Anyway, to get Unicode codepoints out of an arbitrary string, string.encode('UTF-8').codepoints will always do the job.

(Also, there's an unrelated change in that revision.)

Yes, thanks for noticing, fixed.

[InfraRuby (InfraRuby Vision)]

Thanks!