Bug #13321
closedString#codepoints for one-byte encodings
Description
On many versions of Ruby, including 2.4.0:
"\x80".force_encoding("WINDOWS-1252").codepoints.first # => 0x80
I expected 0x20AC: https://en.wikipedia.org/wiki/Windows-1252
See:
https://github.com/ruby/ruby/blob/v2_4_0/string.c#L7817-L7818
https://github.com/ruby/ruby/blob/v2_4_0/string.c#L422-L424
Updated by InfraRuby (InfraRuby Vision) over 7 years ago
- Description updated (diff)
Updated by nobu (Nobuyoshi Nakada) over 7 years ago
- Status changed from Open to Rejected
0x20AC is euro sign in Unicode, it is 0x80 in Windows-1252.
Updated by InfraRuby (InfraRuby Vision) over 7 years ago
That's surprising to me but I can see that. Thanks!
Updated by duerst (Martin Dürst) over 7 years ago
I tried to improve the documentation with r58000. Please tell me if that helps, or if further explanations are needed.
Updated by InfraRuby (InfraRuby Vision) over 7 years ago
Please update the documentation for String#codepoints
too.
String#codepoints
does return (Unicode) codepoints for US-ASCII and ISO-8859-1 as those encodings are the basis of Unicode.
Maybe add Encoding#unicode_codepoints?
which returns true
for these encodings: US-ASCII, ISO-8859-1, UTF-8, UTF-16(BE|LE), UTF-32(BE|LE).
(Also, there's an unrelated change in that revision.)
Updated by stomar (Marcus Stollsteimer) over 7 years ago
@duerst (Martin Dürst), @normal
r58000 accidentally reverts r57997 ("deduplicate static rb_str_format format strings") for string.c
.
Updated by duerst (Martin Dürst) over 7 years ago
InfraRuby (InfraRuby Vision) wrote:
Please update the documentation for
String#codepoints
too.
That says "This is a shorthand for str.each_codepoint.to_a
".
String#codepoints
does return (Unicode) codepoints for US-ASCII and ISO-8859-1 as those encodings are the basis of Unicode.
Well, yes, and for almost all encodings, the returned values are Unicode code points for the ASCII characters, and for some other encodings, there is a bit more of overlap. I don't think we need to go too much into details.
Maybe add
Encoding#unicode_codepoints?
which returnstrue
for these encodings: US-ASCII, ISO-8859-1, UTF-8, UTF-16(BE|LE), UTF-32(BE|LE).
There are quite a few other cases where behavior of String methods changes depending on the string's Encoding. I think it would be good to have access to this information, but methods with more general names may be needed.
Anyway, to get Unicode codepoints out of an arbitrary string, string.encode('UTF-8').codepoints
will always do the job.
(Also, there's an unrelated change in that revision.)
Yes, thanks for noticing, fixed.
Updated by InfraRuby (InfraRuby Vision) over 7 years ago
Thanks!