Strange behavior of String#encode('UTF-8', 'UTF-8', ...) when the encoding of the source string is not UTF-8
When the dst_encoding and src_encoding options of String#encode are the same, it appears to ignore the encoding given and instead operate on the actual encoding of the string. Examples:
"abcdÁ".force_encoding('ASCII').encode('UTF-8', 'UTF-8', invalid: :replace, undef: :replace) => "abcd??" "abcdÁ".force_encoding('ASCII').encode('UTF-8', 'UTF-8', invalid: :replace, undef: :replace, replace: '�') Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-8 "abcdÁ\xff".encode('ASCII', 'ASCII', invalid: :replace, undef: :replace).force_encoding('UTF-8') => "abcdÁ�"
Also, without the "replace" options, exceptions are not raised as they should be:
"\xff".force_encoding('ASCII').encode('UTF-8', 'UTF-8') => "\xFF"
I looked a little at the code, and I think the problem might be in this block where the given string is passed to
rb_str_scrub without any other encoding information.
What I would expect is for
s.dup.force_encoding('X').encode('Y', opts) to behave identically to
s.encode('Y', 'X', opts), but that is clearly not the case.
Verified on Ruby 2.1.5, 2.3.0, and 2.3.1.
Updated by nobu (Nobuyoshi Nakada) about 4 years ago
- Status changed from Open to Closed
Applied in changeset r55181.
transcode.c: scrub in the given encoding
- transcode.c (str_transcode0): scrub in the given encoding when the source encoding is given, not in the encoding of the receiver. [ruby-core:75732] [Bug #12431]