Backport #6190: String#encode return string containing invalid chars but marked as valid

• Status: Closed
• Assignee: naruse (Yui NARUSE)

[pplr (Pierre PLR)]

a = " \xE9 ".encode('UTF-8', 'UTF-8', :invalid => :replace, :replace => "?")
a.valid_encoding?
=> true
a
=> " \xE9 "
a.squeeze
ArgumentError: invalid byte sequence in UTF-8
from (irb):32:in squeeze' from (irb):32 from /usr/bin/irb:12:in '

The expected string is " ? ", as the documentation for the ":replace" options says :
If the value is :replace, encode replaces invalid byte sequences in str with the replacement character.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r35112.
Pierre, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• transcode.c (str_encode_bang, encoded_dup): if nothing was
  transcoded, just set encoding but leave coderange unchanged as
  forcee_encoding. [ruby-core:43557][Bug #6190]

[duerst (Martin Dürst)]

• Description updated (diff)

pplr (Pierre PLR) wrote:

a = " \xE9 ".encode('UTF-8', 'UTF-8', :invalid => :replace, :replace => "?")
a.valid_encoding?
=> true

Nobu fixed this so it won't return true anymore, which would be a lie.

a
=> " \xE9 "

The expected string is " ? ", as the documentation for the ":replace" options says :
If the value is :replace, encode replaces invalid byte sequences in str with the replacement character.

I added documentation to say that encoding from encoding A to the same encoding A is a no-op. Changing this would not be impossible, but would involve quite some work, and would make these operations slower.

[naruse (Yui NARUSE)]

• Tracker changed from Bug to Backport
• Project changed from Ruby master to Backport193

[naruse (Yui NARUSE)]

• Status changed from Closed to Assigned

[zzak (zzak _)]

• Assignee set to naruse (Yui NARUSE)

naruse-san what do you want for this ticket?

[usa (Usaku NAKAMURA)]

• Status changed from Assigned to Closed

This issue was solved with changeset r40056.
Pierre, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

merge revision(s) 35112,35121: [Backport #6190]

* transcode.c (str_encode_bang, encoded_dup): if nothing was
  transcoded, just set encoding but leave coderange unchanged as
  forcee_encoding.  [ruby-core:43557][Bug #6190]

* transcode.c (documentation for str_encode): Explain
  that transcoding to the same encoding is a no-op
  (i.e. no exceptions, no replacements,...).
  [ruby-core:43557][Bug #6190]

[duerst (Martin Dürst)]

• Has duplicate Bug #19342: String#encode does not always throw exceptions for invalid source encodings added