Bug #18338
openEncoding.default_external = Encoding::UTF_16BE may add a wrongly-encoded string to $LOADED_FEATURES
Description
# coding: US-ASCII
Encoding.default_external = Encoding::UTF_16BE
"abc".encode(Encoding.default_external)
p $LOADED_FEATURES.last.encoding #=> #<Encoding:UTF-16BE>
p $LOADED_FEATURES.last
#=> "\u2F68\u6F6D\u652F\u6D61\u6D65\u2F77\u6F72\u6B2F\u7275\u6279\u2F6C\u6F63\u616C\u2F6C\u6962\u2F72\u7562\u792F\u332E\u312E\u302F\u7838\u365F\u3634\u2D6C\u696E\u7578\u2F65\u6E63\u2F74\u7261\u6E73\u2F75\u7466\u5F31\u365F\u3332\u2E73\x6F"
This weird string seems "/home/mame/work/ruby/local/lib/ruby/3.1.0/x86_64-linux/enc/trans/utf_16_32.s\u0000o".force_encoding("UTF-16BE").
Note that the code may raise a "code converter not found" error depending on the length of install path (or build path?). Maybe it works if it is even due to UTF-16.
# works
mame$ /Users/mame/ruby2/exe/ruby --disable-gems test.rb
#<Encoding:UTF-16BE>
"\u2F55\u7365\u7273\u2F6D\u616D\u652F\u7275\u6279\u322F\u2E65\u7874\u2F78\u3836\u5F36\u342D\u6461\u7277\u696E\u3139\u2F65\u6E63\u2F74\u7261\u6E73\u2F73\u696E\u676C\u655F\u6279\u7465\u2E62\u756E\u646C\x65"
# raises an exception
mame$ /Users/mame/ruby22/exe/ruby --disable-gems test.rb
0ec: 0x0
test.rb:4:in `encode': code converter not found (US-ASCII to UTF-16BE) (Encoding::ConverterNotFoundError)
from test.rb:4:in `<main>'
test.rb:4:in `encode': No such file or directory @ rb_check_realpath_internal - ⽕獥牳⽭慭支牵批㈲⼮數琯砸㙟㘴ⵤ慲睩渱㤯敮振瑲慮猯獩湧汥形祴攮扵湤汥 (Errno::ENOENT)
from test.rb:4:in `<main>'
Updated by mame (Yusuke Endoh) about 5 years ago
This issue causes a random failure of make test-all.
http://ci.rvm.jp/results/trunk-test@ruby-sky1/3711615
/tmp/ruby/v3/src/trunk-test/test/io/console/test_io_console.rb:11:in `===': incompatible encoding regexp match (US-ASCII regexp with UTF-16BE string) (Encoding::CompatibilityError)
When test_object_inspect_external in test/ruby/test_m17n.rb is executed first, it adds a wrong UTF-16 path to $LOADED_FEATURES. And then, test/io/console/test_io_console.rb applies a regexp to strings in $LOADED_FEATURES. It highly depends on the order of tests, which made the failure inconsistent.
@naruse-san said that we don't pay effort for this issue soon, so I'd like to skip test_object_inspect_external to prevent the random test failure.
Updated by jasonmaxlee (Jason Lee) 19 days ago
I'd like to fix this. The cause is that on Unix the filesystem encoding is set from default_external (Init_enc_set_filesystem_encoding in localeinit.c, called from enc_set_default_encoding), so an ASCII-incompatible default_external such as UTF-16BE becomes the encoding of every path, including transcoder paths added to $LOADED_FEATURES by String#encode.
Proposal: when default_external is ASCII-incompatible, keep using the locale encoding (or ASCII-8BIT) for the filesystem encoding, since such an encoding can't describe a file path anyway. ASCII-compatible encodings keep following default_external as now. Then test_object_inspect_external can run again.
Pull request: https://github.com/ruby/ruby/pull/19628