ENV data yield ASCII-8BIT encoded strings under Windows with unicode username
My testing scenario:
English Windows, Unicode username: てすと
Home directory: C:\Users\てすと\
The values returned from ENV have different encoding depending on their content. It appear to be OEM encoding label to most value, except when they contain characters not included in the OEM codepage. When they are not, for instance
ENV['HOME'] when the username is "てすと" will have ASCII-8BIT.
(I find the "ASCII-8BIT" name for an encoding confusing, as ASCII is 7bit - byte range 0-127)
But it appear that "ASCII-8BIT" is also aliased as "binary"? So Ruby is here returning a binary string when ENV contain byte characters not included in the OEM code page?
Reading the docs for Encoding:
Returns default internal encoding. Strings will be transcoded to the default internal encoding in the following places if the default internal encoding is not nil:
::default_internal is initialized by the source file's internal_encoding or -E option.
ENV - but, even when I run ruby with the
-E flag the
ENV encoding doesn't change. It's still using the OEM code page - or ASCII-8BIT.
However, regardless of having set
-E or not, ENV do appear to return UTF-8 bytes in the strings that contain the Unicode username.
This is one of several areas where I have found -E to have no effect on Ruby's string handling. I understand that some of Ruby's file handling is for backwards compatibility reasons, but I'm finding it difficult to set up a system which can properly handle Unicode files under Windows. Is this deliberate due to backwards compatibility decisions? Or have I simply not found the correct configuration flags for it? To me it appear bugged - inconsistent with what the documentation says. But please enlighten me if I am incorrect. My ideal situation would be for all strings to default to UTF-8.
C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].encoding" #<Encoding:CP850> C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].bytes" [67, 58, 92, 80, 114, 111, 103, 114, 97, 109, 32, 70, 105, 108, 101, 115, 32, 40, 120, 56, 54, 41]
C:\ruby-220\usr\bin>ruby -e "p ENV['HOME']" "C:/Users/\xE3\x81\xA6\xE3\x81\x99\xE3\x81\xA8" C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].encoding" #<Encoding:ASCII-8BIT> C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].bytes" [67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168] C:\ruby-220\usr\bin>ruby -e "p __ENCODING__" #<Encoding:CP850> C:\ruby-220\usr\bin>ruby -e "p Encoding.default_internal" nil C:\ruby-220\usr\bin>ruby -e "p Encoding.default_external" #<Encoding:CP850> C:\ruby-220\usr\bin>ruby -e "p Encoding.find('filesystem')" #<Encoding:Windows-1252> C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].encoding" #<Encoding:ASCII-8BIT> C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].bytes" [67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168]
Updated by thomthom (Thomas Thomassen) about 5 years ago
Updated by spatulasnout (B Kelly) about 5 years ago
Agreed: although Ruby 2.2 appears to be trying to use locale to determine the encoding for environment vars, it's not producing reasonable results on Windows. (E.g. w32_getenv() in hash.c, and also putenv() in hash.c)
Its current behavior seems to mangle UTF-8 data in environment vars so badly that the data can't be reconstructed from within ruby. (As such, simply passing through ASCII-8BIT data untouched would be preferable to the current behavior, as one could at least then force_encoding from within ruby.)
Updated by Iristyle (Ethan Brown) over 4 years ago
I agree that ENV corruption continues to be a problem in Ruby. The expectation is that regardless of current locale, I should get UTF-8 strings when using ENV on Windows. Here is a simple reproduction of the failure on
C:\Users\Administrator> $env:unicode = 'taskᚠᛇᚻ' C:\Users\Administrator> dir Env:\unicode Name Value ---- ----- unicode taskᚠᛇᚻ C:\Users\Administrator> ruby --version ruby 2.3.0p0 (2015-12-25 revision 53290) [x64-mingw32] C:\Users\Administrator> chcp Active code page: 437 C:\Users\Administrator> irb irb(main):001:0> RUBY_VERSION => "2.3.0" irb(main):002:0> Encoding.default_internal => nil irb(main):003:0> Encoding.default_external => #<Encoding:IBM437> irb(main):004:0> str = ENV['unicode'] => "task???" irb(main):005:0> str.encoding => #<Encoding:IBM437>
Again, when I access
ENV on Windows, I should receive a UTF-8 string with the correct data. The expected string in this case is:
irb(main):036:0> str2 = "task\u16A0\u16C7\u16BB" => "task\u16A0\u16C7\u16BB" irb(main):037:0> str2.encoding => #<Encoding:UTF-8>
Note that some browsers, like Chrome on OSX, may fail to render the Rune characters correctly, but if you copy into a proper editor or use another browser you should see the characters fine.