Feature #16604
Updated by larskanis (Lars Kanis) about 4 years ago
This issue is related to https://bugs.ruby-lang.org/issues/13488 where we already discussed the topic an postponed the change for ruby-3. Patch is here: https://github.com/ruby/ruby/pull/2877 Currently `Encoding.default_external` is initialized to the local console encoding of the Windows installation unless changed per option `-E`. This is e.g. cp850 for Western Europe. It should be changed to UTF-8. RubyInstaller provided a checkbox for `RUBYOPT=-Eutf-8` since version 2.4. This checkbox was disabled per default, but I noticed from bug reports, that many people enabled it. With RubyInstaller-2.7.0 this checkbox is [enabled per default](https://rubyinstaller.org/2020/01/05/rubyinstaller-2.7.0-1-released.html). So we already have a steady migration towards UTF-8 on Windows. Changing to UTF-8 fixes various inconsistencies within ruby and with external tools. A very annoying case is that writing a text to file writes the file content in UTF-8, since this is the default ruby source encoding. But reading the content back, tags the string with the wrong encoding. But not in `irb` since it already set `Encoding.default_external = "utf-8"` on it's own. ``` s = "äöü" File.write("x", s) # => 6 bytes File.read("x") == s # => true in irb but false in .rb file ``` Another issue is that many non-asian regions have distinct legacy encodings for OEM-ANSI (aka `Encoding.find('locale')` ) and ASCII (aka `Encoding.find('filesystem')` ), so that a file written in current default external encoding `Encoding.find('locale')` is not properly interpret in Windows GUI tools like notepad. It is therefore uncommon to store files in OEM-ANSI encoding and doing so is almost certainly wrong. RubyInstaller ships the MSYS2 environment, which defaults to UTF-8 as well. Powershell made the switch to UTF-8 (without BOM) in [Powershell-6.0](https://docs.microsoft.com/en-us/powershell/scripting/whats-new/what-s-new-in-powershell-core-60?view=powershell-7#default-encoding-is-utf-8-without-a-bom-except-for-new-modulemanifest) and even more in 6.1. Changing the default of `Encoding.default_external` to UTF-8 is a trade-off. It doesn't fit to every case, but in my experience this is the best overall option. There are some alternatives to it: Changing the Windows console to codepage 65001: * The Windows implementation of 65001 is buggy in the console. I didn't verify it lately but `chcp 65001` didn't work reliable years ago. * It is not the default and input methods like IME are incompatible. Setting `Encoding.default_internal` in addition: * This triggers transcoding of output strings, which is not enabled on other systems, causing unexpected results and incompatibilities. Change ruby to use `Encoding.find("filesystem")` as encoding for file operations: * That would fix the compatibility with some builtin Windows tools, but doesn't fix encoding issues due to increased use of UTF-8. Please note that changing `Encoding.default_external` doesn't affect file or IO output, unless `Encoding.default_internal` is set as well (which is not the default). So inspecting ruby's output with Windows builtin `more` will most likely result in garbage (since strings are usually UTF-8 in ruby) regardless of the particular `default_external` setting. On the other hand output inspected with MSYS2 `less` is most likely correct, since it expects UTF-8 input. The patch is currently about Windows only, because I would like to focus on that question for now. Possibly it's a subsequent question whether Encoding.default_external should default to UTF-8 on all operating systems or at least in case of `LANG=C` locale (which currently triggers US-ASCII).