Bug #20101: rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set

• Status: Closed
• Assignee: kjtsanaktsidis (KJ Tsanaktsidis)
• Target version: -
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[kjtsanaktsidis (KJ Tsanaktsidis)]

When opening a file with File.open, as long as 'b' is not set in the mode, Ruby will perform CRLF -> LF conversion on Windows when reading text files - i.e. CRLF line endings on disk get converted to Ruby strings with only "\n" in them. If you explicitly set the encoding with IO#set_encoding, this still works properly.

If you open the file in C with either the rb_io_fdopen or rb_file_open APIs in text mode, CRLF -> LF conversion also works. However, if you then call IO#set_encoding on this file, the CRLF -> LF conversion stops happening.

Concretely, this means that the conversion doesn't happen in the following circumstances:

• When loading ruby files with require (that calls rb_io_fdopen)
• When parsing ruuby files with RubyVM::AbstractSyntaxTree (that calls rb_file_open).
  This then causes the ErrorHighlight tests to fail on windows if git has checked them out with CRLF line endings - the error messages it's testing wind up with literal \r\n sequences in them because the iseq text from the parser contains un-newline-converted strings.

This seems to happen because, in File.open, the file's encflags get the flag ECONV_DEFAULT_NEWLINE_DECORATOR in rb_io_extract_modeenc; however, this method isn't called for rb_io_fdopen or rb_file_open, so encflags doesn't get set to ECONV_DEFAULT_NEWLINE_DECORATOR. Without that flag, the underlying file descriptor's mode gets changed to binary mode by the NEED_NEWLINE_DECORATOR_ON_READ_CHECK macro.

[kjtsanaktsidis (KJ Tsanaktsidis)]

I opened a PR for this: https://github.com/ruby/ruby/pull/9380

[kjtsanaktsidis (KJ Tsanaktsidis)]

• Status changed from Open to Closed

Fix merged in 31371b2e24b03ccb0a03b622faf8c65e6cf6a31a