Misc #22230: Should Ruby specs define byte oriented read behavior when the character buffer is not empty?

• Status: Open
• Assignee: -

[javanthropus (Jeremy Bopp)]

In https://github.com/ruby/spec/pull/1384, there was a discussion regarding whether or not raising an IOError when performing a byte oriented read operation while the character buffer is nonempty should be part of the Ruby spec. @Eregon (Benoit Daloze) asked me to remove the new tests I was requesting to add and also remove existing tests that would define this behavior since it was unclear if the behavior in CRuby was an intentional design decision or a consequence of its specific implementation.

I intend to remove the tests in the above MR via a single commit that is easy to revert pending the outcome of this ticket as requested.

In all cases I tested, CRuby raises IOError for all byte oriented read operations whenever the character buffer is nonempty. IO.getc is used in all these cases to ensure that the character buffer has data in it, but it was argued that other implementations could instead make getc reverse the encoding conversion (if one is in play) and put the bytes into the byte buffer instead, eliminating the character buffer entirely. If that conversion would fail, getc could raise a conversion error of some kind and leave the buffer unchanged. It's unclear to me though if there are other ways that content could be left in the character buffer during normal read operations which would be harder to avoid while preserving performance and semantics.

CRuby's implementation appears to be consistent, so I don't think these tests are attempting to enforce buggy behavior. Should this behavior be codified as part of the spec, or should other implementations be free to behave differently?

[YO4 (Yoshinao Muramatsu)]

This may overlap with the discussion in ruby/spec, but I'll share my understanding of the current situation with CRuby.
The following illustrates the read model for IO in CRuby's implementation.

device -> [rbuf] -> encoding converter -> [cbuf] -> character read -> [ruby string]
             |
             + - > byte oriented read -> [ruby string]

When reading bytes after reading characters, it is necessary to calculate the current position in rbuf corresponding to the current position in cbuf; however, the reverse conversion from cbuf to rbuf is not straightforward.
For example, in the universal newline conversion, \n, \r, and \r\n are converted to \n, so a single \n may correspond to either 1 byte or 2 bytes in rbuf.
The current implementation appears to have opted to avoid this reverse conversion. To prevent characters from remaining in cbuf, it performs encoding conversion one character at a time and immediately consumes cbuf during reads that are not whole reads.
This is why byte-oriented reads are prohibited when ungetc was called, and it is also the cause of the slowness when encoding conversion is enabled.

When the encoding converter is not used, the [rbuf] -> encoding converter step is omitted, and character operations handle rbuf directly; therefore, there is no difference between character and byte operations.
In this case, byte reads are still possible after ungetc.

The difference in ungetc behavior depending on the presence or absence of an encoding converter seems to be a compromise resulting from the fact that the encoding converter was added later while preserving the previous behavior.

[matz (Yukihiro Matsumoto)]

Please leave this implementation defined. Other implementations should be free to behave differently.

#note-1 makes the case well. The IOError is a consequence of CRuby having a character buffer and avoiding the reverse conversion back to bytes, not a decision about what the language promises. An implementation that converts back in getc and keeps no character buffer at all would have no reason to raise here, and I do not want to force it to emulate a buffer it does not have.

CRuby is not even consistent with itself: after ungetc, byte reads work when no converter is in play and fail when one is. That is not a rule to hold every implementation to.

Matz.