Feature #13626: Add String#byteslice!

• Status: Open
• Priority: Normal
• Assignee: -
• Target version: -

[ioquatix (Samuel Williams)]

It's a common pattern in IO buffering, to read a part of a string while leaving the remainder.

# Consume only part of the read buffer:
result = @read_buffer.byteslice(0, size)
@read_buffer = @read_buffer.byteslice(size, @read_buffer.bytesize)

It would be nice if this code could be simplified to:

result = @read_buffer.byteslice!(size)

Additionally, this allows a significantly improved implementation by the interpreter.

[normalperson (Eric Wong)]

[e-mail] wrote:

https://bugs.ruby-lang.org/issues/13626

I used to want this, too; but then I realized IO#read and
similar methods will always return a binary string when given a
length limit.

So String#slice! should be enough.

(And IO#read and friends without a length limit is suicidal, anyways :)

[ioquatix (Samuel Williams)]

Thanks for that idea.

If that's the case, when appending to the write buffer:

write_buffer = String.new.b
unicode_string = "\u1234".force_encoding("UTF-8")
write_buffer << unicode_string
write_buffer.encoding # Changed from ASCII-8BIT to Encoding:UTF-8

The only way I can think to fix this is to run +force_encoding+ on the write buffer after every append but this seems hugely inefficient.

Ideas?

[normalperson (Eric Wong)]

[e-mail] wrote:

Thanks for that idea.

If that's the case, when appending to the write buffer:

write_buffer = String.new.b
unicode_string = "\u1234".force_encoding("UTF-8")
write_buffer << unicode_string
write_buffer.encoding # Changed from ASCII-8BIT to Encoding:UTF-8

The only way I can think to fix this is to run +force_encoding+ on the write buffer after every append but this seems hugely inefficient.

Ideas?

String#force_encoding is done in-place so it should not be
that slow, the String#<< would be the slow part since it
involves at least one memcpy (worst case is realloc + 2 memcpy)

But I'm not sure why you would want to be setting data to
UTF-8; I guess you got it from some 3rd-party library?

Maybe String#b! could be shorter alias for
force_encoding(Encoding::UTF_8); but yeah, exposing writev via
[Feature #9323] is probably the best option, anyways.

Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part... But I don't know much about human
language encodings, I am just a *nix plumber where a byte is a
byte.

[ioquatix (Samuel Williams)]

Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part...

Agreed.

[matz (Yukihiro Matsumoto)]

Sounds OK to me.

Matz.

[akr (Akira Tanaka)]

At the developer meeting, we discuss that byteslice! and byteslice method should take same arguments.

[duerst (Martin Dürst)]

normalperson (Eric Wong) wrote:

Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part... But I don't know much about human
language encodings, I am just a *nix plumber where a byte is a
byte.

This behavior may not be the best for this specific case, but in general, if one string is US-ASCII, and the other is UTF-8, then UTF-8 is a superset of US-ASCII, and concatenating the two will produce a string in UTF-8. Dropping the encoding would loose important information.

Please also note that you are actually on dangerous ground here. The above only works because the string doesn't contain any non-ASCII (high bit set) bytes. As soon as there is such a byte, there will be an error.

s = "abcde".b
s.encoding   # => #<Encoding:ASCII-8BIT>
s << "αβγδε" # => "abcdeαβγδε"
s.encoding   # => #<Encoding:UTF-8>

but:

t = "αβγδε".b # => "\xCE\xB1\xCE\xB2\xCE\xB3\xCE\xB4\xCE\xB5"
t.encoding    # => #<Encoding:ASCII-8BIT>
t << "λμπρ"   # => Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8

So if you have an ASCII-8BIT buffer, and want to append something, always make sure you make the appended stuff also ASCII-8BIT.

[ioquatix (Samuel Williams)]

If you round trip UTF-8 to ASCII-8BIT and back again, the result should be the same IMHO. It's just the interpretation of the bytes which is different, but the underlying data should be the same. I still think adding String#byteslice! is a good idea. Has there been any progress?

[ioquatix (Samuel Williams)]

By the way, I ended up implementing https://github.com/socketry/async-io/blob/master/lib/async/io/binary_string.rb which I guess is okay but it's not ideal.

[janko (Janko Marohnić)]

I support adding String#byteslice!. I've been using String#byteslice in custom IO-like objects that implement IO#read semantics, as the strings I work with don't necessarily have to be in binary encoding (otherwise I'd just use String#slice), they can also be in UTF-8. Since IO#read needs to work in terms of bytes, that's why I needed String#byteslice.

I've used the exact idiom from Samuel's original description in three different projects already:

• https://github.com/janko-m/down/blob/ac4a32f296cb9cd8c12fc46a01a7e2f7c5fcd1b2/lib/down/chunked_io.rb#L169-L170
• https://github.com/janko-m/goliath-rack_proxy/blob/7b359ff3ddfa3cba23c32220389abb39481735a9/lib/goliath/rack_proxy.rb#L134-L135
• https://github.com/socketry/falcon/blob/12b8818812b23c920e545e6b4c91e08e5348ee04/lib/falcon/adapters/input.rb#L80-L81

String#byteslice! would allow reducing the code and probably end up with fewer strings at the end.