Feature #13626
openAdd String#byteslice!
Description
It's a common pattern in IO buffering, to read a part of a string while leaving the remainder.
# Consume only part of the read buffer: result = @read_buffer.byteslice(0, size) @read_buffer = @read_buffer.byteslice(size, @read_buffer.bytesize)
It would be nice if this code could be simplified to:
result = @read_buffer.byteslice!(size)
Additionally, this allows a significantly improved implementation by the interpreter.
Updated by normalperson (Eric Wong) almost 4 years ago
samuel@oriontransfer.org wrote:
I used to want this, too; but then I realized IO#read and
similar methods will always return a binary string when given a
length limit.
So String#slice! should be enough.
(And IO#read and friends without a length limit is suicidal, anyways :)
Updated by ioquatix (Samuel Williams) almost 4 years ago
Thanks for that idea.
If that's the case, when appending to the write buffer:
write_buffer = String.new.b unicode_string = "\u1234".force_encoding("UTF-8") write_buffer << unicode_string write_buffer.encoding # Changed from ASCII-8BIT to Encoding:UTF-8
The only way I can think to fix this is to run +force_encoding+ on the write buffer after every append but this seems hugely inefficient.
Ideas?
Updated by normalperson (Eric Wong) almost 4 years ago
samuel@oriontransfer.org wrote:
Thanks for that idea.
If that's the case, when appending to the write buffer:
write_buffer = String.new.b unicode_string = "\u1234".force_encoding("UTF-8") write_buffer << unicode_string write_buffer.encoding # Changed from ASCII-8BIT to Encoding:UTF-8The only way I can think to fix this is to run +force_encoding+ on the write buffer after every append but this seems hugely inefficient.
Ideas?
String#force_encoding is done in-place so it should not be
that slow, the String#<< would be the slow part since it
involves at least one memcpy (worst case is realloc + 2 memcpy)
But I'm not sure why you would want to be setting data to
UTF-8; I guess you got it from some 3rd-party library?
Maybe String#b! could be shorter alias for
force_encoding(Encoding::UTF_8); but yeah, exposing writev via
[Feature #9323] is probably the best option, anyways.
Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part... But I don't know much about human
language encodings, I am just a *nix plumber where a byte is a
byte.
Updated by ioquatix (Samuel Williams) almost 4 years ago
Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part...
Agreed.
Updated by matz (Yukihiro Matsumoto) over 3 years ago
Sounds OK to me.
Matz.
Updated by akr (Akira Tanaka) over 3 years ago
At the developer meeting, we discuss that byteslice! and byteslice method should take same arguments.
Updated by duerst (Martin Dürst) over 3 years ago
normalperson (Eric Wong) wrote:
Fwiw, I'm also not convinced String#<< behavior about changing
write_buffer to Encoding::UTF-8 in your above example is good
behavior on Ruby's part... But I don't know much about human
language encodings, I am just a *nix plumber where a byte is a
byte.
This behavior may not be the best for this specific case, but in general, if one string is US-ASCII, and the other is UTF-8, then UTF-8 is a superset of US-ASCII, and concatenating the two will produce a string in UTF-8. Dropping the encoding would loose important information.
Please also note that you are actually on dangerous ground here. The above only works because the string doesn't contain any non-ASCII (high bit set) bytes. As soon as there is such a byte, there will be an error.
s = "abcde".b s.encoding # => #<Encoding:ASCII-8BIT> s << "αβγδε" # => "abcdeαβγδε" s.encoding # => #<Encoding:UTF-8>
but:
t = "αβγδε".b # => "\xCE\xB1\xCE\xB2\xCE\xB3\xCE\xB4\xCE\xB5" t.encoding # => #<Encoding:ASCII-8BIT> t << "λμπρ" # => Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
So if you have an ASCII-8BIT buffer, and want to append something, always make sure you make the appended stuff also ASCII-8BIT.
Updated by ioquatix (Samuel Williams) about 3 years ago
If you round trip UTF-8 to ASCII-8BIT and back again, the result should be the same IMHO. It's just the interpretation of the bytes which is different, but the underlying data should be the same. I still think adding String#byteslice!
is a good idea. Has there been any progress?
Updated by ioquatix (Samuel Williams) about 3 years ago
By the way, I ended up implementing https://github.com/socketry/async-io/blob/master/lib/async/io/binary_string.rb which I guess is okay but it's not ideal.
Updated by janko (Janko Marohnić) almost 3 years ago
I support adding String#byteslice!
. I've been using String#byteslice
in custom IO-like objects that implement IO#read
semantics, as the strings I work with don't necessarily have to be in binary encoding (otherwise I'd just use String#slice
), they can also be in UTF-8. Since IO#read
needs to work in terms of bytes, that's why I needed String#byteslice
.
I've used the exact idiom from Samuel's original description in three different projects already:
- https://github.com/janko-m/down/blob/ac4a32f296cb9cd8c12fc46a01a7e2f7c5fcd1b2/lib/down/chunked_io.rb#L169-L170
- https://github.com/janko-m/goliath-rack_proxy/blob/7b359ff3ddfa3cba23c32220389abb39481735a9/lib/goliath/rack_proxy.rb#L134-L135
- https://github.com/socketry/falcon/blob/12b8818812b23c920e545e6b4c91e08e5348ee04/lib/falcon/adapters/input.rb#L80-L81
String#byteslice!
would allow reducing the code and probably end up with fewer strings at the end.