Feature #22431: Add `IO#unpack` and `IO#unpack1`

• Status: Open
• Assignee: -
• Target version: -

[ntkme (Natsuki Natsume)]

I would like to have ruby add IO#unpack and IO#unpack1 that effectively does both IO#read(...) and String#unpacking(...) in one call.

This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable IO stream (e.g. network/stdio).

In a streaming IO with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have IO#unpack/IO#unpack1, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation.

---

Here is a read world example:

LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785

A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream.

For example:

length, payload, length, payload, length, payload...

In pseudo code, reading and writing such stream would look like:

def write(payload, io)
  io.write([payload.length].pack('R'))
  io.write(payload)
end

def read(io)
  length = io.unpack1('R')
            # ^ This method does not exist, which makes the real world implementation slower and more complicated.
            #   See attached benchmark.rb for working implementations.
  io.read(length)
end

---

See attached benchmark.rb, which demonstrates the limitation of native unpacking for LEB128:

ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27]
Warming up --------------------------------------
write varint (pure ruby)    12.167k i/100ms
write leb128 (native)       15.710k i/100ms
Calculating -------------------------------------
write varint (pure ruby)    124.060k (± 1.6%) i/s    (8.06 μs/i) -    620.517k in   5.001746s
write leb128 (native)       157.036k (± 1.1%) i/s    (6.37 μs/i) -    785.500k in   5.002040s

Comparison:
write leb128 (native)   :   157035.9 i/s
write varint (pure ruby):   124060.1 i/s - 1.27x  slower

ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27]
Warming up --------------------------------------
read varint (pure ruby)    42.025k i/100ms
read leb128 (native)       20.246k i/100ms
Calculating -------------------------------------
read varint (pure ruby)    423.775k (± 0.8%) i/s    (2.36 μs/i) -      2.143M in   5.057579s
read leb128 (native)       201.781k (± 1.4%) i/s    (4.96 μs/i) -      1.012M in   5.016818s

Comparison:
read varint (pure ruby):   423774.9 i/s
read leb128 (native)   :   201781.3 i/s - 2.10x  slower

On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected.

However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked:

buffer = []
loop do
  byte = io.readbyte
  buffer << byte
  break if byte < 0x80 # this is the last byte of a LEB128 number
end
buffer.pack('C*').unpack1('R')

Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop:

value = bits = 0
loop do
  byte = io.readbyte
  value |= (byte & 0x7f) << bits
  bits += 7
  break if byte < 0x80
end
value

Therefore, if we can have a native ruby method that does scanning and decoding from IO in one shot, it would be a huge improvement.

io.unpack1('R')

[mame (Yusuke Endoh)]

I understand the motivation and have wanted this myself, but there seem to be many things to consider. unpack has specifiers like X and @ that do not fit streams. We would also need to decide how to handle specifiers with unbounded length, such as a*.

Also, the protobuf use case looks risky to me. Reading LEB128 from an untrusted stream without a limit can make us read endlessly and build a huge Integer. Protobuf itself avoids this by imposing its own limit of 10 bytes when reading a varint. But LEB128 itself has no length limit, so it would be unnatural for unpack to impose one. So a protobuf implementation would likely need its own limit-aware code anyway, and could not use this method.

[ntkme (Natsuki Natsume)]

The current String#unpack has its signature as:

String#unpack(template, offset: 0, &block)

The concern for unbounded IO read can be potentially solved by having a maxlen option:

IO#unpack(template, maxlen = nil, &block)

I agree that maxlen does not make sense for String#unpack, however in IO#unpack it would be the limit for the internal read operation as IO#unpack is effectively a method that combines blocking read and unpack.

On the other hand, the offset option in String#unpack does not make sense for IO due to IO#seek.

As for X and @ in template, it should behave like IO#seek, which raises Errno::ESPIPE if a IO does not support positioning/rewinding.

[matz (Yukihiro Matsumoto)]

I understand the need. Reading length-prefixed messages from a stream is common, and doing it efficiently in Ruby is hard today.

But I don't want to bring the whole unpack template to IO. As @mame (Yusuke Endoh) pointed out, X, @ and * assume random access to a finite string, and giving them stream semantics makes the same template behave differently on String and IO. We would also have to define what happens at EOF in the middle of a value, with nonblocking IO, and in text mode.

For fixed-size data, io.read(n).unpack already works. The real problem is variable-length data. Could you narrow the proposal to that? It could be a method that reads one variable-length value, or a lower-level primitive that lets Ruby code do it efficiently.

Matz.