Feature #22185: Add alignment directives to `Array#pack` and `String#unpack`

• Status: Open
• Assignee: -
• Target version: -

[nobu (Nobuyoshi Nakada)]

Array#pack and String#unpack can express byte skips and absolute positions with x, X, and @, but they do not currently have a direct way to align the current position to a byte boundary or to the ABI alignment of another directive.

This makes it awkward to describe binary formats that follow C-like structure layout rules.
Users need to calculate padding manually, and the required padding can depend on the platform ABI.

I propose adding alignment forms to pack templates.

Alignment Directive

[1, 2].pack("C x!4 C")
# => "\x01\x00\x00\x00\x02"

x!N advances to the next N-byte boundary, filling with NUL bytes when packing and skipping bytes when unpacking.

The alignment can also be specified by another directive:

[1, 2].pack("C x!i i")

Here x!i aligns to the native alignment of the i directive.
This lets templates describe platform-dependent native layouts without spelling out the padding size.

Alignment Mode

For formats that need repeated C-structure-style alignment, add an alignment mode:

[1, 2].pack("!{C i}")

This is equivalent to inserting the necessary alignment before each numeric or pointer directive, for example:

[1, 2].pack("C x!i i")

The alignment mode is intentionally not nested.

Relative and Absolute Alignment

There is also a distinction between relative and absolute alignment when buffer: or offset: is used.
x! aligns relative to the starting position of the current pack/unpack operation, while @! aligns relative to the beginning of the output or input string:

buffer = +"z"

[1, 2].pack("C x!4 C", buffer: buffer)
buffer
# => "z\x01\x00\x00\x00\x02"

buffer = +"z"

[1, 2].pack("C @!4 C", buffer: buffer)
buffer
# => "z\x01\x00\x00\x02"

Similarly for unpacking:

"z\x01\x00\x00\x00\x02".unpack("C x!4 C", offset: 1)
# => [1, 2]

"z\x01\x00\x00\x02".unpack("C @!4 C", offset: 1)
# => [1, 2]

This keeps x! useful for templates that describe a sub-layout starting at the given buffer:/offset: position, while @! covers formats whose alignment is based on the absolute position in the whole binary string.

Caveats

The directive-alignment form rejects endian modifiers, since byte order does not affect ABI alignment.
For example, x!i is meaningful, but x!i< should not be accepted as an alignment specifier.

The alignment is for the pack/unpack offset, not a guarantee about the native address of the underlying String buffer.
Callers that need to pass the buffer to C as a typed pointer must ensure separately that the actual address is suitably aligned.
Shared substrings, for example, may start at an arbitrary byte offset.

Examples

A concrete use case is building or reading binary records whose layout is shared with C code.
For example, a file format, IPC message, device command buffer, or FFI payload may contain records equivalent to:

struct entry {
    uint8_t tag;
    uint32_t size;
    void *data;
};

Today, a Ruby template has to spell out the padding manually:

[tag, size, data].pack("C xxx L J")

This is both error-prone and ABI-dependent.
The padding after tag depends on the native alignment of uint32_t, and the next field may depend on pointer alignment.
With alignment directives, the template can express the intended layout directly:

[tag, size, data].pack("!{C L J}")

or, when only one boundary is needed:

[tag, size].pack("C x!L L")

One target use case is binary buffers that contain several C-like record types.
For example, an IPC or device buffer may store a sequence of records, each with its own layout.
It is natural to pack or unpack each record with a separate template at a given buffer: or offset: position.
In that case x! is useful for alignment relative to the start of the current record, while @! is useful for formats whose alignment is defined relative to the start of the whole buffer.

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)