Project

General

Profile

Actions

Feature #16352

open

Modify Marshal to dump objects larger than 2 GiB

Added by seoanezonjic (Pedro Seoane) about 2 years ago. Updated over 1 year ago.

Status:
Open
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:95898]
Tags:

Description

Using a gem called Numo-array to handle matrix operations, I found the following error while saving a large matrix:

in `dump': long too big to dump (TypeError)

Github thread is https://github.com/ruby-numo/numo-narray/issues/144. Digging with the authors, I found the following code that reproduces the error:

ruby -e 'Marshal.dump(" "*2**31)'

Executed in:
ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]

The marshal library has a limit based on constant SIZEOF_LONG. This check is performed in here. I don't understand the motivation of this limit. It has a great impact on libraries that need to serialize large objects such as numeric matrix. In this case, the limit >= 2 GiB is reached easily, and it blocks ruby development. I found another related bug report: #1560, but the Marshal problem was not addressed in it.

Actions

Also available in: Atom PDF