Project

General

Profile

Feature #16352

Updated by sawa (Tsuyoshi Sawada) over 1 year ago

Hi 
 Using a gem called Numo-array to handle matrix operations, operations called Numo-array I found the following error while saving a when save large matrix: 

 ``` 
 in `dump': long too big to dump (TypeError) 
 ``` 

 Github thread is https://github.com/ruby-numo/numo-narray/issues/144. thread: https://github.com/ruby-numo/numo-narray/issues/144 
 Digging with the authors, I we found the following code that reproduces the error: 

 
 ``` 
 ruby -e 'Marshal.dump(" "*2**31)' 
 ``` 

 
 Executed in: in : 
 ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux] 

 The marshal library    has a limit based on constant `SIZEOF_LONG`. that is checked with the SIZEOF_LONG constant. This check is performed in [here](https://github.com/ruby/ruby/blob/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301L321). this line https://github.com/ruby/ruby/blob/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Marshal.c file. I don't understand the motivation of this limit. It limit and has a great impact on in libraries that need to serialize large objects such as numeric matrix. In this case, the limit of    >= 2 GiB is it's reached easily, easily and it blocks the ruby development. development in scientifical projects as cited. I found another related other bug report: related: #1560, but the Marshal problem itself was not addressed in it. this case. 
 Thank you in advance 
 PEdro Seoane

Back