Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin

• Status: Closed
• Assignee: -
• Target version: -

[hsbt (Hiroshi SHIBATA)]

On x64-mswin64, long is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in long, so none of them can reach 2GiB.

"a" * 2**31                              # RangeError: bignum too big to convert into 'long'
File.open(path, "rb") { it.read(2**31) } # RangeError
Array.new(2**31)                         # RangeError

I propose giving these lengths and indices a type of their own, rb_long_t, in two steps.

The first step is https://github.com/ruby/ruby/pull/18862. It defines typedef long rb_long_t everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding LONGT2NUM, NUM2LONGT and PRIdLONGT. A static assert keeps it long, so nothing changes, and NUM2LONG and FIX2LONG still return long.

The second step makes rb_long_t 64 bits on mswin, as rb_off_t already does for off_t there. mingw keeps long, and Fixnum (#14378) stays as is. This breaks extensions that pass a long * to an out-parameter such as rb_range_beg_len, because Ruby then writes 8 bytes into a 4-byte variable. -we4133 makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can typedef long rb_long_t when RB_LONGT_MAX is undefined.

No existing type fits. ssize_t and intptr_t are int on ILP32 and would change long * parameters there. SIGNED_VALUE is already 64 bits on mswin, so it could not land as a no-op first.

To catch new code that adds a plain long for a length, I would like a CI job that builds with rb_long_t set to long long on Linux, where %ld and long * mismatches fail.

First, do you agree with this direction and with carrying it out? If so, is rb_long_t a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think?

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #14378: Increase Fixnum range on Windows from 31 bits to 63 bits added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #20614: Integer#size returns incorrect values on 64-bit Windows added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #11235: [BUG] Segmentation fault added

[byroot (Jean Boussier)]

What do you think?

I think I'd be worth trying for a preview (there was no 4.1.0preview this year?) as to see what the blast radius really is.

Other than that, I wonder if one avenue could be to have this be a build flag first, allowing gem maintainers 1 year to test and handle the change (bonus point if it's available in ruby/setup-ruby.

[matz (Yukihiro Matsumoto)]

I agree with the direction. A 2GiB limit on a 64-bit platform should go away, and the name rb_long_t is fine with me since it follows rb_off_t.

Step 1 can go into 4.1. For step 2, I am worried about switching the default in 4.1 with no trial period. Besides the long * out-parameters, existing extensions that do long len = RSTRING_LEN(str) will still compile and silently truncate. I prefer @byroot's idea: make it a build option in 4.1 and switch the default in 4.2, unless a preview shows the impact is small.

Matz.

[matz (Yukihiro Matsumoto)]

One more thing: as far as mswin is concerned, please feel free to go ahead as you see fit.

Matz.

[rhenium (Kazuki Yamaguchi)]

I think this is a good step, but the conversion macro NUM2LONG(val) vs NUM2LONGT(val) seems incredibly confusing.

Were completely different names considered, such as len?

[byroot (Jean Boussier)]

Were completely different names considered, such as len?

Also the natural type to go for would be size_t, so rb_size_t would make more sense than rb_long_t? But then there's the sign difference...

[hsbt (Hiroshi SHIBATA)]

Thank you both. I'd still like to keep rb_long_t, since it replaces long one for one, like rb_off_t and OFFT2NUM. rb_size_t would sit next to SIZET2NUM for size_t, which seems even easier to confuse.

[rhenium (Kazuki Yamaguchi)]

My suggestion would be rb_len_t, or possibly index or count instead of len.

I don't think this is really comparable to rb_off_t ([Bug #5317]), which appears to have been introduced as an equivalent of POSIX-like off_t on Windows.

Besides LONG and LONGT seem too similar, I think naming the new type after long is misleading when the plan is to eventually decouple it from long. Since we're going to expect extensions to eventually use this new type and the helpers, why not give it a more descriptive name?

[hsbt (Hiroshi SHIBATA)]

Thanks. Your comment reminded me of what matz said at the developers' meeting: the representation should say 64 bits instead of relying on long or long long, whose meaning can change. So this type may well stop being long someday, and I agree a name based on long would mislead. I've renamed it to rb_len_t in https://github.com/ruby/ruby/pull/18862, with LEN2NUM, NUM2LEN and PRIdLEN.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Closed

Applied in changeset git|672cb48d09c4e0df8e2fd9868628fb825d849f58.

---

[Feature #22400] Mention rb_len_t in NEWS

Co-Authored-By: Claude Opus 5.5 [e-mail]

[hsbt (Hiroshi SHIBATA)]

• Status changed from Closed to Open

[byroot (Jean Boussier)]

@hsbt (Hiroshi SHIBATA) what's the migration path for C extensions? I don't see a HAVE_RB_LEN_T, should we provide one so that extensions can declare:

#ifndef HAVE_RB_LEN_T
typedef long rb_len_t;
typedef unsigned long rb_ulen_t;
#endif

[hsbt (Hiroshi SHIBATA)]

#ifndef SIZEOF_RB_LEN_T works for that today, since it is defined together with rb_len_t. Still, I agree a HAVE_ style macro like HAVE_RB_IO_T is the usual convention and easier to find, so I'll add HAVE_RB_LEN_T.

[larskanis (Lars Kanis)]

Both MSWIN and MINGW intentionally use the same ABI. They only have a different toolset. So I don't think we should introduce a ABI difference on the ruby C interface for them.

Likewise there's no good reason to handle MINGW and MSWIN differently because of compatibility, IMHO. The same source gems are used for MSWIN and for MINGW. So they need to be changed regarding rb_len_t in order to be compatible to MSWIN anyway. I think this is manageable for the time frame until Ruby-4.2 comes out, so that they are compatible to 64 bit rb_len_t on both MSWIN and MINGW then.

There are currently more binary gems available for MINGW than for MSWIN. But it doesn't matter, since already released binaries are bound to the Ruby ABI 3.3, 3.4, 4.0, etc. and Ruby-4.1/4.2 need new binaries anyway.

I understand that the ruby core developers prefer MSWIN and that testing on MINGW is cumbersome for you and slows down development. So I'm fine with checking the MSWIN related changes and port them to MINGW where necessary. You could give me a hint to do so, if there's some need. Anyway it's great that the Windows port receives some attention recently and I really appreciate all the efforts!

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Closed

Step 2 is merged in https://github.com/ruby/ruby/pull/19312, so String, Array and MatchData can now be 2GiB or longer on mswin. Next I'll work toward a 64-bit Fixnum on mswin (#14378) step by step, targeting Ruby 4.2.