Bug #20184: Ruby segfaults on Fly.io with 256 MB RAM

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
• Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONE

[aalin (Andreas Alin)]

I've been running a Ruby 3.2 app on Fly.io with 256 MB RAM and it has been working fine.
When I upgraded to Ruby 3.3.0, any Ruby script would crash immediately with a segfault.

I haven't been able to reproduce it outside of fly.io.

I created a repository with a basic rack app which crashes on fly.io.
The readme includes the full output and a GDB backtrace.
https://github.com/aalin/ruby-3-3-0-fly-crash

[nobu (Nobuyoshi Nakada)]

In shape.c:Init_default_shapes(), the result of mmap for rb_shape_tree_ptr->shape_cache seems unchecked.

I'm not sure if it can continue without shape_cache.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed
• Backport changed from 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED

8b8dcc7af174def5216044019c1d3e42edfdc7cf should have fixed this.

[ioquatix (Samuel Williams)]

I was also investigating this issue.

On low-memory systems (< ~400MB RAM+Swap), Init_default_shapes will fail and cause Ruby to segfault later.

Even something as simple as ruby -e "puts 123" will fail.

The 2nd mmap in shape.c tries to allocate 384MB as a cache. This fails with MAP_FAILED which is stored into a shape_cache pointer and later this pointer is derefenced causing a segfault.

The current merged fix skips the cache if the mmap fails, but might I suggest we explore the following:

• Divide and conquer strategy for finding an allocation size that works (e.g. divide by half and try again a few times).
• Expose an environment variable to control the size, so that those on memory constrained systems can at least enable a smaller cache.

[ioquatix (Samuel Williams)]

I also wonder if we should add some CI for running ruby on a low memory/virtual memory constrained system.

[anton-stepanof (Anton Stepanov)]

Backport PR for 3.3: https://github.com/ruby/ruby/pull/9805

[mame (Yusuke Endoh)]

• Has duplicate Bug #20263: Backport https://github.com/ruby/ruby/pull/9498 to Ruby 3.3 added

[naruse (Yui NARUSE)]

• Backport changed from 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONE

ruby_3_3 53f0c5a4e8834f11af0f903d2c59754d9be2a7f2.