Bug #22487: Allocation between GCs and peak RSS keep growing on master when large String buffers are dropped

• Status: Open
• Assignee: eightbitraptor (Matt V-H)
• Target version: 4.1
• ruby -v: ruby 4.1.0dev (2026-10-28T21:25:41Z master 637bd7cba1) +PRISM [x86_64-linux]
• Backport: 3.3: DONTNEED, 3.4: DONTNEED, 4.0: DONTNEED

[watson1978 (Shizuo Fujita)]

Summary

On master (637bd7cba1), a loop that repeatedly allocates a String with a 1 MiB buffer, without keeping any reference to the previous Strings, triggers GC after larger and larger amounts of allocation. Peak RSS keeps rising as the loop runs. On Ruby 4.0.9, allocation between GCs settles at about 32 MiB, and peak RSS stays about 65 MiB above the starting RSS in every run we tried, up to 32 GiB allocated in total.

Since the loop does not keep the Strings, we would expect memory usage to stay bounded however long it runs.

Peak RSS below is VmHWM minus the RSS right after the GC.start that precedes the loop.

total allocated	4.0.9	master (637bd7cba1)
1 GiB	32 GCs, peak RSS +65 MiB	10 GCs, peak RSS +194 MiB
8 GiB	256 GCs, peak RSS +65 MiB	25 GCs, peak RSS +674 MiB
32 GiB	1024 GCs, peak RSS +65 MiB	48 GCs, peak RSS +1410 MiB

The behavior started with ec114e5701, and since e648c81dce the allocation between GCs no longer comes back down. It needs a heap with fragmented pages, which a normal start with RubyGems already has.

We found this through the test suite of Cumo, a CUDA array library. Cumo reports device memory with rb_gc_adjust_memory_usage, and its test that drops 1 GiB of 1 MiB arrays now grows the memory pool past its 256 MiB limit on master.

The behavior shows on the latest master we tried (637bd7cba1). The bisect results below come from separate builds of each listed commit. The proposed patch was tested against c2b6f7f0a7, which gives the same numbers as 637bd7cba1.

Environment

• ruby 4.1.0dev (2026-10-28T21:25:41Z master 637bd7cba1) +PRISM [x86_64-linux]
• ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [x86_64-linux]
• Linux 7.3.0 x86_64, glibc 2.44, GCC 16.2.1, Intel Core Ultra 9 275HX

For the bisect and the patch, each commit was built from source and run with make runruby RUN_OPTS= RUNOPT0=, so that RubyGems is loaded.

Reproduction

repro.rb (attached) allocates Strings with 1 MiB buffers one at a time and drops them. It prints the allocation between consecutive GCs, in MiB. An optional argument sets the total in GiB.

rss = ->(key) { File.read("/proc/self/status")[/^#{key}:\s+(\d+)/, 1].to_i / 1024 }
total_gib = Integer(ARGV[0] || 1)

GC.start
before = rss.("VmRSS")
gaps = []
since = 0
last = GC.count
(total_gib * 1024).times do
  String.new(capacity: 1 << 20) << "x"
  since += 1
  next if GC.count == last

  gaps << since
  since = 0
  last = GC.count
end

puts "#{RUBY_DESCRIPTION}"
puts "#{total_gib} GiB: #{gaps.size} GCs, peak RSS +#{rss.("VmHWM") - before} MiB"
puts "MiB allocated between GCs: #{gaps.inspect}"

$ ruby repro.rb
ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [x86_64-linux]
1 GiB: 32 GCs, peak RSS +65 MiB
MiB allocated between GCs: [16, 23, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32]

$ ruby repro.rb
ruby 4.1.0dev (2026-10-28T21:25:41Z master 637bd7cba1) +PRISM [x86_64-linux]
1 GiB: 10 GCs, peak RSS +194 MiB
MiB allocated between GCs: [17, 40, 72, 32, 33, 65, 97, 129, 161, 193]

$ ruby --disable-gems repro.rb
ruby 4.1.0dev (2026-10-28T21:25:41Z master 637bd7cba1) +PRISM [x86_64-linux]
1 GiB: 32 GCs, peak RSS +65 MiB
MiB allocated between GCs: [17, 24, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32]

After the first few GCs, the allocation between GCs on master grows by about 32 MiB (malloc_limit) at each GC.

RubyGems itself is not the cause. require "net/http" under --disable-gems shows the same growth. repro_fragmented.rb (attached) shows it with no library at all. It keeps every other one of 3,500 Strings before running the same loop:

$ ruby --disable-gems repro_fragmented.rb
ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [x86_64-linux]
1 GiB: 32 GCs, peak RSS +65 MiB

$ ruby --disable-gems repro_fragmented.rb
ruby 4.1.0dev (2026-10-28T21:25:41Z master 637bd7cba1) +PRISM [x86_64-linux]
1 GiB: 10 GCs, peak RSS +226 MiB
MiB allocated between GCs: [22, 31, 32, 33, 65, 97, 129, 161, 193, 225]

Bisect

The same loop (1 MiB x 1024, RubyGems loaded) on each commit. Max allocation between GCs is the largest allocation between two GCs in the loop.

commit		GCs	Max allocation between GCs
d375bcc965	branch point of ruby_4_0	32	32 MiB
cb9f2ba804	parent of ec114e5701	32	38 MiB
ec114e5701	Use monotonic add/sub counters for malloc_increase	15	161 MiB
8b7c4342af	Snapshot malloc counters at end of sweep	19	97 MiB
45fd5d58f1	Fix malloc counters during GC	15	161 MiB
a9c656ebd2	parent of e631daf17b	15	161 MiB
e631daf17b	Don't let GC's own frees pay for the next cycle's malloc budget	25	97 MiB
132d22571b	parent of e648c81dce	24	97 MiB
e648c81dce	Clamp oldmalloc_increase at zero instead of snapshotting it	10	193 MiB
c2b6f7f0a7	master as of 2026-10-24	10	193 MiB

The figure in the summary plots every GC for d375bcc965, 132d22571b and e648c81dce. Before e648c81dce, the allocation between GCs grew and then came back to 32 MiB from time to time. Since e648c81dce it does not come back. c2b6f7f0a7 gives exactly the same points as e648c81dce.

Cause

Why a debt appears

Since ec114e5701, malloc_increase is (malloc - malloc_at_last_gc) - (free - free_at_last_gc), and gc_reset_malloc_info records both baselines when a GC starts. The lazy sweep that follows frees the previous cycle's garbage, and those frees count against the new cycle. Once they exceed what the cycle has allocated so far, the increase goes below zero. Later allocations have to pay that back before malloc_limit is reached. The next cycle therefore lasts the previous cycle's garbage plus malloc_limit, so the allocation between GCs grows by 32 MiB at each GC.

4.0 kept one counter that was decremented on free and clamped at zero, so frees could not create a debt.

Why the sweep does not finish

e631daf17b updates free_at_last_gc in gc_sweep_finish to drop those frees. That only helps when the sweep finishes within the cycle.

A sweep step adds whole pages until it has the pool budget plus GC_INCREMENTAL_SWEEP_BYTES worth of free slots, at least about 3,000 40-byte slots. Only pages that hold both live objects and free slots count toward it. Fully empty and fully live pages are passed over.

Another step runs when a heap runs out of free slots. This loop allocates one object per MiB, so that rarely happens. gc_sweep_step_for_malloc also runs a step, but only once malloc_increase is already over the limit, at the end of the cycle. So when one step is not enough, the sweep spans the whole cycle.

We checked the state when each GC began, in the fragmented case without libraries. On master, 8 of 10 GCs began while the previous sweep was still running. When the sweep fits in one step, none of 32 did. On 4.0.9, 26 of 32 GCs began while the previous sweep was still running, so a sweep spanning cycles is normal. It only became a problem when frees started to create a debt.

Why it needs a fragmented heap

One step finishes the sweep unless there are enough pages that mix live objects and free slots.

After a normal start with RubyGems, the 40-byte heap has 12 pages that hold live objects, with about 9,100 free slots among them (counted with ObjectSpace.dump_all). With --disable-gems it has 5 such pages and about 1,700 free slots. With repro_fragmented.rb and other counts of Strings, the growth starts between about 4,000 and 6,800 free slots in such pages.

Why it stopped coming back down at e648c81dce

Before e648c81dce, majors triggered by oldmalloc recorded fresh baselines from time to time, which brought the allocation between GCs back to 32 MiB. Between e648c81dce and its parent, the loop goes from 15 minors and 9 majors (all by oldmalloc) to 9 minors and 1 major (by oldgen). e648c81dce fixed the extra majors as intended. It also removed the reset that had been hiding the debt.

Without a reset, cycle k allocates about 32 MiB x k, so after a total of T the last cycle is about sqrt(2 * 32 MiB * T). The measured peak RSS follows this up to 32 GiB: +674 MiB at 8 GiB, compared with an estimate of 724 MiB, and +1410 MiB at 32 GiB, compared with an estimate of 1448 MiB. We saw no reset in those runs.

Who is affected

Code that allocates large malloc'd buffers while creating few objects, for example reading or writing large files, building large strings, compression, or C extensions that report memory with rb_gc_adjust_memory_usage.

Code that creates many small objects keeps the sweep moving and is not affected much. In the fragmented case with 1 MiB buffers, allocating 100 small objects per buffer caps the allocation between GCs at 49 MiB, and 1,000 per buffer keeps it at 32 MiB. We have not measured a real application. Other majors in a real application may also reset the debt from time to time.

A possible fix

Go back to counters that stop at zero, the same shape e648c81dce used for oldmalloc_increase. The patch keeps two of them:

• increase counts every malloc and free and drives the malloc trigger. Frees subtract with atomic_sub_nounderflow, so they can no longer create a debt. The frees of a lazy sweep still lower it while the sweep runs, which the retry in objspace_malloc_increase_body relies on.
• mutator_increase counts every malloc, but subtracts a free only when during_gc is false. It also stops at zero.

gc_reset_malloc_info takes increase and sets both to zero. gc_sweep_finish replaces increase with mutator_increase. The sweep's own frees then stop counting, as e631daf17b intended, and the mutator's frees keep their order relative to its mallocs. The monotonic counters stay as they are, for GC.stat and gc_check_oldmalloc.

This is an experimental patch. It does not handle a malloc committed by another thread while gc_reset_malloc_info or gc_sweep_finish rewrites the counters, as described in "What changes and what we have not checked" below.

The patch (attached, against c2b6f7f0a7) changes gc/default/default.c and adds four tests to test/ruby/test_gc.rb. Its core:

/* malloc_increase_commit */
        gc_counter_add(&objspace->malloc_counters.counters.malloc, delta);
        RUBY_ATOMIC_SIZE_ADD(objspace->malloc_counters.increase, delta);
        RUBY_ATOMIC_SIZE_ADD(objspace->malloc_counters.mutator_increase, delta);
        ...
        gc_counter_add(&objspace->malloc_counters.counters.free, delta);
        atomic_sub_nounderflow(&objspace->malloc_counters.increase, delta);
        if (!during_gc) atomic_sub_nounderflow(&objspace->malloc_counters.mutator_increase, delta);

/* gc_reset_malloc_info */
        size_t inc = RUBY_ATOMIC_SIZE_EXCHANGE(objspace->malloc_counters.increase, 0);
        RUBY_ATOMIC_SIZE_EXCHANGE(objspace->malloc_counters.mutator_increase, 0);

/* called from gc_sweep_finish */
static inline void
gc_malloc_increase_drop_sweep_frees(rb_objspace_t *objspace)
{
    size_t mutator_increase = rbimpl_atomic_size_load(&objspace->malloc_counters.mutator_increase, RBIMPL_ATOMIC_RELAXED);
    RUBY_ATOMIC_SIZE_EXCHANGE(objspace->malloc_counters.increase, mutator_increase);
}

The rest of the patch reads the new counters in GC.stat, objspace_absorb and gc_marks_check, and updates comments.

Results

Builds from source, RubyGems loaded:

check	c2b6f7f0a7	with the patch
1 GiB of 1 MiB buffers	10 GCs, peak RSS +194 MiB	32 GCs, peak RSS +65 MiB
8 GiB of 1 MiB buffers	25 GCs, peak RSS +674 MiB	256 GCs, peak RSS +64 MiB
fragmented heap, --disable-gems	Max allocation between GCs 224 MiB	Max allocation between GCs 32 MiB
e631daf17b's benchmark (64 KiB x 200,000)	186 GCs, Max allocation between GCs 67 MiB	391 GCs, Max allocation between GCs 32 MiB
45fd5d58f1's script	count=196 minor=152 major=44	count=196 minor=152 major=44
e648c81dce's script	majors=1 minors=243	majors=1 minors=243
test/ruby/test_gc.rb, with the four new tests	103 tests, 4 failures	103 tests, 0 failures
test/ruby/test_gc_compact.rb	31 tests, 0 failures	31 tests, 0 failures
test/ruby/test_ractor.rb	82 tests, 0 failures	82 tests, 0 failures

The new tests fix malloc_limit at 16 MiB and check the following. All four fail without the patch. They are listed in the order they appear in the patch.

scenario	checks	c2b6f7f0a7	with the patch
Free before allocation	GCs while allocating 64 MiB right after freeing 64 MiB, at least 2	0	4
Allocation during an unfinished sweep	GCs while allocating 256 MiB on a fragmented heap, at least 12	6	16
Allocate and free repeatedly	malloc GCs while allocating and freeing a 1 MiB buffer 10,000 times, none	2	0
Frees during a lazy sweep	GCs when 48 MiB is allocated right after freeing 64 MiB during a lazy sweep, at least 1	0	1

4.0.9 gives the same numbers as the patch in all four.

Cost

Measured on one pinned core with perf stat -r 5, in two sets. Each value is the change from c2b6f7f0a7 in the mean of the 5 runs. The instruction counts round to the same change in both sets, so the table shows one value. Cycles show one value per set.

workload	instructions	cycles
e648c81dce's script (4 KiB x 2,000,000)	-0.6%	-4.0%, +2.8%
2 KB Strings x 2,000,000	-0.6%	+1.3%, +1.7%
20,000,000 small Arrays, no malloc	0.0%	-0.2%, +0.3%
e631daf17b's benchmark, --disable-gems	+0.2%	+0.6%, +1.0%

The trigger now reads one counter where master computed the increase from four, so the first two workloads run 0.6% fewer instructions. The 2 KB Strings still take about 1.5% more cycles in both sets, probably from the extra atomic add on each malloc commit. e631daf17b's benchmark runs 0.2% more instructions and takes 0.6% and 1.0% more cycles in the two sets. The other two workloads show changes in both directions between the sets.

The workloads that hit this issue get faster. On the same pinned core with RubyGems loaded, the 8 GiB run takes 0.257 s instead of 0.770 s, and e631daf17b's benchmark takes 0.400 s instead of 0.703 s (median of 5 runs, faster in all 5). The 8 GiB run does 256 GCs instead of 25, but reuses memory instead of growing.

What changes and what we have not checked

• GC runs about as often as on 4.0 in these workloads, which also changes when objects get old. In test_retained_malloc_triggers_oldmalloc_major_gc_despite_lazy_sweep_churn with 64 MB of extra garbage per iteration, oldmalloc majors go from 2 to 19 (20 on 4.0.9). The garbage Array now survives a few minors while it is built and is promoted before it is dropped. The test as it is still gives 1.
• A free is counted as the GC's or the mutator's by during_gc when it is committed. From reading the code, some can land on the wrong side: small frees buffered in malloc_increase_local across a GC boundary, frees by a thread without the GVL while another thread runs GC, dfree of zombies in finalize_deferred, and the gc_rest in objspace_absorb. We tried zombies with File.open("/dev/zero").getc in a loop. With malloc_limit fixed at 4 MiB, the patch matches 4.0.9 (241 GCs, 3 MiB apart) while master runs 30 GCs, up to 83 MiB apart. With the default limits all three stretch: 75 MiB on master, 76 MiB with the patch and 115 MiB on 4.0.9.
• gc_sweep_finish replaces increase, and gc_reset_malloc_info resets the two counters one after the other. From reading the code, a malloc committed by another thread in between can be lost from increase, which delays the next GC by its size. We have not tested frees from several threads or Ractors at once.
• A global GC calls gc_reset_malloc_info for the other objspaces before it settles their lazy sweeps, so the value it uses to grow malloc_limit can already be lowered by their sweep frees. This is from reading the code.
• We ran test_gc.rb, test_gc_compact.rb and test_ractor.rb on x86_64 Linux. We have not built it for 32-bit or Windows.

Workaround

Calling GC.start(full_mark: false) every 256 MiB keeps the peak of the 8 GiB run at +104 MiB, against +674 MiB without it and +65 MiB on 4.0.9. The GC has to sweep immediately. With immediate_sweep: false the debt stays, and the allocation between GCs keeps growing until the next forced GC.

We used Claude Code, an AI coding assistant, for the investigation, the measurements and the patch.

@eightbitraptor (Matt V-H), the bisect points at your ec114e5701 and e648c81dce, so could you take a look? @ko1 (Koichi Sasada), the patch tries to keep what e631daf17b intended.

[nobu (Nobuyoshi Nakada)]

• Assignee set to eightbitraptor (Matt V-H)
• Backport changed from 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN to 3.3: DONTNEED, 3.4: DONTNEED, 4.0: DONTNEED