Bug #18134: Memory leak with master

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.1.0dev (2021-08-26T10:47:48Z master ef10e8a1eb) [x64-mingw32]
• Backport: 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONTNEED

[MSP-Greg (Greg L)]

While working on code to test Puma’s performance, I noticed a memory leak using WSL2 Ubuntu 20.04 and Ruby master. The leak seemed proportional to the number of requests made, and independent of the response size, either headers or body. Locally, I was running up to 2 million requests. The code used @ioquatix’s version of wrk and smem for measuring memory use. I ran GC.start and GC.compact before checking memory. The Puma configuration was using fork.

I’ve got a lot of Ruby versions on Windows, but haven’t created the same setup for Ubuntu. So, I took the above code and ran it on GitHub Actions Ubuntu 20.04. Ruby 2.5.9, 2.7.4, and 3.0.2 did not have the memory leak, but master had the same leak I saw locally.

Puma and one of its dependencies (nio4r) are both extension gems. Has there been an ABI change that might affect the code?

I can try to bisect it if that would be helpful.

Thanks, Greg

[MSP-Greg (Greg L)]

• Subject changed from Memory leak in master to Memory leak with master

[MSP-Greg (Greg L)]

The leak first appears after 0ef2923c2b Avoid rehashing in Hash#replace/dup/initialize_copy [Bug #16996].

Log I kept while bisecting:

FAIL  b9908ea666  ruby 3.1.0dev (2021-03-18T19:03:14Z master b9908ea666) [x86_64-linux]
FAIL  0ef2923c2b  ruby 3.1.0dev (2021-03-18T11:34:40Z master 0ef2923c2b) [x86_64-linux]
PASS  d094c3ef04  ruby 3.1.0dev (2021-03-18T11:34:40Z master d094c3ef04) [x86_64-linux]
PASS  e0dd072978  ruby 3.1.0dev (2021-03-18T06:20:41Z master e0dd072978) [x86_64-linux]

GitHub Commit History

Assuming this is caused by c code, Puma's only c code when running the test is the http parser, which has been relatively static, and is derived from the old Mongrel parser. I'm not familiar with the Nio4r code used...

[MSP-Greg (Greg L)]

I'm not that familiar with the best way to measure Ruby memory use.

The following code:

loops = ARGV[0] || 500_000

smem = "smem -c 'pid pss rss uss command'"

env = ENV.to_h
env_c = nil

GC.start; GC.compact
system smem

100_000.times { env_c = env.dup }

puts '', "Duped hash #{loops} times", ''

GC.start; GC.compact
system smem

Yielded the following. Both Ruby 3.0.2 and master show an increase in memory, but master uses 10 times as much...

  PID      PSS      RSS      USS
  865    20609    22484    20072  Ruby master start
  865   120927   122808   120680   "     "    end
                          100608              increase

  870    20468    22216    19872  Ruby 3.0.2 start
  870    30492    32268    29912   "     "   end
                           10040   "     "   increase

[MSP-Greg (Greg L)]

Sorry, code should be:

loops = ARGV[0].to_i || 500_000

smem = "smem -c 'pid pss rss uss command'"

env = ENV.to_h
env_c = nil

GC.start; GC.compact
system smem

loops.times { env_c = env.dup }

puts '', "Duped hash #{loops} times", ''

GC.start; GC.compact
system smem

[xtkoba (Tee KOBAYASHI)]

I can confirm this, and also that the leak stops by reverting 0ef2923c2b9 (#16996).

$ ./miniruby bug18134.rb
USER      PID   PPID  VSIZE  RSS   WCHAN            PC  NAME
u0_a116   12366 10224 50636  7388  poll_sched b6e3cd28 S ./miniruby

Duped hash 500000 times

USER      PID   PPID  VSIZE  RSS   WCHAN            PC  NAME
u0_a116   12366 10224 168728 125492 poll_sched b6e3cd28 S ./miniruby

$ ./miniruby.revert bug18134.rb
USER      PID   PPID  VSIZE  RSS   WCHAN            PC  NAME
u0_a116   12374 10224 50636  7388  poll_sched b6e34d28 S ./miniruby.revert

Duped hash 500000 times

USER      PID   PPID  VSIZE  RSS   WCHAN            PC  NAME
u0_a116   12374 10224 51608  8376  poll_sched b6e34d28 S ./miniruby.revert

BTW, the first line of the repro should be something like:

loops = (ARGV[0] || 500_000).to_i

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|a615885f1e87f4bfbc5398b060fd3a64d5de8c4a.

---

Free previously used tables [Bug #18134]

[nobu (Nobuyoshi Nakada)]

• Backport changed from 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONTNEED