Bug #9541: Markedly increased Hash memory usage if Hash is cleared before reinsertion.

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]

[hanke (Florian Hanke)]

After a recent upgrade to Ruby 2.1.0 I noticed a significant increase in memory usage when using hashes of any size when clearing the hashes before inserting.

The compared versions are:
ruby 1.9.3dev (2011-09-23 revision 33323) [x86_64-darwin12.2.1]
ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0]

To test this behaviour I repeatedly (10x) filled in 1 million different string keys, each with a nil value.

Results:

• With Ruby 1.9.3, RSS used stays mostly constant (to 157952 when clearing before inserting).
• With Ruby 2.1.0, RSS used grows far more rapidly when the Hash is cleared before inserting (to 241972, mostly right after refilling for the first time), but also jumps up once when this is not done.

I've uploaded my measurements and test setup here:
https://gist.github.com/floere/9111529

My assumption is that the GC is not run at appropriate times due to more lax GC settings.

Note: I was not sure whether this is a bug or not. It does lead to a marked increase in memory usage for example on Heroku.

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed
• Backport deleted (1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN)

This difference is probably due to addition of the generational garbage collector introduced in Ruby 2.1, in combination with the specifics of the malloc implementation you are using. hashmemtest2 is expected to perform worse as you are resizing the hash on each iteration, as opposed to hashmemtest1, which does not change the hash size after the first iteration.

For hashmemtest1.rb with ruby 2.1 in my environment:

    0 jeremy   11138 14.3  0.8 125752 136396 p5  S+     7:51PM    0:03.33 ruby21 hm1.rb
    1 jeremy   11138 22.1  1.0 162168 172808 p5  S+     7:51PM    0:05.40 ruby21 hm1.rb
    2 jeremy   11138 29.3  1.0 162168 172804 p5  S+     7:51PM    0:07.25 ruby21 hm1.rb
    3 jeremy   11138 35.9  1.0 162168 172804 p5  S+     7:51PM    0:09.12 ruby21 hm1.rb
    4 jeremy   11138 42.2  1.0 162168 172808 p5  S+     7:51PM    0:10.98 ruby21 hm1.rb
    5 jeremy   11138 44.7  1.0 162168 172804 p5  S+     7:51PM    0:12.85 ruby21 hm1.rb
    6 jeremy   11138 49.7  1.0 162168 172804 p5  S+     7:51PM    0:14.75 ruby21 hm1.rb
    7 jeremy   11138 54.1  1.0 162168 172804 p5  S+     7:51PM    0:16.70 ruby21 hm1.rb
    8 jeremy   11138 60.4  1.0 162168 172804 p5  S+     7:51PM    0:18.74 ruby21 hm1.rb
    9 jeremy   11138 63.4  1.0 162168 172804 p5  S+     7:51PM    0:20.67 ruby21 hm1.rb

and hashmemtest2.rb:

    0 jeremy   64509 13.8  0.8 125732 136384 p5  S+     7:52PM    0:03.52 ruby21 hm2.rb
    1 jeremy   64509 29.6  1.0 162148 172792 p5  S+     7:52PM    0:07.06 ruby21 hm2.rb
    2 jeremy   64509 39.2  1.0 162144 172788 p5  S+     7:52PM    0:10.30 ruby21 hm2.rb
    3 jeremy   64509 49.8  1.0 162144 172788 p5  S+     7:52PM    0:13.60 ruby21 hm2.rb
    4 jeremy   64509 56.6  1.0 162376 173020 p5  S+     7:52PM    0:17.11 ruby21 hm2.rb
    5 jeremy   64509 63.8  1.0 162368 173012 p5  S+     7:52PM    0:20.62 ruby21 hm2.rb
    6 jeremy   64509 70.2  1.0 162364 173008 p5  S+     7:52PM    0:24.20 ruby21 hm2.rb
    7 jeremy   64509 73.8  0.9 147212 157856 p5  S+     7:52PM    0:27.46 ruby21 hm2.rb
    8 jeremy   64509 78.6  0.9 140328 150976 p5  S+     7:52PM    0:30.66 ruby21 hm2.rb
    9 jeremy   64509 81.4  0.9 133444 144088 p5  S+     7:52PM    0:34.00 ruby21 hm2.rb

In short, I do not think this is a bug, or if it was, it was addressed in a later release of Ruby 2.1 (I tested 2.1.9).