Bug #9518

Objects in large arrays are leaked

Added by Charlie Somerville over 1 year ago. Updated about 1 year ago.

[ruby-core:60699]
Status:Closed
Priority:Normal
Assignee:-
ruby -v:ruby 2.1.0p0 (2013-12-25 revision 44422) [x86_64-darwin13.0] Backport:1.9.3: DONTNEED, 2.0.0: DONTNEED, 2.1: DONE

Description

a = [nil] * 131071
loop { a << Object.new; a.pop }

process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }

process RSS grows quickly and never falls

It seems to be related to this bit of code: https://github.com/github/ruby/blob/2.1/gc.c#L4764-4766

Associated revisions

Revision 45638
Added by normal about 1 year ago

gc.c: drop special case for big hash/array

  • gc.c (rb_gc_writebarrier): drop special case for big hash/array [Bug #9518]

Revision 45638
Added by normal about 1 year ago

gc.c: drop special case for big hash/array

  • gc.c (rb_gc_writebarrier): drop special case for big hash/array [Bug #9518]

Revision 45818
Added by Tomoyuki Chikanaga about 1 year ago

merge revision(s) r45638: [Backport #9518]

* gc.c (rb_gc_writebarrier): drop special case for big hash/array
  [Bug #9518]

Revision 45819
Added by Tomoyuki Chikanaga about 1 year ago

merge revision(s) r45298,r45311: [Backport #9518]

test/ruby/envutil.rb: compare RSS to check memory leak

* test/ruby/envutil.rb (Test::Test#assert_no_memory_leak): compare

also RSS if available.

* test/ruby/memory_status.rb (Memory::Status.parse): read string

form of an instance.

Revision 45820
Added by Tomoyuki Chikanaga about 1 year ago

merge revision(s) r45291,r45299,r45314,r45325: [Backport #9518]

* ext/dl/cptr.c (dlptr_free), ext/dl/handle.c (dlhandle_free),
  ext/fiddle/handle.c (fiddle_handle_free),
  ext/fiddle/pointer.c (fiddle_ptr_free): fix memory leak.
  based on the patch Heesob Park at  [Bug #9599].

History

#1 Updated by Charlie Somerville over 1 year ago

Oops, the markdown rendering messed up the code I pasted above.

a = [nil] * 131071
loop { a << Object.new; a.pop }
# process RSS stays stable

a = [nil] * 131072
loop { a << Object.new; a.pop }
# process RSS grows quickly and never falls

#2 Updated by Koichi Sasada over 1 year ago

Yes, you are right. WB (write barrier) strategy doesn't care such case.

How many such program?
If it is popular case, we can care about it.

#3 Updated by Eric Wong over 1 year ago

The yahns HTTP server uses a long-lived fdmap array to map
Fixnum(fileno) -> IO connections.
This array exists prevent GC from sweeping IOs (because IOs are watched
by Linux epoll and not markable w/o an Array to store them).

64K+ connections (array size) is attainable with yahns. The long-lived
fdmap array lifetime should not infect the short lived client
connections.

If I were to allow >=64K client connections on my public yahns instance,
clients will cause memory leaks/waste.

(yahns supports infinitely-lived connections, but in
practice, clients connections tend to be short-lived).

#4 Updated by Eric Wong about 1 year ago

ko1: may I commit the removal of special case for large array/hash?

  • gc.c (rb_gc_writebarrier): remove special case for large hash/array

http://bogomips.org/ruby.git/patch?id=40849ff1bc881
git://80x24.org/ruby.git gc-nomagic

I strongly believe there should no magic based on size.

#5 Updated by Koichi Sasada about 1 year ago

(2014/03/29 15:48), normalperson@yhbt.net wrote:

I strongly believe there should no magic based on size.

I need to ask tarui-san, who introduce this mechanism.

Tarui-san, could you give me a comment?

--
// SASADA Koichi at atdot dot net

#6 Updated by Masaya Tarui about 1 year ago

Sorry for such a late response.

I agree.

At the first time, I was introduled for reducing minor GC's long time
by rescan of huge size array.
But in real case, if there are huge size array, there may be other huge data too.
No matter how it do, it have to take time.

#7 Updated by Anonymous about 1 year ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Applied in changeset r45638.


gc.c: drop special case for big hash/array

  • gc.c (rb_gc_writebarrier): drop special case for big hash/array [Bug #9518]

#8 Updated by Eric Wong about 1 year ago

Thanks all! r45638

#9 Updated by Tomoyuki Chikanaga about 1 year ago

  • Backport changed from 1.9.3: UNKNOWN, 2.0.0: UNKNOWN, 2.1: UNKNOWN to 1.9.3: DONTNEED, 2.0.0: DONTNEED, 2.1: DONE

r45638 was backported into ruby_2_1 at r45818.
according to #9796

Also available in: Atom PDF