Bug #5437
Using fibers leads to huge memory leak
| Status: | Assigned | Start date: | 10/12/2011 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: |  ko1 (Koichi Sasada) |
% Done: | 0% |
|
| Category: | core | |||
| Target version: | - | |||
| ruby -v: | ruby 1.9.3dev (2011-10-11 revision 33457) [x86_64-linux] |
Description
It appears to me that there is something wrong with reallocating (reusing?) memory used by fibers. Here is a little script:
<code>
require 'fiber'
fibers = 10_000.times.map do
Fiber.new do
f = Fiber.current
Fiber.yield
end
end; nil
fibers.each(&:resume); nil
fibers.each(&:resume); nil
fibers = []
GC.start
</code>
Running this code in IRB multiple times leads to consuming more and more memory every time. However every time I execute this code the amount of newly consumed memory is lower. I was able to repeat this bug with ruby 1.9.2.p290 and 1.9.3-head using 32 and 64 bit architecture.
Here are some memory stats:
a) 32bit
Iter | VSZ | RSS
1 186268 63508
2 360196 119580
3 360340 147600
4 360484 178552
5 360616 199968
6 360612 210716
7 360612 221920
8 360612 224544
9 360612 229976
b) 64bit
1 395152 107600
2 739924 176532
3 739924 187972
4 739924 203300
5 739924 217716
6 739924 218344
7 739924 218540
8 739924 234040
9 739924 234256
We can see on our production instance that the more Fibers are in use the more memory is never reclaimed back to OS and the bigger the leak is.
History
Updated by normalperson (Eric Wong) 7 months ago
Robert Pankowecki <robert.pankowecki@gmail.com> wrote:
> ruby -v: ruby 1.9.3dev (2011-10-11 revision 33457) [x86_64-linux]
> fibers = []
>
> GC.start
GC.start is never guaranteed to release memory (especially not with
lazy sweep). Even if GC.start always releases memory (via free(3),
malloc(3)/free(3) implementations are not guaranteed to release memory
back to the kernel.
> We can see on our production instance that the more Fibers are in use
> the more memory is never reclaimed back to OS and the bigger the leak
> is.
Since you're on Linux, I assume you're using glibc/eglibc and hitting
this problem with its malloc. The comment starting with
"Update in 2006" in malloc/malloc.c (git://sourceware.org/git/glibc.git)
can give you an explanation of why memory isn't always given back to
the kernel.
To workaround this glibc malloc issue, you can try setting
MALLOC_MMAP_THRESHOLD_=131072 (or another suitable value) in the env and
possibly trade performance for less memory use.
Updated by rupert (Robert Pankowecki) 7 months ago
- File cieknace_fibery added
- File 65536.txt added
- File 131072.txt added
- File 262144.txt added
I tried to use your setting but as you can see in the results files it does not change very much. Also I added finalizers to those fibers to make sure that they are garbage collected and most of them are. So why is the memory increasing when new fibers can use the memory allocated by the old, unused, garbagecollected fibers ? Am I misunderstanding something?
Updated by ko1 (Koichi Sasada) 2 months ago
- Category set to core
- Assignee set to ko1 (Koichi Sasada)
I'll review it later. however, i think it is libc (malloc) issue...
Updated by shyouhei (Shyouhei Urabe) 2 months ago
- Status changed from Open to Assigned