https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112013-11-15T14:45:36ZRuby Issue Tracking SystemRuby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429522013-11-15T14:45:36Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>kosaki (Motohiro KOSAKI)</i></li></ul><p>Could you comment this?</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429542013-11-15T14:49:06Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Third Party's Issue</i></li></ul><p>If system malloc is replaced with those newer libraries, ruby will use it.<br>
Otherwise, configure with LIBS=-ljemalloc.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429552013-11-15T14:49:43Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Category</strong> set to <i>build</i></li><li><strong>Assignee</strong> deleted (<del><i>kosaki (Motohiro KOSAKI)</i></del>)</li></ul> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429562013-11-15T15:23:17Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p>On one level, this feels like a non-brainer. But then the question is<br>
why the standard memory allocator in libc hasn't been improved.</p>
<p>I can imagine all kinds of reasons, from "alternatives use too much<br>
memory" to "not invented here". Any background info?</p>
<p>Regards, Martin.</p>
<p>On 2013/11/15 12:08, sam.saffron (Sam Saffron) wrote:</p>
<blockquote>
<p><a href="https://bugs.ruby-lang.org/issues/9113" class="external">https://bugs.ruby-lang.org/issues/9113</a></p>
</blockquote>
<blockquote>
<hr>
<p>I recently benched Discourse with tcmalloc / jemalloc and default and noticed 2 very important thing:</p>
<p>median request time reduce by up to 10% (under both)<br>
PSS (proportional share size) is reduced by 10% under jemalloc and 8% under tcmalloc.</p>
<p>We can always use LD_PRELOAD to yank these in, but my concern is that standard distributions are using a far from optimal memory allocator. It would be awesome if the build, out-of-the-box, just checked if it was on Linux (eg: <a href="https://github.com/antirez/redis/blob/unstable/src/Makefile#L30-L34" class="external">https://github.com/antirez/redis/blob/unstable/src/Makefile#L30-L34</a> ) and then used jemalloc instead.</p>
</blockquote> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429592013-11-15T16:16:58Zsam.saffron (Sam Saffron)sam.saffron@gmail.com
<ul></ul><p>@martin this is a great "oldish" article by facebook about this <a href="http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919" class="external">http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919</a></p>
<p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/4">@nobu (Nobuyoshi Nakada)</a> I guess my suggestion here is to include jemalloc source in the the repo, and compile on demand for linux (by default with an option to opt-out) that way everyone will pick this change up and it becomes "officially blessed" allocator.</p>
<p>At Github <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/390">@tmm1 (Aman Karmani)</a> has been using tcmalloc for years now, fragmentation is less good than jemalloc (tmm1 said perf is better, but I think its time to re-test cause I found jemalloc to perform better)</p>
<p>Regardless, libc allocator is a problem and default compiles should not use it.</p>
<p>Firefox has been using jemalloc for years and years, it is safe for production <a href="http://glandium.org/blog/?p=2581" class="external">http://glandium.org/blog/?p=2581</a> ..</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=429642013-11-15T21:53:29Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Third Party's Issue</i> to <i>Rejected</i></li></ul><p>Then it is a task of package maintainers.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=430692013-11-22T11:40:11Zkosaki (Motohiro KOSAKI)kosaki.motohiro@gmail.com
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/50">@duerst (Martin Dürst)</a> It is not correct. I and glibc folks are working on several improvement about malloc. Moreover, each allocator has different pros/cons. jemalloc can retrieve some workload better and glibc allocator can retrieve some other workload. There is no single perfect allocator. That's our difficulty.</p>
<p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/5660">@sam.saffron (Sam Saffron)</a> The Facebook's page you pointed out is out of date. It compare glibc 2.5 vs jemalloc 2.1.0. But latest are glibc 2.18 and jemalloc 3.4.1. And, glibc malloc and jemalloc shares a lot of basic design. So, this documentation is completely useless. If<br>
you have several workload which glibc doesn't work well, please make and share benchmark instead of rumor. Then, we can improve several bottlenecks.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=430702013-11-22T11:41:19Zkosaki (Motohiro KOSAKI)kosaki.motohiro@gmail.com
<ul></ul><p>It does NOT mean jemalloc has no chance. But we don't discuss performance issue if nobody has a number.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=430752013-11-22T14:44:23Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Status</strong> changed from <i>Rejected</i> to <i>Feedback</i></li></ul> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=445232014-01-22T22:19:43Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>Btw, jemalloc 3.5 includes an updated non-standard experimental API.</p>
<p>It looks like it has the ability to specify different arenas for<br>
allocation (via MALLOCX_ARENA(a)). Perhaps could be used to<br>
distinguish long/short-lived allocations.<br>
Probably worth experimenting on some day...</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=446042014-01-26T02:11:02Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>I tried jemalloc 3.5.0 vs eglibc 2.13-38 (Debian x86_64)</p>
<p><a href="http://80x24.org/bmlog-20140126-003136.7320.gz" class="external">http://80x24.org/bmlog-20140126-003136.7320.gz</a></p>
<p>Mostly close results, but I think our "make benchmark" suite is<br>
incomplete and we need more fork/concurrency-intensive benchmarks of<br>
large apps.</p>
<p>io_file_read and vm2_bigarray seem to be big losses because jemalloc<br>
tends to release large allocations back to the kernel more aggressively<br>
(and the kernel must zero that memory).</p>
<p>[1] I have applied two patches for improved benchmark consistency:<br>
<a href="https://bugs.ruby-lang.org/issues/5985#change-44442" class="external">https://bugs.ruby-lang.org/issues/5985#change-44442</a><br>
<a href="https://bugs.ruby-lang.org/issues/9430" class="external">https://bugs.ruby-lang.org/issues/9430</a><br>
(Note: I still don't trust the vm_thread* benchmarks too much,<br>
they seem very inconsistent even with no modifications)</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=452602014-02-18T23:39:59Zsam.saffron (Sam Saffron)sam.saffron@gmail.com
<ul></ul><p>I can confirm 2 findings.</p>
<p>When heaps are small you barely notice a different.<br>
When heaps grow and general memory fragmentation grows, jemalloc is far better.</p>
<p>I see a 6% reduction of RSS running discourse bench on 2.1.0 <a href="https://github.com/discourse/discourse/blob/master/script/bench.rb" class="external">https://github.com/discourse/discourse/blob/master/script/bench.rb</a></p>
<p>An artificial test is:</p>
<pre><code>@retained = []
MAX_STRING_SIZE = 100
def stress(allocate_count, retain_count, chunk_size)
chunk = []
while retain_count > 0 || allocate_count > 0
if retain_count == 0 || (Random.rand < 0.5 && allocate_count > 0)
chunk << " " * (Random.rand * MAX_STRING_SIZE).to_i
allocate_count -= 1
if chunk.length > chunk_size
chunk = []
end
else
@retained << " " * (Random.rand * MAX_STRING_SIZE).to_i
retain_count -= 1
end
end
end
start = Time.now
stress(1_000_000, 600_000, 200_000)
puts "Duration: #{(Time.now - start).to_f}"
puts `ps aux | grep #{Process.pid} | grep -v grep`
</code></pre>
<a name="For-glibc"></a>
<h2 >For glibc<a href="#For-glibc" class="wiki-anchor">¶</a></h2>
<pre><code>sam@ubuntu ~ % time ruby stress_mem.rb
Duration: 0.705922489
sam 17397 73.0 2.5 185888 156884 pts/10 Sl+ 10:37 0:00 ruby stress_mem.rb
ruby stress_mem.rb 0.78s user 0.08s system 100% cpu 0.855 total
</code></pre>
<a name="For-jemalloc-350"></a>
<h2 >For jemalloc 3.5.0<a href="#For-jemalloc-350" class="wiki-anchor">¶</a></h2>
<pre><code>Duration: 0.676871705
sam 17428 70.0 2.3 186248 144800 pts/10 Sl+ 10:37 0:00 ruby stress_mem.rb
LD_PRELOAD=/home/sam/Source/jemalloc-3.5.0/lib/libjemalloc.so ruby 0.68s user 0.09s system 100% cpu 0.771 total
</code></pre>
<p>--</p>
<p>You can see the 8% or so better RSS with jemalloc</p>
<p>Note the more iterations you add the better jemalloc does. up allocations to 10 million</p>
<p>jemalloc 200mb rss vs glibc 230mb rss</p>
<p>glibc gets fragmented at a far faster rate than jemalloc</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=452622014-02-18T23:53:38Zsam.saffron (Sam Saffron)sam.saffron@gmail.com
<ul></ul><p>Note, this pattern of</p>
<ol>
<li>Retaining large number of objects</li>
<li>Allocating a big chunk of objects (and releasing)</li>
<li>Repeating (2)</li>
</ol>
<p>Is very representative of web apps / rails apps. For our application requests will range between 20k allocations and 200k allocations.</p>
<p>It is very much a scenario we want to optimise for.</p>
<p>on another note Rust lang just picked jemalloc, golang uses a fork of tcmalloc <a href="http://golang.org/src/pkg/runtime/malloc.h?h=tcmalloc" class="external">http://golang.org/src/pkg/runtime/malloc.h?h=tcmalloc</a></p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=452632014-02-19T00:00:51Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:sam.saffron@gmail.com" class="email">sam.saffron@gmail.com</a> wrote:</p>
<blockquote>
<p>An artificial test is:</p>
<p>@retained = []</p>
<p>MAX_STRING_SIZE = 100</p>
<p>def stress(allocate_count, retain_count, chunk_size)</p>
</blockquote>
<p>Note: I think we should seed the RNG to a constant to have<br>
consistent data between runs</p>
<pre><code> srand(123)
</code></pre>
<blockquote>
<p>chunk = []<br>
while retain_count > 0 || allocate_count > 0<br>
if retain_count == 0 || (Random.rand < 0.5 && allocate_count > 0)<br>
chunk << " " * (Random.rand * MAX_STRING_SIZE).to_i<br>
allocate_count -= 1<br>
if chunk.length > chunk_size<br>
chunk = []<br>
end<br>
else<br>
@retained << " " * (Random.rand * MAX_STRING_SIZE).to_i<br>
retain_count -= 1<br>
end<br>
end<br>
end</p>
</blockquote>
<p>Sam: Thank you!</p>
<p>I think we should integrate this test into the mainline benchmark suite.<br>
Perhaps even provide an option to run with all the existing tests with<br>
the big @retained array.</p>
<p>ko1: what do you think?</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=452642014-02-19T00:05:37Zsam.saffron (Sam Saffron)sam.saffron@gmail.com
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/2491">@eric (Eric Anderson)</a></p>
<p>sure bench needs a bit more love to be totally representative of a rails request. Also this test will do ko1 lots of help improving the promotion to oldgen algorithm, we are talking about changing oldgen promotion to either use additional flags (as a counter) or only promote on major GC.</p>
<p>Either change will slash RSS in this test.</p>
<pre><code>sam@ubuntu ~ % rbenv shell 2.1.0
sam@ubuntu ~ % ruby stress_mem.rb
Duration: 5.459891703
sam 17870 109 3.8 267076 238732 pts/10 Sl+ 11:03 0:05 ruby stress_mem.rb
sam@ubuntu ~ % rbenv shell 2.0.0-p353
sam@ubuntu ~ % ruby stress_mem.rb
Duration: 7.616282557
sam 17986 95.6 2.0 151120 125684 pts/10 Sl+ 11:04 0:07 ruby stress_mem.rb
sam@ubuntu ~ %
</code></pre>
<p>This is basically a repro of the memory growth under 2.1.0 people are seeing.</p>
<p>238mb in Ruby 2.1 vs 125mb in 2.0</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=453162014-02-21T00:57:05Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>I'm absolutely against including external libraries into ruby repository itself, e.g., libyaml.<br>
It may not be the worst idea to bundle them with the tarballs.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=454292014-02-22T21:07:24Zsam.saffron (Sam Saffron)sam.saffron@gmail.com
<ul></ul><p>@nobusan I think that would be a reasonable approach</p>
<p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/2491">@eric (Eric Anderson)</a> / <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a> / everyone</p>
<p>here are the results of running that script across every 5 builds in the last year (with seeded rand)</p>
<p><a href="https://gist.github.com/SamSaffron/9162366" class="external">https://gist.github.com/SamSaffron/9162366</a></p>
<p>I think big jumps should be investigated</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=454782014-02-25T17:38:48Zko1 (Koichi Sasada)
<ul></ul><p>(2014/02/19 9:08), Eric Wong wrote:</p>
<blockquote>
<p>Btw, I also hope to experiment with a slab allocator since many internal<br>
objects are around the same size (like an OS kernel). This idea is<br>
originally from the Solaris kernel, but also in Linux and FreeBSD. One<br>
benefit with slab allocators over a general purpose malloc is malloc<br>
has too little context/information make some decisions:</p>
<ul>
<li>long-lived vs short-lived (good for CoW)</li>
<li>shared between threads or not</li>
<li>future allocations of the same class</li>
</ul>
<p>Notes on slab: I don't think caching constructed objects like the<br>
reference Solaris implementation does is necessary (or even good),<br>
since it should be possible to transparently merge objects of different<br>
classes (like SLUB in Linux, I think).</p>
<p>Anyways, I think jemalloc is a great general-purpose malloc for things<br>
that don't fit well into slabs. And it should be easy to let a slab<br>
implementation switch back to general-purpose malloc for<br>
testing/benching.</p>
</blockquote>
<p>Recently I'm working around this topic.</p>
<p>(1) Life-time oriented, similar to Copying GC<br>
(2) CoW frindly (read only) memories</p>
<p>More detail about (2):<br>
The following figure shows the stacked memory usage (snapshot) collected<br>
by valgrind/massif, on discorse benchmark by @sam's help.<br>
<a href="http://www.atdot.net/fp_store/f.69bk1n/file.copipa-temp-image.png" class="external">http://www.atdot.net/fp_store/f.69bk1n/file.copipa-temp-image.png</a></p>
<p>Interestingly, 50MB is consumed by iseq (iseq.c, compile.c). Most of<br>
data are read only, so it can be more CoW frindly. Now, we mixes<br>
read-only data and r/w data such as inline cahce.</p>
<p>There are several ideas. And I belive it is good topic to consider for<br>
Ruby 2.2.</p>
<p>--<br>
// SASADA Koichi at atdot dot net</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=454792014-02-25T18:48:25Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:ko1@atdot.net" class="email">ko1@atdot.net</a> wrote:</p>
<blockquote>
<p>(2014/02/19 9:08), Eric Wong wrote:</p>
<blockquote>
<p>Btw, I also hope to experiment with a slab allocator since many internal<br>
objects are around the same size (like an OS kernel). This idea is<br>
originally from the Solaris kernel, but also in Linux and FreeBSD. One<br>
benefit with slab allocators over a general purpose malloc is malloc<br>
has too little context/information make some decisions:</p>
<ul>
<li>long-lived vs short-lived (good for CoW)</li>
<li>shared between threads or not</li>
<li>future allocations of the same class</li>
</ul>
<p>Notes on slab: I don't think caching constructed objects like the<br>
reference Solaris implementation does is necessary (or even good),<br>
since it should be possible to transparently merge objects of different<br>
classes (like SLUB in Linux, I think).</p>
<p>Anyways, I think jemalloc is a great general-purpose malloc for things<br>
that don't fit well into slabs. And it should be easy to let a slab<br>
implementation switch back to general-purpose malloc for<br>
testing/benching.</p>
</blockquote>
<p>Recently I'm working around this topic.</p>
<p>(1) Life-time oriented, similar to Copying GC<br>
(2) CoW frindly (read only) memories</p>
</blockquote>
<p>Yes. We should be able to do moving/defragmentation of long-lived<br>
internal allocations, even.</p>
<blockquote>
<p>Interestingly, 50MB is consumed by iseq (iseq.c, compile.c). Most of<br>
data are read only, so it can be more CoW frindly. Now, we mixes<br>
read-only data and r/w data such as inline cahce.</p>
</blockquote>
<p>Yes, also the iseq struct is huge (300+ bytes on 64-bit). I think we<br>
can shrink it (like I did with struct vtm/time_object) and move r/w data<br>
off to a different area.</p>
<blockquote>
<p>There are several ideas. And I belive it is good topic to consider for<br>
Ruby 2.2.</p>
</blockquote>
<p>OK; especially since this should have no public API breakage.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=454902014-02-26T09:12:54Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>Sam: btw, if you have time, can you prepare a patch which integrates<br>
jemalloc with the build/tarball dist?</p>
<p>We should also see if using the non-standard jemalloc API is worth it.<br>
(with fallbacks to standard APIs on systems where jemalloc is unsupported).</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=467052014-05-13T02:58:31Znormalperson (Eric Wong)normalperson@yhbt.net
<ul><li><strong>File</strong> <a href="/attachments/4424">0001-configure.in-add-with-jemalloc-option.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/4424/0001-configure.in-add-with-jemalloc-option.patch">0001-configure.in-add-with-jemalloc-option.patch</a> added</li></ul><p>Attached patch to use jemalloc by default on GNU/Linux iff<br>
jemalloc is already installed. Maybe we can integrate/force<br>
it later, but this is a first step (and some distros like<br>
Debian already ship jemalloc).</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=467062014-05-13T03:22:46Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>"<code>linux-gnu</code>" in "<code>target_os</code>" is substituted by "<code>linux</code>", so your <code>AS_CASE</code> never match.<br>
You should check for <code>libjemalloc</code> availability instead, I think.</p>
<pre><code class="diff syntaxhl" data-language="diff"><span class="gh">diff --git a/configure.in b/configure.in
index cf317af..a9037d6 100644
</span><span class="gd">--- a/configure.in
</span><span class="gi">+++ b/configure.in
</span><span class="p">@@ -1170,6 +1170,17 @@</span> AS_IF([test "x$with_gmp" != xno],
with_gmp="$ac_cv_lib_gmp___gmpz_init"
AS_IF([test -z "$with_gmp"], [with_gmp=no])])
<span class="gi">+dnl use jemalloc on GNU/Linux, assume non-GNU has a better malloc than glibc
+AC_ARG_WITH([jemalloc],
+ [AS_HELP_STRING([--with-jemalloc],
+ [use jemalloc allocator. Default is yes on GNU/Linux, no elsewhere])],
+ )
+AS_IF([test "x$with_jemalloc" != xno],[
+ AC_CHECK_LIB([jemalloc],[malloc_conf],
+ [LIBS="-ljemalloc $LIBS"],
+ [AS_IF([test "x$with_jemalloc" = xyes],
+ [AC_MSG_WARN([No jemalloc found, using system malloc])])])
+
</span> dnl check for large file stuff
mv confdefs.h confdefs1.h
: > confdefs.h
</code></pre> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=467072014-05-13T04:09:48Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:nobu@ruby-lang.org" class="email">nobu@ruby-lang.org</a> wrote:</p>
<blockquote>
<p>"linux-gnu" in "target_os" is substituted by "linux", so your AS_CASE never match.<br>
You should check for libjemalloc availability instead, I think.</p>
</blockquote>
<p>I used $target, not $target_os. An early version of my patch used<br>
$target_os, but I noticed it was a noop so I changed it to use $target.</p>
<p>I've tested this with bare "./configure" and also --without-jemalloc.<br>
Both work as expected on my GNU/Linux systems.</p>
<p>I worry this leads to platform-specific problems on non-GNU/Linux<br>
systems. It seems varnish/redis only tries jemalloc on Linux, too.</p>
<p>Also, I get one failure on both 32-bit and 64-bit:</p>
<p>[1/1] Test_StringModifyExpand#test_modify_expand_memory_leak = 0.02 s</p>
<ol>
<li>Failure:<br>
Test_StringModifyExpand#test_modify_expand_memory_leak<br>
[/home/ew/ruby/test/-ext-/string/test_modify_expand.rb:7]:<br>
rb_str_modify_expand().<br>
size: 14307328 => 31084544..<br>
Expected 2.1726309762381906 to be < 1.5.</li>
</ol>
<p>It looks like certain allocation patterns are worse with jemalloc, but<br>
I think real-world apps use less memory.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468112014-05-19T20:09:54Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>[1/1] Test_StringModifyExpand#test_modify_expand_memory_leak = 0.02 s</p>
<ol>
<li>Failure:<br>
Test_StringModifyExpand#test_modify_expand_memory_leak<br>
[/home/ew/ruby/test/-ext-/string/test_modify_expand.rb:7]:<br>
rb_str_modify_expand().<br>
size: 14307328 => 31084544..<br>
Expected 2.1726309762381906 to be < 1.5.</li>
</ol>
<p>It looks like certain allocation patterns are worse with jemalloc, but<br>
I think real-world apps use less memory.</p>
</blockquote>
<p>I can update this to limit=2.2 to workaround the failure and commit my<br>
original patch soon. Any comment/objections?</p>
<p>Platform maintainers for non-GNU/Linux may enable jemalloc detection if<br>
they feel comfortable, but I don't want to potentially break things<br>
behind their backs. For example, I cannot test/fix OSX problems, but I<br>
have read using non-standard malloc is a difficult problem on that<br>
platform (but possible).</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468132014-05-19T21:18:57Zko1 (Koichi Sasada)
<ul></ul><p>(2014/05/20 5:09), <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>I can update this to limit=2.2 to workaround the failure and commit my<br>
original patch soon. Any comment/objections?</p>
</blockquote>
<p>which one?</p>
<p>--<br>
// SASADA Koichi at atdot dot net</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468142014-05-19T21:25:53Zko1 (Koichi Sasada)
<ul></ul><p>Sorry I found <a href="https://bugs.ruby-lang.org/attachments/download/4424/0001-configure.in-add-with-jemalloc-option.patch" class="external">https://bugs.ruby-lang.org/attachments/download/4424/0001-configure.in-add-with-jemalloc-option.patch</a></p>
<p>I want to suggest only add "--enable-jemalloc" (not default).</p>
<p>I'm afraid that it makes performance prediction more difficult.<br>
(someone use it and someone doesn't use)</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468152014-05-19T22:09:35Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:ko1@atdot.net" class="email">ko1@atdot.net</a> wrote:</p>
<blockquote>
<p>I want to suggest only add "--enable-jemalloc" (not default).</p>
</blockquote>
<p>I am considering it, but I am not sure non-default is worth it over<br>
nobu's original suggestion of using "./configure LIBS=-ljemalloc"</p>
<p>Sam: what do you think?</p>
<blockquote>
<p>I'm afraid that it makes performance prediction more difficult.<br>
(someone use it and someone doesn't use)</p>
</blockquote>
<p>I understand. Maybe a big warning at the end of ./configure?</p>
<p>Fwiw, I am mildly against bundling jemalloc in the tarball. It is<br>
more to maintain and keep up-to-date. Varnish no longer bundles<br>
jemalloc for this reason, either.</p>
<p>I chose to detect + enable jemalloc by default since it can encourage<br>
distributors and packagers to use and depend on it (like gmp).</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468362014-05-23T21:59:18Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>Warning moved to the end of output where it may be more visible:<br>
<a href="http://yhbt.net/ruby-9113-jemalloc-v2.patch" class="external">http://yhbt.net/ruby-9113-jemalloc-v2.patch</a><br>
Comments?</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468472014-05-24T04:07:57Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>I'm still negative to enable it by default.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468542014-05-24T18:21:00Zkosaki (Motohiro KOSAKI)kosaki.motohiro@gmail.com
<ul></ul><p>On Sat, May 24, 2014 at 1:07 PM, <a href="mailto:nobu@ruby-lang.org" class="email">nobu@ruby-lang.org</a> wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Ship Ruby for Linux with jemalloc out-of-the-box (Closed)" href="https://bugs.ruby-lang.org/issues/9113">#9113</a> has been updated by Nobuyoshi Nakada.</p>
<p>I'm still negative to enable it by default.</p>
</blockquote>
<p>Do we have only one benchmark provided by Sam? I don't think it is<br>
enough much comparison to make decision.</p>
<p>Anyway, I don't think Linux distros enable jemalloc because of their<br>
packaging policy even if we enable by default.<br>
So we need, at least</p>
<ol>
<li>A way to disable it</li>
<li>Easy to detect which malloc is used from bug reports. For our maintenance.</li>
</ol> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=468592014-05-25T01:48:25Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>KOSAKI Motohiro <a href="mailto:kosaki.motohiro@gmail.com" class="email">kosaki.motohiro@gmail.com</a> wrote:</p>
<blockquote>
<p>Do we have only one benchmark provided by Sam? I don't think it is<br>
enough much comparison to make decision.</p>
</blockquote>
<p>Empirical evidence on my 32-bit yahns server after running several days<br>
shows ~40M RSS w/ jemalloc 3.0/3.6 vs 60-80M RSS for eglibc malloc on<br>
Debian stable. I'll work on gathering more data for other systems.</p>
<blockquote>
<p>Anyway, I don't think Linux distros enable jemalloc because of their<br>
packaging policy even if we enable by default.</p>
</blockquote>
<p>I know distros do not like bundling extra libs, but I'm not aware of<br>
policies against extra linkage for already packaged libraries.</p>
<p>Looking at Debian stable packages, both redis and varnish packages[1]<br>
use jemalloc on common architectures. For redis, Debian just favors the<br>
system jemalloc instead of the bundled one.</p>
<blockquote>
<p>So we need, at least</p>
<ol>
<li>A way to disable it</li>
</ol>
</blockquote>
<p>My patch respects the --without-jemalloc option</p>
<blockquote>
<ol start="2">
<li>Easy to detect which malloc is used from bug reports. For our maintenance.</li>
</ol>
</blockquote>
<p>rb_bug already shows dynamically loaded libraries. My patch only enables dynamic<br>
link by default.</p>
<p>[1] <a href="http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14-1.dsc" class="external">http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14-1.dsc</a><br>
<a href="http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14.orig.tar.gz" class="external">http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14.orig.tar.gz</a><br>
<a href="http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14-1.debian.tar.gz" class="external">http://http.us.debian.org/debian/pool/main/r/redis/redis_2.4.14-1.debian.tar.gz</a><br>
<a href="http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2-2+deb7u1.dsc" class="external">http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2-2+deb7u1.dsc</a><br>
<a href="http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2.orig.tar.gz" class="external">http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2.orig.tar.gz</a><br>
<a href="http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2-2+deb7u1.debian.tar.gz" class="external">http://http.us.debian.org/debian/pool/main/v/varnish/varnish_3.0.2-2+deb7u1.debian.tar.gz</a></p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470002014-06-01T19:27:02Zhansdegraaff (Hans de Graaff)hans@degraaff.org
<ul></ul><p>Eric Wong wrote:</p>
<blockquote>
<p><a href="mailto:ko1@atdot.net" class="email">ko1@atdot.net</a> wrote:</p>
<blockquote>
<p>I want to suggest only add "--enable-jemalloc" (not default).</p>
</blockquote>
<p>I am considering it, but I am not sure non-default is worth it over<br>
nobu's original suggestion of using "./configure LIBS=-ljemalloc"</p>
</blockquote>
<p>Speaking from the Gentoo perspective: with "--enable-jemalloc" I could give my Gentoo users a choice of enabling jemalloc via our "USE flag" mechanism. I'd rather not do that with the LIBS mechanism since then I cannot be sure that this is a maintained and supported configuration.</p>
<blockquote>
<p>Fwiw, I am mildly against bundling jemalloc in the tarball. It is<br>
more to maintain and keep up-to-date. Varnish no longer bundles<br>
jemalloc for this reason, either.</p>
</blockquote>
<p>In Gentoo we would always remove or ignore the bundled version and build against our system version.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470112014-06-02T02:56:18Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>Hans de Graaff wrote:</p>
<blockquote>
<p>I'd rather not do that with the LIBS mechanism since then I cannot be sure that this is a maintained and supported configuration.</p>
</blockquote>
<p>It's a common method for autoconfisticated programs.<br>
Did you have troubles with it?</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470162014-06-02T17:11:50Zhansdegraaff (Hans de Graaff)hans@degraaff.org
<ul></ul><p>Nobuyoshi Nakada wrote:</p>
<blockquote>
<p>It's a common method for autoconfisticated programs.<br>
Did you have troubles with it?</p>
</blockquote>
<p>I have not tried. My point was that, being one of the Gentoo ruby maintainers, I would not want to use this mechanism to officially support jemalloc or tmalloc for the ruby packages that we offer in Gentoo.</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470172014-06-03T00:42:25Zkosaki (Motohiro KOSAKI)kosaki.motohiro@gmail.com
<ul></ul><p>On Sat, May 24, 2014 at 9:48 PM, <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Ship Ruby for Linux with jemalloc out-of-the-box (Closed)" href="https://bugs.ruby-lang.org/issues/9113">#9113</a> has been updated by Eric Wong.</p>
<p>KOSAKI Motohiro <a href="mailto:kosaki.motohiro@gmail.com" class="email">kosaki.motohiro@gmail.com</a> wrote:</p>
<blockquote>
<p>Do we have only one benchmark provided by Sam? I don't think it is<br>
enough much comparison to make decision.</p>
</blockquote>
<p>Empirical evidence on my 32-bit yahns server after running several days<br>
shows ~40M RSS w/ jemalloc 3.0/3.6 vs 60-80M RSS for eglibc malloc on<br>
Debian stable. I'll work on gathering more data for other systems.</p>
</blockquote>
<p>So, I'd suggest two phase action.</p>
<ol>
<li>Commit your patch, but disable by default.</li>
<li>Gather more use-case and performance data.<br>
Note: They should be reproducible. We need measure them again and<br>
again when libs updated.</li>
<li>Change the default when the community is convinced the benefit.</li>
</ol> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470202014-06-03T01:10:09Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>Hans de Graaff wrote:</p>
<blockquote>
<p>My point was that, being one of the Gentoo ruby maintainers, I would not want to use this mechanism to officially support jemalloc or tmalloc for the ruby packages that we offer in Gentoo.</p>
</blockquote>
<pre><code>$ ./configure --help
(snip)
Some influential environment variables:
CC C compiler command
CFLAGS C compiler flags
LDFLAGS linker flags, e.g. -L<lib dir> if you have libraries in a
nonstandard directory <lib dir>
LIBS libraries to pass to the linker, e.g. -l<library>
CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
you have headers in a nonstandard directory <include dir>
CXX C++ compiler command
CXXFLAGS C++ compiler flags
CPP C preprocessor
Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.
</code></pre> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470412014-06-04T19:51:23Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>KOSAKI Motohiro <a href="mailto:kosaki.motohiro@gmail.com" class="email">kosaki.motohiro@gmail.com</a> wrote:</p>
<blockquote>
<p>So, I'd suggest two phase action.</p>
<ol>
<li>Commit your patch, but disable by default.</li>
</ol>
</blockquote>
<p>OK, r46349</p>
<blockquote>
<ol start="2">
<li>Gather more use-case and performance data.<br>
Note: They should be reproducible. We need measure them again and<br>
again when libs updated.</li>
<li>Change the default when the community is convinced the benefit.</li>
</ol>
</blockquote>
<p>I wouldn't mind changing the default. However small applications<br>
take up more memory with jemalloc. Using MALLOC_CONF=lg_chunk:21<br>
or smaller numbers can reduce it (at performance cost).</p>
<p>Users may also try the Lockless Inc allocator. However we<br>
cannot ship it with Ruby since Ruby is not GPLv3.<br>
<a href="http://locklessinc.com/downloads/" class="external">http://locklessinc.com/downloads/</a></p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470422014-06-05T01:06:37Zhsbt (Hiroshi SHIBATA)hsbt@ruby-lang.org
<ul></ul><p>I tried to build with --with-jemalloc option.</p>
<ul>
<li>OS X 10.9.4 pre-release</li>
<li>jemalloc-3.6.0 via homebrew</li>
<li>r46350</li>
</ul>
<p>but it got build failure.</p>
<pre><code>checking for malloc_conf in -ljemalloc... no
configure: error: jemalloc requested but not found
</code></pre>
<p>libjemalloc.dylib is following location:</p>
<pre><code>% ls -l /usr/local/lib/libjemalloc.dylib
lrwxr-xr-x 1 hsbt admin 46 6 5 09:18 /usr/local/lib/libjemalloc.dylib -> ../Cellar/jemalloc/3.6.0/lib/libjemalloc.dylib
</code></pre>
<p>config.log: <a href="https://gist.github.com/hsbt/87f2748dd4117d3cd139" class="external">https://gist.github.com/hsbt/87f2748dd4117d3cd139</a></p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470432014-06-05T01:58:11Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>jemalloc prefixes "je_" to each public symbols at Mach-O (Mac OS X) and PE-COFF (Windows), so it can't be drop-in by default.</p>
<p><a href="https://github.com/jemalloc/jemalloc/blob/dev/configure.ac#L494L498" class="external">https://github.com/jemalloc/jemalloc/blob/dev/configure.ac#L494L498</a></p>
<p>Or you have to install it with <code>--with-jemalloc-prefix</code> option unless compiling with "jemalloc/jemalloc.h".</p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=470442014-06-05T05:16:58Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset r46354.</p>
<hr>
<p>configure.in, missing.h: jemalloc mangling</p>
<ul>
<li>configure.in (with-jemalloc): also check for header, for ABIs<br>
which JEMALLOC_MANGLE is needed, i.e., Mach-O and PE-COFF<br>
platforms. <a href="/issues/9113">[ruby-core:62939]</a> [Feature <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Ship Ruby for Linux with jemalloc out-of-the-box (Closed)" href="https://bugs.ruby-lang.org/issues/9113">#9113</a>]</li>
<li>include/ruby/missing.h: include alternative malloc header to<br>
replace memory management functions.</li>
<li>dln.c, io.c, parse.y, st.c: undef malloc family before<br>
re-definition to suppress warnings.</li>
</ul> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=484292014-08-20T19:48:28Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>SASADA Koichi <a href="mailto:ko1@atdot.net" class="email">ko1@atdot.net</a> wrote:</p>
<blockquote>
<p>Recently I'm working around this topic.</p>
<p>(1) Life-time oriented, similar to Copying GC<br>
(2) CoW frindly (read only) memories</p>
<p>More detail about (2):<br>
The following figure shows the stacked memory usage (snapshot) collected<br>
by valgrind/massif, on discorse benchmark by @sam's help.<br>
<a href="http://www.atdot.net/fp_store/f.69bk1n/file.copipa-temp-image.png" class="external">http://www.atdot.net/fp_store/f.69bk1n/file.copipa-temp-image.png</a></p>
<p>Interestingly, 50MB is consumed by iseq (iseq.c, compile.c). Most of<br>
data are read only, so it can be more CoW frindly. Now, we mixes<br>
read-only data and r/w data such as inline cahce.</p>
<p>There are several ideas. And I belive it is good topic to consider for<br>
Ruby 2.2.</p>
</blockquote>
<p>ko1: any progress on this front? I may use dlmalloc mspace API[1] to<br>
make a special CoW-friendly heap for read-only parts of the iseq<br>
structure (and probably other read-only data such as frozen pathnames).</p>
<p>And continue using regular malloc (e.g. jemalloc) for r/w heap.</p>
<p>[1] <a href="ftp://gee.cs.oswego.edu/pub/misc/malloc.c" class="external">ftp://gee.cs.oswego.edu/pub/misc/malloc.c</a></p> Ruby master - Feature #9113: Ship Ruby for Linux with jemalloc out-of-the-boxhttps://bugs.ruby-lang.org/issues/9113?journal_id=1066272024-02-07T09:04:06Zhackeron (Roman Gaufman)hackeron@gmail.com
<ul></ul><p>We've been using jemalloc since 2018 and it cut our memory use to 1/5, it also stopped us needing to restart our app every 3-4 days as the memory slowly leaked. It is staggering to me why this isn't the default, until we discovered jemalloc we were actively looking to stop using Ruby for its memory bloating and leaking.</p>