https://bugs.ruby-lang.org/
https://bugs.ruby-lang.org/favicon.ico?1711330511
2019-10-07T22:07:12Z
Ruby Issue Tracking System
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81944
2019-10-07T22:07:12Z
sam.saffron (Sam Saffron)
sam.saffron@gmail.com
<ul></ul><p>An alternative design could be to add 1 extra object to the heap</p>
<p>RubyVM::NonMaterializedInstructionSequences (sizeof all IMEMOS that are not wrapped)</p>
<p>Then <code>each_object</code> could include it and we could use that to measure size of all IMEMOs that are not materialized yet. Advantage here is that naive <code>memsize_of</code> all objects will return all the memory.</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81948
2019-10-07T22:52:47Z
methodmissing (Lourens Naudé)
lourens@bearmetal.eu
<ul></ul><p>sam.saffron (Sam Saffron) wrote:</p>
<blockquote>
<p>Koichi introduced an experimental gem: <a href="https://github.com/ko1/iseq_collector" class="external">https://github.com/ko1/iseq_collector</a></p>
<p>It allows:</p>
<p>ObjectSpace.each_iseq{|iseq| ...}<br>
ObjectSpace.count_iseq #=> Integer<br>
ObjectSpace.memsize_of_all_iseq (should not generate RubyVM::InstructionSequence wrappers for IMEMOs)</p>
<p>Since the wrapper object RubyVM::InstructionSequence is lazily allocated, ObjectSpace.each_object does not find these IMEMOs unless they have been wrapped. This design is good and conserves memory.</p>
<p><code>count_iseq</code> and <code>memsize_of_all_iseq</code> are very powerful metrics most large Ruby deployments can use to automatically detect method leaks introduced via meta programming. These issues are invisible now short of walking a heap dump.</p>
<p>Can we add the new interface into 2.7?</p>
</blockquote>
<p>I worked on <code>imemo_memsize</code> some time ago to correctly reflect the type sizes in <a href="https://github.com/ruby/ruby/commit/90c4bd2d2bd10b19c2b09834396553742bc7e8a4" class="external">https://github.com/ruby/ruby/commit/90c4bd2d2bd10b19c2b09834396553742bc7e8a4</a> which makes heap dumps more accurate. I understand the API proposal, but also I believe the intention was for these objects to be internal and not necessarily to be exposed through API. However I do suspect for large Rails applications their combined footprint can add up, especially for the types that can allocate heap memory too:</p>
<ul>
<li>imemo_ment (method entries)</li>
<li>imemo_iseq (as per your description above)</li>
<li>imemo_env (bindings)</li>
<li>imemo_tmpbuf (tried to support these on the transient heap but found them to be almost never used much in practice as it appears to be a fallback for <code>ALLOCA</code> under some circumstances.</li>
<li>imemo_ast</li>
</ul>
<p>I have not had any free time to investigate further, but I think this is an interesting storage class to explore further and I'd be interesting in helping, whichever way the proposal goes.</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81959
2019-10-08T17:55:41Z
shevegen (Robert A. Heiler)
shevegen@gmail.com
<ul></ul><p>Personally I love introspection so I am all in favour of giving ruby people<br>
lots of tools to play with internal. I also liked oldschool evi.rb. :)</p>
<p>I guess this is for koichi to comment e. g. how stable he considers the<br>
gem/code; and possibly also whether the API is wanted in the first place.<br>
And perhaps also whether the name iseq is already an official name or<br>
not, ruby-internal wise (I really don't know, just pointing that out).</p>
<p>As for the name <strong>NonMaterializedInstructionSequences</strong> - I think that name is<br>
too long and complicated. Ideally accessing should be simple, whenever possible,<br>
in my opinion. I am not even sure what a "non-materialized instruction sequence"<br>
is - is that ruby's version of a monoid-endofunctor monad?</p>
<p>IMO, simpler names would be better. Although I guess if the functionality is<br>
what matters, then I guess we may agree that the functionality can be useful.</p>
<p>methodmissing wrote:</p>
<blockquote>
<p>I understand the API proposal, but also I believe the intention was for these<br>
objects to be internal and not necessarily to be exposed through API.</p>
</blockquote>
<p>Yeah, I think I have read similar discussions in the past, also comments made<br>
by matz, koichi and shyouhei, in a different context. Which I guess makes<br>
sense too - less exposure may mean less problems. I am also neutral about the<br>
proposal really, don't mind either way - guess it may be for sam to reason<br>
in favour of it. :-)</p>
<p>Even then, though, I love introspection in general. Ruby is like a closed box<br>
initially, just like on xmas (and the xmas release), and you get the tools to<br>
poke inside and try to find out how it works! \o/</p>
<p>Perhaps if it may help the discussion (not that I contribute much to it),<br>
there could be a discussion for potential problems in this regard, e. g.<br>
pitfalls, problems etc... or it may remain a separate gem, and it may be<br>
evaluated how useful it may be to integrate it into ruby directly. That<br>
discussion has also happened with other code elements / gems in the past,<br>
e. g. martin duerst pointed this out a few times before. But as said, I<br>
am really neutral either way here.</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81960
2019-10-08T19:28:09Z
Eregon (Benoit Daloze)
<ul></ul><p>Do you think it would be possible for this new API to not rely on whether there is bytecode/iseqs?<br>
That way, it could be implemented on other Ruby implementations.</p>
<p>Is the main purpose to be able to estimate memory used by loaded Ruby code (methods)?<br>
A count of reachable methods' internal representations (iseq, AST, etc) would be a metric that is likely easy to provide for all Ruby implementations.</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81969
2019-10-09T10:49:22Z
sam.saffron (Sam Saffron)
sam.saffron@gmail.com
<ul></ul><p>To be honest I think the best spot for this is <code>RubyVM.stat</code></p>
<p>perhaps:</p>
<pre><code>RubyVM.stat
{
:global_method_state=>143,
:global_constant_state=>1369,
:class_serial=>8768,
:imemo_ment_count,
:imemo_iseq_count,
:imemo_env_count,
:imemo_tmpbuf_count,
:imemo_ast_count,
:imemo_ment_size,
:imemo_iseq_size,
:imemo_env_size,
:imemo_tmpbuf_size,
:imemo_ast_size
}
</code></pre>
<p>Since <code>RubyVM.stat(:class_serial)</code> is already supported as an efficient way to grab a single metric this interface fits nicely. It does not expand the signature surface of Ruby and is something that would be very simple to add for 2.7.</p>
<p>Additionally for extra bonus points:</p>
<p>RubyVM.stat(:total_allocated_bytes): all the bytes Ruby xmalloc and family allocated since process start<br>
RubyVM.stat(:total_freed_bytes): all the bytes freed</p>
<p>This comprehensive set of changes would make introspection of "why is my Ruby size XYZ?" really easy and provide some extremely powerful metrics for graphing.</p>
<p>Thoughts?</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=81971
2019-10-09T11:00:42Z
methodmissing (Lourens Naudé)
lourens@bearmetal.eu
<ul></ul><p>I like this API more, however <code>RubyVM</code> has been under discussion in <a href="https://bugs.ruby-lang.org/issues/15752" class="external">https://bugs.ruby-lang.org/issues/15752</a> regarding implementation specific exposure of experimental API and / or insights. These 2 issues are in a way strongly coupled.</p>
<p>sam.saffron (Sam Saffron) wrote:</p>
<blockquote>
<p>To be honest I think the best spot for this is <code>RubyVM.stat</code></p>
<p>perhaps:</p>
<pre><code>RubyVM.stat
{
:global_method_state=>143,
:global_constant_state=>1369,
:class_serial=>8768,
:imemo_ment_count,
:imemo_iseq_count,
:imemo_env_count,
:imemo_tmpbuf_count,
:imemo_ast_count,
:imemo_ment_size,
:imemo_iseq_size,
:imemo_env_size,
:imemo_tmpbuf_size,
:imemo_ast_size
}
</code></pre>
<p>Since <code>RubyVM.stat(:class_serial)</code> is already supported as an efficient way to grab a single metric this interface fits nicely. It does not expand the signature surface of Ruby and is something that would be very simple to add for 2.7.</p>
<p>Additionally for extra bonus points:</p>
<p>RubyVM.stat(:total_allocated_bytes): all the bytes Ruby xmalloc and family allocated since process start<br>
RubyVM.stat(:total_freed_bytes): all the bytes freed</p>
<p>This comprehensive set of changes would make introspection of "why is my Ruby size XYZ?" really easy and provide some extremely powerful metrics for graphing.</p>
<p>Thoughts?</p>
</blockquote>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=82061
2019-10-16T05:52:33Z
ko1 (Koichi Sasada)
<ul></ul><p>Already we have:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'objspace'</span>
<span class="n">pp</span> <span class="no">ObjectSpace</span><span class="p">.</span><span class="nf">count_imemo_objects</span>
<span class="c1">#=></span>
<span class="p">{</span><span class="ss">:imemo_env</span><span class="o">=></span><span class="mi">42</span><span class="p">,</span>
<span class="ss">:imemo_cref</span><span class="o">=></span><span class="mi">177</span><span class="p">,</span>
<span class="ss">:imemo_ment</span><span class="o">=></span><span class="mi">3662</span><span class="p">,</span>
<span class="ss">:imemo_iseq</span><span class="o">=></span><span class="mi">1194</span><span class="p">,</span>
<span class="ss">:imemo_tmpbuf</span><span class="o">=></span><span class="mi">117</span><span class="p">,</span>
<span class="ss">:imemo_ast</span><span class="o">=></span><span class="mi">22</span><span class="p">,</span>
<span class="ss">:imemo_svar</span><span class="o">=></span><span class="mi">40</span><span class="p">,</span>
<span class="ss">:imemo_throw_data</span><span class="o">=></span><span class="mi">55</span><span class="p">,</span>
<span class="ss">:imemo_ifunc</span><span class="o">=></span><span class="mi">35</span><span class="p">,</span>
<span class="ss">:imemo_memo</span><span class="o">=></span><span class="mi">32</span><span class="p">,</span>
<span class="ss">:imemo_parser_strterm</span><span class="o">=></span><span class="mi">118</span><span class="p">}</span>
</code></pre>
<p>There is no size version. do you want to introduce it?</p>
Ruby master - Feature #16245: Add interfaces to count and measure size all IMEMO objects
https://bugs.ruby-lang.org/issues/16245?journal_id=82131
2019-10-17T21:16:38Z
sam.saffron (Sam Saffron)
sam.saffron@gmail.com
<ul></ul><p>Yes!</p>
<p>ObjectSpace.memsize_of_imemo_objects sounds perfect to me.</p>
<p>I also support adding <code>ObjectSpace.each_iseq</code> which seems the simplest way to get iteration working.</p>
<p>I get the concern about not wanting to pollute MRI with MRI specific logic directly in ObjectSpace, but given we already have ObjectSpace.count_imemo_objects it feels a bit too late for this case.</p>