https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112015-02-19T21:48:43ZRuby Issue Tracking SystemRuby master - Feature #10869: Add support for option to pre-compile Ruby fileshttps://bugs.ruby-lang.org/issues/10869?journal_id=515612015-02-19T21:48:43Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>I profiled startup time a bit last year and much of it was in the<br>
parser and malloc; so having this should be a win.</p>
<p>I've pondered having something like a self-managing cache in $HOME. It<br>
would be similar to what ccache uses for caching gcc output, but for<br>
Ruby iseqs. I've used ccache for over a decade it's never failed me.</p>
<p>Since 2.2 fixed iseq dumping/loading, we're closer to being able to<br>
support this. We'll need to be careful and correctly invalidate the<br>
cache on any internal ABI changes, of course, and I don't want us to be<br>
stuck on an old/stable ABI for a cache.</p>
<p>I don't think this will make distributing most Ruby apps any easier<br>
(because C extensions are usually the biggest problem)</p> Ruby master - Feature #10869: Add support for option to pre-compile Ruby fileshttps://bugs.ruby-lang.org/issues/10869?journal_id=515682015-02-20T13:11:52Zrosenfeld (Rodrigo Rosenfeld Rosas)rr.rosas@gmail.com
<ul></ul><p>Good to know that you think this could speed up requires :) At least this seems like something feasible in case someone would be interested in giving it a try.</p>
<p>For the ABI case, maybe the easiest would be to separate the database location by ABI version, for example:</p>
<p>~/.mri/abi-xx/cache-database<br>
~/.mri/abi-xy/cache-database</p>
<p>Or it could use the exact Ruby version/patch as the subdirectory name. I believe a first step towards precompiling would be great to allow us to evaluate how faster require could get with such approach. If the results are great then it would worth exploring more efficient solutions for caching format, invalidation rules and how to support most cases. But if we could get an initial version that only took care of the common cases it would already allow us to evaluate how faster we would be able to require files or gems with such approach...</p>
<p>Initially the database could be even some sort of Marshal dump if it makes it easier to implement...</p> Ruby master - Feature #10869: Add support for option to pre-compile Ruby fileshttps://bugs.ruby-lang.org/issues/10869?journal_id=516072015-02-23T10:28:16Zko1 (Koichi Sasada)
<ul></ul><p>We did trials.</p>
<ul>
<li>AOT compiler from YARV code to C
<ul>
<li><a href="https://github.com/shyouhei/ruby/tree/shyouhei/yarvaot/ext/yarvaot?utm_source=twitterfeed&utm_medium=twitter" class="external">https://github.com/shyouhei/ruby/tree/shyouhei/yarvaot/ext/yarvaot?utm_source=twitterfeed&utm_medium=twitter</a></li>
<li>
<a href="https://github.com/soba1104/CastOff" class="external">https://github.com/soba1104/CastOff</a><br>
...</li>
</ul>
</li>
<li><a href="https://blade.ruby-lang.org/ruby-core/46896">[ruby-core:46896]</a> to define protocl to make your own loading framework</li>
</ul>
<p>It is my favorite topic.<br>
Recent days, go-language is popular. I heard one of the reasons is packaging codes into one binary file. For Ruby, if we have AOT compiled code (special format or C extensions) and packaging them, we can provide similar feature.</p> Ruby master - Feature #10869: Add support for option to pre-compile Ruby fileshttps://bugs.ruby-lang.org/issues/10869?journal_id=516092015-02-23T11:08:42Zrosenfeld (Rodrigo Rosenfeld Rosas)rr.rosas@gmail.com
<ul></ul><p>I understand there are different reasons why one would want to pre-compile Ruby code but not all kind of pre-compilation would meet all goals.</p>
<p>Personally, I'm not currently worried about packing an application in a single file, or to hide code in some binary form and similar concerns. With this particular request I'm mostly interested in providing an alternative to autoload with regards to performance improvent over the application boot process.</p>
<p>I understand the other goals are also interesting ones but maybe it would be much simpler to implement some kind of solution that would simply take care of caching the Parser result to avoid reparsing the files everytime we require them in case such cache could improve a lot the load time of all files required by an application.</p>
<p>Eric believes the Parser take a significant amount of time when requiring files and such a simple and imperfect implementation of parser results caching could at least give us some numbers to evaluate how faster loading an application with thousands of files containing hundreds of methods each could be with such approach.</p>
<p>Even if such patch couldn't be used production-wide due to lots of edge cases which are not handled it would at least give us an idea on how much faster loading Ruby code could be...</p>
<p>I think it might be too early to think about defining a framework loading protocol or something like that before we play a bit with this idea for a little while...</p>
<p>If we try to handle all goals that pre-compilation support could help it may become too complex for an initial implementation.</p>
<p>I'm really interested in having an idea of how much faster code loading could be without autoload and a pretty basic implementation could get us such idea without requiring too much development time, thinking about how to handle all possible cases that pre-compilation could solve or which would be the best database format to store the cached parsing result and so on.</p>
<p>For instance, suppose it would be easier to just create some path/to/.source.rb.precompiled file for each path/to/source.rb than to think about a more complex database. Of course this wouldn't be acceptable as a final implementation for many reasons, including read-only locations, different MRI versions in the same computer and so on. But if this is the fastest to implement, maybe it would be enough to give us some numbers on how faster we could get by using a cache for the parser results... We could then continue the discussion based on some real numbers and the expected performance improvements with such approach.</p>