https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112022-11-24T18:26:01ZRuby Issue Tracking SystemRuby master - Misc #19149: Minimal covered tests with the --enable-yjit case?https://bugs.ruby-lang.org/issues/19149?journal_id=1002432022-11-24T18:26:01Zmaximecb (Maxime Chevalier-Boisvert)maxime.chevalierboisvert@shopify.com
<ul></ul><p>If you want to thoroughly test everything, you could run make check two times, once with YJIT, and once without.</p>
<p>If you just run:</p>
<pre><code>$ make check RUN_OPTS="--yjit"
</code></pre>
<p>This will run YJIT with its default options, which is what most people would use when running YJIT. If you specify <code>--yjit-call-threshold=1 --yjit-verify-ctx</code>, it will make YJIT compile every single method and run extra verifications. This is what we do on the CI to be extra thorough, but it will be slower.</p>
<p>If you just want to quickly verify that YJIT works on the host platform, you could just run <code>make btest</code> with <code>--yjit</code> for some quick minimal verification. We do already test YJIT pretty well on the GitHub CI and on <code>rubyci.org</code>.</p> Ruby master - Misc #19149: Minimal covered tests with the --enable-yjit case?https://bugs.ruby-lang.org/issues/19149?journal_id=1002582022-11-25T10:41:00Zjaruga (Jun Aruga)
<ul></ul><p>maximecb (Maxime Chevalier-Boisvert) wrote in <a href="#note-1">#note-1</a>:</p>
<blockquote>
<p>If you want to thoroughly test everything, you could run make check two times, once with YJIT, and once without.</p>
<p>If you just run:</p>
<pre><code>$ make check RUN_OPTS="--yjit"
</code></pre>
<p>This will run YJIT with its default options, which is what most people would use when running YJIT. If you specify <code>--yjit-call-threshold=1 --yjit-verify-ctx</code>, it will make YJIT compile every single method and run extra verifications. This is what we do on the CI to be extra thorough, but it will be slower.</p>
<p>If you just want to quickly verify that YJIT works on the host platform, you could just run <code>make btest</code> with <code>--yjit</code> for some quick minimal verification. We do already test YJIT pretty well on the GitHub CI and on <code>rubyci.org</code>.</p>
</blockquote>
<p>Thank you for clarifying it. I understood it. I am still trying to find a good balance of running the YJIT tests between the running time and the coverage.</p>
<p>In the case of "MJIT", while the MJIT is supported as a default. And we don't run the tests with <code>RUN_OPTS="--mjit"</code> in the Feodra Ruby's RPM recipe file, while I notice there are tests with <code>RUN_OPTS="--mjit"</code> now at <a href="https://github.com/ruby/ruby/blob/4ab89d57bbc569143e9833addb88b91db86ad057/.github/workflows/mjit.yml#L35" class="external">https://github.com/ruby/ruby/blob/4ab89d57bbc569143e9833addb88b91db86ad057/.github/workflows/mjit.yml#L35</a>.</p>
<p>Because we can run the the following MJIT tests within the <code>make check</code> without <code>RUN_OPTS="--mjit"</code>. Here are the test results on the yesterday's latest master <code>66e5200ba435361624caa3e23db7962d906b70db</code>.</p>
<pre><code>$ grep MJIT_SUPPORT rbconfig.rb
CONFIG["MJIT_SUPPORT"] = "yes"
$ make test-all TESTS=test/ruby/test_mjit.rb
revision.h updated
Run options:
--seed=16704
"--ruby=./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems"
--excludes-dir=./test/excludes
--name=!/memory_leak/
# Running tests:
Finished tests in 132.081114s, 0.7950 tests/s, 4.2020 assertions/s.
105 tests, 555 assertions, 0 failures, 0 errors, 4 skips
ruby -v: ruby 3.2.0dev (2022-11-24T06:13:00Z master 66e5200ba4) [x86_64-linux]
</code></pre>
<p>However, we cannot run the following YJIT tests without <code>RUN_OPTS="--yjit"</code>. The tests were skipped.</p>
<pre><code>$ grep YJIT_SUPPORT rbconfig.rb
CONFIG["YJIT_SUPPORT"] = "dev"
$ make test-all TESTS=test/ruby/test_yjit.rb
revision.h updated
Run options:
--seed=38677
"--ruby=./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems"
--excludes-dir=./test/excludes
--name=!/memory_leak/
# Running tests:
Finished tests in 0.017483s, 0.0000 tests/s, 0.0000 assertions/s.
0 tests, 0 assertions, 0 failures, 0 errors, 0 skips
ruby -v: ruby 3.2.0dev (2022-11-24T06:13:00Z master 66e5200ba4) [x86_64-linux]
</code></pre>
<p>What does the difference of the ideas come from?</p> Ruby master - Misc #19149: Minimal covered tests with the --enable-yjit case?https://bugs.ruby-lang.org/issues/19149?journal_id=1002592022-11-25T15:22:25Zjaruga (Jun Aruga)
<ul></ul><blockquote>
<blockquote>
<p>This will run YJIT with its default options, which is what most people would use when running YJIT. If you specify <code>--yjit-call-threshold=1 --yjit-verify-ctx</code>, it will make YJIT compile every single method and run extra verifications. This is what we do on the CI to be extra thorough, but it will be slower.</p>
</blockquote>
</blockquote>
<p>Checking the commit <a href="https://github.com/ruby/ruby/commit/f90549cd38518231a6a74432fe1168c943a7cc18" class="external">https://github.com/ruby/ruby/commit/f90549cd38518231a6a74432fe1168c943a7cc18</a> about YJIT, and reading <a href="https://developer.squareup.com/blog/rubys-new-jit/" class="external">a Ruby JIT's document</a>, the main feature of the YJIT is to create iseq (Instruction Sequence) uniquely for YJIT, and compile it into YARV code, right? The top main functions are below?</p>
<ul>
<li>
<code>rb_yjit_iseq_mark</code> at <code>iseq.c</code>
</li>
<li>
<code>rb_yjit_compile_iseq</code> at <code>vm.c</code>
</li>
</ul>
<p>Running the command below with the Ruby above, the <code>compiled_iseq_count</code> was <code>0</code>, the <code>rb_yjit_iseq_mark</code> and <code>rb_yjit_compile_iseq</code> were not executed.</p>
<pre><code>$ ruby --yjit --yjit-stats -e 'puts "a"'
a
***YJIT: Printing YJIT statistics on exit***
method call exit reasons:
(all relevant counters are zero)
invokeblock exit reasons:
(all relevant counters are zero)
invokesuper exit reasons:
(all relevant counters are zero)
leave exit reasons:
(all relevant counters are zero)
getblockparamproxy exit reasons:
(all relevant counters are zero)
getinstancevariable exit reasons:
(all relevant counters are zero)
setinstancevariable exit reasons:
(all relevant counters are zero)
opt_aref exit reasons:
(all relevant counters are zero)
expandarray exit reasons:
(all relevant counters are zero)
opt_getinlinecache exit reasons:
(all relevant counters are zero)
invalidation reasons:
(all relevant counters are zero)
bindings_allocations: 73
bindings_set: 0
compiled_iseq_count: 0
compiled_block_count: 0
compiled_branch_count: 0
freed_iseq_count: 0
invalidation_count: 0
constant_state_bumps: 0
inline_code_size: 0
outlined_code_size: 84
freed_code_size: 0
code_region_size: 12288
yjit_alloc_size: 1763
live_page_count: 1
freed_page_count: 0
code_gc_count: 0
num_gc_obj_refs: 0
object_shape_count: 236
side_exit_count: 0
total_exit_count: 0
total_insns_count: 236517
vm_insns_count: 236517
yjit_insns_count: 0
ratio_in_yjit: 0.0%
avg_len_in_yjit: NaN
total_exits: 0
</code></pre>
<p>Running the command below, the 2 functions above were executed. The <code>compiled_iseq_count</code> was <code>4</code> non-zero. I feel that the <code>--yjit-call-threshold=1</code> is necessary to avoid skipping the functions unintentionally in the tests.</p>
<pre><code>$ ruby --yjit --yjit-stats --yjit-call-threshold=1 -e 'puts "a"'
a
***YJIT: Printing YJIT statistics on exit***
method call exit reasons:
(all relevant counters are zero)
invokeblock exit reasons:
(all relevant counters are zero)
invokesuper exit reasons:
(all relevant counters are zero)
leave exit reasons:
interp_return 1 (100.0%)
getblockparamproxy exit reasons:
(all relevant counters are zero)
getinstancevariable exit reasons:
(all relevant counters are zero)
setinstancevariable exit reasons:
(all relevant counters are zero)
opt_aref exit reasons:
(all relevant counters are zero)
expandarray exit reasons:
(all relevant counters are zero)
opt_getinlinecache exit reasons:
(all relevant counters are zero)
invalidation reasons:
(all relevant counters are zero)
bindings_allocations: 73
bindings_set: 0
compiled_iseq_count: 4
compiled_block_count: 8
compiled_branch_count: 12
freed_iseq_count: 0
invalidation_count: 0
constant_state_bumps: 0
inline_code_size: 1074
outlined_code_size: 603
freed_code_size: 0
code_region_size: 12288
yjit_alloc_size: 21097
live_page_count: 1
freed_page_count: 0
code_gc_count: 0
num_gc_obj_refs: 9
object_shape_count: 236
side_exit_count: 0
total_exit_count: 1
total_insns_count: 236325
vm_insns_count: 236313
yjit_insns_count: 12
ratio_in_yjit: 0.0%
avg_len_in_yjit: 12.0
total_exits: 0
</code></pre>
<p>And it seems that the <code>--yjit-verify-ctx</code> option is defined at <a href="https://github.com/ruby/ruby/blob/e15cd01149afe4924460f81cb6e27dd96de06657/yjit/src/options.rs#L66" class="external">https://github.com/ruby/ruby/blob/e15cd01149afe4924460f81cb6e27dd96de06657/yjit/src/options.rs#L66</a>. However the option is not defined at <a href="https://github.com/ruby/ruby/blob/e15cd01149afe4924460f81cb6e27dd96de06657/ruby.c#L339-L347" class="external">https://github.com/ruby/ruby/blob/e15cd01149afe4924460f81cb6e27dd96de06657/ruby.c#L339-L347</a>. Is it intentionally hidden for users? in the <code>--help</code> document?</p>
<pre><code>$ ruby -v
ruby 3.2.0dev (2022-11-24T06:13:00Z master 66e5200ba4) [x86_64-linux]
$ ruby --yjit-call-threshold=1 --yjit-verify-ctx -e 'puts "a"'
a
$ ruby --help | grep 'yjit'
--jit enable JIT for the platform, same as --yjit (experimental)
--yjit enable in-process JIT compiler (experimental)
yjit in-process JIT compiler (default: disabled)
--yjit-stats Enable collecting YJIT statistics
--yjit-exec-mem-size=num
--yjit-call-threshold=num
--yjit-max-versions=num
--yjit-greedy-versioning
</code></pre> Ruby master - Misc #19149: Minimal covered tests with the --enable-yjit case?https://bugs.ruby-lang.org/issues/19149?journal_id=1002672022-11-25T22:15:38Zalanwu (Alan Wu)
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="YJIT: Run test-all tests without requiring RUN_OPTS Most tests in test_yjit.rb use a sub process..." href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/1d64a5a7c09d1029508b6b3cb3d04e5a939bc8f8">git|1d64a5a7c09d1029508b6b3cb3d04e5a939bc8f8</a>.</p>
<hr>
<p>YJIT: Run test-all tests without requiring RUN_OPTS</p>
<p>Most tests in test_yjit.rb use a sub process, so we can run them even<br>
when the parent process is not running with YJIT. Run them so simply<br>
running <code>make check</code> tests YJIT a bit.</p>
<p>[Misc <a class="issue tracker-5 status-5 priority-4 priority-default closed" title="Misc: Minimal covered tests with the --enable-yjit case? (Closed)" href="https://bugs.ruby-lang.org/issues/19149">#19149</a>]</p>