Misc #20739: Test suite does not carry over CLI options

• Status: Open
• Assignee: -

[kddnewton (Kevin Newton)]

I'm not sure if anything needs to change here, but I wanted to report this in case anyone had any particular ideas. When running the ruby/ruby test suite, lots of tests call EnvUtil.invoke_ruby directly or indirectly through assert_in_out_err. That spawns a subprocess that runs Ruby and reports its status back through status, stdout, and stderr. The issue is, the subprocess does not always use the same options, which means there are lots of different ways it can be run.

For RJIT/YJIT, we have -DRJIT_FORCE_ENABLE and -DYJIT_FORCE_ENABLE, which forces the subprocess to have it enabled. Still, this isn't always used, so when running the test suite for RJIT/YJIT, a lot of tests are not actually running with those compilers.

Prism and parse.y have a similar problem. When running in the same process, it will respect the --parser option. But when running in a subprocess it will fall back to whatever the default parser is. This presents a problem for some tests that assert against RUBY_DESCRIPTION in a subprocess, as it may have +PRISM in the description.

We have tried to fix it before by adding RUN_OPTS/RUBYOPT to the subprocess options, but this causes lots of other problems. I am not sure of the correct solution, or if we should even bother fixing this. Please let me know what you think.

[Eregon (Benoit Daloze)]

I think it's a real issue and sometimes valuable to fix (it hurts the coverage of tests * flags).
#6648 could be helpful to automatically inherit passed flags.
Alternatively, we could ensure we only pass options for tests through RUBYOPT, and that RUBYOPT is never cleared/overridden without considering the previous value.

MSpec currently handles this by having an extra subprocess, so all test processes always have all the -T-flag options consistently.
(#6648 would be nicer)

[Eregon (Benoit Daloze)]

• Related to Feature #6648: Provide a standard API for retrieving all command-line flags passed to Ruby added