Misc #20661: Stop retrying tests in `make test-all` command by default

• Status: Open
• Assignee: ono-max (Naoto Ono)

[ono-max (Naoto Ono)]

Summary

Currently, tests are retried once if they fail when tests are executed as parallel tests in the make test-all command by default. This is to prevent test failures caused by parallel testing. That's why failed tests are executed serially, not in parallel.
Source code: https://github.com/ruby/ruby/blob/master/tool/lib/test/unit.rb#L728-L751

However, there is a problem, which might hide a real issue like https://bugs.ruby-lang.org/issues/20314. To prevent missing this "real" problem, stopping the retry of tests in the make test-all command by default would be better.

Concerns when stopping the retry of tests in the make test-all command

Here are some concerns about stopping the retry mechanism in make test-all command.

1. There are some flaky tests in the test suite. Are we okay with that?

Yes, all flaky tests are tracked by Launchable, and they're mainly monitored by @mame (Yusuke Endoh). Thus, they'll be fixed in the future.

2. When a test that is not related to my change fails, what should we do?

Just click the "Re-run jobs" button via the GitHub UI.

[kjtsanaktsidis (KJ Tsanaktsidis)]

I have no objection to disabling automatic re-runs, but one thing:

Just click the "Re-run jobs" button via the GitHub UI.

Is this available to people who aren't a member of the Ruby organisation on Github? I think it might not be... I certainly remember pushing empty commits to poke CI because of this.