Bug #14064: test-all with and without -j - incorrect assertions and missing test methods

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.5.0dev (2017-10-29 trunk 60531) [x64-mingw32]
• Backport: 2.3: UNKNOWN, 2.4: UNKNOWN
• Tags: test

[MSP-Greg (Greg L)]

I always run test-all with a -v parameter, and although both Travis & Appveyor scripts do not, many of the https://rubyci.org builds seem to. With -v, one has a log of all test methods.

For quite a while, I ran ruby-loco (MinGW trunk build) with test-all having no -j parameter. I've now changed to using -j, and I'm surprised at what I'm finding.

When run without a -j parameter, the total test count is correct, but the total assertions count is high.

When run with a -j parameter (even set to 1), the test count drops (along with tests not appearing in the log), and the total assertions count is correct. Often, whole test files (classes) disappear from the log.

Re the test count variance, I wrote code to parse the logs (the total test count variance), which showed which test methods were present in non-parallel test logs that were missing in parallel test logs.

As to the assertions, I ran test-all on a few single files, and added the assertion count to each test method line, then compared with and without -j. Finally, by checking individual methods for their respective number of assertions, it was clear that the numbers shown in a non-parallel run were high. At first, it seemed that they were high by a count of three, but that only held for a few files.

First of all, I'm wondering if anyone else has found similar results.

Secondly, I've spent some time verifying this, but I can't seem to find where the code needs corrections to function properly. I hoped it might be a timeout issue, but that doesn't seem to be the case. Hence, suggestions as to where to start are welcome.

Finally, in Travis, the line for test-all is:

make -s $JOBS test-all -o exts TESTOPTS='-q --color=never --job-status=normal'

Using MinGW, adding $JOBS as a make parameter seems to have no effect.

[MSP-Greg (Greg L)]

While looking for a solution to the 'tests' variance, as mentioned above, there were test classes that were missing, especially when running with -j. Also, these missing classes varied from build to build.

I then noticed that the string test class in test/ruby was named TestString, and the string class in test-ext-\string was named Test_String. Proceeded to do similar renames for several classes.

I've posted GitHub PR #1739, and it seems to result in higher test numbers for both Appveyor mswin (two tests on my fork) and Travis. I also saw an increase in my ruby-loco MinGW numbers. I suspect these tests were not consistently being run previously. Summary below:

Appveyor
PR    a 16835 tests, 2183934 assertions, 0 failures, 0 errors, 312 skips
PR    b 16836 tests, 2182259 assertions, 0 failures, 0 errors, 320 skips
PR    c 16842 tests, 2184360 assertions, 0 failures, 0 errors, 320 skips
60628   16565 tests, 2174808 assertions, 0 failures, 0 errors, 312 skips
60625   16314 tests, 2162820 assertions, 0 failures, 0 errors, 317 skips

Travis
PR 1739 17455 tests, 2191876 assertions, 0 failures, 5 errors, 114 skips
60630   17048 tests, 2185008 assertions, 0 failures, 5 errors, 112 skips
60628   17044 tests, 2183920 assertions, 0 failures, 5 errors, 112 skips
60625   16853 tests, 2172202 assertions, 0 failures, 5 errors, 112 skips

I did try to locate the issue in code, but I couldn't find the cause...

Thanks, Greg

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed

It looks like a fix for this was committed in dd3851d2786412de019350a11e749c56fa5a07cc