https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112020-06-05T23:15:19ZRuby Issue Tracking SystemRuby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860022020-06-05T23:15:19Zjaruga (Jun Aruga)
<ul><li><strong>Subject</strong> changed from <i>`make check TESTS="-n !/Foo#method/"` not skipping the test case</i> to <i>`make test-all TESTS="-n !/Foo#method/"` not skipping the test case</i></li></ul> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860052020-06-06T07:45:46Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Is duplicate of</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/16935">Bug #16935</a>: Syntax error with `make check TESTS="-n /Foo#method/"`</i> added</li></ul> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860072020-06-06T07:46:31Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="Hash marks in Makefile need to be escaped [Bug #16935]" href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/0c00a4176ba353d59d8c991428574ef2c2676674">git|0c00a4176ba353d59d8c991428574ef2c2676674</a>.</p>
<hr>
<p>Hash marks in Makefile need to be escaped [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Syntax error with `make check TESTS="-n /Foo#method/"` (Closed)" href="https://bugs.ruby-lang.org/issues/16935">#16935</a>]</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860192020-06-08T13:22:44Zjaruga (Jun Aruga)
<ul></ul><blockquote>
<p>Hash marks in Makefile need to be escaped</p>
</blockquote>
<p>Thanks for the answer. In this case of the ticket, when running with the escaped hash <code>\#</code>, the tests are not skipped. Is this way wrong?</p>
<pre><code>$ make V=1 test-all TESTS="test/-ext-/bug_reporter/test_bug_reporter.rb test/ruby/test_process.rb -v -n \!/\^TestBugReporter\#test_bug_reporter_add\$$/ -n \!/\^TestProcess\#test_status_quit\$$/" 2>&1 | tee make.log
exec ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems "./test/runner.rb" --ruby="./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/ test/-ext-/bug_reporter/test_bug_reporter.rb test/ruby/test_process.rb -v -n \!/\^TestBugReporter\#test_bug_reporter_add$/ -n \!/\^TestProcess\#test_status_quit$/
...
Run options:
--seed=66754
"--ruby=./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems"
--excludes-dir=./test/excludes
--name=!/memory_leak/
-v
-n
"!/^TestBugReporter#test_bug_reporter_add$/"
-n
"!/^TestProcess#test_status_quit$/"
# Running tests:
[ 1/142] TestBugReporter#test_bug_reporter_add = 0.42 s
...
[128/142] TestProcess#test_status_quit = 0.33 s
...
</code></pre>
<p>I am reading and debugging the source code <code>tool/lib/test/unit.rb</code> <code>filter</code> logic, seeing the following command result.</p>
<pre><code>$ ruby tool/test/runner.rb --help
...
-n, --name PATTERN Filter test method names on pattern: /REGEXP/, !/REGEXP/ or STRING
...
</code></pre> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860212020-06-08T14:12:52Zjaruga (Jun Aruga)
<ul></ul><p>Could you show me the actual command based on the following one that was able to skip the tests on your local environment?</p>
<pre><code>$ make V=1 test-all TESTS="test/-ext-/bug_reporter/test_bug_reporter.rb test/ruby/test_process.rb -v -n \!/\^TestBugReporter\#test_bug_reporter_add\$$/ -n \!/\^TestProcess\#test_status_quit\$$/" 2>&1 | tee make.log
</code></pre>
<p>Thanks.</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860232020-06-08T15:17:26Zjaruga (Jun Aruga)
<ul></ul><p><a href="https://github.com/ruby/ruby/blob/465b5dc124917b828a5964c50c4e470a0c03dcf4/tool/lib/test/unit.rb#L742" class="external">https://github.com/ruby/ruby/blob/465b5dc124917b828a5964c50c4e470a0c03dcf4/tool/lib/test/unit.rb#L742</a></p>
<p>The following part matches only <code>type</code> (= <code>test_methods</code>) of the suites (<code>Array</code> of testing class such as <code>TestBugReporter</code>) It seems the testing class in <code>filter</code> is not considered.</p>
<pre><code> total = if filter
suites.inject(0) {|n, suite| n + suite.send(type).grep(filter).size}
else
suites.inject(0) {|n, suite| n + suite.send(type).size}
end
</code></pre>
<p>However if it matches only testing method, why the following command with the positive PATTERN matches the 2 test methods as expected? Actually I think <code>0</code> in <code>[1/0]</code> (<code>tool/lib/test/unit.rb#_prepare_run</code> <code>total</code> = <code>0</code>) is wrong number in the case. That should be <code>2</code>.</p>
<pre><code>$ make test-all TESTS="-v -n /\^TestBugReporter#test_bug_reporter_add\$$/ -n /\^TestProcess#test_status_quit\$$/"
...
[1/0] TestBugReporter#test_bug_reporter_add = 0.41 s
[2/0] TestProcess#test_status_quit = 0.35 s
Finished tests in 9.392046s, 0.2129 tests/s, 1.5971 assertions/s.
2 tests, 15 assertions, 0 failures, 0 errors, 0 skips
...
</code></pre>
<p>Why the 2 tests are executed?<br>
Possibly when printing and running the test, the <code>TestClass#test_method</code> is considered in the following part.</p>
<p><a href="https://github.com/ruby/ruby/blob/465b5dc124917b828a5964c50c4e470a0c03dcf4/tool/lib/test/unit.rb#L833" class="external">https://github.com/ruby/ruby/blob/465b5dc124917b828a5964c50c4e470a0c03dcf4/tool/lib/test/unit.rb#L833</a></p>
<pre><code> def print(s)
case s
when /\A(.*\#.*) = \z/ # <= Testing class is considered to run it.
runner.new_test($1)
</code></pre>
<p>I want to see the testing class is supported for the pattern matching.</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860242020-06-08T15:32:06Zjaruga (Jun Aruga)
<ul></ul><p>It seems there is an issue <code>[142/140]</code> when using <code>\^</code> in the pattern.</p>
<pre><code>$ make V=1 test-all TESTS="test/-ext-/bug_reporter/test_bug_reporter.rb test/ruby/test_process.rb -v -n \!/\^test_bug_reporter_add\$$/ -n \!/\^test_status_quit\$$/" 2>&1 | tee make.log
...
[ 1/140] TestBugReporter#test_bug_reporter_add = 0.44 s
...
[128/140] TestProcess#test_status_quit = 0.31 s
...
[142/140] TestProcess#test_waitall = 0.18 s
Finished tests in 25.888517s, 5.4851 tests/s, 39.3997 assertions/s.
142 tests, 1020 assertions, 0 failures, 0 errors, 0 skips
</code></pre>
<p>Here is okay case without using <code>\^</code>.</p>
<pre><code>$ make V=1 test-all TESTS="test/-ext-/bug_reporter/test_bug_reporter.rb test/ruby/test_process.rb -v -n \!/test_bug_reporter_add\$$/ -n \!/test_status_quit\$$/" 2>&1 | tee make.log
...
[140/140] TestProcess#test_waitall = 0.18 s
Finished tests in 25.745323s, 5.4379 tests/s, 39.0362 assertions/s.
140 tests, 1005 assertions, 0 failures, 0 errors, 0 skips
</code></pre> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860372020-06-09T14:59:23Zjaruga (Jun Aruga)
<ul></ul><p>Though I do not know how Ruby project's test/unit has been implemented.<br>
I am asking test-unit project about the how to ignore only a test name in the specific test suite.<br>
<a href="https://github.com/test-unit/test-unit/issues/169" class="external">https://github.com/test-unit/test-unit/issues/169</a></p>
<p>I would like to align the way with the test-unit project if we improve something for test/unit in Ruby project.</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860402020-06-09T20:42:59Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>test-unit in Ruby is thin wrapper above old version of minitest and does not have too much in common with test-unit project. Changing this would be quite big feat.</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860412020-06-09T21:06:36Zjaruga (Jun Aruga)
<ul></ul><blockquote>
<p>test-unit in Ruby is thin wrapper above old version of minitest and does not have too much in common with test-unit project. Changing this would be quite big feat.</p>
</blockquote>
<p>Thanks for the info.</p>
<p>I did not intend to implement the new features of test/unit in Ruby project such as new command options <code>--ignore-foo</code> used in the test-unit project.<br>
My intent was to look for the best direction or way to ignore only a test name in the specific test suite in test/unit of Ruby.</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=860512020-06-10T09:54:59Zjaruga (Jun Aruga)
<ul></ul><p>Seeing the current minitest and unit-test updated by the above ticket, it seems the both implementation supports "TestCase#name" perfect matching only if <code>filter</code> is String.</p>
<p><a href="https://github.com/seattlerb/minitest/blob/32d49db55d80b8479237898f07d161bb52ef905c/lib/minitest.rb#L317-L322" class="external">https://github.com/seattlerb/minitest/blob/32d49db55d80b8479237898f07d161bb52ef905c/lib/minitest.rb#L317-L322</a></p>
<pre><code> filter = options[:filter] || "/./"
filter = Regexp.new $1 if filter.is_a?(String) && filter =~ %r%/(.*)/%
filtered_methods = self.runnable_methods.find_all { |m|
filter === m || filter === "#{self}##{m}"
}
</code></pre>
<p><a href="https://github.com/test-unit/test-unit/blob/74f2ddce2fe7c54bde281082acbfd90c0c953485/lib/test/unit/autorunner.rb#L521-L529" class="external">https://github.com/test-unit/test-unit/blob/74f2ddce2fe7c54bde281082acbfd90c0c953485/lib/test/unit/autorunner.rb#L521-L529</a></p>
<pre><code> def match_test_name(test, pattern)
return true if pattern === test.method_name
return true if pattern === test.local_name
if pattern.is_a?(String)
return true if pattern === "#{test.class}##{test.method_name}"
return true if pattern === "#{test.class}##{test.local_name}"
end
false
end
</code></pre> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=861732020-06-15T17:24:20Zjaruga (Jun Aruga)
<ul></ul><p>I noticed the test/unit negative filter was implemented as Ruby project's original code in 2016.<br>
<a href="https://github.com/ruby/ruby/commit/83e36bb5a6e02b747d10c1baf5e1b7ff2b4c49fe" class="external">https://github.com/ruby/ruby/commit/83e36bb5a6e02b747d10c1baf5e1b7ff2b4c49fe</a></p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=943252021-10-26T10:21:12Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>Open</i></li><li><strong>ruby -v</strong> set to <i>ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]</i></li></ul><p>This was somehow swept under the carpet, but it is still unresolved. I have opened PR with a fix:</p>
<p><a href="https://github.com/ruby/ruby/pull/5026" class="external">https://github.com/ruby/ruby/pull/5026</a></p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=943452021-10-27T10:56:55ZAnonymous
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="Properly exclude test cases. Lets consider the following scenario: ~~~ irb(#<Test::Unit::AutoRu..." href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/5086c25f6015558877f85c3f1c014780b08fd3ce">git|5086c25f6015558877f85c3f1c014780b08fd3ce</a>.</p>
<hr>
<p>Properly exclude test cases.</p>
<p>Lets consider the following scenario:</p>
<pre><code>irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):001:0> p suite
OpenSSL::TestEC
=> OpenSSL::TestEC
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):002:0> p all_test_methods
["test_ECPrivateKey", "test_ECPrivateKey_encrypted", "test_PUBKEY", "test_check_key", "test_derive_key", "test_dh_compute_key", "test_dsa_sign_asn1_FIPS186_3", "test_ec_group", "test_ec_key", "test_ec_point", "test_ec_point_add", "test_ec_point_mul", "test_generate", "test_marshal", "test_sign_verify", "test_sign_verify_raw"]
=>
["test_ECPrivateKey",
"test_ECPrivateKey_encrypted",
"test_PUBKEY",
"test_check_key",
"test_derive_key",
"test_dh_compute_key",
"test_dsa_sign_asn1_FIPS186_3",
"test_ec_group",
"test_ec_key",
"test_ec_point",
"test_ec_point_add",
"test_ec_point_mul",
"test_generate",
"test_marshal",
"test_sign_verify",
"test_sign_verify_raw"]
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):003:0> p filter
/\A(?=.*)(?!.*(?-mix:(?-mix:memory_leak)|(?-mix:OpenSSL::TestEC.test_check_key)))/
=> /\A(?=.*)(?!.*(?-mix:(?-mix:memory_leak)|(?-mix:OpenSSL::TestEC.test_check_key)))/
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):004:0> method = "test_check_key"
=> "test_check_key"
</code></pre>
<p>The intention here is to exclude the <code>test_check_key</code> test case.<br>
Unfortunately this does not work as expected, because the negative filter<br>
is never checked:</p>
<pre><code>irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):005:0> filter === method
=> true
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):006:0> filter === "#{suite}##{method}"
=> false
irb(#<Test::Unit::AutoRunner::Runner:0x0000560f68afc3c8>):007:0> filter === method || filter === "#{suite}##{method}"
=> true
</code></pre>
<p>Therefore always filter against the fully qualified method name<br>
<code>#{suite}##{method}</code>, which should provide the expected result.</p>
<p>However, if plain string filter is used, keep checking also only the<br>
method name.</p>
<p>This resolves [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: `make test-all TESTS="-n !/Foo#method/"` not skipping the test case (Closed)" href="https://bugs.ruby-lang.org/issues/16936">#16936</a>].</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=943622021-10-27T18:05:39Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul><li><strong>Backport</strong> changed from <i>2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN</i> to <i>2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: REQUIRED</i></li></ul><p>Setting the backport flag, because it is useful to disable some test cases in Fedora package, if needed (and it will be included there soonish). But it is not super important. So feel free to ignore this backport request. Thx</p> Ruby master - Bug #16936: `make test-all TESTS="-n !/Foo#method/"` not skipping the test casehttps://bugs.ruby-lang.org/issues/16936?journal_id=956132021-12-24T06:06:08Znagachika (Tomoyuki Chikanaga)nagachika00@gmail.com
<ul><li><strong>Backport</strong> changed from <i>2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: REQUIRED</i> to <i>2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: DONE</i></li></ul><p>ruby_3_0 1d29740c1b101db4bd8fc2d05f929a9e37471a0f merged revision(s) 5086c25f6015558877f85c3f1c014780b08fd3ce,3ff0a0b40c2e1fbdad2286f1dafe837f822d0e0d.</p>