https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112018-10-04T14:13:18ZRuby Issue Tracking SystemRuby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743072018-10-04T14:13:18Zjaruga (Jun Aruga)
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/74307/diff?detail_id=50014">diff</a>)</li></ul> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743082018-10-04T14:17:08Zjaruga (Jun Aruga)
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/74308/diff?detail_id=50015">diff</a>)</li></ul> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743092018-10-04T15:06:09Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>I had run the Coverity Scan analysis on CI (twice a week), and I had checked the result only when I felt like. But recently I forgot it completely. By this ticket, I have just noticed that the analysis has not worked since Feb. 2018 :-)</p>
<p>Personally, I no longer like to spend effort for Coverity Scan. The analysis result includes too many false warnings. I have no intention to blame Coverity Scan; many of the false alarms are really subtle and even hard for human to understand that they are actually false. Anyway, it really makes me tired. Indeed, it sometimes tells us actual bugs, but I don't see that the advantage is worth the cost.</p>
<p>If anyone wants to use Coverity Scan again, I'd like to grant her/him to restore the build analysis. It would be preferable that s/he is a Ruby committer according to <a href="https://scan.coverity.com/faq#who-can-have-access" class="external">Coverity Scan FAQ "Who may be granted access to a Registered Project?"</a>.</p> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743102018-10-04T21:34:54Zshevegen (Robert A. Heiler)shevegen@gmail.com
<ul></ul><blockquote>
<p>However there are around 1000 "not important" issues.</p>
</blockquote>
<p>I think the ruby core team always likes to remove bugs from ruby, if these reports<br>
constitute real bugs that is. At the github page of mruby one can see that some sort of<br>
systematically try to find ways to break ruby, if you look at some of the issues there. :P</p>
<p>Many such reported examples are really quite contrived. I think that makes a huge difference<br>
towards bugs that are caused by a "regular" use of ruby, by a real person - that<br>
is, you write ruby code in some application and discover that things are failing<br>
or breaking, as opposed to automatic/systematic testing of syntax that can crash<br>
ruby, like fuzzing in C/C++ and such.</p>
<p>I remember being able to do so (that is, causing segfaults) with the ruby-gtk bindings -<br>
not on purpose, but simply because some of the code I used is quite long, many thousand<br>
lines of code, and very old too (much of which was written by me too, many years ago), and<br>
it is hard to make changes (I got a bit lazy from gtk2 to gtk3 so I don't write anywhere<br>
near as much gtk-code in ruby these days); and a bit cumbersome to find and report bugs<br>
too. (Even if Kouhei Sutou does a great job improving, fixing bugs and maintaining the<br>
ruby-gnome bindings.) But I think developer manpower is still limited, in numbers alone.<br>
Perhaps not only in numbers but in knowledge too. I am sure there are many people who<br>
know ruby quite well, but significantly fewer who know both ruby and C very well.</p>
<p>I also don't think there is any ... realistic way that the ruby bug tracker could<br>
handle reports like +1000 bugs started. :)</p>
<p>The total amount of bugs reported on the issue tracker so far here is 8307. Even<br>
though nobu is the patch monster from Mount Fuji with 20 arms, even he may have<br>
a hard time tackling 1000 different bugs (even if they are well documented and<br>
reproducible),</p>
<p>What may be useful, perhaps, is if there could be some way to not only find "real"<br>
bugs, but those that seem to be more "promising" and worthwhile to fix. Like to<br>
somehow semi-automatically "promote" the more outstanding bugs from Coverity Scan<br>
or any other method that may have a real, larger impact. And then say that you may<br>
distribute a few of these onto the bug tracker here somehow, distributed somewhat<br>
slowly so that they would not take too much time away (I think people also prefer<br>
to add features since it is something new, whereas fixing bugs is not quite as fun<br>
as playing around with new features :D ).</p>
<p>Developer time is limited and I think different members of the ruby core team said<br>
it before, that often priority will be given to "real" problems, as opposed to<br>
theoretical problems or ... not sure about the word ... very unlikely bugs?</p>
<p>Anyway, perhaps some people like to look at CI to discover "worthwhile" bugs to<br>
fix. A few bug reports in the last ~3 months really amazed me, how they were found.<br>
Some other bugs (or "surprising behaviour") are interesting too, like the one where<br>
<< can change the encoding of Strings (<a href="https://bugs.ruby-lang.org/issues/14975" class="external">https://bugs.ruby-lang.org/issues/14975</a>),<br>
even though that one was found by regular use of ruby rather than any automated<br>
or semi-automated CI (may not be a bug and perhaps just a specific behaviour but<br>
it still surprised me when I read it).</p> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743122018-10-05T09:09:11Zjaruga (Jun Aruga)
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/74312/diff?detail_id=50016">diff</a>)</li></ul> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=743172018-10-05T14:59:47Zjaruga (Jun Aruga)
<ul></ul><p>Yusuke and Robert, thank you for sharing current status and thoughts.</p>
<blockquote>
<p>I had run the Coverity Scan analysis on CI (twice a week), and I had checked the result only when I felt like. But recently I forgot it completely. By this ticket, I have just noticed that the analysis has not worked since Feb. 2018 :-)</p>
</blockquote>
<p>Alright :-) The CI is with Travis CI or a CI in rubyci.org?</p>
<blockquote>
<p>I think the ruby core team always likes to remove bugs from ruby, if these reports<br>
constitute real bugs that is. At the github page of mruby one can see that some sort of<br>
systematically try to find ways to break ruby, if you look at some of the issues there. :P</p>
</blockquote>
<p>I found the issues on mruby project. They are doing something :)</p>
<p><a href="https://github.com/mruby/mruby/commit/8da787b" class="external">https://github.com/mruby/mruby/commit/8da787b</a><br>
<a href="https://github.com/mruby/mruby/commit/a4f63ca" class="external">https://github.com/mruby/mruby/commit/a4f63ca</a><br>
<a href="https://github.com/mruby/mruby/commit/b071dcd" class="external">https://github.com/mruby/mruby/commit/b071dcd</a></p>
<blockquote>
<p>But I think developer manpower is still limited, in numbers alone.<br>
Perhaps not only in numbers but in knowledge too. I am sure there are many people who<br>
know ruby quite well, but significantly fewer who know both ruby and C very well.</p>
</blockquote>
<p>One of the benefits to access the result of Coverity Scan casually is that it can be good opportunity for people like me who do not know C very well, to make friends with C in Ruby project.<br>
Because fixing the some issues is easier than implementing new features. Though it might not be a real important issue.</p>
<p>That might increase future potential developer with C in Ruby project.<br>
Recently I fixed a GCC warning issue for Ruby, it was the good opportunity for me.</p>
<blockquote>
<p>What may be useful, perhaps, is if there could be some way to not only find "real"<br>
bugs, but those that seem to be more "promising" and worthwhile to fix. Like to<br>
somehow semi-automatically "promote" the more outstanding bugs from Coverity Scan<br>
or any other method that may have a real, larger impact.</p>
</blockquote>
<p>I think that it might be possible to pick up only "real" bugs from the result of Coverity Scan semi-automatically.</p>
<blockquote>
<p>Developer time is limited and I think different members of the ruby core team said<br>
it before, that often priority will be given to "real" problems, as opposed to<br>
theoretical problems or ... not sure about the word ... very unlikely bugs?</p>
</blockquote>
<p>I can understand the meaning.</p> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=744382018-10-13T10:33:27Zjaruga (Jun Aruga)
<ul></ul><p>I like to share a good example that they only run Coverity Scan for Travis's cron job.<br>
<a href="https://github.com/systemd/systemd/blob/master/.travis.yml" class="external">https://github.com/systemd/systemd/blob/master/.travis.yml</a></p> Ruby master - Misc #15202: Adding Coverity Scan to CI to see the result casuallyhttps://bugs.ruby-lang.org/issues/15202?journal_id=746092018-10-25T10:50:35Zjaruga (Jun Aruga)
<ul></ul><blockquote>
<p>Instead of that, It looks good to me that someone could see the result of coverity scan casually anytime to fix those in long term.</p>
</blockquote>
<p>Current Travis CI "pedanticism" test case to output compiler warnings with <code>-Wall</code> and <code>-Wextra</code> is very helpful to see issues included in a result of Coverity Scan possibly.<br>
Thank you for adding the test case!</p>
<p>See also <a href="https://github.com/ruby/ruby/blob/trunk/.travis.yml" class="external">https://github.com/ruby/ruby/blob/trunk/.travis.yml</a></p>
<blockquote>
<pre><code> - name: pedanticism
...
warnflags_array=(
-Wall
-Wextra
...
</code></pre>
</blockquote>