Project

General

Profile

Actions

Bug #20158

open

Ractor affects Coverage results

Added by janosch-x (Janosch Müller) 9 months ago. Updated 9 months ago.

Status:
Assigned
Target version:
-
ruby -v:
ruby 3.3.0preview1 (2023-05-12 master a1b01e7701) [arm64-darwin22]
[ruby-core:116061]

Description

I have a large rspec test suite. I found that if I call a Ractor, the Coverage results are strongly affected, i.e. almost all files appear to be uncovered. This happens even if I only ever call a Ractor before the library or rspec are required.

Unfortunately, I was not able to build a simple repro yet.

I assume it is a timing thing and only affects larger suites, or it only happens if there are multiple files, and maybe if the library lazily requires its sub-modules?

However, I guess this should produce the same results when added to the spec_helper.rb of other large suites:

# Ractor.new { nil } # uncomment this to affect coverage results

require 'coverage'
Coverage.start

# require library, set up rspec etc. 

RSpec.configuration.after(:suite) do
  # this number is greatly reduced and unstable when calling Ractor above
  p Coverage.result.values.sum { |arr| arr.sum(&:to_i) }
end

I had this problem in this library. The problem affects simplecov users as well, as described here.


Related issues 1 (1 open0 closed)

Has duplicate Ruby master - Bug #20167: Code execution isn't recorded in RactorOpenActions

Updated by mame (Yusuke Endoh) 9 months ago

@janosch-x (Janosch Müller) Could you explain the complete reproduction procedure? I couldn't reproduce the issue by the following configuration.

test.rb

Ractor.new { nil }

require "coverage"

Coverage.start

load "test2.rb"
foo

pp Coverage.result

test2.rb

def foo
  1
end
$ ruby test.rb
test.rb:1: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues.
{"test2.rb"=>[1, 1, nil]}

Indeed, coverage does not support Ractor (#20167). However, I don't understand why just creating a Ractor affects coverage.

Actions #2

Updated by jeremyevans0 (Jeremy Evans) 9 months ago

  • Has duplicate Bug #20167: Code execution isn't recorded in Ractor added

Updated by janosch-x (Janosch Müller) 9 months ago

Hi @mame (Yusuke Endoh)!

As mentioned in the ticket, i could not reproduce it with a smaller setup.

Maybe problems only begin at a certain size, or when there is some require hierarchy?

I've now forked my affected repository to demonstrate the problem: https://github.com/jaynetics/ractor_coverage_repro

Maybe you have some idea how to debug it based on this?

Updated by mame (Yusuke Endoh) 9 months ago

  • Status changed from Open to Assigned
  • Assignee set to ko1 (Koichi Sasada)

Thanks, I reproduce the problem successfully with rspec + Ractor + TracePoint (without coverage).

# test_spec.rb
if ENV["RACTOR"] == "1"
  Ractor.new { nil }
  p :ractor_enabled
else
  p :ractor_disabled
end

$tp = TracePoint.new(:line) {|t| pp t }
$tp.enable

describe "foo" do
  $tp.disable
end

Expected behavior: The last output should be the line of $tp.disable as follows.

$ RACTOR=0 rspec t_spec.rb
:ractor_disabled
...
#<TracePoint:line /home/mame/.rbenv/versions/ruby-dev/lib/ruby/gems/3.4.0+0/gems/rspec-core-3.12.2/lib/rspec/core/example_group.rb:398 in `subclass'>
#<TracePoint:line /home/mame/work/ractor_coverage_repro/t_spec.rb:12>
#<TracePoint:line <internal:trace_point>:297 in `disable'>
No examples found.


Finished in 0.00022 seconds (files took 7.97 seconds to load)
0 examples, 0 failures

Actual result: Once Ractor mode is enabled, TracePoint stops firing in the middle of the process.

$ RACTOR=1 rspec t_spec.rb
:ractor_enabled
...
#<TracePoint:line /home/mame/.rbenv/versions/ruby-dev/lib/ruby/3.4.0+0/rubygems/basic_specification.rb:208 in `internal_init'>
#<TracePoint:line /home/mame/.rbenv/versions/ruby-dev/lib/ruby/3.4.0+0/rubygems/basic_specification.rb:209 in `internal_init'>
#<TracePoint:line /home/mame/.rbenv/versions/ruby-dev/lib/ruby/3.4.0+0/rubygems/stub_specification.rb:73 in `initialize'>
#<TracePoint:line /home/mame/.rbenv/versions/ruby-dev/lib/ruby/3.4.0+0/rubygems/stub_specification.rb:74 in `initialize'>
No examples found.


Finished in 0.00027 seconds (files took 0.101 seconds to load)
0 examples, 0 failures

Assigning to @ko1 (Koichi Sasada)

Updated by luke-gru (Luke Gruber) 9 months ago

It sounds like this bug is related to https://bugs.ruby-lang.org/issues/19112

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like1