Feature #20187: Bundled gems at Ruby 3.4 - Ruby master - Ruby Issue Tracking System

I think bundled gems should keep testing their code in their repository, not ruby repository.
We only test them with make test-bundled-gems for supporting development version of ruby interpreter.

Actions

Copy link

#7 [ruby-core:116737]

Updated by Eregon (Benoit Daloze) 2 months ago

hsbt (Hiroshi SHIBATA) wrote in #note-6:

@Eregon (Benoit Daloze) reverts my changes at ruby/spec without any discussion.

There is some discussion at https://github.com/ruby/ruby/commit/44d74f22c8da3c13aa5363769418e2f5fd29f65a#r138276491.
From that I understood that running these specs was failing under some circumstances, and I fixed that.

What is the problem with my fix, which adds extract-gems as a dependency of test-spec?

We shouldn't test bundled gems in make check. But he disagree it.

Yes, I think we should not stop running these specs, because it is likely to introduce regressions.
Your commits stopped running (33062-32468) = 594 specs/tests (1.8%), while they were passing and they do pass on a proper Ruby installation of 3.4 dev.

Whether a gem is default gem or bundled gem does not matter when running without Bundler (as it needs to be, otherwise any script doing e.g. require "bigdecimal" would break).
And so IMO ruby/spec should run the same whether a gem is a default or bundled gem (since it doesn't use Bundler).
When a gem is no longer bundled then of course we need to stop testing it in ruby/spec, but this is not the case here.

The current task environment is not intentional behavior from https://github.com/ruby/ruby/pull/6130

As there is more and more bundled gems I think it is valuable to test them and have them available without extra steps.

I propose disable them again at https://github.com/ruby/ruby/pull/9960.

Is there a concrete problem that this would solve?
I am not keen to disable so many specs, if it's only to avoid an extra extract-gems for test-spec, which seems pretty fast.
If it solves a serious and concrete problem then I am OK that this gets merged.

I think bundled gems should keep testing their code in their repository, not ruby repository.

That is a fair point, I agree the specs/tests should ideally live in the same repository as the implementation.
However there are still older Ruby versions and alternative Rubies where the implementation is in stdlib or in their repositories, so in that case having the specs in ruby/spec is good.
This can even be the case after a gem becomes bundled, because alternative Rubies might have their own implementation of them, e.g. digest implemented as Ruby/Java instead of a C extension.

I think so far it was never discussed in details how to move these specs to the bundled gem repository.
It probably deserves its own ticket to figure it out.
My main concern there is it would be a lot of effort to keep running the specs in the many bundled gems repositories vs just running ruby/spec (would need to integrate this as part of test-bundled-gems but also for all other Ruby implementations, which is lots of work. Would also need a way to sync these spec files from bundled gems to ruby/ruby).
And then of course upstreaming all this would be a lot of work too.
I guess the first thing we need is ruby/mspec as a gem, so it could easily be reused by bundled gems, like test-unit-ruby-core.
And then we can try upstreaming the specs to one of the bundled gems and see how it goes.

We only test them with make test-bundled-gems for supporting development version of ruby interpreter.

Maybe we could make test-spec part of test-bundled-gems or so.
But there also seems to be no need since make test-spec works as-is on master.

I think it would remove a lot of workarounds if ruby/spec was always run on a complete ruby (i.e., like a real Ruby installation, so after make install or very similar layout). But I understand some people have concerns about that because e.g. make install is not so fast/convenient (discussed previously on some other ticket).

Actions

Copy link

#8 [ruby-core:116739]

Updated by hsbt (Hiroshi SHIBATA) 2 months ago

Status changed from Open to Assigned

I changed test policy for this with nalsh's suggestion.

https://github.com/ruby/ruby/pull/9960#issuecomment-1943371439

We will separate ruby/spec to two groups.

core and library for default gems
library for bundled gems

I will make test-spec for core and default gems and disable to load .bundle directory. After that, I will create new mspec definition for bundled gems like bigdecimal etc. test-bundled-gems invoke that new definition.

Actions

Copy link

#9 [ruby-core:116776]