Feature #16012: Add a (small) test-install suite?

• Status: Closed
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -

[MSP-Greg (Greg L)]

At various times there has been discussion about whether testing should require make install. Although I prefer to do testing using install (vs build), I see it as a choice, and not a requirement.

From time to time various issues have arisen that cannot be found with 'build' testing. Often, these issues cause CI test failure with master/trunk/ruby-head in external repos. Sometimes people blame 'Core', other times Travis, or rvm. Regardless, it doesn't look good.

So, might a small set of tests that check install functionality be added? It may need to be two separate (but equivalent) scripts. One for *nix, one for Windows.

In ruby-loco, I'm using a ps1 script to check that CLI bin files work. As soon as the update is pushed here, I'll add a test for nested bundler commands...

[nobu (Nobuyoshi Nakada)]

What do you mean by to "check install functionality" concretely?
Something other than make install with DESTDIR?

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada),

What do you mean by to "check install functionality" concretely?

Anything that cannot be checked in the build folder.

Sometime recently, at least some of the bin stubs broke. So checking that all bin stubs run correctly would be one set of tests.

See https://ci.appveyor.com/project/MSP-Greg/ruby-loco/builds/26107569#L2961 for what I added to ruby-loco for cli. The build/job will fail if they don't pass.

Also, Bundler can be installed as a gem, it's also tested in it's own repo. But, some of the logic used when it is a default gem (as in trunk) was incorrect, and repos that had nested 'bundle' commands had CI failures.

This test suite would be small, as there aren't a lot of things to check, and it would only be usable/informative when used after make install.

One thing in particular it would help with is verifying that 'ruby-head' functions corrrectly when used in external repo/gem's CI testing...

I remember a while ago I was helping a popular gem with Appveyor, and I suggested that they test on 'ruby-head'. The response was essentially 'no, it fails too often, we don't want to bother". Not a common opinion, and I've also seen failures blamed on Travis, rvm, etc.

Personally, I want the Ruby community to think that 'ruby-head' is solid. The more testing it's exposed to, the better...

[MSP-Greg (Greg L)]

An example test (no assert) that's currently failing, run from a trunk install

require 'tmpdir'

Dir.mktmpdir 'test_bundler' do |dir|
  File.write 'Gemfile', '', mode:'wb'
  str = `bundle exec "bundle exec 'ruby -v'"`
  puts str
  puts str.strip == RUBY_DESCRIPTION
end

[hsbt (Hiroshi SHIBATA)]

Recently, I added test-tool target for the common library under the tool directory. We can test rbinstall.rb with it with the installation specification.

I know your concern. But your ruby-head only points Travis environment especially RVM. The some CI uses docker image provided the docker inc or the original images. The traditional CI like Travis or Azure Pipelines uses their internal environment or original distribution etc. We couldn't support all of the environments.

But I would like to add the system test for the ruby version managers with Docker like https://github.com/oracle/truffleruby/blob/master/doc/contributor/docker.md.

The system test was:

• install ruby master used by ruby managers or make install.
• assert installation files with the expected structures.
• assert binstub behavior with ruby managers or not.

The additional works, We can test them with --program-suffix, --prefix and DESTDIR etc.

[hsbt (Hiroshi SHIBATA)]

• Related to Misc #16013: Bundler... added

[MSP-Greg (Greg L)]

I understand the issue with several packagers/builders, etc.

But, getting things checked in the build here is a start, and we would know that the internal make install worked correctly. We don't know that now.

The tool could also be used as a check to be used by the other packagers/builders...

Should it be part of the make system, or a bash/shell/ps1 script, or a ruby script?

Lastly, you make a good point about '--program-suffix, --prefix and DESTDIR etc.' I'm suggesting testing things that are difficult to test without the 'struture' of a true install.

You're also suggesting adding checking for the install itself, which I've never had issue with. But I've never used '--program-suffix' or '--prefix'. Nor have I seen the folder structure or contents fail. But they could...

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned
• Assignee set to hsbt (Hiroshi SHIBATA)

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #13534: Checking installation results of default gems added

[hsbt (Hiroshi SHIBATA)]

I'm considering this request again. I'll add small check into https://github.com/ruby/chkbuild.

[Eregon (Benoit Daloze)]

FYI there is this script to check bin/ files, which could be reused: https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb

[Eregon (Benoit Daloze)]

I think this could become part of ruby/spec, and then we'd run ruby/spec on both built and on installed CRuby with chkbuild.

BTW there are similar specs in https://github.com/oracle/truffleruby/blob/master/spec/truffle/launcher_spec.rb

[hsbt (Hiroshi SHIBATA)]

• Status changed from Assigned to Closed

We have this at https://github.com/ruby/actions.