Project

General

Profile

Feature #16012

Add a (small) test-install suite?

Added by MSP-Greg (Greg L) 2 months ago. Updated about 2 months ago.

Status:
Assigned
Priority:
Normal
Target version:
-
[ruby-core:93845]

Description

At various times there has been discussion about whether testing should require make install. Although I prefer to do testing using install (vs build), I see it as a choice, and not a requirement.

From time to time various issues have arisen that cannot be found with 'build' testing. Often, these issues cause CI test failure with master/trunk/ruby-head in external repos. Sometimes people blame 'Core', other times Travis, or rvm. Regardless, it doesn't look good.

So, might a small set of tests that check install functionality be added? It may need to be two separate (but equivalent) scripts. One for *nix, one for Windows.

In ruby-loco, I'm using a ps1 script to check that CLI bin files work. As soon as the update is pushed here, I'll add a test for nested bundler commands...


Related issues

Related to Ruby master - Bug #16013: Bundler...FeedbackActions

History

Updated by nobu (Nobuyoshi Nakada) about 2 months ago

What do you mean by to "check install functionality" concretely?
Something other than make install with DESTDIR?

Updated by MSP-Greg (Greg L) about 2 months ago

nobu (Nobuyoshi Nakada),

What do you mean by to "check install functionality" concretely?

Anything that cannot be checked in the build folder.

Sometime recently, at least some of the bin stubs broke. So checking that all bin stubs run correctly would be one set of tests.

See https://ci.appveyor.com/project/MSP-Greg/ruby-loco/builds/26107569#L2961 for what I added to ruby-loco for cli. The build/job will fail if they don't pass.

Also, Bundler can be installed as a gem, it's also tested in it's own repo. But, some of the logic used when it is a default gem (as in trunk) was incorrect, and repos that had nested 'bundle' commands had CI failures.

This test suite would be small, as there aren't a lot of things to check, and it would only be usable/informative when used after make install.

One thing in particular it would help with is verifying that 'ruby-head' functions corrrectly when used in external repo/gem's CI testing...

I remember a while ago I was helping a popular gem with Appveyor, and I suggested that they test on 'ruby-head'. The response was essentially 'no, it fails too often, we don't want to bother". Not a common opinion, and I've also seen failures blamed on Travis, rvm, etc.

Personally, I want the Ruby community to think that 'ruby-head' is solid. The more testing it's exposed to, the better...

Updated by MSP-Greg (Greg L) about 2 months ago

An example test (no assert) that's currently failing, run from a trunk install

require 'tmpdir'

Dir.mktmpdir 'test_bundler' do |dir|
  File.write 'Gemfile', '', mode:'wb'
  str = `bundle exec "bundle exec 'ruby -v'"`
  puts str
  puts str.strip == RUBY_DESCRIPTION
end

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago

Recently, I added test-tool target for the common library under the tool directory. We can test rbinstall.rb with it with the installation specification.

I know your concern. But your ruby-head only points Travis environment especially RVM. The some CI uses docker image provided the docker inc or the original images. The traditional CI like Travis or Azure Pipelines uses their internal environment or original distribution etc. We couldn't support all of the environments.

But I would like to add the system test for the ruby version managers with Docker like https://github.com/oracle/truffleruby/blob/master/doc/contributor/docker.md.

The system test was:

  • install ruby master used by ruby managers or make install.
  • assert installation files with the expected structures.
  • assert binstub behavior with ruby managers or not.

The additional works, We can test them with --program-suffix, --prefix and DESTDIR etc.

#5

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago

Updated by MSP-Greg (Greg L) about 2 months ago

I understand the issue with several packagers/builders, etc.

But, getting things checked in the build here is a start, and we would know that the internal make install worked correctly. We don't know that now.

The tool could also be used as a check to be used by the other packagers/builders...

Should it be part of the make system, or a bash/shell/ps1 script, or a ruby script?

Lastly, you make a good point about '--program-suffix, --prefix and DESTDIR etc.' I'm suggesting testing things that are difficult to test without the 'struture' of a true install.

You're also suggesting adding checking for the install itself, which I've never had issue with. But I've never used '--program-suffix' or '--prefix'. Nor have I seen the folder structure or contents fail. But they could...

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago

  • Assignee set to hsbt (Hiroshi SHIBATA)
  • Status changed from Open to Assigned

Also available in: Atom PDF