Misc #14902: Replace benchmark/driver.rb with benchmark_driver.gem

• Status: Closed
• Assignee: k0kubun (Takashi Kokubun)

[k0kubun (Takashi Kokubun)]

What's benchmark_driver.gem?

benchmark_driver.gem is a gem created by a project accepted as Ruby Association Grant 2017.
https://github.com/benchmark-driver/benchmark-driver

This is built as a general-purpose benchmark driver for Ruby and allows to collect any metric and to output in any way by writing plugins.

Why to replace benchmark/driver.rb?

First of all, the benchmark_driver.gem is created as a successor of benchmark/driver.rb, requested by ko1 (author of benchmark/driver.rb).
After the yesterday's project report of Ruby Association Grant 2017, ko1 and I agreed to replace benchmark/driver.rb with benchmark_driver.gem.
So I'll start working on this.

How to replace benchmark/driver.rb?

Some metrics supported by benchmark/driver.rb can't be collected by benchmark_driver.gem by default for now, but I'll leave them as plugins of benchmark_driver.gem.
It's a gem implemented outside this repository, but I'm going to implement make task to clone the gem repository like simplecov.

Unfortunately the scripts in benchmark directory can't be run by benchmark_driver.gem by default. So I'll implement some wrapper script to allow benchmark_driver.gem to run them at first, rewrite benchmark scripts to benchmark_driver.gem format one by one, drop old scripts and benchmark/driver.gem in the end.

During the migration, I'm going to keep the interface of make benchmark and make benchmark-each, but directly running benchmark/driver.rb will be broken at some moment.

[k0kubun (Takashi Kokubun)]

• Status changed from Assigned to Closed

Applied in changeset trunk|r63888.

---

benchmark: introduce benchmark_driver.gem

Makefile.in: Clone benchmark-driver repository in benchmark/benchmark-driver
make update-benchmark-driver, like simplecov.
win32/Makefile.sub: Roughly do the same thing.
.gitignore: Ignore the cloned repository.
common.mk: Trigger make update-benchmark-driver to run make benchmark
and adjust arguments for benchmark_driver.gem.

benchmark/require.yml: renamed from benchmark/bm_require.rb, benchmark/prepare_require.rb
benchmark/require_thread.yml: renamed from benchmark/bm_require_thread.rb, benchmark/prepare_require_thread.rb
benchmark/so_count_words.yml: renamed from benchmark/bm_so_count_words.rb, benchmark/prepare_so_count_words.rb,
benchmark/wc.input.base
benchmark/so_k_nucleotide.yml: renamed from benchmark/bm_so_k_nucleotide.rb, benchmark/prepare_so_k_nucleotide.rb,
benchmark/make_fasta_output.rb
benchmark/so_reverse_complement.yml: renamed from benchmark/bm_so_reverse_complement.rb, benchmark/prepare_so_reverse_complement.rb,
benchmark/make_fasta_output.rb

I'm sorry but I made some duplications between benchmark/require.yml and benchmark/require_thread.yml,
and between benchmark/so_k_nucleotide.yml and benchmark/so_reverse_complement.yml.

If you're not comfortable with it, please combine these YAMLs to share
the same prelude. One YAML file can have multiple benchmark definitions
sharing prelude.

benchmark/driver.rb: Replace its core feature with benchmark_driver.gem.
Some old features are gone for now, but I'll add them again later.

[Misc #14902]

[MSP-Greg (Greg L)]

@k0kubun (Takashi Kokubun)

Thank you for your work on this. Somewhat off topic...

I ran the suite on ruby-loco, and the only change needed was adding the following line to file_rename.yml, before the max.times do line:

tmp.each { |t| t.close } if RUBY_PLATFORM[/mingw|mswin/]

Windows won't allow renaming an open file. Thanks, Greg