Bug #5985: miniruby skews "make benchmark" results

• Status: Closed
• Assignee: ko1 (Koichi Sasada)
• Target version: 2.0.0
• ruby -v: -

[normalperson (Eric Wong)]

miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v
--executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems"
--pattern=vm3_gc --directory=../benchmark
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]

---

vm3_gc

#! /usr/bin/ruby
5000.times do
100.times do
{"xxxx"=>"yyyy"}
end
GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .1.857623815536499

---

raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)

benchmark results:
name ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] average difference
vm3_gc 2.752 1.858 -0.895

average total difference is -0.894646406173706

[ko1 (Koichi Sasada)]

• Assignee set to ko1 (Koichi Sasada)

I'll change it.

[shyouhei (Shyouhei Urabe)]

• Status changed from Open to Assigned

[ko1 (Koichi Sasada)]

• Status changed from Assigned to Closed

I changed to accept multiple -e' like "-e path1 -e path2". or "-e label1::path1 -e label2::path2". You don't need to use ;' character.

[ko1 (Koichi Sasada)]

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead of miniruby'.

The difference is `--disable-gems':

$ make benchmark COMPARE_RUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3_gc'
...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

[normalperson (Eric Wong)]

"ko1 (Koichi Sasada)" [e-mail] wrote:

Issue #5985 has been updated by ko1 (Koichi Sasada).

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead of miniruby'.

The difference is `--disable-gems':

$ make benchmark COMPARE_RUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3_gc'
...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.

I think it's unrealistic to use/benchmark Ruby without gems nowadays.

[normalperson (Eric Wong)]

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bm_so_reverse_complement

This is running the same COMPARE_RUBY as what was just installed via
"make install" Strangely, miniruby is the one that is slow, here.

 $ make benchmark-each ITEM=bm_so_reverse_complement
 ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]   1.530159944
 built-ruby      2.93252692

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.530159944], [2.93252692]]]]

 Elapsed time: 4.464871542 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]
 built-ruby
 so_reverse_complement   1.530   2.933

 Speedup ratio: compare with the result of `ruby 2.2.0dev (2014-01-14
 trunk 44595) [x86_64-linux]' (greater is better)
 name    built-ruby
 so_reverse_complement   0.522

Compare with the following, the numbers only differ in noise:

 $ ruby --disable=gems ./benchmark/driver.rb -v  --executables="a::ruby \
 --disable=gems; b::ruby --disable=gems" \
 --pattern=bm_so_reverse_complement --directory=./benchmark

 a       1.547410982
 b       1.537930862

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.547410982], [1.537930862]]]]

 Elapsed time: 3.087557723 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    a       b
 so_reverse_complement   1.547   1.538

 Speedup ratio: compare with the result of `a' (greater is better)
 name    b
 so_reverse_complement   1.006

[Eregon (Benoit Daloze)]

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.

[Eregon (Benoit Daloze)]

• ruby -v changed from ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] to -

Issue #5985 has been updated by Benoit Daloze.

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.

---

Bug #5985: miniruby skews "make benchmark" results
https://bugs.ruby-lang.org/issues/5985#change-44341

• Author: Eric Wong
• Status: Closed
• Priority: Low
• Assignee: Koichi Sasada
• Category: build
• Target version: 2.0.0
• ruby -v: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
• Backport:

---

miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v
--executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems"
--pattern=vm3_gc --directory=../benchmark
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]

---

vm3_gc

#! /usr/bin/ruby
5000.times do
100.times do
{"xxxx"=>"yyyy"}
end
GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .1.857623815536499

---

raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)

benchmark results:
name ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] average difference
vm3_gc 2.752 1.858 -0.895

average total difference is -0.894646406173706

--
http://bugs.ruby-lang.org/

[normalperson (Eric Wong)]

[e-mail] wrote:

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby
versions more brittle as the gems might change, particularly how much
allocations they do, no?

Good point. Perhaps keeping --disable-gems globally is better.

I have found as well some benchmarks in benchmark/ are very sensitive
to the heap usage.

Exactly.

[normalperson (Eric Wong)]

Eric Wong [e-mail] wrote:

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bm_so_reverse_complement

Actually, this benchmark is all over the place because
benchmark/driver.rb slurps the gigantic output of this benchmark.

Patch at https://bugs.ruby-lang.org/issues/9430

[normalperson (Eric Wong)]

Btw, I'm still getting more consistent results with normal ruby instead
of miniruby, probably because the encodings still affect heap size and
memory layout.

I suggest defining BENCHRUBY in my patch:
http://bogomips.org/ruby.git/patch?id=1ec0da52b7ce2e0e16f17

The following changes since commit 0558b8a5c40b93c01f5724fe8a3ab409d4374490:

• ext/psych/lib/psych.rb: fix typo by @jwworth [fix GH-500] * lib/rake/file_list.rb: ditto (2014-01-19 05:48:33 +0000)

are available in the git repository at:

git://80x24.org/ruby.git benchmark

for you to fetch changes up to 1ec0da52b7ce2e0e16f1729933ea795b12fea7ae:

common.mk: define BENCHRUBY to avoid miniruby (2014-01-19 07:02:16 +0000)

---

Eric Wong (1):
common.mk: define BENCHRUBY to avoid miniruby

common.mk | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

[normalperson (Eric Wong)]

I would like to commit this soon:

 * common.mk: define BENCHRUBY to avoid miniruby heap size difference
   [ruby-core:59887] [Bug #5985]

[ko1 (Koichi Sasada)]

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use installed libruby?).

[normalperson (Eric Wong)]

[e-mail] wrote:

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common
$(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use
installed libruby?).

Good catch, I tested out-of-tree build, but not --enable-shared.
I can force the use of STATIC_RUBY, it looks like.
I have this working both with and without --enable-shared.

http://bogomips.org/ruby.git/patch/?id=7495b0fdc7

--- a/common.mk
+++ b/common.mk
@@ -1001,14 +1001,14 @@ COMPARE_RUBY = $(BASERUBY)
ITEM =
OPTS =

-BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)
+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

-benchmark: $(PROGRAM) PHONY
+benchmark: $(PROGRAM) PHONY $(STATIC_RUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v
--executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)"
--pattern='bm_' --directory=$(srcdir)/benchmark $(OPTS)

-benchmark-each: $(PROGRAM) PHONY
+benchmark-each: $(PROGRAM) PHONY $(STATIC_RUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v
--executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)"
--pattern=$(ITEM) --directory=$(srcdir)/benchmark $(OPTS)

[ko1 (Koichi Sasada)]

+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPT

We need to discuss which version we want to compare. static_ruby and dynamic (using libruby) ruby show different performance (basically, dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly to compare installed rubies. To do so, I install non-modified trunk and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to compare installed "ruby 1.8" and built miniruby. it is not fair comparison but it is enough to check the rough performance.

[normalperson (Eric Wong)]

[e-mail] wrote:

+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPT

We need to discuss which version we want to compare. static_ruby and
dynamic (using libruby) ruby show different performance (basically,
dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly
to compare installed rubies. To do so, I install non-modified trunk
and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to
compare installed "ruby 1.8" and built miniruby. it is not fair
comparison but it is enough to check the rough performance.

I can count 3 options right now for "make benchmark":

a) "make benchmark" becomes a help message to tell people
to only benchmark with installed Rubies.

b) use LD_PRELOAD for shared Rubies (not portable?)

c) ignore dynamic for benchmark, it will always
be slower than static (neither is nearly as fast as miniruby)

I think c) is the easiest path right now, static-ruby is closer to
dynamic ruby. miniruby is too far different and unrealistic.

In all cases, we should probably add a warning if any benchmarked
ruby is dynamic.

[normalperson (Eric Wong)]

May I commit my current STATIC_RUBY version?

It is not perfect for dynamic ruby, but it is better than current
"make benchmark". We can make it perfect later.