Bug #5985

miniruby skews "make benchmark" results

Added by Eric Wong about 2 years ago. Updated about 1 month ago.

[ruby-core:42443]
Status:Closed
Priority:Low
Assignee:Koichi Sasada
Category:build
Target version:2.0.0
ruby -v:- Backport:

Description

miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3gc
ruby ../benchmark/driver.rb -v \
--executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems" \
--pattern=vm3
gc --directory=../benchmark
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86
64-linux]


vm3_gc

#! /usr/bin/ruby
5000.times do
100.times do
{"xxxx"=>"yyyy"}
end
GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] .2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86
64-linux] .1.857623815536499


raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)

benchmark results:
name ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] average difference

vm3_gc 2.752 1.858 -0.895

average total difference is -0.894646406173706

History

#1 Updated by Koichi Sasada about 2 years ago

  • Assignee set to Koichi Sasada

I'll change it.

#2 Updated by Shyouhei Urabe about 2 years ago

  • Status changed from Open to Assigned

#3 Updated by Koichi Sasada over 1 year ago

  • Status changed from Assigned to Closed

I changed to accept multiple -e' like "-e path1 -e path2". or "-e label1::path1 -e label2::path2".
You don't need to use
;' character.

#4 Updated by Koichi Sasada over 1 year ago

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead ofminiruby'.

The difference is `--disable-gems':

$ make benchmark COMPARERUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3gc'

...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

#5 Updated by Eric Wong over 1 year ago

"ko1 (Koichi Sasada)" redmine@ruby-lang.org wrote:

Issue #5985 has been updated by ko1 (Koichi Sasada).

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead ofminiruby'.

The difference is `--disable-gems':

$ make benchmark COMPARERUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3gc'

...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.

I think it's unrealistic to use/benchmark Ruby without gems nowadays.

#6 Updated by Eric Wong 3 months ago

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bmsoreverse_complement

This is running the same COMPARE_RUBY as what was just installed via
"make install" Strangely, miniruby is the one that is slow, here.

 $ make benchmark-each ITEM=bm_so_reverse_complement
 ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]   1.530159944
 built-ruby      2.93252692

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.530159944], [2.93252692]]]]

 Elapsed time: 4.464871542 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]
 built-ruby
 so_reverse_complement   1.530   2.933

 Speedup ratio: compare with the result of `ruby 2.2.0dev (2014-01-14
 trunk 44595) [x86_64-linux]' (greater is better)
 name    built-ruby
 so_reverse_complement   0.522

Compare with the following, the numbers only differ in noise:

 $ ruby --disable=gems ./benchmark/driver.rb -v  --executables="a::ruby \
 --disable=gems; b::ruby --disable=gems" \
 --pattern=bm_so_reverse_complement --directory=./benchmark

 a       1.547410982
 b       1.537930862

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.547410982], [1.537930862]]]]

 Elapsed time: 3.087557723 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    a       b
 so_reverse_complement   1.547   1.538

 Speedup ratio: compare with the result of `a' (greater is better)
 name    b
 so_reverse_complement   1.006

#7 Updated by Benoit Daloze 3 months ago

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.

#8 Updated by Benoit Daloze 3 months ago

  • ruby -v changed from ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] to -

Issue #5985 has been updated by Benoit Daloze.

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.


Bug #5985: miniruby skews "make benchmark" results
https://bugs.ruby-lang.org/issues/5985#change-44341

  • Author: Eric Wong
  • Status: Closed
  • Priority: Low
  • Assignee: Koichi Sasada
  • Category: build
  • Target version: 2.0.0
  • ruby -v: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
  • Backport:

    miniruby has fewer objects and a smaller heap than the normal "ruby",
    benchmarks that are affected by GC performance is skewed heavily.
    This is most noticeable with the vm3_gc benchmark using the same
    Ruby revision/build but different executables (miniruby vs ruby):

    $ make
    $ make install
    $ make benchmark-each ITEM=vm3gc
    ruby ../benchmark/driver.rb -v \
    --executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems" \
    --pattern=vm3
    gc --directory=../benchmark
    total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
    2012-02-08 18:57:12 +0000
    target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux]
    target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86
    64-linux]


    vm3_gc

    #! /usr/bin/ruby
    5000.times do
    100.times do
    {"xxxx"=>"yyyy"}
    end
    GC.start
    end

    ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] .2.752270221710205
    ruby 2.0.0dev (2012-02-08 trunk 34493) [x86
    64-linux] .1.857623815536499


    raw data:

    [["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

    Elapesed time: 4.611220872 (sec)

    benchmark results:
    name ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x8664-linux] average difference

    vm3_gc 2.752 1.858 -0.895

    average total difference is -0.894646406173706

    http://bugs.ruby-lang.org/

#9 Updated by Eric Wong 3 months ago

eregontp@gmail.com wrote:

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby
versions more brittle as the gems might change, particularly how much
allocations they do, no?

Good point. Perhaps keeping --disable-gems globally is better.

I have found as well some benchmarks in benchmark/ are very sensitive
to the heap usage.

Exactly.

#10 Updated by Eric Wong 3 months ago

Eric Wong normalperson@yhbt.net wrote:

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bmsoreverse_complement

Actually, this benchmark is all over the place because
benchmark/driver.rb slurps the gigantic output of this benchmark.

Patch at https://bugs.ruby-lang.org/issues/9430

#11 Updated by Eric Wong 3 months ago

Btw, I'm still getting more consistent results with normal ruby instead
of miniruby, probably because the encodings still affect heap size and
memory layout.

I suggest defining BENCHRUBY in my patch:
http://bogomips.org/ruby.git/patch?id=1ec0da52b7ce2e0e16f17

The following changes since commit 0558b8a5c40b93c01f5724fe8a3ab409d4374490:

  • ext/psych/lib/psych.rb: fix typo by @jwworth [fix GH-500] * lib/rake/file_list.rb: ditto (2014-01-19 05:48:33 +0000)

    are available in the git repository at:

    git://80x24.org/ruby.git benchmark

    for you to fetch changes up to 1ec0da52b7ce2e0e16f1729933ea795b12fea7ae:

    common.mk: define BENCHRUBY to avoid miniruby (2014-01-19 07:02:16 +0000)


    Eric Wong (1):
    common.mk: define BENCHRUBY to avoid miniruby

    common.mk | 6 ++++--
    1 file changed, 4 insertions(+), 2 deletions(-)

#12 Updated by Eric Wong 3 months ago

I would like to commit this soon:

 * common.mk: define BENCHRUBY to avoid miniruby heap size difference
    [Bug #5985]

#13 Updated by Koichi Sasada 3 months ago

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use installed libruby?).

#14 Updated by Eric Wong 3 months ago

ko1@atdot.net wrote:

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common
$(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use
installed libruby?).

Good catch, I tested out-of-tree build, but not --enable-shared.
I can force the use of STATIC_RUBY, it looks like.
I have this working both with and without --enable-shared.

http://bogomips.org/ruby.git/patch/?id=7495b0fdc7

--- a/common.mk
+++ b/common.mk
@@ -1001,14 +1001,14 @@ COMPARE_RUBY = $(BASERUBY)
ITEM =
OPTS =

-BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUNOPTS)
+BENCHRUBY = ./$(STATIC
RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

-benchmark: $(PROGRAM) PHONY
+benchmark: $(PROGRAM) PHONY $(STATICRUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v \
--executables="$(COMPARE
RUBY); built-ruby::$(BENCHRUBY)" \
--pattern='bm_' --directory=$(srcdir)/benchmark $(OPTS)

-benchmark-each: $(PROGRAM) PHONY
+benchmark-each: $(PROGRAM) PHONY $(STATICRUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v \
--executables="$(COMPARE
RUBY); built-ruby::$(BENCHRUBY)" \
--pattern=$(ITEM) --directory=$(srcdir)/benchmark $(OPTS)

#15 Updated by Koichi Sasada 3 months ago

+BENCHRUBY = ./$(STATICRUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUNOPT

We need to discuss which version we want to compare. static_ruby and dynamic (using libruby) ruby show different performance (basically, dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly to compare installed rubies. To do so, I install non-modified trunk and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to compare installed "ruby 1.8" and built miniruby. it is not fair comparison but it is enough to check the rough performance.

#16 Updated by Eric Wong 3 months ago

ko1@atdot.net wrote:

+BENCHRUBY = ./$(STATICRUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUNOPT

We need to discuss which version we want to compare. static_ruby and
dynamic (using libruby) ruby show different performance (basically,
dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly
to compare installed rubies. To do so, I install non-modified trunk
and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to
compare installed "ruby 1.8" and built miniruby. it is not fair
comparison but it is enough to check the rough performance.

I can count 3 options right now for "make benchmark":

a) "make benchmark" becomes a help message to tell people
to only benchmark with installed Rubies.

b) use LD_PRELOAD for shared Rubies (not portable?)

c) ignore dynamic for benchmark, it will always
be slower than static (neither is nearly as fast as miniruby)

I think c) is the easiest path right now, static-ruby is closer to
dynamic ruby. miniruby is too far different and unrealistic.

In all cases, we should probably add a warning if any benchmarked
ruby is dynamic.

#17 Updated by Eric Wong about 1 month ago

May I commit my current STATIC_RUBY version?

It is not perfect for dynamic ruby, but it is better than current
"make benchmark". We can make it perfect later.

Also available in: Atom PDF