Feature #8339

Introducing Geneartional Garbage Collection for CRuby/MRI

Added by Koichi Sasada 12 months ago. Updated 12 months ago.

[ruby-core:54621]
Status:Closed
Priority:Normal
Assignee:Koichi Sasada
Category:core
Target version:2.1.0

Description

| One day a Rubyist came to Koichi and said, "I understand how to improve
| CRuby's performance. We must use a generational garbage collector." Koichi
| patiently told the Rubyist the following story: "One day a Rubyist came
| to Koichi and said, 'I understand how to improve CRuby's performance..."
| [This story is an homage of an introduction in a paper:
| "A real-time garbage collector based on the lifetimes of objects"
| (by Henry Lieberman, Carl Hewitt)
| http://dl.acm.org/citation.cfm?id=358147&CFID=321285546&CFTOKEN=10963356]

We Heroku Matz team developed a new generational mark&sweep garbage
collection algorithm RGenGC for CRuby/MRI.
(correctly speaking, it is generational marking algorithm)

What goods are:

  • Reduce marking time (yay!)
  • My algorithm doesn't introduce any incompatibility into normal C-exts.
  • Easy to development

Please read more details in attached PDF file.
Code is: https://github.com/ko1/ruby/tree/rgengc

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?

Thanks,
Koichi

gc-strategy-en.pdf (717 KB) Koichi Sasada, 04/28/2013 03:19 AM


Related issues

Related to ruby-trunk - Bug #8399: Remove usage of RARRAY_PTR in C extensions when not needed Closed 05/13/2013

Associated revisions

Revision 40689
Added by Koichi Sasada 12 months ago

  • include/ruby/ruby.h: add new utility macros to access Array's element.
  • RARRAY_AREF(a, i) returns i-th element of an array `a'
  • RARRAY_ASET(a, i, v) set i-th element of a' tov' This change is a part of RGENGC branch [ruby-trunk - Feature #8339].

Revision 40691
Added by Koichi Sasada 12 months ago

  • include/ruby/ruby.h: constify RBasic::klass and add RBASIC_CLASS(obj) macro which returns a class of `obj'. This change is a part of RGENGC branch [ruby-trunk - Feature #8339].
  • object.c: add new function rbobjreveal(). This function reveal interal (hidden) object by rbobjhide(). Note that do not change class before and after hiding. Only permitted example is: klass = RBASICCLASS(obj); rbobjhide(obj); .... rbobjreveal(obj, klass); TODO: API design. rbobjreveal() should be replaced with others. TODO: modify constified variables using cast may be harmful for compiler's analysis and optimizaton. Any idea to prohibt inserting RBasic::klass directly? If rename RBasic::klass and force to use RBASICCLASS(obj), then all codes such as `RBASIC(obj)->klass' will be compilation error. Is it acceptable? (We have similar experience at Ruby 1.9, for example "RARRAY(ary)->ptr" to "RARRAY_PTR(ary)".
  • internal.h: add some macros.
  • RBASICCLEARCLASS(obj) clear RBasic::klass to make it internal object.
  • RBASICSETCLASS(obj, cls) set RBasic::klass.
  • RBASICSETCLASSRAW(obj, cls) same as RBASICSET_CLASS without write barrier (planned).
  • RCLASSSETSUPER(a, b) set super class of a.
  • array.c, class.c, compile.c, encoding.c, enum.c, error.c, eval.c, file.c, gc.c, hash.c, io.c, iseq.c, marshal.c, object.c, parse.y, proc.c, process.c, random.c, ruby.c, sprintf.c, string.c, thread.c, transcode.c, vm.c, vm_eval.c, win32/file.c: Use above macros and functions to access RBasic::klass.
  • ext/coverage/coverage.c, ext/readline/readline.c, ext/socket/ancdata.c, ext/socket/init.c,
  • ext/zlib/zlib.c: ditto.

Revision 40698
Added by Koichi Sasada 12 months ago

  • include/ruby/ruby.h: constify RRational::(num,den) and RComplex::(real,imag). Add macro to set these values:
  • RRATIONALSETNUM()
  • RRATIONALSETDEN()
  • RCOMPLEXSETREAL()
  • RCOMPLEXSETIMAG() This change is a part of RGENGC branch [ruby-trunk - Feature #8339]. TODO: API design. RRATIONAL_SET(rat,num,den) is enough? TODO: Setting constify variable with cast has same issue of r40691.
  • complex.c, rational.c: use above macros.

Revision 40703
Added by Koichi Sasada 12 months ago

  • gc.c: support RGENGC. [ruby-trunk - Feature #8339] See this ticet about RGENGC.
  • gc.c: Add several flags:
  • RGENGC_DEBUG: if >0, then prints debug information.
  • RGENGCCHECKMODE: if >0, add assertions.
  • RGENGC_PROFILE: if >0, add profiling features. check GC.stat and GC::Profiler.
  • include/ruby/ruby.h: disable RGENGC by default (USE_RGENGC == 0).
  • array.c: add write barriers for T_ARRAY and generate sunny objects.
  • include/ruby/ruby.h (RARRAYPTRUSE): added. Use this macro if you want to access raw pointers. If you modify the contents which pointer pointed, then you need to care write barrier.
  • bignum.c, marshal.c, random.c: generate T_BIGNUM sunny objects.
  • complex.c, include/ruby/ruby.h: add write barriers for T_COMPLEX and generate sunny objects.
  • rational.c (nuratsnewinternal), include/ruby/ruby.h: add write barriers for TRATIONAL and generate sunny objects.
  • internal.h: add write barriers for RBasic::klass.
  • numeric.c (rbfloatnewinheap): generate sunny T_FLOAT objects.
  • object.c (rbclassallocateinstance), range.c: generate sunny TOBJECT objects.
  • string.c: add write barriers for T_STRING and generate sunny objects.
  • variable.c: add write barriers for ivars.
  • vminsnhelper.c (vmsetivar): ditto.
  • include/ruby/ruby.h, debug.c: use two flags FLWBPROTECTED and FL_OLDGEN.
  • node.h (NODEFLCREFPUSHEDBYEVAL, NODEFLCREFOMOD_SHARED): move flag bits.

History

#1 Updated by Boris Stitnicky 12 months ago

=begin
Great, even a simpleton like me understood. So (({Dog.new.speak})) will get incinerated as soon as it finishes its ((%"Bow wow"%)), while senior objects have tenure. Btw. let me express thanks for 2.0, it's noticeably faster than 1.9.
=end

#2 Updated by Narihiro Nakamura 12 months ago

Great work!!! Does RGenGC Ruby pass test-all?

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?

+1

#3 Updated by Koichi Sasada 12 months ago

(2013/04/28 9:23), authorNari (Narihiro Nakamura) wrote:

Great work!!! Does RGenGC Ruby pass test-all?

Sure.

--
// SASADA Koichi at atdot dot net

#4 Updated by Konstantin Haase 12 months ago

Wow, nice! Is there a patch/branch around somewhere already to take a look at?

Keep up the good work!

#5 Updated by Koichi Sasada 12 months ago

(2013/04/28 20:27), rkh (Konstantin Haase) wrote:

Wow, nice! Is there a patch/branch around somewhere already to take a look at?

Code is: https://github.com/ko1/ruby/tree/rgengc

--
// SASADA Koichi at atdot dot net

#6 Updated by Magnus Holm 12 months ago

On Sat, Apr 27, 2013 at 8:19 PM, ko1 (Koichi Sasada)
redmine@ruby-lang.orgwrote:

We Heroku Matz team developed a new generational mark&sweep garbage
collection algorithm RGenGC for CRuby/MRI.
(correctly speaking, it is generational marking algorithm)

What goods are:

  • Reduce marking time (yay!)
  • My algorithm doesn't introduce any incompatibility into normal C-exts.
  • Easy to development

Please read more details in attached PDF file.
Code is: https://github.com/ko1/ruby/tree/rgengc

Great work!

Are instances of user-level Ruby classes (class MyClass; end) marked as
sunny by default as well? If not, would it be difficult?

Also, I notice that generation/wb are stored as flags on the object.
Wouldn't this undo the work of the new bitmap mark flag improvements in 2.0?

#7 Updated by Magnus Holm 12 months ago

On Sat, Apr 27, 2013 at 8:19 PM, ko1 (Koichi Sasada)
redmine@ruby-lang.orgwrote:

We Heroku Matz team developed a new generational mark&sweep garbage
collection algorithm RGenGC for CRuby/MRI.
(correctly speaking, it is generational marking algorithm)

What goods are:

  • Reduce marking time (yay!)
  • My algorithm doesn't introduce any incompatibility into normal C-exts.
  • Easy to development

Please read more details in attached PDF file.
Code is: https://github.com/ko1/ruby/tree/rgengc

How about to introduce this new GC algorithm/implementation into Ruby
2.1.0?

Thanks,
Koichi

Another thing: What about passing the old value to OBJWB too? OBJWB(from,
to, old)? In this implementation it would just be a no-op. There are some
nice garbage collectors that you can implement if your write-barriers have
old values too (e.g "An On-the-Fly Mark and Sweep Garbage Collector Based
on Sliding Views").

#8 Updated by Koichi Sasada 12 months ago

(2013/04/28 21:40), Magnus Holm wrote:

Are instances of user-level Ruby classes (class MyClass; end) marked
as sunny by default as well? If not, would it be difficult?

Not difficult (I think). I'll try ASAP.

Also, I notice that generation/wb are stored as flags on the object.
Wouldn't this undo the work of the new bitmap mark flag improvements in 2.0?

Good question. I believe it is only a small affection (w/ some
modification, I considered about that).

Another thing: What about passing the old value to OBJWB too? OBJWB(from, to, old)? In this implementation it would just be a no-op. There are some nice garbage collectors that you can implement if your write-barriers have old values too (e.g "An On-the-Fly Mark and Sweep Garbage Collector Based on Sliding Views").

Interesting. But I doubt that we can implement them (w/o incomplete
WBs). Anyway, I need to read a paper. If there are good resources (slide
pdf, etc) to know them, please tell them.

--
// SASADA Koichi at atdot dot net

#9 Updated by Koichi Sasada 12 months ago

(2013/04/28 23:34), SASADA Koichi wrote:

Another thing: What about passing the old value to OBJWB too? OBJWB(from, to, old)? In this implementation it would just be a no-op. There are some nice garbage collectors that you can implement if your write-barriers have old values too (e.g "An On-the-Fly Mark and Sweep Garbage Collector Based on Sliding Views").
Interesting. But I doubt that we can implement them (w/o incomplete
WBs). Anyway, I need to read a paper. If there are good resources (slide
pdf, etc) to know them, please tell them.

Maybe I understand the snapshot idea.
I don't think it is acceptable for CRuby (because CRuby moves memory
areas!). But not for CRuby, it may considerable.

--
// SASADA Koichi at atdot dot net

#10 Updated by Magnus Holm 12 months ago

On Sun, Apr 28, 2013 at 6:07 PM, SASADA Koichi ko1@atdot.net wrote:

(2013/04/28 23:34), SASADA Koichi wrote:

Another thing: What about passing the old value to OBJWB too?
OBJ
WB(from, to, old)? In this implementation it would just be a no-op.
There are some nice garbage collectors that you can implement if your
write-barriers have old values too (e.g "An On-the-Fly Mark and Sweep
Garbage Collector Based on Sliding Views").
Interesting. But I doubt that we can implement them (w/o incomplete
WBs). Anyway, I need to read a paper. If there are good resources (slide
pdf, etc) to know them, please tell them.

Maybe I understand the snapshot idea.
I don't think it is acceptable for CRuby (because CRuby moves memory
areas!). But not for CRuby, it may considerable.

In hindsight I think that the sliding views-technique is only required if
you have multiple threads running at the same time. In CRuby there will
only be a single thread running Ruby code so this becomes easier. For a
concurrent tracer (tracing/marking objects as the Ruby thread runs) the
most important thing is to get a consistent view of the heap.

Example:

(1) Tracer starts running in the background
(2) a = object.thing; object.thing = nil
(3) Tracer reaches object, doesn't find a, frees it
(4) object.thing = a, dangling pointer

If you have a WB that records previous values you just need to add the
previous value to the marklist (e.g. object.thing = nil will add a to
the marklist).

If we have the sunny/shady-distinction it might be possible to (1) stop the
world, (2) mark all roots, trace all shady objects (3) start the world (4)
trace sunny objects concurrently. We just need a strategy for objects that
become shady during the concurrent tracing.

// Magnus Holm

#11 Updated by Koichi Sasada 12 months ago

(2013/04/29 1:19), Magnus Holm wrote:

In hindsight I think that the sliding views-technique is only required
if you have multiple threads running at the same time. In CRuby there
will only be a single thread running Ruby code so this becomes easier.
For a concurrent tracer (tracing/marking objects as the Ruby thread
runs) the most important thing is to get a consistent view of the heap.

Please read my previous message: "I don't think it is acceptable for
CRuby (because CRuby moves memory areas!)".

Concurrent tracing needs an assumption that "do not move (free) memory
area except sweeping timing". Current CRuby does.
For example: "ary << obj". Yes, the CRuby's memory management strategy
(assumption) is different from normal interpreters.

--
// SASADA Koichi at atdot dot net

#12 Updated by Magnus Holm 12 months ago

On Sun, Apr 28, 2013 at 6:29 PM, SASADA Koichi ko1@atdot.net wrote:

(2013/04/29 1:19), Magnus Holm wrote:

In hindsight I think that the sliding views-technique is only required
if you have multiple threads running at the same time. In CRuby there
will only be a single thread running Ruby code so this becomes easier.
For a concurrent tracer (tracing/marking objects as the Ruby thread
runs) the most important thing is to get a consistent view of the heap.

Please read my previous message: "I don't think it is acceptable for
CRuby (because CRuby moves memory areas!)".

Concurrent tracing needs an assumption that "do not move (free) memory
area except sweeping timing". Current CRuby does.
For example: "ary << obj". Yes, the CRuby's memory management strategy
(assumption) is different from normal interpreters.

I was not aware of that.

What exactly does "moving memory areas" mean? Do you have any links?

#13 Updated by Eric Wong 12 months ago

"ko1 (Koichi Sasada)" redmine@ruby-lang.org wrote:

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?

What is the expected performance impact for short-lived scripts and
one-liners? I hope there is no (or very little) regression.

#14 Updated by Koichi Sasada 12 months ago

(2013/04/29 5:00), Eric Wong wrote:

What is the expected performance impact for short-lived scripts and
one-liners? I hope there is no (or very little) regression.

Maybe it is a little (up to 10% down).
We need more measurements.

--
// SASADA Koichi at atdot dot net

#15 Updated by Koichi Sasada 12 months ago

(2013/04/29 2:19), Magnus Holm wrote:

What exactly does "moving memory areas" mean? Do you have any links?

This is pseudo code:
arypush(VALUE ary, VALUE item)
{
old
size = RARRALEN(ary);
realloc(RARRAY
PTR(ary), oldsize + 1); <- change memory area
RARRAY
PTR(ary)[oldsize] = item;
RARRAY
SETLEN(ary) = oldsize + 1;
}

--
// SASADA Koichi at atdot dot net

#16 Updated by Koichi Sasada 12 months ago

(2013/04/29 11:41), SASADA Koichi wrote:

(2013/04/29 5:00), Eric Wong wrote:

What is the expected performance impact for short-lived scripts and
one-liners? I hope there is no (or very little) regression.

Maybe it is a little (up to 10% down).
We need more measurements.

FYI: Measurement on my environment:
http://www.atdot.net/sp/raw/e4uylm

Observation:
- Works!! :)
- SOme of benchmarks are slow down because of
- not enough WBs to reduce GC (marking time)
- but increase WB overheads (for Array access)
- vm3gc speed up! , but it is mystery because
there are no old objects and
GC.start run full-GC.
It seems that full-GC (== current M&S) is faster
with my some my modification (removing mark
counter, etc).
- Computing intensive simple benchmarks are not affected.

--
// SASADA Koichi at atdot dot net

#17 Updated by Charles Nutter 12 months ago

I like the technique. I have some observations.

  • Most of the benchmarks do not have enough old data to make a difference.

Small benchmarks, in particular, will not show anything useful. These benchmarks could be made more interesting by creating a large amount of old data first, and then running the benchmark a few times. If dealing with a benchmark that's all young data I can't imagine there's going to be any gain (and probably loss, instead).

  • This seems like a nice segway toward more explicit reference management, as in JNI.

A "sunny" reference is rather like a downcall-local reference in JNI, where you guarantee (or else you are forced) to only hold a reference for the duration of the downcall. I could see adding APIs that would say "give me this reference, but only consider it "shady" until the downcall is done". Somewhat like explicit GIL release, but explicit "I won't use this reference outside this call". I'd really like to see that enter the C API, since it's a key reason why JNI libraries work well without limiting JVM GC implementation. A "global" reference in JNI would be like explicitly saying "I want a shady reference". This is, honestly, the way the Ruby C API needs to go to truly enable smart GC. What you have done with RGenGC is a halfway step, marking specific C API operations as grabbing "global" or "shady" references. We can make this more explicit, and it would be a good thing to do so.

For example, if I want to get access to an array's internals, I could say "I want a local reference to array internals". For the duration of that downcall, the array would be pinned (JVM does not guarantee this, but does guarantee that the reference is good for the downcall's lifetime). It would open up the possibility for C ext authors to use "shady" APIs in a limited scope.

...

Nice work either way. What you have done here may be applicable to moving-GC implementations that want to implement the Ruby C API, in that it would provide a way for us to map implicit C API operations to explicit VM operations without a lot of nasty weak references and such.

#18 Updated by Koichi Sasada 12 months ago

(2013/04/29 15:46), headius (Charles Nutter) wrote:

  • Most of the benchmarks do not have enough old data to make a difference.

Yes.

We can see
- overhead of WBs
- overhead of generational gc (increasing dead but un-collected objects)

And this observation is for trivial optimization (WB implementation, and
so on). I want to reduce such overhead. Ideally, it should be same time
with current M&S gc.

  • This seems like a nice segway toward more explicit reference management, as in JNI.

Yes. And what "our technique is cool" is we can go this way gradually.
We don't change all of them at a time.

--
// SASADA Koichi at atdot dot net

#19 Updated by Koichi Sasada 12 months ago

(2013/04/29 11:45), SASADA Koichi wrote:

(2013/04/29 2:19), Magnus Holm wrote:

What exactly does "moving memory areas" mean? Do you have any links?

This is pseudo code:
arypush(VALUE ary, VALUE item)
{
old
size = RARRALEN(ary);
realloc(RARRAY
PTR(ary), oldsize + 1); <- change memory area
RARRAY
PTR(ary)[oldsize] = item;
RARRAY
SETLEN(ary) = oldsize + 1;
}

Sorry, I assume only parallel tracing.
(a tracing thread and a mutator thread conflict (GVL protected), for
example, above code)

But concurrent, realtime GC can be implemented (not parallel,
synchronizing a tracing thread and a mutator thread).

--
// SASADA Koichi at atdot dot net

#20 Updated by Koichi Sasada 12 months ago

Now, I add "disable RGENGC feature".
A macro USE_RGENGC in include/ruby/ruby.h enable/disable RGENGC completely.

If performance of RGENGC is not enough, then it can be disabled easily.

It is a big change. So I hope someone verify our algorithm and approach.
And merge it ASAP.


Note:
After discussion with Magnus Holm (*1), I changed OBJWB(a, b) interface
to OBJ
WRITE(a, ptr, b). This interface do two:
(1) *ptr = b
(2) make write barrier with a and b

In some cases (*2), we can't get ptr to store b, so I add another
interface OBJ_CONNECT(a, oldval, b). This only makes wb.

(*1) Re: [ruby-trunk - Feature #8339][Open]
Introducing Geneartional Garbage Collection for CRuby/MRI

(*2) st_insert()

Thank you,
Koichi

(2013/04/28 3:19), ko1 (Koichi Sasada) wrote:

Issue #8339 has been reported by ko1 (Koichi Sasada).


Feature #8339: Introducing Geneartional Garbage Collection for CRuby/MRI
https://bugs.ruby-lang.org/issues/8339

Author: ko1 (Koichi Sasada)
Status: Open
Priority: Normal
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: current: 2.1.0

| One day a Rubyist came to Koichi and said, "I understand how to improve
| CRuby's performance. We must use a generational garbage collector." Koichi
| patiently told the Rubyist the following story: "One day a Rubyist came
| to Koichi and said, 'I understand how to improve CRuby's performance..."
| [This story is an homage of an introduction in a paper:
| "A real-time garbage collector based on the lifetimes of objects"
| (by Henry Lieberman, Carl Hewitt)
| http://dl.acm.org/citation.cfm?id=358147&CFID=321285546&CFTOKEN=10963356]

We Heroku Matz team developed a new generational mark&sweep garbage
collection algorithm RGenGC for CRuby/MRI.
(correctly speaking, it is generational marking algorithm)

What goods are:

  • Reduce marking time (yay!)
  • My algorithm doesn't introduce any incompatibility into normal C-exts.
  • Easy to development

Please read more details in attached PDF file.
Code is: https://github.com/ko1/ruby/tree/rgengc

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?

Thanks,
Koichi

--
// SASADA Koichi at atdot dot net

#21 Updated by Sam Saffron 12 months ago

Having trouble testing this against Discourse, can't get it to install ruby gems

/home/sam/.rvm/gems/ruby-head-rgengc@global/bin/bundle: gc.c:587: allocbitmap: Assertion `objspace->heap.freebitmap != ((void *)0)' failed.
Aborted (core dumped)

This is possibly platform related, on ubuntu x64 latest
$ make install
...
load mapsrc enc/trans/JIS/UCS%JISX0208VDC@NEC.src
load mapsrc enc/trans/CP/UCS%CP932VDC@NEC_IBM.src
load mapsrc enc/trans/CP/UCS%CP932VDC@IBM.src
load mapsrc enc/trans/JIS/UCS%JISX0208@MS.src
converter from UTF8-KDDI to stateless-ISO-2022-JP-KDDI
miniruby: gc.c:587: alloc_bitmap: Assertion objspace->heap.free_bitmap != ((void *)0)' failed.
Aborted (core dumped)
make[1]: *** [enc/trans/emoji_iso2022_kddi.c] Error 134
make[1]: Leaving directory
/home/sam/Source/koruby'
make: *** [srcs-enc] Error 2

#22 Updated by Koichi Sasada 12 months ago

(2013/04/30 15:59), sam.saffron (Sam Saffron) wrote:

Having trouble testing this against Discourse, can't get it to install ruby gems

/home/sam/.rvm/gems/ruby-head-rgengc@global/bin/bundle: gc.c:587: allocbitmap: Assertion `objspace->heap.freebitmap != ((void *)0)' failed.
Aborted (core dumped)

Thanks. How to reproduce?

--
// SASADA Koichi at atdot dot net

#23 Updated by Sam Saffron 12 months ago

Install latest ubuntu x64, grab latest source, make install seems to do the trick for me, will try to install on my mac and see what happens.

gcc --version
gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

export
...
declare -x RUBYFREEMIN="600000"
declare -x RUBYGCMALLOCLIMIT="1000000000"
declare -x RUBY
HEAPMINSLOTS="800000"
declare -x RUBYHEAPSLOTSGROWTHFACTOR="1.25"

sam@ubuntu:~ uname -a
Linux ubuntu 3.8.0-19-generic #29-Ubuntu SMP Wed Apr 17 18:16:28 UTC 2013 x8664 x8664 x86_64 GNU/Linux

#24 Updated by Koichi Sasada 12 months ago

(2013/04/30 16:03), SASADA Koichi wrote:

Thanks. How to reproduce?

And can you pass all tests by `make test-all' on your environment?

--
// SASADA Koichi at atdot dot net

#25 Updated by Sam Saffron 12 months ago

And can you pass all tests by `make test-all' on your environment?

Nope, its core dumping with the exact same assertion

#26 Updated by Koichi Sasada 12 months ago

(2013/04/30 15:59), sam.saffron (Sam Saffron) wrote:

Having trouble testing this against Discourse, can't get it to install ruby gems

/home/sam/.rvm/gems/ruby-head-rgengc@global/bin/bundle: gc.c:587: allocbitmap: Assertion `objspace->heap.freebitmap != ((void *)0)' failed.
Aborted (core dumped)

OMG!

I received only this part I quoted above via e-mail (mailing list).

However, on the redmine,

https://bugs.ruby-lang.org/issues/8339#change-39053> has more details!

With your log, I can't understand where is related to Gem. It seems ruby
installing log, not a gem.

--
// SASADA Koichi at atdot dot net

#27 Updated by Koichi Sasada 12 months ago

(2013/04/30 17:05), sam.saffron (Sam Saffron) wrote:

And can you pass all tests by `make test-all' on your environment?
Nope, its core dumping with the exact same assertion

Thank you for your report. I can reproduce with `make test-all'.

ko1debugfiber.resume

--
// SASADA Koichi at atdot dot net

#28 Updated by Koichi Sasada 12 months ago

(2013/04/30 17:41), SASADA Koichi wrote:

ko1debugfiber.resume

Maybe I can yield non-bug version.

--
// SASADA Koichi at atdot dot net

#29 Updated by Sam Saffron 12 months ago

Confirmed the recent push resolves it, will try to gather some benchmarks against Discourse tomorrow.

#30 Updated by Sam Saffron 12 months ago

@ko1 I ran some basic benches against Discourse, results are here:

http://meta.discourse.org/t/ruby-may-be-getting-a-generational-gc-what-this-means-to-you/6289

was surprised that the tuned stack performed better for requests not triggering GCs

#31 Updated by Narihiro Nakamura 12 months ago

2013/4/28 SASADA Koichi ko1@atdot.net:

(2013/04/28 9:23), authorNari (Narihiro Nakamura) wrote:

Great work!!! Does RGenGC Ruby pass test-all?

Sure.

Great!

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?

I agree :)

--
Narihiro Nakamura (nari)

#32 Updated by Koichi Sasada 12 months ago

(2013/05/04 12:08), Narihiro Nakamura wrote:

How about to introduce this new GC algorithm/implementation into Ruby 2.1.0?
I agree :)

Thank you for your positive response.
I will merge it soon.

--
// SASADA Koichi at atdot dot net

#33 Updated by Sam Saffron 12 months ago

@ko1 can we backport the GC optimisations (non rgengc ones in this branch into 2.0? )

#34 Updated by Akira Tanaka 12 months ago

2013/5/4 SASADA Koichi ko1@atdot.net:

Thank you for your positive response.
I will merge it soon.

Is it ABI compatible?

This is just a question.
I don't against it.
--
Tanaka Akira

#35 Updated by Koichi Sasada 12 months ago

(2013/05/06 11:50), Tanaka Akira wrote:

Is it ABI compatible?

(1) ABI compatbible for most of part.

You don't need to consider if you don't

(2) ABI incompatibility fo exceptional C-exts

If C api manages RBASIC(obj)->klass directly, it is incompatibile (WB is
needed, if assigned value is new and obj is old). However, I don't think
no such C-exts.

Only one example "C-exts manipulate RBASIC(obj)->klass directly" is, to
hide klass to make it internal object and restore it (restore to normal
object). We will provide new C api to do it.

Also, C-exts manipulate RArray's ptr directly without RARRAYPTR(), it
should be brokne. However, I believe no such exts (because of embeded
array feature introducing from 1.9) and everyone use RARRAY
PTR().

Another issue is flags. RGenGC patch uses two reserved flag (KEEP_WB and
OLDGEN). However, I believe no one use them (and C-exts which use these
flags should be re-considered).

So, answer is:
(a) ABI compatibility will be braek
(b) But normal C-exts can work without any modification
(and most of case, without any re-build)
(c) If modification needed, then (C level) compile erros will occure

--
// SASADA Koichi at atdot dot net

#36 Updated by Koichi Sasada 12 months ago

(2013/05/06 11:47), sam.saffron (Sam Saffron) wrote:

@ko1 can we backport the GC optimisations (non rgengc ones in this branch into 2.0? )

I think nagachika-san (release manager of 2.0.0 series) don't admint
`optimization' patch.

--
// SASADA Koichi at atdot dot net

#37 Updated by Akira Tanaka 12 months ago

2013/5/6 SASADA Koichi ko1@atdot.net:

(a) ABI compatibility will be braek
(b) But normal C-exts can work without any modification
(and most of case, without any re-build)

I wonder here.

I think many extension libraries uses RARRAY_PTR macro.
I feel such libraries are normal.
I guessed such libraries should be re-compiled but you said that
such libraries works without re-compilation.
Really?
--
Tanaka Akira

#38 Updated by Koichi Sasada 12 months ago

(2013/05/06 21:23), Tanaka Akira wrote:

I think many extension libraries uses RARRAY_PTR macro.
I feel such libraries are normal.
I guessed such libraries should be re-compiled but you said that
such libraries works without re-compilation.
Really?

You are right. It was my mistake.

--
// SASADA Koichi at atdot dot net

#39 Updated by Akira Tanaka 12 months ago

2013/5/6 SASADA Koichi ko1@atdot.net:

You are right. It was my mistake.

RGenGC is ABI incompatible.
I understood.
--
Tanaka Akira

#40 Updated by Uriel Katz 12 months ago

Amazing work!

Quick question,wouldn't it be better to add a generation number instead of just a flag?
The idea is that in the future you could add more generations(most gen GC I know use 3 generations)

#41 Updated by Koichi Sasada 12 months ago

(2013/05/08 20:56), urielka (Uriel Katz) wrote:

Quick question,wouldn't it be better to add a generation number instead of just a flag?
The idea is that in the future you could add more generations(most gen GC I know use 3 generations)

Current limitation (1 gen0 is due to flags (no space to memory
generation). However, we can extend generations with reconsidering data
structure.

--
// SASADA Koichi at atdot dot net

#42 Updated by Koichi Sasada 12 months ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r40689.
Koichi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • include/ruby/ruby.h: add new utility macros to access Array's element.
  • RARRAY_AREF(a, i) returns i-th element of an array `a'
  • RARRAY_ASET(a, i, v) set i-th element of a' tov' This change is a part of RGENGC branch [ruby-trunk - Feature #8339].

Also available in: Atom PDF