Project

General

Profile

Actions

Bug #19436

closed

Call Cache for singleton methods can lead to "memory leaks"

Added by byroot (Jean Boussier) about 1 year ago. Updated 6 months ago.


Description

Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads.

Minimal Reproduction

module Foo
  def bar
  end
end

def call_bar(obj)
  # Here the call cache we'll keep a ref on the method_entry
  # which then keep a ref on the singleton_class, making that
  # instance immortal until the method is called again with
  # another instance.

  # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT
  obj.bar
end

obj = Object.new
obj.extend(Foo)

call_bar(obj)

id = obj.object_id
obj = nil

4.times { GC.start }
p ObjectSpace._id2ref(id)

Explanation

Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class
and in the cache of a singleton class, it keeps a strong reference onto the attached_object (instance).

This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver.
If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak.
And if the attached_object is big, the wasted memory can be very substantial.

Practical Implications

Once relative common API impacted by this is Rails' extending API. This API allow to extend a "query result set" with a module.
These query results set can sometimes be very big, especially since they keep references to the instantiated ActiveRecord::Base instances etc.

Possible Solutions

Only keep a weak reference to the CME

The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work.
However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is.

Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class:

# vm_callinfo.h:275

struct rb_callcache {
    const VALUE flags;

    /* inline cache: key */
    const VALUE klass; // should not mark it because klass can not be free'd
                       // because of this marking. When klass is collected,
                       // cc will be cleared (cc->klass = 0) at vm_ccs_free().

So it appears that the class being also marked through the CME is some kind of oversight?

Don't cache based on some heuristics

If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a Class or Module.

It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long.

Make Class#attached_object a weak reference

Alternatively we could make the attached_object a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario.

The downside is that Class#attached_object was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit.

cc @peterzhu2118 (Peter Zhu) @ko1 (Koichi Sasada)


Related issues 1 (0 open1 closed)

Related to Ruby master - Feature #19783: Weak References in the GCClosedActions

Updated by Eregon (Benoit Daloze) about 1 year ago

I think making Class#attached_object a weak reference is a better and easier solution.

Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM, as it would mean one extra WeakReference object for each of these call caches.
I suppose there could be workarounds like storing the WeakReference on the singleton class, and use a direct reference otherwise, but then it'd make method lookup more complex and slower in interpreter or when polymorphic (need to handle cached method lookup on regular and singleton class differently).
FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.
That approach is probably also quite difficult to get right on CRuby, also considering ractors and multithreading when the class GCs.

As a general note, creating a singleton class is not cheap, this should only be used for class objects (which always have one) and for a few rare global objects where it's convenient.
Using Object#extend objects often/on the fast path is just "making programs slow and uncached".

Updated by byroot (Jean Boussier) about 1 year ago

FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.

I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a Module or Class.

Using Object#extend objects often/on the fast path is just "making programs slow and uncached".

Yeah I agree. I tried to avoid this in Rails by caching these dynamic classes, unfortunately that's not possible because of the API: https://github.com/rails/rails/pull/47314. So I'll try to see if this API can be replaced by a better one and deprecated.

Updated by byroot (Jean Boussier) about 1 year ago

Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM

Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.

Updated by Eregon (Benoit Daloze) about 1 year ago

byroot (Jean Boussier) wrote in #note-2:

I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a Module or Class.

I missed that sorry.
I think there are quite a few cases where a global object (which is not a module/class) with a singleton class is used, it'd be unfortunate and slow to not inline cache lookup in those cases.

I think it makes sense to only try caching one (or a few) singleton class(es), and give up when seeing a second one or more (already happens on TruffleRuby, but for a higher limit, 8 currently).
That means we still need to make Class#attached_object a weak reference.

Updated by Eregon (Benoit Daloze) about 1 year ago

byroot (Jean Boussier) wrote in #note-3:

Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.

Yes, since TruffleRuby caches the the Class object in method lookup (LookupMethodNode), and currently the Class object has a strong reference to the attached_object.

Updated by byroot (Jean Boussier) about 1 year ago

That means we still need to make Class#attached_object a weak reference.

I agree that conceptually, it's the simplest solution. That being said, in MRI weak references generally require a finalizer, which is annoying. But I'll see if I can implement that.

Updated by ko1 (Koichi Sasada) 12 months ago

Matz asked me to solve this memory leaking issue, so I'll consider about it before Ruby 3.3.
One idea is re-introduce class serial.

Updated by byroot (Jean Boussier) 12 months ago

One idea is re-introduce class serial.

Does that mean the method cache could be global again?

Updated by Eregon (Benoit Daloze) 12 months ago

@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make Class#attached_object (internally) a weak reference.
So I would strongly advise against changing the method lookup for this, it's an independent problem.
For instance, if someone does some caching in a Hash based on the Class, they don't expect holding onto a Class also holds onto the singleton object forever.

So Class#attached_object (the internal field used for it) is the leak, not the inline cache itself which needs to hold onto the class.

Updated by Eregon (Benoit Daloze) 12 months ago

Also as mentioned in https://bugs.ruby-lang.org/issues/19436#note-1 I believe TruffleRuby and JRuby (and probably more) would never use a weak reference in the inline cache, it's too much of a footprint overhead and also an extra indirection at least in the interpreter.
So then the only other solution I see is Class#attached_object (internally) being weak.
I think CRuby should follow the same idea, otherwise CRuby would be knowingly hurting compatibility between Ruby implementations here.

A per-class serial as far as I understand it is no good for a JIT, if e.g. methods are defined in some class late or repeatedly, it would invalidate all inline cache of method lookup on that class and all related JITed code, needlessly.
https://medium.com/graalvm/precise-method-and-constant-invalidation-in-truffleruby-4dd56c6bac1a has more details on that.

Updated by ko1 (Koichi Sasada) 12 months ago

byroot (Jean Boussier) wrote in #note-8:

One idea is re-introduce class serial.

Does that mean the method cache could be global again?

No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.

Updated by ko1 (Koichi Sasada) 12 months ago

Eregon (Benoit Daloze) wrote in #note-9:

@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make Class#attached_object (internally) a weak reference.

o = Object.new
c = o.singleton_class
o = nil

# do some task

p c.attached_object #=> ???

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

Updated by Eregon (Benoit Daloze) 12 months ago

ko1 (Koichi Sasada) wrote in #note-12:

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

Yes, this is indeed a consequence.
I think it's fine, Class#attached_object is very recent and this is necessary to solve a fundamental leak.

Updated by Eregon (Benoit Daloze) 12 months ago

ko1 (Koichi Sasada) wrote in #note-11:

No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.

That would mean an extra indirection/memory load on every cached method lookup, i.e. obj->klass == CACHED_CLASS (current) vs obj->klass->serial == CACHED_SERIAL.
Also it would require these serials to be unique, otherwise that's not a correct check (so maybe more of a "unique ID" than serial).

Updated by ufuk (Ufuk Kayserilioglu) 12 months ago

Let me add a few points in this conversation:

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

I agree with @Eregon (Benoit Daloze) on this point that this is totally acceptable. When I was proposing Class#attached_object the argument was that it would be helpful in cases where the objects are still supposed to be in memory for introspection purposes. There is very little value in trying to introspect an attached object that was already garbage collected, and it would not be great to retain objects just because they had a singleton class defined for them.

So, I am +1 for slightly changing the semantics of Class#attached_object before it becomes more widely used.

I should also say, though, that the purist in me still considers that the cache should be the one holding weak references, since, after all it is a cache and a miss should be relatively inconsequential, as opposed to the link between a singleton class and its attached object. I do understand, however, that it might not be the desired implementation and pragmatism over purism might be more warranted here.

Actions #16

Updated by byroot (Jean Boussier) 12 months ago

  • Target version set to 3.3

Updated by byroot (Jean Boussier) 12 months ago

  • Assignee set to ko1 (Koichi Sasada)
Actions #18

Updated by byroot (Jean Boussier) 7 months ago

Actions #19

Updated by Ruby (Elena Mangano) 7 months ago

  • Status changed from Open to Closed

Applied in changeset git|c330037c1a38cc257dbe21022fcc7b13159c2821.


cc->cme should not be marked.

cc is callcache.

cc->klass (klass) should not be marked because if the klass is
free'ed, the cc->klass will be cleared by vm_cc_invalidate().

cc->cme (cme) should not be marked because if cc is invalidated
when cme is free'ed.

  • klass marks cme if klass uses cme.
  • caller classe's ccs->cme marks cc->cme.
  • if cc is invalidated (klass doesn't refer the cc),
    cc is invalidated by vm_cc_invalidate() and cc->cme is
    not be accessed.
  • On the multi-Ractors, cme will be collected with global GC
    so that it is safe if GC is not interleaving while accessing
    cc and cme.

fix [Bug #19436]

10_000.times{|i|
  # p i if (i%1_000) == 0

  str = "x" * 1_000_000
  def str.foo = nil
  eval "def call#{i}(s) = s.foo"
  send "call#{i}", str
}

Without this patch:

real    1m5.639s
user    0m6.637s
sys     0m58.292s

and with this patch:

real    0m2.045s
user    0m1.627s
sys     0m0.164s

Updated by byroot (Jean Boussier) 7 months ago

@ko1 (Koichi Sasada) thanks for the patch, however we're running into some assertion failures on our CI:

Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)

/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x563d185aeb55] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x563d1859d954] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x563d1877a9b8] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_insnhelper.c:2181

Which I suspect are related?

Updated by ko1 (Koichi Sasada) 7 months ago

Thanks.
Could you give some more lines of C backtrace?

Updated by byroot (Jean Boussier) 7 months ago

@ko1 (Koichi Sasada) of course, here you go:

Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)
ruby 3.3.0dev (2023-07-30T08:05:58Z shopify b5c74d5488) [x86_64-linux]

-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 5

-- C level backtrace information -------------------------------------------
/usr/local/ruby/bin/ruby(rb_print_backtrace+0xd) [0x55dcf6947c3a] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:772
/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x55dcf6751c85] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x55dcf6740995] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x55dcf691db28] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:2181
/usr/local/ruby/bin/ruby(vm_search_super_method) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:4637
/usr/local/ruby/bin/ruby(vm_sendish+0xac) [0x55dcf6938e40] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5520
/usr/local/ruby/bin/ruby(vm_exec_core) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:939
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(catch_i+0xa3) [0x55dcf692da33] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_catch_protect+0xfd) [0x55dcf69266bd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2360
/usr/local/ruby/bin/ruby(rb_catch_obj+0x28) [0x55dcf692697d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2386
/usr/local/ruby/bin/ruby(rb_f_catch) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2336
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_funcallv_scope+0x6f) [0x55dcf6932fac] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1062
/usr/local/ruby/bin/ruby(rb_funcallv) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1077
/usr/local/ruby/bin/ruby(autoload_feature_require+0x70) [0x55dcf690bee6] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2739
/usr/local/ruby/bin/ruby(autoload_try_load) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2753
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_autoload_load+0x12f) [0x55dcf690a94f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2814
/usr/local/ruby/bin/ruby(rb_const_lookup+0x0) [0x55dcf690ab9b] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2916
/usr/local/ruby/bin/ruby(rb_const_search_from) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2903
/usr/local/ruby/bin/ruby(rb_const_search+0x7) [0x55dcf690b097] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2938
/usr/local/ruby/bin/ruby(rb_const_get_0) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2865
/usr/local/ruby/bin/ruby(rb_mod_const_get+0x1f5) [0x55dcf67ec8e5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/object.c:2401
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield_values2+0xa0) [0x55dcf692dae0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(each_pair_i_fast+0x37) [0x55dcf6780cf7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3046
/usr/local/ruby/bin/ruby(RHASH_ST_TABLE+0x0) [0x55dcf678111f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1253
/usr/local/ruby/bin/ruby(hash_foreach_iter) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1255
/usr/local/ruby/bin/ruby(st_general_foreach+0x3b) [0x55dcf68ab030] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1516
/usr/local/ruby/bin/ruby(rb_st_foreach_check) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1621
/usr/local/ruby/bin/ruby(hash_foreach_call+0x3d) [0x55dcf678431d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1382
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_hash_foreach+0x91) [0x55dcf6784519] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1406
/usr/local/ruby/bin/ruby(rb_hash_foreach) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1392
/usr/local/ruby/bin/ruby(rb_hash_each_pair+0x47) [0x55dcf67845b7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3081
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0x11d) [0x55dcf692e0ad] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(yield_under+0x234) [0x55dcf692ceb4] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call0_cfunc_with_frame+0xe8) [0x55dcf692e7d0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:170
/usr/local/ruby/bin/ruby(vm_call0_cfunc) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:184
/usr/local/ruby/bin/ruby(vm_call0_body) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:230
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x35) [0x55dcf67bff94] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require_relative) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:989
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(raise_load_if_failed+0x0) [0x55dcf67bd63d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(rb_load_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:791
/usr/local/ruby/bin/ruby(rb_f_load+0x89) [0x55dcf67bdef9] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:864
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_ec_exec_node+0xc1) [0x55dcf67530b1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:287
/usr/local/ruby/bin/ruby(ruby_run_node+0x56) [0x55dcf6758406] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:328
/usr/local/ruby/bin/ruby(rb_main+0x21) [0x55dcf6752637] ./main.c:39
/usr/local/ruby/bin/ruby(main) ./main.c:58
/lib/x86_64-linux-gnu/libc.so.6(0x7f686651ad90) [0x7f686651ad90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f686651ae40]
[0x55dcf6752685]

Updated by ko1 (Koichi Sasada) 7 months ago

Thanks,
Could you try current master (I hope it was fixed).

Updated by byroot (Jean Boussier) 7 months ago

@ko1 (Koichi Sasada) I don't see any more failures, thank you very much.

So overall the patch is all these commits?

c330037c1a38cc257dbe21022fcc7b13159c2821 - cc->cme should not be marked. 
6391132c03ac08da0483adb986ff9a54e41f9e14 - fix typo (CACH_ -> CACHE_)
6dc15cc8895b28800d5c187929d846ac4eb7cd3f - do not clear cme but invalidate cc 
7a7aba755d8da34555445510dc568380d0d96791 - check liveness of cc->klass and cc->cme_ 
087a2deccfb9d99961f1ce8526b80c5f72ee9a5d - check cc->* liveness strictly
36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super 
280419d0e0ba3e96e19551c70cba789fbedd80e1 - calling->cd instead of calling->ci 

I'm asking because I may test it more extensively in production which involve backporting it to our custom 3.2.

Updated by ko1 (Koichi Sasada) 7 months ago

36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super

for this SEGV. But other patches can be required.

Updated by cedricm (Cédric Maïon) 6 months ago

Hi,

Will this be backported to 3.2?
I'm currently seeing this in production on a large Rails app: when YJIT is enabled, after usual initial warmup, unicorn workers memory usage continue to grows linearly until reaching a significant amount of RAM -- at which point they are either killed by a monitoring tool due to unreasonable memory usage (>650MB), or some cache gets flushed and a big chunk of memory gets released (and then it goes back for another round).
Memory usage is perfectly stable without YJIT (~420MB).
I believe that this is linked to this bug.

Cheers

Updated by byroot (Jean Boussier) 6 months ago

This bug is unrelated to YJIT, it happens with or without YJIT enabled. It thus cannot explain what you are seeing.

As for memory usage with YJIT, what you are describing isn't out of the ordinary. The default --yjit-exec-mem-size is 64MB, and the rule of thumb is that if you had the metadata YJIT need to track, you will find an overhead of ~4x yjit-exec-mem-size, so roughly 250MB.

So you probably want to raise your Unicorn memory limit a bit more, or reduce the --yjit-exec-mem-size.

Actions

Also available in: Atom PDF

Like1
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0