Bug #19436: Call Cache for singleton methods can lead to "memory leaks"

• Status: Closed
• Assignee: ko1 (Koichi Sasada)
• Target version: 3.3
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[byroot (Jean Boussier)]

Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads.

Minimal Reproduction

module Foo
  def bar
  end
end

def call_bar(obj)
  # Here the call cache we'll keep a ref on the method_entry
  # which then keep a ref on the singleton_class, making that
  # instance immortal until the method is called again with
  # another instance.

  # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT
  obj.bar
end

obj = Object.new
obj.extend(Foo)

call_bar(obj)

id = obj.object_id
obj = nil

4.times { GC.start }
p ObjectSpace._id2ref(id)

Explanation

Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class
and in the cache of a singleton class, it keeps a strong reference onto the attached_object (instance).

This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver.
If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak.
And if the attached_object is big, the wasted memory can be very substantial.

Practical Implications

Once relative common API impacted by this is Rails' extending API. This API allow to extend a "query result set" with a module.
These query results set can sometimes be very big, especially since they keep references to the instantiated ActiveRecord::Base instances etc.

Possible Solutions

Only keep a weak reference to the CME

The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work.
However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is.

Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class:

# vm_callinfo.h:275

struct rb_callcache {
    const VALUE flags;

    /* inline cache: key */
    const VALUE klass; // should not mark it because klass can not be free'd
                       // because of this marking. When klass is collected,
                       // cc will be cleared (cc->klass = 0) at vm_ccs_free().

So it appears that the class being also marked through the CME is some kind of oversight?

Don't cache based on some heuristics

If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a Class or Module.

It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long.

Make Class#attached_object a weak reference

Alternatively we could make the attached_object a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario.

The downside is that Class#attached_object was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit.

cc @peterzhu2118 (Peter Zhu) @ko1 (Koichi Sasada)

[Eregon (Benoit Daloze)]

I think making Class#attached_object a weak reference is a better and easier solution.

Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM, as it would mean one extra WeakReference object for each of these call caches.
I suppose there could be workarounds like storing the WeakReference on the singleton class, and use a direct reference otherwise, but then it'd make method lookup more complex and slower in interpreter or when polymorphic (need to handle cached method lookup on regular and singleton class differently).
FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.
That approach is probably also quite difficult to get right on CRuby, also considering ractors and multithreading when the class GCs.

As a general note, creating a singleton class is not cheap, this should only be used for class objects (which always have one) and for a few rare global objects where it's convenient.
Using Object#extend objects often/on the fast path is just "making programs slow and uncached".

[byroot (Jean Boussier)]

FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.

I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a Module or Class.

Using Object#extend objects often/on the fast path is just "making programs slow and uncached".

Yeah I agree. I tried to avoid this in Rails by caching these dynamic classes, unfortunately that's not possible because of the API: https://github.com/rails/rails/pull/47314. So I'll try to see if this API can be replaced by a better one and deprecated.

[byroot (Jean Boussier)]

Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM

Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.

[Eregon (Benoit Daloze)]

byroot (Jean Boussier) wrote in #note-2:

I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a Module or Class.

I missed that sorry.
I think there are quite a few cases where a global object (which is not a module/class) with a singleton class is used, it'd be unfortunate and slow to not inline cache lookup in those cases.

I think it makes sense to only try caching one (or a few) singleton class(es), and give up when seeing a second one or more (already happens on TruffleRuby, but for a higher limit, 8 currently).
That means we still need to make Class#attached_object a weak reference.

[Eregon (Benoit Daloze)]

byroot (Jean Boussier) wrote in #note-3:

Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.

Yes, since TruffleRuby caches the the Class object in method lookup (LookupMethodNode), and currently the Class object has a strong reference to the attached_object.

[byroot (Jean Boussier)]

That means we still need to make Class#attached_object a weak reference.

I agree that conceptually, it's the simplest solution. That being said, in MRI weak references generally require a finalizer, which is annoying. But I'll see if I can implement that.

[ko1 (Koichi Sasada)]

Matz asked me to solve this memory leaking issue, so I'll consider about it before Ruby 3.3.
One idea is re-introduce class serial.

[byroot (Jean Boussier)]

One idea is re-introduce class serial.

Does that mean the method cache could be global again?

[Eregon (Benoit Daloze)]

@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make Class#attached_object (internally) a weak reference.
So I would strongly advise against changing the method lookup for this, it's an independent problem.
For instance, if someone does some caching in a Hash based on the Class, they don't expect holding onto a Class also holds onto the singleton object forever.

So Class#attached_object (the internal field used for it) is the leak, not the inline cache itself which needs to hold onto the class.

[Eregon (Benoit Daloze)]

Also as mentioned in https://bugs.ruby-lang.org/issues/19436#note-1 I believe TruffleRuby and JRuby (and probably more) would never use a weak reference in the inline cache, it's too much of a footprint overhead and also an extra indirection at least in the interpreter.
So then the only other solution I see is Class#attached_object (internally) being weak.
I think CRuby should follow the same idea, otherwise CRuby would be knowingly hurting compatibility between Ruby implementations here.

A per-class serial as far as I understand it is no good for a JIT, if e.g. methods are defined in some class late or repeatedly, it would invalidate all inline cache of method lookup on that class and all related JITed code, needlessly.
https://medium.com/graalvm/precise-method-and-constant-invalidation-in-truffleruby-4dd56c6bac1a has more details on that.

[ko1 (Koichi Sasada)]

byroot (Jean Boussier) wrote in #note-8:

One idea is re-introduce class serial.

Does that mean the method cache could be global again?

No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.

[ko1 (Koichi Sasada)]

Eregon (Benoit Daloze) wrote in #note-9:

@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make Class#attached_object (internally) a weak reference.

o = Object.new
c = o.singleton_class
o = nil

# do some task

p c.attached_object #=> ???

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

[Eregon (Benoit Daloze)]

ko1 (Koichi Sasada) wrote in #note-12:

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

Yes, this is indeed a consequence.
I think it's fine, Class#attached_object is very recent and this is necessary to solve a fundamental leak.

[Eregon (Benoit Daloze)]

ko1 (Koichi Sasada) wrote in #note-11:

No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.

That would mean an extra indirection/memory load on every cached method lookup, i.e. obj->klass == CACHED_CLASS (current) vs obj->klass->serial == CACHED_SERIAL.
Also it would require these serials to be unique, otherwise that's not a correct check (so maybe more of a "unique ID" than serial).

[ufuk (Ufuk Kayserilioglu)]

Let me add a few points in this conversation:

making weak reference, attached_objec may return nil for the collected object?

I'm not sure it is acceptable or not.

I agree with @Eregon (Benoit Daloze) on this point that this is totally acceptable. When I was proposing Class#attached_object the argument was that it would be helpful in cases where the objects are still supposed to be in memory for introspection purposes. There is very little value in trying to introspect an attached object that was already garbage collected, and it would not be great to retain objects just because they had a singleton class defined for them.

So, I am +1 for slightly changing the semantics of Class#attached_object before it becomes more widely used.

I should also say, though, that the purist in me still considers that the cache should be the one holding weak references, since, after all it is a cache and a miss should be relatively inconsequential, as opposed to the link between a singleton class and its attached object. I do understand, however, that it might not be the desired implementation and pragmatism over purism might be more warranted here.

[byroot (Jean Boussier)]

• Target version set to 3.3

[byroot (Jean Boussier)]

• Assignee set to ko1 (Koichi Sasada)

[byroot (Jean Boussier)]

• Related to Feature #19783: Weak References in the GC added

[Ruby (Elena Mangano)]

• Status changed from Open to Closed

Applied in changeset git|c330037c1a38cc257dbe21022fcc7b13159c2821.

---

cc->cme should not be marked.

cc is callcache.

cc->klass (klass) should not be marked because if the klass is
free'ed, the cc->klass will be cleared by vm_cc_invalidate().

cc->cme (cme) should not be marked because if cc is invalidated
when cme is free'ed.

• klass marks cme if klass uses cme.
• caller classe's ccs->cme marks cc->cme.
• if cc is invalidated (klass doesn't refer the cc),
  cc is invalidated by vm_cc_invalidate() and cc->cme is
  not be accessed.
• On the multi-Ractors, cme will be collected with global GC
  so that it is safe if GC is not interleaving while accessing
  cc and cme.

fix [Bug #19436]

10_000.times{|i|
  # p i if (i%1_000) == 0

  str = "x" * 1_000_000
  def str.foo = nil
  eval "def call#{i}(s) = s.foo"
  send "call#{i}", str
}

Without this patch:

real    1m5.639s
user    0m6.637s
sys     0m58.292s

and with this patch:

real    0m2.045s
user    0m1.627s
sys     0m0.164s

[byroot (Jean Boussier)]

@ko1 (Koichi Sasada) thanks for the patch, however we're running into some assertion failures on our CI:

Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)

/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x563d185aeb55] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x563d1859d954] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x563d1877a9b8] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_insnhelper.c:2181

Which I suspect are related?

[ko1 (Koichi Sasada)]

Thanks.
Could you give some more lines of C backtrace?

[byroot (Jean Boussier)]

@ko1 (Koichi Sasada) of course, here you go:

Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)
ruby 3.3.0dev (2023-07-30T08:05:58Z shopify b5c74d5488) [x86_64-linux]

-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 5

-- C level backtrace information -------------------------------------------
/usr/local/ruby/bin/ruby(rb_print_backtrace+0xd) [0x55dcf6947c3a] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:772
/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x55dcf6751c85] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x55dcf6740995] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x55dcf691db28] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:2181
/usr/local/ruby/bin/ruby(vm_search_super_method) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:4637
/usr/local/ruby/bin/ruby(vm_sendish+0xac) [0x55dcf6938e40] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5520
/usr/local/ruby/bin/ruby(vm_exec_core) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:939
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(catch_i+0xa3) [0x55dcf692da33] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_catch_protect+0xfd) [0x55dcf69266bd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2360
/usr/local/ruby/bin/ruby(rb_catch_obj+0x28) [0x55dcf692697d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2386
/usr/local/ruby/bin/ruby(rb_f_catch) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2336
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_funcallv_scope+0x6f) [0x55dcf6932fac] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1062
/usr/local/ruby/bin/ruby(rb_funcallv) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1077
/usr/local/ruby/bin/ruby(autoload_feature_require+0x70) [0x55dcf690bee6] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2739
/usr/local/ruby/bin/ruby(autoload_try_load) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2753
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_autoload_load+0x12f) [0x55dcf690a94f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2814
/usr/local/ruby/bin/ruby(rb_const_lookup+0x0) [0x55dcf690ab9b] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2916
/usr/local/ruby/bin/ruby(rb_const_search_from) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2903
/usr/local/ruby/bin/ruby(rb_const_search+0x7) [0x55dcf690b097] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2938
/usr/local/ruby/bin/ruby(rb_const_get_0) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2865
/usr/local/ruby/bin/ruby(rb_mod_const_get+0x1f5) [0x55dcf67ec8e5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/object.c:2401
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield_values2+0xa0) [0x55dcf692dae0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(each_pair_i_fast+0x37) [0x55dcf6780cf7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3046
/usr/local/ruby/bin/ruby(RHASH_ST_TABLE+0x0) [0x55dcf678111f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1253
/usr/local/ruby/bin/ruby(hash_foreach_iter) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1255
/usr/local/ruby/bin/ruby(st_general_foreach+0x3b) [0x55dcf68ab030] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1516
/usr/local/ruby/bin/ruby(rb_st_foreach_check) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1621
/usr/local/ruby/bin/ruby(hash_foreach_call+0x3d) [0x55dcf678431d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1382
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_hash_foreach+0x91) [0x55dcf6784519] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1406
/usr/local/ruby/bin/ruby(rb_hash_foreach) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1392
/usr/local/ruby/bin/ruby(rb_hash_each_pair+0x47) [0x55dcf67845b7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3081
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0x11d) [0x55dcf692e0ad] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(yield_under+0x234) [0x55dcf692ceb4] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call0_cfunc_with_frame+0xe8) [0x55dcf692e7d0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:170
/usr/local/ruby/bin/ruby(vm_call0_cfunc) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:184
/usr/local/ruby/bin/ruby(vm_call0_body) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:230
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x35) [0x55dcf67bff94] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require_relative) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:989
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(raise_load_if_failed+0x0) [0x55dcf67bd63d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(rb_load_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:791
/usr/local/ruby/bin/ruby(rb_f_load+0x89) [0x55dcf67bdef9] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:864
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_ec_exec_node+0xc1) [0x55dcf67530b1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:287
/usr/local/ruby/bin/ruby(ruby_run_node+0x56) [0x55dcf6758406] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:328
/usr/local/ruby/bin/ruby(rb_main+0x21) [0x55dcf6752637] ./main.c:39
/usr/local/ruby/bin/ruby(main) ./main.c:58
/lib/x86_64-linux-gnu/libc.so.6(0x7f686651ad90) [0x7f686651ad90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f686651ae40]
[0x55dcf6752685]

[ko1 (Koichi Sasada)]

Thanks,
Could you try current master (I hope it was fixed).

[byroot (Jean Boussier)]

@ko1 (Koichi Sasada) I don't see any more failures, thank you very much.

So overall the patch is all these commits?

c330037c1a38cc257dbe21022fcc7b13159c2821 - cc->cme should not be marked. 
6391132c03ac08da0483adb986ff9a54e41f9e14 - fix typo (CACH_ -> CACHE_)
6dc15cc8895b28800d5c187929d846ac4eb7cd3f - do not clear cme but invalidate cc 
7a7aba755d8da34555445510dc568380d0d96791 - check liveness of cc->klass and cc->cme_ 
087a2deccfb9d99961f1ce8526b80c5f72ee9a5d - check cc->* liveness strictly
36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super 
280419d0e0ba3e96e19551c70cba789fbedd80e1 - calling->cd instead of calling->ci 

I'm asking because I may test it more extensively in production which involve backporting it to our custom 3.2.

[ko1 (Koichi Sasada)]

36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super

for this SEGV. But other patches can be required.

[cedricm (Cédric Maïon)]

Hi,

Will this be backported to 3.2?
I'm currently seeing this in production on a large Rails app: when YJIT is enabled, after usual initial warmup, unicorn workers memory usage continue to grows linearly until reaching a significant amount of RAM -- at which point they are either killed by a monitoring tool due to unreasonable memory usage (>650MB), or some cache gets flushed and a big chunk of memory gets released (and then it goes back for another round).
Memory usage is perfectly stable without YJIT (~420MB).
I believe that this is linked to this bug.

Cheers

[byroot (Jean Boussier)]

This bug is unrelated to YJIT, it happens with or without YJIT enabled. It thus cannot explain what you are seeing.

As for memory usage with YJIT, what you are describing isn't out of the ordinary. The default --yjit-exec-mem-size is 64MB, and the rule of thumb is that if you had the metadata YJIT need to track, you will find an overhead of ~4x yjit-exec-mem-size, so roughly 250MB.

So you probably want to raise your Unicorn memory limit a bit more, or reduce the --yjit-exec-mem-size.