Bug #21438: use-after-free when resizing exivars

• Status: Closed
• Assignee: -
• Target version: -
• Backport: 3.2: WONTFIX, 3.3: REQUIRED, 3.4: REQUIRED

[byroot (Jean Boussier)]

Here's a semi-reliable reproduction:

objs = 10_000.times.map do
  a = []
  a.instance_variable_set(:@a, 1)
  a
end

GC.stress = true
GC.auto_compact = true

steps = 1000.times.map do
  a = []
  a.instance_variable_set(:@a, 1)
  a.instance_variable_set(:@b, 2)
  a.instance_variable_set(:@c, 3)
  a.instance_variable_set(:@d, 4)
  a.instance_variable_set(:@e, 5)
  a.instance_variable_set(:@f, 6)
  a.instance_variable_set(:@g, 7)
  a.instance_variable_set(:@h, 8)
  # resize
  a.instance_variable_set(:@i, 9)
  a.instance_variable_set(:@j, 10)
  a
end
objs.clear

GC.stress = false
GC.auto_compact = false

The Exivar codepath uses st_update and allocate within the codebase.

If GC trigger, it may remove entires from the table, or delete+insert in case of compaction, and this can trigger a table rebuild of the generic fields st_table in the middle of calling the st_update callback.
This can cause entries to be reallocated or rearranged and the update to be for the wrong entry.

Auto compaction isn't strictly required to trigger the bug, but makes it more likely.

[jhawthorn (John Hawthorn)]

• Status changed from Open to Closed

Applied in changeset git|5342d9130beb44f9aa1dddbb7f6276bf01c7404f.

---

Fix generic_ivar_set_shape_field for table rebuild

[Bug #21438]

Previously GC could trigger a table rebuild of the generic fields
st_table in the middle of calling the st_update callback. This could
cause entries to be reallocated or rearranged and the update to be for
the wrong entry.

This commit adds an assertion to make that case easier to detect, and
replaces the st_update with a separate st_lookup and st_insert.

Co-authored-by: Aaron Patterson [e-mail]
Co-authored-by: Jean Boussier [e-mail]

[byroot (Jean Boussier)]

• Status changed from Closed to Open

Reopening because the fix we merged need some more work:

ruby(rb_vm_bugreport) ../src/vm_dump.c:1175
ruby(rb_bug_for_fatal_signal+0xf5) [0x55800a5c2d45] ../src/error.c:1130
ruby(sigsegv+0x46) [0x55800a31d176] ../src/signal.c:934
ruby(RVALUE_WB_UNPROTECTED+0x14) [0x55800a20e3cd] ../src/gc/default/default.c:1195
ruby(rgengc_check_relation) ../src/gc/default/default.c:4300
ruby(RVALUE_MARKED+0x0) [0x55800a217f31] ../src/gc/default/default.c:4369
ruby(gc_mark_set) ../src/gc/default/default.c:4311
ruby(gc_mark) ../src/gc/default/default.c:4370
ruby(rb_mark_generic_ivar+0xa8) [0x55800a397828] ../src/variable.c:1249
ruby(rb_gc_mark_children+0x108) [0x55800a218618] ../src/gc.c:3145
ruby(gc_mark_stacked_objects+0x76) [0x55800a21c58e] ../src/gc/default/default.c:4563
ruby(gc_mark_stacked_objects_all) ../src/gc/default/default.c:4622
ruby(gc_marks_rest) ../src/gc/default/default.c:5639
ruby(gc_marks+0x55a) [0x55800a220ff2] ../src/gc/default/default.c:5753
ruby(gc_start) ../src/gc/default/default.c:6425
ruby(rb_multi_ractor_p+0x0) [0x55800a222569] ../src/gc/default/default.c:6307
ruby(rb_vm_lock_leave) ../src/vm_sync.h:101
ruby(rb_gc_vm_unlock) ../src/gc.c:144
ruby(garbage_collect) ../src/gc/default/default.c:6309
ruby(objspace_malloc_gc_stress+0x2a) [0x55800a223dce] ../src/gc/default/default.c:7930
ruby(rb_gc_impl_malloc) ../src/gc/default/default.c:7921
ruby(handle_malloc_failure+0x0) [0x55800a224313] ../src/gc.c:5231
ruby(ruby_xmalloc) ../src/gc.c:5221
ruby(get_allocated_entries+0x0) [0x55800a328af2] ../src/st.c:551
ruby(rb_st_init_existing_table_with_size) ../src/st.c:559
ruby(rebuild_table+0xb6) [0x55800a328c46] ../src/st.c:585
ruby(rebuild_table_if_necessary+0x8) [0x55800a3298b0] ../src/st.c:1125
ruby(rb_st_insert) ../src/st.c:1143
ruby(generic_ivar_set_shape_fields+0x228) [0x55800a39ac38] ../src/variable.c:1846
ruby(general_ivar_set+0x17f) [0x55800a39306f] ../src/variable.c:1760
ruby(generic_ivar_set+0x0) [0x55800a39f02d] ../src/variable.c:1909
ruby(ivar_set) ../src/variable.c:2094
ruby(rb_ivar_set) ../src/variable.c:2103

https://github.com/ruby/ruby/actions/runs/15663763782/job/44125172775#step:12:385

[byroot (Jean Boussier)]

• Status changed from Open to Closed

The remaining issue was fixed in 055fef00a1c27fdc8293114dc134ca7910b1dc79.

Backport PRs:

• 3.4: https://github.com/ruby/ruby/pull/13637
• 3.3: https://github.com/ruby/ruby/pull/13638

Note that the bug while possible is quite unlikely on older branches as it requires 9 ivars instead of 4, so maintainers should feel free to not backport.