Your patch uses RB_VM_LOCK_ENTER_NO_BARRIER but it should block normal use of rb_bug() (using rb_bug() is irregular case though).
So I think it should use simpler mechanism to synchronize rb_bug() calling. For example, introducing a global variable to avoid multiple rb_bug() calls.
(btw VM_ASSERT() calls rb_bug() if RUBY_DEBUG (or other macros) is defined, so rb_bug() is suitable for the example)
Thanks for your comment. I can make it simpler, but I am a bit confused as to what I should do instead. If the first thread gets to the global variable first and enters rb_vm_bugreport, my thinking was that other threads that also try to enter this function should be blocked (mutex, sleep, etc.). Are you saying just return from the function and let the other thread continue anyway?
Also when you say use a global variable, do you mean an atomic global? I'm open to doing whatever you want, because maybe I'm overthinking it for just a debug case anyway.