Feature #20273: Disable callcc when compiled with ASAN

• Status: Closed
• Assignee: -
• Target version: -

[kjtsanaktsidis (KJ Tsanaktsidis)]

Currently, all of the callcc related tests in the Ruby test suite fail when Ruby is compiled with ASAN (CFLAGS="-fsanitize=address").

Unfortunately, I don't believe it's possible for calcc to be supported with ASAN at all. callcc is implemented by:

• When saving a continuation, we take a copy of C the stack, and a jmpbuf with setjmp.
• When calling a continuation, we memcpy the C stack back from the saved copy (using alloca to move the current top-of-stack out of the way of this copying), and then longjmp'ing back to the saved state

There are a few reasons this can't work with ASAN - some fixable, some not:

• The copying of the original stack when creating a continuation requires reading stack memory from outside the current function, which ASAN does not allow (and, likewise with copying the stack back). This can be fixed by using a custom memcpy implementation which has __attribute__((no_sanitize("address"))).
• When we restore the original stack, we also need to restore the shadow-words for that memory range (which record the state of what memory is and is not valid according to ASAN). ASAN does not expose a pointer to the shadow stack to allow it to be copied like this, but this is in theory something which we could patch in LLVM if we wanted to.
• With ASAN, many local variables are not actually on the stack - rather, they're stored in a special "fake stack" (i think to enable detecting cases of stack-use-after-return). (I think) ASAN essentially malloc's memory for each function frame to hold these fake values, and frees them when the function returns. With callcc, some functions whose fake frames have already been freed might be re-entered, but the "fake stack" holding their local variables will have been freed. I don't believe there's a way to fix this that I can think of.

---

Therefore, I propose that:

• When cruby is compiled with -fsanitize=address in CFLAGS, Kernel#callcc is rb_f_notimplement.
• That means Kernel#respond_to?(:callcc) == false, and Kernel#callcc raises NotImplementedError
• Tests in the cruby test suite which use callcc check respond_to?(:callcc) and skip themselves if so.

Since callcc is deprecated and meant only to be used for... um... "fun" things (https://bugs.ruby-lang.org/issues/10548), I think this is an acceptable tradeoff.

[mame (Yusuke Endoh)]

I think it makes sense basically. I would like to see details in PR. Do you already have a patch?

[kjtsanaktsidis (KJ Tsanaktsidis)]

What about something like this? https://github.com/ruby/ruby/pull/10012

This is the "maximal" way - it adds a new define RUBY_CALLCC_ENABLED, which we set to on normally but off if asan is enabled. Then, all the callcc related code gets guarded by this macro (including the bits spread around in vm.c for example).

I can do something less invasive (like this? https://github.com/ruby/ruby/commit/b3500503478e81985ac8d95b3576135f9787bcf6) but that generates unused function warnings that would have to be suppressed (with some ignore_unused attributes or some such)

• Status changed from Open to Closed

Applied in changeset git|5621d794a2fa9293d1da489d6b0ee0e7c73c9128.

---

Disable callcc when ASAN is enabled

callcc's implementation is fundamentally incompatible with ASAN. Since
callcc is deprecated and almost never used, it's probably OK to disable
callcc when ruby is compiled with ASAN.

[Bug #20273]

[kjtsanaktsidis (KJ Tsanaktsidis)]

• Related to Misc #20387: Meta-ticket for ASAN support added