Feature #20249: Introduce a backtrace-only mode for rb_bug()

• Status: Open
• Assignee: -
• Target version: -

[osyoyu (Daisuke Aritomo)]

Background

When a segfault or some unexpected situation occurs, rb_bug() is called and prints some few hundred to thousands of lines.

The most helpful parts are (arguably) "Ruby level backtrace information" and "C-level backtrace information", but those parts are buried in the very lengthy report.
In particular, the "Other runtime information" which contains the list of loaded features (scripts?) and the process memory map could be extremely long despite it does not come very useful, at least when developing C extensions.

Even a minimal report from a simple script would consist of 250 lines and require 7 PgUps on my MacBook Air (13 inch) to reach the backtrace part, which contains all the information I need.

Proposal

My proposal is to default to a "minimal report" mode with a limited set of sections, perhaps only "Ruby level backtrace information" and "C level backtrace information" only

When a full report is desired (i.e. for bug reports), the user could re-run the script with an special environment variable, such as RUBY_FULL_CRASH_REPORT=1.
Rust implmements a similar pattern. It doesn't print the full backtrace on panics by default; instead, it guides the user to re-run the program with RUST_BACKTRACE=1.

It might be hard to reproduce some crashes and segfaults, especially in long-running daemons. It might be nice to default to the "full" mode when stdout is not a tty, since daemons tend to run in non-tty environments.

Appendix

A typical crash report would look like this:

../../example.rb: [BUG]
ruby 3.4.0dev (2024-01-20T15:27:19Z master 366b14c0cd) [arm64-darwin23]

-- Crash Report log information --------------------------------------------
(5 lines)

-- Control frame information -----------------------------------------------
(~50 lines)

-- Ruby level backtrace information ----------------------------------------
(depends on program; typically ~50 lines in Rails)

-- C level backtrace information -------------------------------------------
(50+-ish lines, depends on program)

-- Machine register context ------------------------------------------------
(~10 lines)

-- Threading information ---------------------------------------------------
(2 lines)

-- Other runtime information -----------------------------------------------
* Loaded script (1 line)
* Loaded features (depends on program; 800+ lines in Rails)
* Process memory map (depends on environment; around 200 lines?)

[osyoyu (Daisuke Aritomo)]

I've created a PoC in https://github.com/ruby/ruby/pull/9900.

[byroot (Jean Boussier)]

I agree that the crash reports are very long and some very long part aren't really that useful. 99% of the time only the C-level backtrace and Ruby backtrace is all you need.

So I'm supportive to such change. What we could do it to default to the more complete report if Ruby is compiled with RUBY_DEBUG=1.

[osyoyu (Daisuke Aritomo)]

Thank you for your comment!

byroot (Jean Boussier) wrote in #note-2:

So I'm supportive to such change. What we could do it to default to the more complete report if Ruby is compiled with RUBY_DEBUG=1.

As the majority of Ruby users won't use a Ruby compiled with RUBY_DEBUG=1, I think that bug reports from the general public won't be the full version. Is that okay?

[byroot (Jean Boussier)]

I think that bug reports from the general public won't be the full version. Is that okay?

In my opinion it is. Most of the time I strip the bug report when posting it here (for example: https://bugs.ruby-lang.org/issues/20016), and I don't remember ever been asked to provide the full report.

The one thing that may be missed once in a while, is "Loaded features" is that a few times it helped confirm the user was running a particular native gem that is causing issue, but it's the exception and often not strictly required.

[rubyFeedback (robert heiler)]

byroot wrote:

I don't remember ever been asked to provide the full report.

Probably because e. g. a full stacktrace is a bit spammy and does not always carry the
most important information "everywhere". To explain that: I often have to scroll several
hundred lines up when some .rb file segfaults or crashes, to get the important information.
In the long run perhaps we can adapt the stacktrace in general; a bit terser, but more
importantly to kind of highlight the exact problem first and foremost, without treating
all other lines as "equally important" as that "nil object, line 852 in foobar.rb". I
don't have a good proposal for a change, though, just that the stacktrace I get from
ruby seem less helpful compared to "did you mean" gem information and other helpers here.

[osyoyu (Daisuke Aritomo)]

I've received feedback that if we switch the default, no one would really use the full output mode, so we shall keep the current behavior for better bug reports. I agree that Ruby users (non-CRuby/C ext developers) won't run into crashes often, and they probably won't mind the long output.

So, my new proposal is the following:

(1) Introduce a new environment variable RUBY_CRASH_REPORT_MODE. When set to minimal, only the Ruby-level backtrace and C-level backtrace is printed. C extension developers (who don't want to compile their own Ruby with RUBY_DEBUG=1) would use this feature. Any other value would imply the current behavior.

(2) Default to RUBY_CRASH_REPORT_MODE=minimal behavior when RUBY_DEBUG=1. This feature is for CRuby core developers who don't want to configure an env var all the time.

RUBY_DEBUG behavior is intentionally flipped from @byroot (Jean Boussier) 's comment. I believe non-RUBY_DEBUG behavior should be consistent with what Ruby users see.

[osyoyu (Daisuke Aritomo)]

• Subject changed from Print only backtraces in rb_bug(), by default to Introduce a backtrace-only mode for rb_bug()

[ko1 (Koichi Sasada)]

• As a MRI developer, I don't need loaded features (LF) and memory maps (MM) and it interrupts me to see the backtraces. In many cases they are not needed.
  • Because it is easy to repro, in many cases.
• As a bug report receiver, it can be valuable in rare cases.
  • Checking LF to find suspicious libraries.
  • Checking SEGV memory area with MM. <- I observed only one time it is utilized.
  • I believe it is difficult to use RUBY_FULL_CRASH_REPORT=1 for ordinal Ruby users because it is hard to reproduce the issue on production environment, and forget to set it.

Ideas:

1. Remove long LF/MM because it is not used most of cases.

• easiest and almost reasoanable.

2. Default on for ordinal Ruby users and provide a way to turn off LF/MM.

• Environment variables on running time
• configure options on build time

3. Generate a detailed error report with LF/MM in a file and show summary of errors on TTY.

• it is similar to RUBY_CRASH_REPORT but generate reports without any configuration.
• maybe the file path is issue. respect TEMP envvals?
• it can cause disk full easily on development phase.

4. Generate detailed error report if RUBY_CRASH_REPORT is given.

For (3), the TTY lines should be:

t.rb:1: [BUG] Segmentation fault
ruby 3.3.0dev (2023-09-15T04:27:19Z master fe0225ff4d) [x64-mswin64_140]

-- Ruby level backtrace information ----------------------------------------
t.rb:1:in `<main>'
t.rb:1:in `kill'

-- C level backtrace information -------------------------------------------
C:\Windows\SYSTEM32\ntdll.dll(ZwWaitForSingleObject+0x14) [0x00007FFF625CF3F4]
C:\Windows\System32\KERNELBASE.dll(WaitForSingleObjectEx+0x8e) [0x00007FFF5FC744EE]
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_print_backtrace+0x34) [0x00007FFEF7D25D78] c:\ko1\ruby\src\wt_master\vm_dump.c:791
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_vm_bugreport+0x1b7) [0x00007FFEF7D25F37] c:\ko1\ruby\src\wt_master\vm_dump.c:1091
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_bug_for_fatal_signal+0x65) [0x00007FFEF7C532E9] c:\ko1\ruby\src\wt_master\error.c:819
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(sigsegv+0x3d) [0x00007FFEF7D4D58D] c:\ko1\ruby\src\wt_master\signal.c:920
C:\Windows\System32\ucrtbase.dll(raise+0x1e5) [0x00007FFF6015E5F5]
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(kill+0x49) [0x00007FFEF7CC0851] c:\ko1\ruby\src\wt_master\win32\win32.c:4993
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_f_kill+0x200) [0x00007FFEF7D4CC6C] c:\ko1\ruby\src\wt_master\signal.c:494
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_cfunc_with_frame_+0x13e) [0x00007FFEF7C67E16] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:3477
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_cfunc_with_frame+0x29) [0x00007FFEF7C67CD1] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:3504
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_cfunc_other+0xb3) [0x00007FFEF7C67C93] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:3531
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_cfunc+0x12c) [0x00007FFEF7C679B4] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:3612
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_method_each_type+0x59c) [0x00007FFEF7C69108] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:4389
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_call_method+0x14f) [0x00007FFEF7C68B3F] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:4553
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_sendish+0x209) [0x00007FFEF7C72FA9] c:\ko1\ruby\src\wt_master\vm_insnhelper.c:5563
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(vm_exec_core+0x1c40) [0x00007FFEF7C6DAFC] c:\ko1\ruby\src\wt_master\vm_exec.c:101
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_vm_exec+0x101) [0x00007FFEF7C62E0D] c:\ko1\ruby\src\wt_master\vm.c:2404
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(rb_ec_exec_node+0x135) [0x00007FFEF7C4C945] c:\ko1\ruby\src\wt_master\eval.c:291
c:\ko1\ruby\install\wt_master\bin\x64-vcruntime140-ruby330.dll(ruby_run_node+0x46) [0x00007FFEF7C4F112] c:\ko1\ruby\src\wt_master\eval.c:328
c:\ko1\ruby\install\wt_master\bin\ruby.exe(main+0x4a) [0x00007FF6BA62104A] c:\ko1\ruby\src\wt_master\main.c:59
c:\ko1\ruby\install\wt_master\bin\ruby.exe(__scrt_common_main_seh+0x10c) [0x00007FF6BA621264] D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x1d) [0x00007FFF60DE257D]

Details: /tmp/ruby-crash-log-1234

In details, we can put additional information, for example backtraces in all threads.

BTW RUBY_FULL_CRASH_REPORT=1 can be useful on Ruby's CI environment.

[Eregon (Benoit Daloze)]

(2) Default to RUBY_CRASH_REPORT_MODE=minimal behavior when RUBY_DEBUG=1. This feature is for CRuby core developers who don't want to configure an env var all the time.

This seems counter-productive because e.g. ruby/setup-ruby has a ruby-debug build, and if that crashes we'd definitely want all the info we can get.
So I think CRuby core developers should set the env var too if they want less verbose bug output.

[ko1 (Koichi Sasada)]

At devmeeting yesterday, we concluded to have a new environment variable to control the rb_bug() crash report(as @Eregon (Benoit Daloze) said on https://bugs.ruby-lang.org/issues/20249#note-9).

There are two proposals:

• Introduce new environment variable
  • RUBY_CRASH_REPORT_MODE, _OPTION, _CONFIG, ...
• Extend RUBY_CRASH_REPORT with options
  • RUBY_CRASH_REPORT=path style keeps current behavior
  • RUBY_CRASH_REPORT=key1=val,key2=val,... (if envstr =~ /^\[a-z]+=/)

I prefer to extend RUBY_CRASH_REPORT but is it possible to use the path name like foo=bar?
@katei (Yuta Saito) told us that swift has SWIFT_BACKTRACE https://github.com/apple/swift/blob/main/docs/Backtracing.rst#how-do-i-configure-backtracing to configurate the backtrace output.
And we also have some keys to control crash report (for developers).

The followings are bikeshed fuel:

If we allow key=value style options, maybe the following:

• items (examples):
  • bt: Ruby-level backtrace
  • vmbt: Ruby-level backtrace with VM information
  • cbt: C-level backtrace
  • parallel
  • regs: Register information
  • lf: loaded features
  • mm: memory maps
  • further possibilities
    • allbts: bakcktraces for all threads
• style (or preset or set?): preset configurations
  • default: current setting?
  • short: only bt and cbt?
  • full: all of them?
• path: output filename

To satisfy the current proposal, we can use path and style.

Further bikeshed:

I also want to show short crash report on TTY but I want to get full report in a file. It can be more complicated.

Another concern is, rb_bug() will be called on critical situation and checking internal information (e.g. traversing backtrace) will cause duplicate SEGV. So the order is important. We need to care the order.