Bug #18286: Universal arm64/x86_84 binary built on an x86_64 machine segfaults/is killed on arm64

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 3.0.2
• Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN

[ccaviness (Clay Caviness)]

A universal arm64/x86_84 ruby binary for macOS built on a x86_64 machine segfaults/is killed when executed on an arm64 machine.

To reproduce:

• On an Intel Mac: git clone https://github.com/ruby/ruby && cd ruby && git checkout v3_0_2 && ./autogen.sh && ./configure --with-arch=arm64,x86_64 && make -j$(sysctl -n hw.ncpu)
• Copy the built ./ruby binary to an Apple Silicon machine
• Attempt to execute it

Expected:
The universal ruby binary works correctly on both devices

Actual:
The universal ruby binary crashes with either Segmentation fault: 11 or Killed: 9 (this seems to occur if arm64e is used instead of arm64).

Details:
I'm attempting to build a universal Ruby for macOS that will run on both Intel (x86_64) and Apple Silicon (arm64) machines.

It seemed initially that this was as easy as adding --with-arch=arm64,x86_64 to ./configure would do it, as it produced a ruby binary that reports as Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]

This ruby works correctly on the Intel machine I built in on, but does not work when copied to an Apple Silicon device. The reverse, however, seems to work. That is, if I build the universal ruby on an Apple Silicon machine, the ruby binary that's built seems to work correctly on both Intel and Apple Silicon machines.

Intel:

$ ./ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [universal.x86_64-darwin21]

Apple Silicon:

$ ./ruby -v
Segmentation fault: 11
$ lldb ./ruby
(lldb) target create "./ruby"
Current executable set to '/Users/crc/ruby' (arm64).
(lldb) run
Process 77071 launched: '/Users/crc/ruby' (arm64)
Process 77071 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x00000001002176b8 ruby`ruby_vm_special_exception_copy + 16
ruby`ruby_vm_special_exception_copy:
->  0x1002176b8 <+16>: ldr    x0, [x0, #0x8]
    0x1002176bc <+20>: bl     0x10011fed8               ; rb_class_real
    0x1002176c0 <+24>: bl     0x10012070c               ; rb_obj_alloc
    0x1002176c4 <+28>: mov    x20, x0
Target 0: (ruby) stopped.
(lldb) ^D

I also attempted the same thing with ruby 2.7.4 source, with the same result.

[nobu (Nobuyoshi Nakada)]

Could you try with the master, and show more backtraces?

[ccaviness (Clay Caviness)]

nobu (Nobuyoshi Nakada) wrote in #note-1:

Could you try with the master, and show more backtraces?

Sure. Similar error, though this time running the universal ruby on Apple Silicon just results in a Killed: 9 message. I'm unable to run this binary under lldb; however, I'm not familiar with debuggers so if there's a different method you'd like me to try I'd be happy to. I did get a backtrace for the segfault on the v3_0_2 build.

ruby built on an Intel machine, from master, running my Apple Silicon device:

$ file ruby 
ruby: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
ruby (for architecture x86_64):	Mach-O 64-bit executable x86_64
ruby (for architecture arm64):	Mach-O 64-bit executable arm64
$ ./ruby -v
Killed: 9
$ lldb ./ruby
(lldb) target create "./ruby"
Killed: 9

ruby built on an Intel machine, from v3_0_2, running my Apple Silicon device:

$ lldb ruby
(lldb) target create "ruby"
Current executable set to '/Users/crc/ruby' (arm64).
(lldb) run
Process 38054 launched: '/Users/crc/ruby' (arm64)
Process 38054 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x00000001002176b8 ruby`ruby_vm_special_exception_copy + 16
ruby`ruby_vm_special_exception_copy:
->  0x1002176b8 <+16>: ldr    x0, [x0, #0x8]
    0x1002176bc <+20>: bl     0x10011fed8               ; rb_class_real
    0x1002176c0 <+24>: bl     0x10012070c               ; rb_obj_alloc
    0x1002176c4 <+28>: mov    x20, x0
Target 0: (ruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x00000001002176b8 ruby`ruby_vm_special_exception_copy + 16
    frame #1: 0x0000000100217788 ruby`ec_stack_overflow + 56
    frame #2: 0x0000000100217708 ruby`rb_ec_stack_overflow + 40
    frame #3: 0x000000010023da90 ruby`rb_call0 + 1828
    frame #4: 0x00000001001213bc ruby`rb_class_new_instance + 88
    frame #5: 0x000000010008a6d8 ruby`rb_exc_new_str + 64
    frame #6: 0x000000010022fee4 ruby`rb_vm_register_special_exception_str + 52
    frame #7: 0x00000001000966cc ruby`Init_eval + 768
    frame #8: 0x00000001000c4d34 ruby`rb_call_inits + 72
    frame #9: 0x0000000100093e58 ruby`ruby_setup + 316
    frame #10: 0x0000000100093ee0 ruby`ruby_init + 12
    frame #11: 0x0000000100001be4 ruby`main + 76
    frame #12: 0x00000001003fd0f4 dyld`start + 520
(lldb) 

[timsutton (Tim Sutton)]

I have been hoping to do the same operation here for my org, as a way to distribute a universal Ruby binary that would be usable on both Intel and Apple Silicon machines, and to be able to build it on Intel. I seem to run into the same problem when building on Intel.

[ecnelises (Chaofan QIU)]

Can you please try codesign -s - ruby? Because Apple's arm chip requires the exectutables signed.

I encountered the same killed 9 error elsewhere, FYI: https://lists.gnu.org/archive/html/bug-gnu-emacs/2020-11/msg01480.html

[timsutton (Tim Sutton)]

Sure. I had suspected that at some point so I checked the signature using codesign -dvvvvv. But I also just repeated that test, and then replaced the built binary with a new ad-hoc signature on the M1. That unfortunately seemed to not help:

# intel-built universal binary copied over
tsutton@tim-m1 ~ % cp /Volumes/ssd/ruby_274 .

tsutton@tim-m1 ~ % codesign -d -vvvvv ruby_274 
Executable=/Users/tsutton/ruby_274
Identifier=-5ac6e2.out
Format=Mach-O universal (x86_64 arm64)
CodeDirectory v=20400 size=30020 flags=0x20002(adhoc,linker-signed) hashes=935+0 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=721664
Hash type=sha256 size=32
CandidateCDHash sha256=63eda95634ac1d1ea6c97467085ec887b45f1dde
CandidateCDHashFull sha256=63eda95634ac1d1ea6c97467085ec887b45f1dde4659262d661eccca13ba17ca
Hash choices=sha256
CMSDigest=63eda95634ac1d1ea6c97467085ec887b45f1dde4659262d661eccca13ba17ca
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=2834432
Executable Segment flags=0x1
Page size=4096
CDHash=63eda95634ac1d1ea6c97467085ec887b45f1dde
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

tsutton@tim-m1 ~ % ./ruby_274 
zsh: segmentation fault  ./ruby_274

tsutton@tim-m1 ~ % cp ruby_274 ruby_274_copy

tsutton@tim-m1 ~ % codesign -s - ruby_274_copy 

tsutton@tim-m1 ~ % ./ruby_274_copy  
zsh: segmentation fault  ./ruby_274_copy

# using -f to force signature replacement
tsutton@tim-m1 ~ % codesign -fs - ruby_274_copy
ruby_274_copy: replacing existing signature

tsutton@tim-m1 ~ % ./ruby_274_copy             
zsh: segmentation fault  ./ruby_274_copy

[ccaviness (Clay Caviness)]

Lack of codesigning on Apple Silicon is an excellent guess, but unfortunately does not seem to be the cause here as Tim's demonstrated above (and I've verified as well). I first noticed this issue when testing a ruby that was fully signed with a public developer cert.

[dentarg (Patrik Ragnarsson)]

Is it this? https://bugs.chromium.org/p/v8/issues/detail?id=11389#c18, https://github.com/nodejs/node/issues/37061#issuecomment-774175983

[ccaviness (Clay Caviness)]

I don't believe any of those bugs are related.

My suspicion is that, when building on x86 and targeting universal, during configure for cross-compilation on arm64 the small test binaries that built cannot be executed on x86, leading to the various hints about the host machine to be wildly incorrect.

When building on arm64 and targeting universal, these test binaries that are built for x86 can actually run on the arm64 machine successfully, due to the Rosetta x86 compatibility layer.

There is no mechanism to run arm64 binaries on x86 Macs, though, so I think to get cross-compilation working on x86 many of the various autoconf hints will need to be manually set.

I'm not that familiar with autoconf or what these values should be, though.

[benhamilton (Ben Hamilton)]

I think I know what the problem is.

During the build, Ruby has special logic to serialize its own builtin module to disk using the binary iseq format during the build (I assume for speed so it doesn't have to parse builtin every time it starts up).

However, since iseq format is architecture-specific, when building on x86_64 for universal x86_64 + arm64, the serialized builtin module is written with the x86_64 architecture of the build machine, which fails this check whenever ruby imports its (serialized to x86_64-specific iseq format) builtin module on arm64:

https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/compile.c#L13243

Thankfully, there's logic to disable this feature for cross-compiled builds:

https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/builtin.c#L6

We just need to enable this for universal builds as well.

[benhamilton (Ben Hamilton)]

Sent PR https://github.com/ruby/ruby/pull/7367 with a fix.

[benhamilton (Ben Hamilton)]

I also reproduced the SIGSEGV from the original bug using a build with debug symbols:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x00000001002444b4 ruby`ruby_vm_special_exception_copy [inlined] RBASIC_CLASS(obj=0) at rbasic.h:155:25
    frame #1: 0x00000001002444b4 ruby`ruby_vm_special_exception_copy(exc=0) at vm_insnhelper.c:52:42
    frame #2: 0x0000000100244584 ruby`ec_stack_overflow(ec=0x0000000100704920, setup=1) at vm_insnhelper.c:65:16
    frame #3: 0x0000000100244504 ruby`rb_ec_stack_overflow(ec=0x0000000100704920, crit=0) at vm_insnhelper.c:97:5
    frame #4: 0x000000010026b544 ruby`rb_call0(ec=0x0000000100704920, recv=4300956680, mid=3121, argc=1, argv=0x000000016fdff3e0, call_scope=<unavailable>, self=4) at rgengc.h:0:9
    frame #5: 0x000000010013ca30 ruby`rb_class_new_instance [inlined] rb_class_new_instance_kw(argc=1, argv=0x000000016fdff3e0, klass=<unavailable>, kw_splat=0) at object.c:2025:5
    frame #6: 0x000000010013c9f0 ruby`rb_class_new_instance(argc=1, argv=0x000000016fdff3e0, klass=<unavailable>) at object.c:2033:12
    frame #7: 0x00000001000953c0 ruby`rb_exc_new_str(etype=<unavailable>, str=4300956720) at error.c:1145:12
    frame #8: 0x000000010025d668 ruby`rb_vm_register_special_exception_str(sp=ruby_error_reenter, cls=<unavailable>, mesg=<unavailable>) at vm.c:2872:17
    frame #9: 0x00000001000a1d50 ruby`Init_eval at eval.c:2091:5
    frame #10: 0x00000001000d51fc ruby`rb_call_inits at inits.c:41:5
    frame #11: 0x000000010009fa14 ruby`ruby_setup at eval.c:89:9
    frame #12: 0x000000010009fa90 ruby`ruby_init at eval.c:101:17
    frame #13: 0x0000000100004610 ruby`main [inlined] rb_main(argc=1, argv=0x000000016fdff938) at main.c:37:5
    frame #14: 0x0000000100004604 ruby`main(argc=1, argv=0x000000016fdff938) at main.c:57:12
    frame #15: 0x000000018599be50 dyld`start + 2544

(snip)

(lldb) print *ec
(rb_execution_context_t) $7 = {
  vm_stack = 0x0000000108028000
  vm_stack_size = 131072
  cfp = 0x0000000108127fc0
  tag = 0x000000016fdff4b0
  interrupt_flag = 0
  interrupt_mask = 0
  fiber_ptr = 0x00000001007048d0
  thread_ptr = 0x0000000100704260
  local_storage = NULL
  local_storage_recursive_hash = 4
  local_storage_recursive_hash_for_trace = 4
  storage = 4
  root_lep = 0x0000000000000000
  root_svar = 0
  ensure_list = NULL
  trace_arg = NULL
  errinfo = 4
  passed_block_handler = 0
  raised_flag = '\0'
  method_missing_reason = MISSING_NOENTRY
  private_const_reference = 0
  machine = {
    stack_start = 0x000000016fdff4ac
    stack_end = 0x000000016fdff2c0
    stack_maxsize = 0

(snip)

(lldb) print ec->machine.stack_start
(VALUE *) $4 = 0x000000016fdff4ac
(lldb) print ec->machine.stack_end
(VALUE *) $5 = 0x000000016fdff2c0
(lldb) print ec->machine.stack_end - ec->machine.stack_start
(long) $6 = -61

(snip)

(lldb) print ruby_stack_length(NULL)
(size_t) $8 = 18446744073709551529

Looks like it's got the stack direction backwards. I checked config.log, and it's incorrectly detecting the stack length as +1 (growing towards larger addresses) for arm64:

| #if defined __x86_64__
| #define STACK_GROW_DIRECTION -1
| #endif /* defined __x86_64__ */
| #if defined __arm64__
| #define STACK_GROW_DIRECTION +1

(snip)

rb_cv_stack_grow_dir_arm64=+1
rb_cv_stack_grow_dir_x86_64=-1

Sent PR https://github.com/ruby/ruby/pull/7373 with this fix as well.

[ccaviness (Clay Caviness)]

Could someone please review and merge Ben's PR https://github.com/ruby/ruby/pull/7367 to fix this? I'd like to see these changes make the next release.

[ccaviness (Clay Caviness)]

https://github.com/ruby/ruby/pull/7367 fixes this, and just needs to be merged.

[nobu (Nobuyoshi Nakada)]

Does this help it?
https://github.com/ruby/ruby/pull/8708

[ccaviness (Clay Caviness)]

@nobu (Nobuyoshi Nakada) Yes, at least in initial tests. A universal ruby built on an x86 Mac with that patch seems to work on an Apple Silicon Mac.

[benhamilton (Ben Hamilton)]

• Status changed from Open to Closed

Applied in changeset git|1d5598fe0d3470e7cab06a756d40a9221fcd501b.

---

Disable iseq-dumped builtin module for universal x86_64/arm64 binaries

During the build, Ruby has special logic to serialize its own builtin
module to disk using the binary iseq format during the build (I assume
for speed so it doesn't have to parse builtin every time it starts
up).

However, since iseq format is architecture-specific, when building on
x86_64 for universal x86_64 + arm64, the serialized builtin module is
written with the x86_64 architecture of the build machine, which fails
this check whenever ruby imports the builtin module on arm64:

https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/compile.c#L13243

Thankfully, there's logic to disable this feature for cross-compiled builds:

https://github.com/ruby/ruby/blob/1fdaa0666086529b3aae2d509a2e71c4247c3a12/builtin.c#L6

This disables the iseq logic for universal builds as well.

Fixes [Bug #18286]