Project

General

Profile

Actions

Bug #19005

closed

Ruby interpreter compiled XCode 14 cannot build some native gems on macOS

Added by stanhu (Stan Hu) over 1 year ago. Updated 7 months ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 2.7.6p219 (2022-04-12 revision 44c8bfa984) [arm64-darwin21]
[ruby-core:109905]

Description

This seems related to https://bugs.ruby-lang.org/issues/18912 and https://bugs.ruby-lang.org/issues/18981 .

Steps to reproduce:

  1. Upgrade to XCode 14.
  2. Compile a new Ruby interpreter. I used the version provided in https://github.com/ruby/ruby/pull/6297 with ./configure --prefix=/tmp/ruby --with-openssl-dir=$(brew --prefix openssl@1.1) --with-readline-dir=$(brew --prefix readline) --enable-shared.
  3. Confirm that -Wl,-undefined,dynamic_lookup is no longer available:
irb(main):001:0> RbConfig::CONFIG['DLDFLAGS']
=> "-Wl,-multiply_defined,suppress"
  1. Ran gem install pg_query (gem install ffi-yajl will also fail).

Error:

linking shared-object pg_query/pg_query.bundle
Undefined symbols for architecture arm64:
  "Init_pg_query", referenced from:
     -exported_symbol[s_list] command line option
     (maybe you meant: _Init_pg_query)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I can workaround the problem by doing:

gem install pg_query -- --with-ldflags="-Wl,-undefined,dynamic_lookup"

Related issues 3 (0 open3 closed)

Related to Ruby master - Bug #18912: Build failure with Xcode 14 and macOS 13 (Ventura) BetaClosedhsbt (Hiroshi SHIBATA)Actions
Related to Ruby master - Bug #18981: Backport bundle loader option to stable versionsClosednagachika (Tomoyuki Chikanaga)Actions
Related to Ruby master - Bug #19082: Recent gRPC gem fails to build from the source in already released versionsThird Party's IssueActions
Actions #1

Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

  • Related to Bug #18912: Build failure with Xcode 14 and macOS 13 (Ventura) Beta added
  • Related to Bug #18981: Backport bundle loader option to stable versions added

Updated by stanhu (Stan Hu) over 1 year ago

I should note that it's also possible to workaround the issue by compiling the interpreter with DLDFLAGS="-Wl,-undefined,dynamic_lookup".

configure no longer adds this flag by default because of the warning:

configure:28712: checking whether -Wl,-undefined,dynamic_lookup is accepted as LDFLAGS
configure:28739: gcc -o conftest -g -O2 -Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT -fstack-protector-strong -Wl,-undefined,dynamic_lookup conftest.c -lpthread -ldl -lobjc >&5
ld: warning: -undefined dynamic_lookup may not work with chained fixups

Updated by stanhu (Stan Hu) over 1 year ago

It's possible this is a problem with the respective gems. In https://github.com/pganalyze/pg_query/issues/255#issuecomment-1248800937, I note that the Init_pg_query is using C++ visibility rules.

However, for ffi-yajl, I'm not sure this is the same problem:

compiling encoder.c
encoder.c:307:14: warning: unused function 'rb_cBignum_ffi_yajl' [-Wunused-function]
static VALUE rb_cBignum_ffi_yajl(VALUE self, VALUE rb_yajl_gen, VALUE state) {
             ^
1 warning generated.
linking shared-object ffi_yajl/ext/encoder.bundle
Undefined symbols for architecture arm64:
  "_yajl_gen_alloc", referenced from:
      _mEncoder_do_yajl_encode in encoder.o
  "_yajl_gen_array_close", referenced from:
      _gen_array_close in encoder.o
  "_yajl_gen_array_open", referenced from:
      _gen_array_open in encoder.o
  "_yajl_gen_bool", referenced from:
      _gen_true in encoder.o
      _gen_false in encoder.o
  "_yajl_gen_config", referenced from:
      _mEncoder_do_yajl_encode in encoder.o
  "_yajl_gen_free", referenced from:
      _mEncoder_do_yajl_encode in encoder.o
  "_yajl_gen_get_buf", referenced from:
      _mEncoder_do_yajl_encode in encoder.o
  "_yajl_gen_map_close", referenced from:
      _gen_map_close in encoder.o
  "_yajl_gen_map_open", referenced from:
      _gen_map_open in encoder.o
  "_yajl_gen_null", referenced from:
      _gen_null in encoder.o
  "_yajl_gen_number", referenced from:
      _gen_number in encoder.o
  "_yajl_gen_string", referenced from:
      _gen_cstring in encoder.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [encoder.bundle] Error 1

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Status changed from Open to Feedback

What is the version of ruby you are using?

Updated by stanhu (Stan Hu) over 1 year ago

nobu (Nobuyoshi Nakada) wrote in #note-5:

What is the version of ruby you are using?

ruby 2.7.6p219 (2022-04-12 revision 44c8bfa984) [arm64-darwin21]

Actions #7

Updated by stanhu (Stan Hu) over 1 year ago

  • ruby -v set to ruby 2.7.6p219 (2022-04-12 revision 44c8bfa984) [arm64-darwin21]

Updated by stanhu (Stan Hu) over 1 year ago

I think we can close this issue, unless someone thinks we might want to relax the configure check so that the -undefined,dynamic_lookup linker flag is again the default for macOS.

Gems should be able to handle building without -undefined,dynamic_lookup. Here are related fixes:

  1. https://github.com/pganalyze/pg_query/pull/256
  2. https://github.com/chef/ffi-yajl/pull/114

Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

  • Status changed from Feedback to Open

ruby_3_1 head already fixed a similar issue with bundle loader. But I could reproduce https://bugs.ruby-lang.org/issues/19005#note-3 with ruby_3_1 head.

We should fix this before 3.1.3 release.

Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

Ah, Sorry. I misunderstood. I could install other native extensions like ffi, hiredis and eventmachine.

this issue happens like ffi-yajl.

Updated by stanhu (Stan Hu) over 1 year ago

Yes, it's only a few gems that previously depended on -undefined,dynamic_lookup being present as a linker argument.

In the pg_query case, it seems that there is a bug: the library exports a symbol that's not actually present.

In the ffi-yajl case, it seems that the gem expects to dlopen() the yayl library, so the symbols are not present until the extension is actually loaded. We can resolve that by explicitly marking the symbols that will be used dynamically.

However, it's interesting to me that the Ruby configure script returned 0 compiling the program, but still marked the flag as being unavailable:

configure:28712: checking whether -Wl,-undefined,dynamic_lookup is accepted as LDFLAGS
configure:28739: gcc -o conftest -g -O2 -Wall -Wextra -Wdeprecated-declarations -Wdivision-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-\
64-to-32 -Wwrite-strings -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -W\
no-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wextra-tokens -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT -fstack-\
protector-strong -Wl,-undefined,dynamic_lookup conftest.c -lpthread -ldl -lobjc >&5
ld: warning: -undefined dynamic_lookup may not work with chained fixups
configure:28739: $? = 0
configure: failed program was:

I think it's because configure expects no output:

  (eval "$ac_link") 2>conftest.err
  ac_status=$?
  if test -s conftest.err; then
    grep -v '^ *+' conftest.err >conftest.er1
    cat conftest.er1 >&5
    mv -f conftest.er1 conftest.err
  fi

I don't know if there's a way to suppress that warning.

Actions #13

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Status changed from Open to Closed

Applied in changeset git|6898984f1cd1b0375c3da44d7832724489e0e470.


[Bug #19005] dynamic_lookup linker option in external libraries

The warning against -undefined dynamic_lookup is just a warning yet,
and many gems seem to pay no attention to warnings. Until it fails
actually, keep it as a migration path, except for standard extension
libraries and bundled extension gems.

Actions #14

Updated by stanhu (Stan Hu) over 1 year ago

Thanks! I've confirmed this solves the problem and that the linker adds this argument.

Could this patch be backported to 2.7 as well? Thanks!

Actions #15

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN to 2.7: REQUIRED, 3.0: REQUIRED, 3.1: REQUIRED

Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

Could this patch be backported to 2.7 as well? Thanks!

@stanhu (Stan Hu) Maybe, yes. I submitted the backport patches for Ruby 2.7 at https://github.com/ruby/ruby/pull/6442.

Updated by nagachika (Tomoyuki Chikanaga) over 1 year ago

  • Backport changed from 2.7: REQUIRED, 3.0: REQUIRED, 3.1: REQUIRED to 2.7: REQUIRED, 3.0: REQUIRED, 3.1: DONE

Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

  • Backport changed from 2.7: REQUIRED, 3.0: REQUIRED, 3.1: DONE to 2.7: DONE, 3.0: DONE, 3.1: DONE

Updated by Eregon (Benoit Daloze) over 1 year ago

I think longer-term it would be really best to avoid using -Wl,-undefined,dynamic_lookup.
The gems which depend on that should be fixed to avoid the need for it, or add -Wl,-undefined,dynamic_lookup themselves if they truly need it (probably extremely rare, it most likely means a missing -lmylib or so).

Updated by kou (Kouhei Sutou) over 1 year ago

FYI: Here is one of my use cases that need -Wl,-undefined,dynamic_lookup:

Updated by kou (Kouhei Sutou) over 1 year ago

Ah, should we use -Wl,-U with explicit listed depending symbols for this?

Updated by stanhu (Stan Hu) over 1 year ago

I think longer-term it would be really best to avoid using -Wl,-undefined,dynamic_lookup.

Yeah, I agree. The problem is that most gems aren't tested in this mode, and it's long been the default everywhere.

Ah, should we use -Wl,-U with explicit listed depending symbols for this?

I did that with https://github.com/chef/ffi-yajl/pull/114 since a dlopen call is used, but in your case you can you just link the library directly?

Updated by kou (Kouhei Sutou) over 1 year ago

but in your case you can you just link the library directly?

I can't do it because cairo.bundle (it's a bundle not a shared library) in the cairo gem can't be linked from cairo_gobject.bundle in the cairo-gobject gem.

Updated by stanhu (Stan Hu) about 1 year ago

I think this problem was "accidentally" fixed in Ruby 2.7.7 and 3.0.5, but it's not working in Ruby 3.1.3 and up due to a simple removal of a dollar sign (https://github.com/ruby/ruby/commit/667aa81219ca080c0a4b9f97d29bb3221bd08a33).

In Ruby 3.1.3, I'm not seeing the ADDITIONAL_DLDFLAGS set with -Wl,-undefined,dynamic_lookup:

irb(main):001:0> RUBY_VERSION
=> "3.1.3"
irb(main):002:0> RbConfig::CONFIG['ADDITIONAL_DLDFLAGS']
=> ""

Whereas 3.0.5 has this:

irb(main):001:0> RUBY_VERSION
=> "3.0.5"
irb(main):002:0> RbConfig::CONFIG['ADDITIONAL_DLDFLAGS']
=> "-Wl,-undefined,dynamic_lookup"

If I look at the ./configure output in Ruby 3.0.5 or 2.7.7, I see something like this:

checking whether -Wl,-undefined,dynamic_lookup is accepted as LDFLAGS... ./configure: line 29806: -Wl,-undefined,dynamic_lookup=: command not found
checking whether -Wl,-undefined,dynamic_lookup is accepted for bundle... no

But with 3.1.3 and up, the LDFLAGS check is a straight no:

checking whether -Wl,-undefined,dynamic_lookup is accepted as LDFLAGS... no

The configure output with Ruby 3.0.5 looks like:

if ac_fn_c_try_link "$LINENO"
then :

    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: yes" >&5
    printf "%s\n" "${msg_result_yes}yes${msg_reset}" >&6 ; }
else $as_nop
  $flag=
    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: no" >&5
    printf "%s\n" "${msg_result_no}no${msg_reset}" >&6 ; }
fi

The failing line in question is $flag=.

In Ruby 3.1.3 and up, it appears $flag= has been replaced with flag= due to https://github.com/ruby/ruby/commit/667aa81219ca080c0a4b9f97d29bb3221bd08a33:

if ac_fn_c_try_link "$LINENO"
then :

    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: yes" >&5
    printf "%s\n" "${msg_result_yes}yes${msg_reset}" >&6 ; }
else $as_nop
  flag=
    { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: no" >&5
    printf "%s\n" "${msg_result_no}no${msg_reset}" >&6 ; }
fi
Actions #26

Updated by stanhu (Stan Hu) about 1 year ago

  • Status changed from Closed to 9

Updated by stanhu (Stan Hu) about 1 year ago

  • Status changed from 9 to Closed

Ok, I see this was reported in https://bugs.ruby-lang.org/issues/19082.

Actions #28

Updated by Eregon (Benoit Daloze) about 1 year ago

  • Related to Bug #19082: Recent gRPC gem fails to build from the source in already released versions added
Actions #29

Updated by stanhu (Stan Hu) about 1 year ago

  • Status changed from Closed to 9

Updated by stanhu (Stan Hu) about 1 year ago

A possible fix is to squelch this warning via -no_fixup_chains:

diff --git a/configure.ac b/configure.ac
index a2a0c1e387..d653a13bb8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3104,7 +3104,7 @@ AS_IF([test "$rb_cv_dlopen" = yes], [
         AC_SUBST(ADDITIONAL_DLDFLAGS, "")
 	for flag in \
 	  "-multiply_defined suppress" \
-	  "-undefined dynamic_lookup" \
+	  "-no_fixup_chains -undefined dynamic_lookup" \
 	  ; do
             test "x${linker_flag}" = x || flag="${linker_flag}`echo ${flag} | tr ' ' ,`"
             RUBY_TRY_LDFLAGS([$flag], [], [flag=])

I should note that bazel dropped -undefined dynamic_lookup entirely in https://github.com/bazelbuild/bazel/issues/16413, but that will break some gems that are not ready for this.

Actions #32

Updated by naruse (Yui NARUSE) about 1 year ago

  • Status changed from 9 to Open

Updated by nobu (Nobuyoshi Nakada) about 1 year ago

Adding -no_fixup_chains -undefined dynamic_lookup together will break old compilers.
-no_fixup_chains should be checked solely before -undefined dynamic_lookup, at least.

Updated by stanhu (Stan Hu) about 1 year ago

nobu (Nobuyoshi Nakada) wrote in #note-33:

Adding -no_fixup_chains -undefined dynamic_lookup together will break old compilers.
-no_fixup_chains should be checked solely before -undefined dynamic_lookup, at least.

Good point. I've updated https://github.com/ruby/ruby/pull/7086.

Updated by stanhu (Stan Hu) 7 months ago

We can close this now that XCode 14.3, which ships with clang 14.0.3, now fixes this issue: https://github.com/python/cpython/issues/97524#issuecomment-1458855301

Actions #36

Updated by jeremyevans0 (Jeremy Evans) 7 months ago

  • Status changed from Open to Closed
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0