Project

General

Profile

Actions

Bug #18000

closed

have_library doesn't work when ruby is compiled with --disable-shared --disable-install-static-library

Added by byroot (Jean Boussier) over 2 years ago. Updated over 2 years ago.

Status:
Closed
Assignee:
-
Target version:
-
[ruby-core:104361]

Description

Related [Feature #12845]

If you compile ruby with --disable-shared --disable-install-static-library, then many C-extension won't compile anymore. For instance RedCloth

# extconf.rb
require 'mkmf'
CONFIG['warnflags'].gsub!(/-Wshorten-64-to-32/, '') if CONFIG['warnflags']
$CFLAGS << ' -O0 -Wall ' if CONFIG['CC'] =~ /gcc/
dir_config("redcloth_scan")
have_library("c", "main")
create_makefile("redcloth_scan")
#mkmf.log 

"gcc -o conftest -I/usr/local/include/ruby-3.1.0/x86_64-linux -I/usr/local/include/ruby-3.1.0/ruby/backward -I/usr/local/include/ruby-3.1.0 -I. -I/usr/local/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef -O0 -Wall  conftest.c  -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L. -L/usr/local/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic     -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby-static -lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm   -lm   -lc"
/usr/bin/ld: cannot find -lruby-static
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return !!argv[argc];
6: }
/* end */

We'd like to use both flags, the first because it provide a small performance improvement, the second because libruby-static.a is enormous (up to 120MiB on debug builds).

@alanwu (Alan Wu) says it's theoretically possible to compile with just the headers.


Files

no-static-no-shared.diff (2.55 KB) no-static-no-shared.diff jeremyevans0 (Jeremy Evans), 08/12/2021 05:26 PM
rb_prefix_hack.diff (1.76 KB) rb_prefix_hack.diff jeremyevans0 (Jeremy Evans), 08/12/2021 05:56 PM

Updated by xtkoba (Tee KOBAYASHI) over 2 years ago

A workaround would be to place a dummy libruby-static.a wherever ld can find it.

!<arch>

Updated by byroot (Jean Boussier) over 2 years ago

Thanks for the workaround, it does work for that very specific gem, however gems checking for specific Ruby APIs are broken by it. e.g. stackprof

require 'mkmf'
if have_func('rb_postponed_job_register_one') &&
   have_func('rb_profile_frames') &&
   have_func('rb_tracepoint_new') &&
   have_const('RUBY_INTERNAL_EVENT_NEWOBJ')
  create_makefile('stackprof/stackprof')
else
  fail 'missing API: are you using ruby 2.1+?'
end
checking for rb_postponed_job_register_one()... no
*** extconf.rb failed ***
"gcc -o conftest -I/usr/local/include/ruby-3.1.0/x86_64-linux -I/usr/local/include/ruby-3.1.0/ruby/backward -I/usr/local/include/ruby-3.1.0 -I. -I/usr/local/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef conftest.c  -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L. -L/usr/local/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic     -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby-static -lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm   -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:14:57: error: ‘rb_postponed_job_register_one’ undeclared (first use in this function)
   14 | int t(void) { void ((*volatile p)()); p = (void ((*)()))rb_postponed_job_register_one; return !p; }
      |                                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
conftest.c:14:57: note: each undeclared identifier is reported only once for each function it appears in
conftest.c: At top level:
cc1: warning: unrecognized command line option ‘-Wno-self-assign’
cc1: warning: unrecognized command line option ‘-Wno-parentheses-equality’
cc1: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: /*top*/
 4: extern int t(void);
 5: int main(int argc, char **argv)
 6: {
 7:   if (argc > 1000000) {
 8:     int (* volatile tp)(void)=(int (*)(void))&t;
 9:     printf("%d", (*tp)());
10:   }
11: 
12:   return !!argv[argc];
13: }
14: int t(void) { void ((*volatile p)()); p = (void ((*)()))rb_postponed_job_register_one; return !p; }
/* end */

"gcc -o conftest -I/usr/local/include/ruby-3.1.0/x86_64-linux -I/usr/local/include/ruby-3.1.0/ruby/backward -I/usr/local/include/ruby-3.1.0 -I. -I/usr/local/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef conftest.c  -L. -L/usr/local/lib -Wl,-rpath,/usr/local/lib -L. -L/usr/local/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic     -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lruby-static -lz -lpthread -lrt -lrt -lgmp -ldl -lcrypt -lm   -lm   -lc"
/usr/bin/ld: /tmp/ccLbP8Oe.o: in function `t':
/usr/local/lib/ruby/gems/3.1.0/gems/stackprof-0.2.17/ext/stackprof/conftest.c:15: undefined reference to `rb_postponed_job_register_one'
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: /*top*/
 4: extern int t(void);
 5: int main(int argc, char **argv)
 6: {
 7:   if (argc > 1000000) {
 8:     int (* volatile tp)(void)=(int (*)(void))&t;
 9:     printf("%d", (*tp)());
10:   }
11: 
12:   return !!argv[argc];
13: }
14: extern void rb_postponed_job_register_one();
15: int t(void) { rb_postponed_job_register_one(); return 0; }
/* end */

Updated by jeremyevans0 (Jeremy Evans) over 2 years ago

While it is possible to compile with only the headers, that only works if you know what the headers should contain. If you aren't sure what the headers contain, and you have to test, you really need to test both compiling and linking to be reasonably sure whether the headers support the function in question. You can only do such testing if you have either the shared or static library available.

The issue here is how the mkmf have_func method works. Currently, it writes a small C program and tries to compile and link it. If it cannot do so correctly, it determines the function doesn't work. You can work around this by only trying to compile and not link. However, that breaks cases that will compile but won't link, or won't compile unless you link. Such cases exist in the openssl extension, at least in my environment. Either I couldn't get Ruby to build with the changes, or the built Ruby wouldn't have a working openssl extension (needed for gem install). I've attached my attempt at this in case you want to keep hacking to make it work. I don't think the approach is feasible, though.

One possible hack that may work for this use case is if have_func is checking for a function that starts with rb_, assume it is already defined. These calls are usually used to check that the Ruby version in use includes a function that wasn't defined in previous Ruby versions. If you are running the latest Ruby version, all of these functions are probably defined. This is obviously a hack we couldn't ship, though from my testing, it does allow gem install stackprof. I've attached this hack as well.

I think the tool/mkconfig.rb changes in both patches should be committed. These remove the -lruby-static from LIBRUBYARG_STATIC. After all, there is no point trying to link to a library that you know you didn't install. I've submitted a pull request for that: https://github.com/ruby/ruby/pull/4736

Updated by nobu (Nobuyoshi Nakada) over 2 years ago

We should disallow that combination, I think.
Otherwise, should drop also the headers.

Updated by jeremyevans0 (Jeremy Evans) over 2 years ago

nobu (Nobuyoshi Nakada) wrote in #note-4:

We should disallow that combination, I think.
Otherwise, should drop also the headers.

Here's a pull request to disallow the combination of --disable-shared --disable-install-static-library: https://github.com/ruby/ruby/pull/4737. Not sure how @byroot (Jean Boussier) or @alanwu (Alan Wu) will feel about this approach, though.

Updated by byroot (Jean Boussier) over 2 years ago

Not sure how byroot (Jean Boussier) or alanwu (Alan Wu) will feel about this approach, though.

At the very least I'd say it's progress as my attempt to ship ruby builds with this combinations of flags would have immediately failed during the build rather than later in production.

But yeah I wish I could --disable-shared without having to ship a huge 100MiB+ static library, but since I don't quite understand what it would take to support this combination of flags, I have no strong opinion about it.

Updated by nobu (Nobuyoshi Nakada) over 2 years ago

byroot (Jean Boussier) wrote in #note-6:

But yeah I wish I could --disable-shared without having to ship a huge 100MiB+ static library, but since I don't quite understand what it would take to support this combination of flags, I have no strong opinion about it.

With that combination, shouldn't we drop the header files too?

When both libraries are not installed, all undefined symbols need to be ignored or dynamic lookup, then have_func and find_library can't work as expected.

Updated by alanwu (Alan Wu) over 2 years ago

No objection from me to drop support for the --disable-shared --disable-install-static-library config. I agree with Jean that it's an
improvement to give a clear message and fail fast.

With that combination, shouldn't we drop the header files too?

If we are keen on supporting that config, it should be possible to add
special handling to have_func and other mkmf methods to work without
libruby. On glibc platforms at least, this could be done by building the test
program as a shared library, and then using dlopen(3) with RTLD_NOW to check
if every symbol resolves. The test program would still need the headers.
This testing process is similar to how Ruby would require the extension
once it's built, so maybe in some ways it's also a more accurate test.

It's already possible today to build an extension without libruby using
only the headers. Here's a small demo for it:

# A small demo for building an extension against a Ruby built with
# --disable-shared without libruby.
require 'rbconfig'

config = RbConfig::CONFIG
hdr_dir = config['rubyhdrdir']
arch_hdr_dir = config['rubyarchhdrdir']
# Note how this doesn't link against libruby at all
compile_command = "cc -shared -fpic -o ext.so -I #{hdr_dir} -I #{arch_hdr_dir} ext.c"

File.write("ext.c", <<~EOM)
#include <ruby/ruby.h>

void
Init_ext(void)
{
    // Using two Ruby symbols. The static Ruby executable exports these.
    rb_raise(rb_eRuntimeError, "hello from ext");
}
EOM

puts " Enabled shared: #{config['ENABLE_SHARED']}"
puts "Compile command: #{compile_command}"

system(compile_command)

require_relative 'ext'

Adding support takes more effort than rejecting the config outright of
course. Another easy option is making mkmf error on that config.
We could still leave the headers for people that want to build
extensions without mkmf. Maybe there are users writing extensions
in languages not supported by mkmf.

Actions #9

Updated by jeremyevans (Jeremy Evans) over 2 years ago

  • Status changed from Open to Closed

Applied in changeset git|06c3e80611b81ec8f251957328486e9b6dd18d3b.


Do not allow configuration where neither static or shared library is installed

Fixes [Bug #18000]

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0