Bug #19115: RubyGems fails to detect OpenSSL in --with-static-linked-ext builds

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 3.1.2, 3.2.0dev
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: DONE

[thomthom (Thomas Thomassen)]

Related discussion: https://bugs.ruby-lang.org/issues/18876

We are seeing OpenSSL failing to autoload with the Ruby build we've been using. (We first observed this when using Gem.install that relies on autoload of OpenSSL). Our last working build was Ruby 2.7.2 with similar config.

This causes RubyGems' HAVE_OPENSSL constant to be incorrectly initialized, and subsequent calls to Gem.install fails. There might be other scenarios that fail as a result of that, but we've not managed to identified that.

 module Gem
   HAVE_OPENSSL = defined? OpenSSL::SSL # :nodoc:

Our Ruby 3.1.2 config:

'--prefix=/Users/user/.conan/data/ruby/3.1.2/sketchup/stable/build/bd95825b99dcb274d0e577fbdc953653a9ea0fb9' 
'--with-openssl-dir=/Users/user/.conan/data/openssl/1.1.1q/sketchup/stable/package/93ae0b0e7eebe0611c04d3e0d9bbf49fbce92332' 
'--with-libyaml-dir=/Users/user/.conan/data/libyaml/0.2.5/sketchup/stable/package/a56a950abed2e10dbdc26845400f0a034d97c454' 
'--disable-install-doc' 
'--disable-install-rdoc' 
'--enable-shared' 
'--enable-load-relative' 
'--with-static-linked-ext' 
'--without-debug' 
'--without-gdbm' 
'--without-gettext' 
'--without-irb' 
'--without-mkmf' 
'--without-rdoc' 
'--without-readline' 
'--without-tk' 
'--bindir=${prefix}/bin' 
'--sbindir=${prefix}/bin' 
'--libexecdir=${prefix}/bin' 
'--libdir=${prefix}/lib' 
'--includedir=${prefix}/include' 
'--oldincludedir=${prefix}/include' 
'--datarootdir=${prefix}/share' 'cflags=-mmacosx-version-min=10.14 -fdeclspec' 'cxxflags=-mmacosx-version-min=10.14 -fdeclspec' 'LDFLAGS=-mmacosx-version-min=10.14 -fdeclspec'

We also tested with latest build from master (November 8th 2022):

./ruby -ve 'p RbConfig::CONFIG["configure_args"]'
ruby 3.2.0dev (2022-11-07T19:35:21Z master b14f133054) [x86_64-darwin20]
" '--prefix=/Users/vmehta/ruby/ruby-master/' '--with-openssl-dir=/Users/vmehta/.conan/data/openssl/1.1.1q/sketchup/stable/package/f2d937af1fa19d5fc4095849a65d1927e9e75ae7/' '--with-libyaml-dir=/Users/vmehta/.conan/data/libyaml/0.2.5/sketchup/stable/package/3fc084e254210603a5c5aece184b2d45e2509b30' '--disable-install-doc' '--disable-install-rdoc' '--enable-shared' '--enable-load-relative' '--with-static-linked-ext' '--without-debug' '--without-gdbm' '--without-gettext' '--without-irb' '--without-mkmf' '--without-rdoc' '--without-readline' '--without-tk'"

Using an RVM of Ruby 3.1.2 this appear to work as expected. But using out configuration, that we've used for the Ruby 2.x versions are now failing. We haven't been able to figure out the reason for this.

[thomthom (Thomas Thomassen)]

After looking into this closer, autoload has been broken with this config at least back to Ruby 2.5 (Did test anything older).

What caused us to detect this in Ruby 3.1 was that RubyGems was using autoload for OpenSSL where it previously use require. We've avoided noticing this bug by happenstance and now it's finally bitten us.

Strangely, I've not been able to reproduce this with any autoload..

# foo.rb
puts 'foo.rb loaded'
module Foo; end

autoload(:Foo, 'C:/Users/tthomas2/Desktop/foo')
=> nil

p defined?(Foo)
"constant"
=> "constant"

Foo
foo.rb loaded
=> Foo

That works.

But not OpenSSL, like Gem is using it. Similarly, the example in https://bugs.ruby-lang.org/issues/18876#note-4 that was using Ripper:

autoload(:Ripper, "ripper"); p defined?(Ripper)
nil
=> nil

I'm not sure why some autoload fails, while some doesn't.

[alanwu (Alan Wu)]

The issue is specific to extensions that are statically linked into the build in --with-static-linked-ext.
autoload? also doesn't work correctly with these extensions:

$ ruby --disable-all -ve 'autoload(:Ripper, "ripper"); p autoload?(:Ripper)'
ruby 3.2.0dev (2022-11-15T17:26:51Z autoload 9751b54971) [arm64-darwin22]
nil

This is coming from the special treatment for native extensions.
They are inserted into vm->loading_table and so are treated as
being in the middle of being required on boot even when they're not actually
required yet. This weird loading state causes autoload? and defined? to
return nil because it looks like querying the state of the constant while
it is autoloading.

I haven't tried making a patch, but it seems wise to stop using vm->loading_table for this purpose.

[thomthom (Thomas Thomassen)]

So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?

[alanwu (Alan Wu)]

thomthom (Thomas Thomassen) wrote in #note-3:

So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?

Right.

[alanwu (Alan Wu)]

I've submitted a PR to fix this issue and I thought I'd also post some updates.

Here is a script to summarize the mechanics of the bug:

puts RUBY_DESCRIPTION # ruby 3.2.0dev (2022-01-20) [wasm32-wasi]

autoload :Ripper, 'ripper' # can resolve to ripper.so, which is builtin
p [autoload?(:Ripper), defined?(Ripper), Object.const_defined?(:Ripper)]
# => [nil, nil, false]

# Compare with a library that's an .rb file
autoload :Abbrev, 'abbrev'
p [autoload?(:Abbrev), defined?(Abbrev), Object.const_defined?(:Abbrev)]
# => ["abbrev", "constant", true]

It's specific to builds configured with --with-static-linked-ext.
Since WASM builds use this option you can reproduce the issue in a browser!

https://try.ruby-lang.org/playground/#code=puts+RUBY_DESCRIPTION+%23+ruby+3.2.0dev+(2022-01-20)+%5Bwasm32-wasi%5D%0A%0Aautoload+%3ARipper%2C+'ripper'+%23+can+resolve+to+ripper.so%2C+which+is+builtin%0Ap+%5Bautoload%3F(%3ARipper)%2C+defined%3F(Ripper)%2C+Object.const_defined%3F(%3ARipper)%5D%0A%23+%3D%3E+%5Bnil%2C+nil%2C+false%5D%0A%0A%23+Compare+with+a+library+that's+an+.rb+file%0Aautoload+%3AAbbrev%2C+'abbrev'%0Ap+%5Bautoload%3F(%3AAbbrev)%2C+defined%3F(Abbrev)%2C+Object.const_defined%3F(%3AAbbrev)%5D%0A%23+%3D%3E+%5B%22abbrev%22%2C+%22constant%22%2C+true%5D&engine=cruby-3.2.0dev

For native builds, the issue impacts RubyGems can make it think OpenSSL
is unavailable even though it is. The error is the same as mentioned in [Bug #18876]:

ERROR:  While executing gem ... (Gem::Exception)
    OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

Also, some corrections for my mistakes in [Bug #18876].
I thought defined? AutoloadConst would trigger loading but it's supposed
to behave more like checking whether loading is necessary.
I also thought the issue was fixed on master but it was actually due to me
not including Ripper in my test build which made ripper resolve to ripper.rb
when the issue could only reproduce with .so features like ripper.so that are statically linked.

[alanwu (Alan Wu)]

• Subject changed from OpenSSL fails to autoload (macOS) to RubyGems fails to detect OpenSSL in --with-static-linked-ext builds

[alanwu (Alan Wu)]

• Status changed from Open to Closed

Applied in changeset git|790cf4b6d0475614afb127b416e87cfa39044d67.

---

Fix autoload status of statically linked extensions

Previously, for statically-linked extensions, we used
vm->loading_table to delay calling the init function until the
extensions are required. This caused the extensions to look like they
are in the middle of being loaded even before they're required.
(rb_feature_p() returned true with a loading path output.) Combined
with autoload, queries like defined?(CONST) and Module#autoload?
were confused by this and returned nil incorrectly. RubyGems uses
defined? to detect if OpenSSL is available and failed when OpenSSL was
available in builds using --with-static-linked-ext.

Use a dedicated table for the init functions instead of adding them to
the loading table. This lets us remove some logic from non-EXTSTATIC
builds.

[Bug #19115]

[thomthom (Thomas Thomassen)]

Will this patch be pack-ported to the 3.1 branch?

[alanwu (Alan Wu)]

thomthom (Thomas Thomassen) wrote in #note-8:

Will this patch be pack-ported to the 3.1 branch?

It's up to the branch maintainer, @nagachika (Tomoyuki Chikanaga), to decide.

[thomthom (Thomas Thomassen)]

I built Ruby 3.1.2 that we use in our product with this patch applied, and it worked on our systems. Would be nice to see it backported.

[hsbt (Hiroshi SHIBATA)]

• Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN to 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: REQUIRED

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: REQUIRED to 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: DONE

ruby_3_1 5c5a1135b2e688eca7dbc7ce085f37a4d9fa3fd1 merged revision(s) 790cf4b6d0475614afb127b416e87cfa39044d67.