Project

General

Profile

Actions

Bug #19115

closed

RubyGems fails to detect OpenSSL in --with-static-linked-ext builds

Added by thomthom (Thomas Thomassen) 3 months ago. Updated 23 days ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
3.1.2, 3.2.0dev
[ruby-core:110670]

Description

Related discussion: https://bugs.ruby-lang.org/issues/18876

We are seeing OpenSSL failing to autoload with the Ruby build we've been using. (We first observed this when using Gem.install that relies on autoload of OpenSSL). Our last working build was Ruby 2.7.2 with similar config.

This causes RubyGems' HAVE_OPENSSL constant to be incorrectly initialized, and subsequent calls to Gem.install fails. There might be other scenarios that fail as a result of that, but we've not managed to identified that.

 module Gem
   HAVE_OPENSSL = defined? OpenSSL::SSL # :nodoc:

Our Ruby 3.1.2 config:

'--prefix=/Users/user/.conan/data/ruby/3.1.2/sketchup/stable/build/bd95825b99dcb274d0e577fbdc953653a9ea0fb9' 
'--with-openssl-dir=/Users/user/.conan/data/openssl/1.1.1q/sketchup/stable/package/93ae0b0e7eebe0611c04d3e0d9bbf49fbce92332' 
'--with-libyaml-dir=/Users/user/.conan/data/libyaml/0.2.5/sketchup/stable/package/a56a950abed2e10dbdc26845400f0a034d97c454' 
'--disable-install-doc' 
'--disable-install-rdoc' 
'--enable-shared' 
'--enable-load-relative' 
'--with-static-linked-ext' 
'--without-debug' 
'--without-gdbm' 
'--without-gettext' 
'--without-irb' 
'--without-mkmf' 
'--without-rdoc' 
'--without-readline' 
'--without-tk' 
'--bindir=${prefix}/bin' 
'--sbindir=${prefix}/bin' 
'--libexecdir=${prefix}/bin' 
'--libdir=${prefix}/lib' 
'--includedir=${prefix}/include' 
'--oldincludedir=${prefix}/include' 
'--datarootdir=${prefix}/share' 'cflags=-mmacosx-version-min=10.14 -fdeclspec' 'cxxflags=-mmacosx-version-min=10.14 -fdeclspec' 'LDFLAGS=-mmacosx-version-min=10.14 -fdeclspec'

We also tested with latest build from master (November 8th 2022):

./ruby -ve 'p RbConfig::CONFIG["configure_args"]'
ruby 3.2.0dev (2022-11-07T19:35:21Z master b14f133054) [x86_64-darwin20]
" '--prefix=/Users/vmehta/ruby/ruby-master/' '--with-openssl-dir=/Users/vmehta/.conan/data/openssl/1.1.1q/sketchup/stable/package/f2d937af1fa19d5fc4095849a65d1927e9e75ae7/' '--with-libyaml-dir=/Users/vmehta/.conan/data/libyaml/0.2.5/sketchup/stable/package/3fc084e254210603a5c5aece184b2d45e2509b30' '--disable-install-doc' '--disable-install-rdoc' '--enable-shared' '--enable-load-relative' '--with-static-linked-ext' '--without-debug' '--without-gdbm' '--without-gettext' '--without-irb' '--without-mkmf' '--without-rdoc' '--without-readline' '--without-tk'"

Using an RVM of Ruby 3.1.2 this appear to work as expected. But using out configuration, that we've used for the Ruby 2.x versions are now failing. We haven't been able to figure out the reason for this.

Updated by thomthom (Thomas Thomassen) 3 months ago

After looking into this closer, autoload has been broken with this config at least back to Ruby 2.5 (Did test anything older).

What caused us to detect this in Ruby 3.1 was that RubyGems was using autoload for OpenSSL where it previously use require. We've avoided noticing this bug by happenstance and now it's finally bitten us.

Strangely, I've not been able to reproduce this with any autoload..

# foo.rb
puts 'foo.rb loaded'
module Foo; end
autoload(:Foo, 'C:/Users/tthomas2/Desktop/foo')
=> nil

p defined?(Foo)
"constant"
=> "constant"

Foo
foo.rb loaded
=> Foo

That works.

But not OpenSSL, like Gem is using it. Similarly, the example in https://bugs.ruby-lang.org/issues/18876#note-4 that was using Ripper:

autoload(:Ripper, "ripper"); p defined?(Ripper)
nil
=> nil

I'm not sure why some autoload fails, while some doesn't.

Updated by alanwu (Alan Wu) 2 months ago

The issue is specific to extensions that are statically linked into the build in --with-static-linked-ext.
autoload? also doesn't work correctly with these extensions:

$ ruby --disable-all -ve 'autoload(:Ripper, "ripper"); p autoload?(:Ripper)'
ruby 3.2.0dev (2022-11-15T17:26:51Z autoload 9751b54971) [arm64-darwin22]
nil

This is coming from the special treatment for native extensions.
They are inserted into vm->loading_table and so are treated as
being in the middle of being required on boot even when they're not actually
required yet. This weird loading state causes autoload? and defined? to
return nil because it looks like querying the state of the constant while
it is autoloading.

I haven't tried making a patch, but it seems wise to stop using vm->loading_table for this purpose.

Updated by thomthom (Thomas Thomassen) 2 months ago

So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?

Updated by alanwu (Alan Wu) 2 months ago

thomthom (Thomas Thomassen) wrote in #note-3:

So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?

Right.

Updated by alanwu (Alan Wu) 2 months ago

I've submitted a PR to fix this issue and I thought I'd also post some updates.

Here is a script to summarize the mechanics of the bug:

puts RUBY_DESCRIPTION # ruby 3.2.0dev (2022-01-20) [wasm32-wasi]

autoload :Ripper, 'ripper' # can resolve to ripper.so, which is builtin
p [autoload?(:Ripper), defined?(Ripper), Object.const_defined?(:Ripper)]
# => [nil, nil, false]

# Compare with a library that's an .rb file
autoload :Abbrev, 'abbrev'
p [autoload?(:Abbrev), defined?(Abbrev), Object.const_defined?(:Abbrev)]
# => ["abbrev", "constant", true]

It's specific to builds configured with --with-static-linked-ext.
Since WASM builds use this option you can reproduce the issue in a browser!

https://try.ruby-lang.org/playground/#code=puts+RUBY_DESCRIPTION+%23+ruby+3.2.0dev+(2022-01-20)+%5Bwasm32-wasi%5D%0A%0Aautoload+%3ARipper%2C+'ripper'+%23+can+resolve+to+ripper.so%2C+which+is+builtin%0Ap+%5Bautoload%3F(%3ARipper)%2C+defined%3F(Ripper)%2C+Object.const_defined%3F(%3ARipper)%5D%0A%23+%3D%3E+%5Bnil%2C+nil%2C+false%5D%0A%0A%23+Compare+with+a+library+that's+an+.rb+file%0Aautoload+%3AAbbrev%2C+'abbrev'%0Ap+%5Bautoload%3F(%3AAbbrev)%2C+defined%3F(Abbrev)%2C+Object.const_defined%3F(%3AAbbrev)%5D%0A%23+%3D%3E+%5B%22abbrev%22%2C+%22constant%22%2C+true%5D&engine=cruby-3.2.0dev

For native builds, the issue impacts RubyGems can make it think OpenSSL
is unavailable even though it is. The error is the same as mentioned in [Bug #18876]:

ERROR:  While executing gem ... (Gem::Exception)
    OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

Also, some corrections for my mistakes in [Bug #18876].
I thought defined? AutoloadConst would trigger loading but it's supposed
to behave more like checking whether loading is necessary.
I also thought the issue was fixed on master but it was actually due to me
not including Ripper in my test build which made ripper resolve to ripper.rb
when the issue could only reproduce with .so features like ripper.so that are statically linked.

Actions #6

Updated by alanwu (Alan Wu) 2 months ago

  • Subject changed from OpenSSL fails to autoload (macOS) to RubyGems fails to detect OpenSSL in --with-static-linked-ext builds
Actions #7

Updated by alanwu (Alan Wu) 2 months ago

  • Status changed from Open to Closed

Applied in changeset git|790cf4b6d0475614afb127b416e87cfa39044d67.


Fix autoload status of statically linked extensions

Previously, for statically-linked extensions, we used
vm->loading_table to delay calling the init function until the
extensions are required. This caused the extensions to look like they
are in the middle of being loaded even before they're required.
(rb_feature_p() returned true with a loading path output.) Combined
with autoload, queries like defined?(CONST) and Module#autoload?
were confused by this and returned nil incorrectly. RubyGems uses
defined? to detect if OpenSSL is available and failed when OpenSSL was
available in builds using --with-static-linked-ext.

Use a dedicated table for the init functions instead of adding them to
the loading table. This lets us remove some logic from non-EXTSTATIC
builds.

[Bug #19115]

Updated by thomthom (Thomas Thomassen) about 2 months ago

Will this patch be pack-ported to the 3.1 branch?

Updated by alanwu (Alan Wu) about 2 months ago

thomthom (Thomas Thomassen) wrote in #note-8:

Will this patch be pack-ported to the 3.1 branch?

It's up to the branch maintainer, @nagachika (Tomoyuki Chikanaga), to decide.

Updated by thomthom (Thomas Thomassen) 23 days ago

I built Ruby 3.1.2 that we use in our product with this patch applied, and it worked on our systems. Would be nice to see it backported.

Actions #11

Updated by hsbt (Hiroshi SHIBATA) 23 days ago

  • Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN to 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: REQUIRED
Actions

Also available in: Atom PDF

Like1
Like0Like0Like0Like0Like1Like0Like0Like0Like0Like0Like0