Project

General

Profile

Actions

Bug #18249

open

The ABI version of dev builds of CRuby does not correspond to the ABI

Added by Eregon (Benoit Daloze) 4 months ago. Updated 6 days ago.

Status:
Open
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:105618]

Description

In fact, it even conflicts with the next release's ABI version:

$ ruby -ve 'p RbConfig::CONFIG["ruby_version"]'
ruby 3.1.0dev (2021-10-11T10:13:16Z master 0c3ac87345) [x86_64-linux]
"3.1.0"

This mismatch can very easily result in segfaults, memory corruption, etc when using dev versions of CRuby,
or when using a dev version and then later the release.

Possible solutions:

  • Include the git commit sha in the ABI. Pros: always correct and simple. Cons: changing more then necessary.
  • Track the ABI explicitly, and bump it whenever the ABI changes. Pros: changes only when needed. Cons: easy to forget bumping it, and if checked in CI already too late.

From https://bugs.ruby-lang.org/issues/18239#note-14:

FWIW TruffleRuby actually tracks the ABI of dev versions through this file which means it is possible to sensibly cache compiled gems even for dev versions details.
Also it stores the ABI version in .so/.bundle files and checks them when loading, so it can be sure the ABI used to compile and runtime ABI match (if not, an exception is raised).

ruby/setup-ruby has no choice for CRuby dev but to use the commit as the ABI version.
This issue is made worse by Ruby switchers like RVM & chruby setting GEM_HOME (so the ABI is effectively ignored by RubyGems in those cases, and those directories need to be cleaned manually).
When GEM_HOME is not set, it would be enough to rebuild CRuby dev and remove the directory before installing (which includes both CRuby & gems), but Ruby installers don't do that yet.
Bundler always includes the ABI version when setting the bundler path (bundle config --local path), but if the ABI version is incorrect like for CRuby dev it's of no use.

Actions #1

Updated by Eregon (Benoit Daloze) 4 months ago

  • Description updated (diff)

Updated by peterzhu2118 (Peter Zhu) 7 days ago

I've implemented ABI checking in development versions of Ruby in this PR: https://github.com/ruby/ruby/pull/5474

Copying the description:

During compilation, the script tool/abi.rb will generate a MD5 hash of
all the header files under the include directory and create
include/ruby/internal/abi.h which contains the generated MD5 hash in a
function called rb_abi_version.

When loading dynamic libraries, Ruby will compare its own
rb_abi_version and the rb_abi_version of the loaded library. If
these two values don't match it will raise a LoadError. This is only
enabled on development Ruby (release Ruby should ensure ABI
compatibility). This feature is not enabled on Windows since Windows
does not support weak symbols.

This feature will prevent cases where previously installed native gems
fail in unexpected ways due to incompatibility of changes in header
files. This will force the developer to recompile their gems to use the
same header files as the built Ruby.

In Ruby, the ABI version is exposed through
RbConfig::CONFIG["rb_abi_version"].

Actions #3

Updated by Eregon (Benoit Daloze) 7 days ago

  • Description updated (diff)

Updated by Eregon (Benoit Daloze) 7 days ago

peterzhu2118 (Peter Zhu) That's great, thank you for this work!

In Ruby, the ABI version is exposed through RbConfig::CONFIG["rb_abi_version"].

I would have a preference for RbConfig::CONFIG["abi_version"], the rb_ prefix seems redundant there.

Updated by Eregon (Benoit Daloze) 7 days ago

Actually, RbConfig::CONFIG["ruby_version"] is what is considered the ABI version by RubyGems, Bundler, some Ruby switchers probably, and more (even though it's not obvious from the name it really means that AFAIK).
So I believe that should be updated to return the new value.

We can also add RbConfig::CONFIG["abi_version"] for clarity, but I don't think it adds much.
I think a sentence in RbConfig documentation that RbConfig::CONFIG["ruby_version"] is the ABI version would be more useful.

Updated by peterzhu2118 (Peter Zhu) 7 days ago

There's assumptions that the format of ruby_version is three, period separated integers. I tried adding a tag with the ABI version (e.g. 3.2.0+12345) and Bundler fails with a Malformed version number string. There might be more places that have this kind of assumption, so I feel like changing ruby_version will be a breaking change.

Updated by byroot (Jean Boussier) 7 days ago

There might be more places that have this kind of assumption, so I feel like changing ruby_version will be a breaking change.

But only for dev builds right? People testing dev builds are more likely to adapt to that kind of small changes.

I feel like it would be easier to sligthly alter bundler to accept these than to have it append another info. (Well I guess not that much harder, so I'm fine with both).

Updated by nobu (Nobuyoshi Nakada) 7 days ago

Is this to tell if an installed gem set is compatible, at switching ruby versions?
If so, https://github.com/ruby/ruby/pull/5474 seems including a different change.
And I don't think it is enough by comparing only files under include without ruby/config.h.

Updated by naruse (Yui NARUSE) 7 days ago

  • Status changed from Open to Rejected

For dev builds, Ruby switchers has responsibility to handle the ABI compatibility breakage.
CRuby developers don't want to manage the compatibility. (We don't invest our resource here)

Updated by nobu (Nobuyoshi Nakada) 7 days ago

Regarding ruby/setup-ruby, can’t it use hashFiles('include/**/*.h') as the cache keys?

Updated by peterzhu2118 (Peter Zhu) 7 days ago

Is this to tell if an installed gem set is compatible, at switching ruby versions?

Yes

If so, https://github.com/ruby/ruby/pull/5474 seems including a different change.
And I don't think it is enough by comparing only files under include without ruby/config.h.

We can also add ruby/config.h to the hash.

For dev builds, Ruby switchers has responsibility to handle the ABI compatibility breakage.

But currently, (AFAIK) no Ruby switcher handles ABI compatibility for dev builds. This results in bad experiences for developers working on Ruby. We often test and perform benchmarks on internal apps and open-source apps like Discourse. Running into bugs, crashes, and odd behavior due to previously compiled native gems that isn't ABI compatible is a bad experience and waste of time. This kind of feature would solve this issue. Remembering to reinstall gems all the time is also a bad experience.

Regarding ruby/setup-ruby, can’t it use hashFiles('include/*/.h') as the cache keys?

We can do this kind of thing on CI, but we can't do something like this on local development machines.

Updated by Eregon (Benoit Daloze) 7 days ago

  • Status changed from Rejected to Open

naruse (Yui NARUSE) wrote in #note-9:

For dev builds, Ruby switchers has responsibility to handle the ABI compatibility breakage.
CRuby developers don't want to manage the compatibility. (We don't invest our resource here)

naruse (Yui NARUSE) There is a PR doing the work, and it requires no effort from CRuby devs, how is this a problem?
Without this fix, it is a common issue for both CRuby devs and for users which try ruby-head (e.g., to try to keep their gem compatible with it).
It's a mistake to close this issue without a good reason, so I reopen, and of course it can be discussed at the dev meeting.

Updated by Eregon (Benoit Daloze) 7 days ago

Also from experience in TruffleRuby I can say this is a solution working very well to avoid users unintentionally reusing gems with an incompatible ABI, which usually ends up in segfaults or other errors.
CRuby did not care enough about this, and lots of people wasted time due to this issue (I saw multiple reports on this subject), so let's solve it.
Ruby switchers/RubyGems/Bundler need to do their part, but they can't if the ABI version CRuby reports is wrong.

There's assumptions that the format of ruby_version is three, period separated integers. I tried adding a tag with the ABI version (e.g. 3.2.0+12345) and Bundler fails with a Malformed version number string. There might be more places that have this kind of assumption, so I feel like changing ruby_version will be a breaking change.

How about 3.2.0.12345 then?
It is a bug of any software to assume exactly 3 digits for ABI version, for instance on TruffleRuby it's 4+ digits (e.g. 3.0.2.9 on current truffleruby-dev version).

Updated by peterzhu2118 (Peter Zhu) 6 days ago

How about 3.2.0.12345 then?
It is a bug of any software to assume exactly 3 digits for ABI version, for instance on TruffleRuby it's 4+ digits (e.g. 3.0.2.9 on current truffleruby-dev version).

I was able to add build metadata to RUBY_VERSION and get build metadata support in bundler with minimal difficulty. It seems to work!

$ bundle show --paths
/Users/peter/.gem/ruby/3.2.0+4439851323553804410/gems/ast-2.4.2
/Users/peter/.gem/ruby/3.2.0+4439851323553804410/gems/benchmark-ips-2.9.2
/Users/peter/src/ruby/install/lib/ruby/gems/3.2.0/gems/bundler-2.4.0.dev
/Users/peter/src/liquid
/Users/peter/.gem/ruby/3.2.0+4439851323553804410/bundler/gems/liquid-c-d38956d76b58
/Users/peter/.gem/ruby/3.2.0+4439851323553804410/gems/memory_profiler-1.0.0
/Users/peter/.gem/ruby/3.2.0+4439851323553804410/gems/minitest-5.15.0
...
Actions

Also available in: Atom PDF