Bug #20857: Ruby 3.4 seems to have backwards compatibility issues more than its predecessors

• Status: Closed
• Assignee: matz (Yukihiro Matsumoto)
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-10-15 master 3da3cabf98) +PRISM [x86_64-linux]
• Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[vo.x (Vit Ondruch)]

Testing Ruby 3.4 in Fedora, I just hit issue like this in AsciiDoctor test suite:

  1) Failure:
Minitest::Test::TestExtensions::TestIntegration#test_should_assign_captures_correctly_for_inline_macros [test/extensions_test.rb:1382]:
--- expected
+++ actual
@@ -1,9 +1,9 @@
 "target=\"\", attributes={}
-target=\"value,key=val\", attributes={1=>\"value\", \"key\"=>\"val\", \"name\"=>\"value\"}
-target=\"\", attributes={\"text\"=>\"\"}
-target=\"[text]\", attributes={\"text\"=>\"[text]\"}
+target=\"value,key=val\", attributes={1 => \"value\", \"key\" => \"val\", \"name\" => \"value\"}
+target=\"\", attributes={\"text\" => \"\"}
+target=\"[text]\", attributes={\"text\" => \"[text]\"}
 target=\"target\", attributes={}
-target=\"target\", attributes={1=>\"value\", \"key\"=>\"val\", \"name\"=>\"value\"}
-target=\"target\", attributes={\"text\"=>\"\"}
-target=\"target\", attributes={\"text\"=>\"[text]\"}
+target=\"target\", attributes={1 => \"value\", \"key\" => \"val\", \"name\" => \"value\"}
+target=\"target\", attributes={\"text\" => \"\"}
+target=\"target\", attributes={\"text\" => \"[text]\"}
 target=\"target\", attributes={}"

This suggest that Hash#inspect formatting was changed and there are additional spaces around hash rocket. Is the space really worth of the troubles? BTW here is AsciiDoctor upstream reaction and I share the sentiment.

[vo.x (Vit Ondruch)]

Just FTR, speaking of testing in Fedora, after initial rebuild with Ruby 3.4, there have failed to build 127 out of 528 packages.

[vo.x (Vit Ondruch)]

• Subject changed from Don't change `Hash#inspect formatting` to Don't change `Hash#inspect` formatting

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned
• Assignee set to matz (Yukihiro Matsumoto)

[mame (Yusuke Endoh)]

I ran the asciidoctor test and he have already fixed it. Awesome.

To my knowledge, this change has never required any substantive modification other than updating expectation values of test cases. And in principle, you know, it is not recommended to use #inspect results for test expectations. Of course, this does not mean that we can break compatibility just on this principle, though.

there have failed to build 127 out of 528 packages.

How many of the failures are due to this change?

[mame (Yusuke Endoh)]

• Related to Bug #20433: Hash.inspect for some hash returns syntax invalid representation added

[byroot (Jean Boussier)]

there have failed to build 127 out of 528 packages.

Could you share the package names and failures? I think it's an important data point.

Personally as stated on https://bugs.ruby-lang.org/issues/20433 I really love this change, and after fixing various applications and libraries, there wasn't any failed test I thought was really legitimate in using Hash#inspect to make assertions.

Of course this is going to create quite a bit of churn, so I'd understand if it was reverted or more limited on that basis. But I don't think the "compatibility" argument is sound, because it doesn't make sense to assume that an API that is meant for debugging (#inspect) would guarantee a stable output forever. Otherwise that means we can't ever improve user experience related to it.

[vo.x (Vit Ondruch)]

byroot (Jean Boussier) wrote in #note-6:

there have failed to build 127 out of 528 packages.

Could you share the package names and failures? I think it's an important data point.

This is the repository I am using:

https://copr.fedorainfracloud.org/coprs/vondruch/mpb/builds/

From there, click on the "Build ID" and then builder-live.log.gz at the bottom page. This would be log for the AsciiDoctor. I hope this is enough, but let me know if something is not as clear as I might think 😇

Please note that this is very much WIP and the repository might significantly change. E.g. working on rubygem-fog-core fix, which might unblock other Fog related packages as well as Vagrant.

Personally as stated on https://bugs.ruby-lang.org/issues/20433 I really love this change, and after fixing various applications and libraries, there wasn't any failed test I thought was really legitimate in using Hash#inspect to make assertions.

Of course this is going to create quite a bit of churn, so I'd understand if it was reverted or more limited on that basis.

Fedora packages don't necessarily follow upstream development at the same speed. Mostly due to maintainers capacity, other times due to compatibility matrix or even due to "it works, don't touch it". So while there are possibly upstream fixes, it is by nature not as straight forward for Fedora.

But I don't think the "compatibility" argument is sound, because it doesn't make sense to assume that an API that is meant for debugging (#inspect) would guarantee a stable output forever. Otherwise that means we can't ever improve user experience related to it.

[byroot (Jean Boussier)]

This is the repository I am using: https://copr.fedorainfracloud.org/coprs/vondruch/mpb/builds/

Thank you. I went over a dozen or so, and couldn't find any other than ASCIIDoctor for which the failure cause was the change in Hash#inspect.

e.g. the puma one was because of the change in Backtrace::Location#to_s:

Expected /:in `dummy_error'/ to match "

The rack one is because of the base64 extraction, some more are because of compilation issues, etc.

So I highly doubt this is causing even the majority of the 127 failures.

Fedora packages don't necessarily follow upstream development at the same speed. Mostly due to maintainers capacity, other times due to compatibility matrix or even due to "it works, don't touch it". So while there are possibly upstream fixes, it is by nature not as straight forward for Fedora.

I know, that's why I'm always puzzled at why distributions are trying to re-package rubygems... Like most of the failure I'm seeing on your CI are things already fixed upstream. But I suppose that's another topic.

[vo.x (Vit Ondruch)]

byroot (Jean Boussier) wrote in #note-8:

This is the repository I am using: https://copr.fedorainfracloud.org/coprs/vondruch/mpb/builds/

Thank you. I went over a dozen or so, and couldn't find any other than ASCIIDoctor for which the failure cause was the change in Hash#inspect.

e.g. the puma one was because of the change in Backtrace::Location#to_s:

Expected /:in `dummy_error'/ to match "

The rack one is because of the base64 extraction, some more are because of compilation issues, etc.

So I highly doubt this is causing even the majority of the 127 failures.

This remark is more in general "compatibility" context the specifically to Hash#inspect. Sorry I have made this remark here, but there is unfortunately not better place to share such remarks. We will certainly try to reduce the other issues so this will yet to show.

Fedora packages don't necessarily follow upstream development at the same speed. Mostly due to maintainers capacity, other times due to compatibility matrix or even due to "it works, don't touch it". So while there are possibly upstream fixes, it is by nature not as straight forward for Fedora.

I know, that's why I'm always puzzled at why distributions are trying to re-package rubygems... Like most of the failure I'm seeing on your CI are things already fixed upstream. But I suppose that's another topic.

I wish we had a place to discuss "why distributions are trying to re-package rubygems."

[austin (Austin Ziegler)]

vo.x (Vit Ondruch) wrote in #note-9:

byroot (Jean Boussier) wrote in #note-8:

vo.x (Vit Ondruch) wrote in #node-7:

Fedora packages don't necessarily follow upstream development at the same speed. Mostly due to maintainers capacity, other times due to compatibility matrix or even due to "it works, don't touch it". So while there are possibly upstream fixes, it is by nature not as straight forward for Fedora.

I know, that's why I'm always puzzled at why distributions are trying to re-package rubygems... Like most of the failure I'm seeing on your CI are things already fixed upstream. But I suppose that's another topic.

I wish we had a place to discuss "why distributions are trying to re-package rubygems."

I think that we should make such a place, because in the past it has caused no small amount of difficulty for upstream maintainers (both in how distributions re-packaged Ruby and in how distributions re-packaged several gems). I’m maintaining several (non-Ruby) packages on MacPorts now, and while we frequently have patches that are applied to sources (sometimes to fix things that are MacPorts build specific, sometimes to make things like command upgrade print a message instead of trying to replace a protected binary), I am looking for ways that we can improve our handling of RubyGems in general to make it work more like Ruby expects (what we have now is…not great).

[jeremyevans0 (Jeremy Evans)]

byroot (Jean Boussier) wrote in #note-8:

I know, that's why I'm always puzzled at why distributions are trying to re-package rubygems... Like most of the failure I'm seeing on your CI are things already fixed upstream. But I suppose that's another topic.

I assume that the reason that most distributions repackage gems is because they have other software they package that depends on those gems, and therefore the gems need to be packaged just as any other dependencies in the packaging system.

For OpenBSD, we also need to re-package gems for some additional reasons:

1. We repackage native gems because OpenBSD chooses a stable API instead of a stable ABI, so upgrading OpenBSD versions can break existing native gems. You could use gem pristine to handle this, but it's easier if it's handled automatically when you run the command to upgrade all packages.

2. Some native gems are not portable (or consider Windows+Mac+Linux as portable), and must be patched to compile or work correctly on OpenBSD (e.g. argon2, rmagick).

3. In general, OpenBSD prefers native gems to link to system libraries, so they automatically pick up security updates in those libraries, and the default for many gems is to build embedded copies of libraries (e.g. sqlite3, nokogiri).

[mame (Yusuke Endoh)]

I took a quick look at all the failures of packages starting with "ruby" in https://copr.fedorainfracloud.org/coprs/vondruch/mpb/packages/, and categorized them. Please forgive me if this is not accurate.

• 38 of them seemed to be dependency issues that did not lead to Ruby test execution. I don't think this is Ruby's fault.
• There were 38 failures like cannot load such file -- base64. I think is related to gemification.
• There were 13 failures that appeared to be due to Hash#inspect incompatibilities. Most of them seemed to be using inspect results for test expectations. rubygem-rspec-expectations seemed to be a bit of a pain, since it is really a gem about test expectations.
• There were 10 that appeared to be affected by the change from backquotes to single quotes in error messages.
• There were 8 that appeared to be due to cucumber incompatibilities and 2 that were due to a change in the default parser for URIs.
• The remaining 18 were not clear. It is possible that one of the above incompatibilities caused the code to behave differently than intended, resulting in an incomprehensible error.

I don't say that the impact of Hash#inspect incompatibility is negligible, but I don't think it is significant as a percentage. Also, I can confirm that most of the failures due to Hash#inspect could be fixed by modifying the expected values of the test.

Should this ticket be kept alive? I think the re-packaging issue is worth discussing, but I think it should be discussed in another ticket.

# 38: maybe dependency issue (Failed to resolve the transaction)
ruby-gnome2
rubygem-actioncable
rubygem-actionpack
rubygem-activerecord
rubygem-activestorage
rubygem-bootsnap
rubygem-bundler_ext
rubygem-cairo
rubygem-cairo-gobject
rubygem-capybara
rubygem-clutter
rubygem-clutter-gstreamer
rubygem-clutter-gtk
rubygem-excon
rubygem-gdk3
rubygem-gdk4
rubygem-gdk_pixbuf2
rubygem-glu
rubygem-goocanvas
rubygem-goocanvas1
rubygem-gtk2
rubygem-gtk3
rubygem-gtksourceview2
rubygem-gtksourceview3
rubygem-importmap-rails
rubygem-jekyll (install issue?)
rubygem-opengl
rubygem-pango
rubygem-poppler
rubygem-rabbit
rubygem-railties
rubygem-rspec-rails
rubygem-rsvg2
rubygem-sass-rails
rubygem-shoulda-matchers
rubygem-vte
rubygem-vte3
rubygem-webkit2-gtk
rubygem-webmock

# 38: gemification
rubygem-abrt: cannot load such file -- syslog
rubygem-activesupport: cannot load such file -- mutex_m
rubygem-asciidoctor: cannot load such file -- base64
rubygem-bson: cannot load such file -- base64
rubygem-coveralls: cannot load such file -- base64
rubygem-em-http-request: cannot load such file -- base64
rubygem-em-websocket: cannot load such file -- base64
rubygem-fakefs: cannot load such file -- csv
rubygem-fog-core: cannot load such file -- base64
rubygem-guard-livereload: cannot load such file -- base64
rubygem-httparty: cannot load such file -- csv
rubygem-httpclient: cannot load such file -- mutex_m
rubygem-i18n: cannot load such file -- mutex_m
rubygem-jekyll-asciidoc: cannot load such file -- csv
rubygem-jekyll-feed: cannot load such file -- csv
rubygem-jekyll-git-authors: cannot load such file -- csv
rubygem-jekyll-sass-converter: cannot load such file -- csv
rubygem-jekyll-toc: cannot load such file -- csv
rubygem-net-scp: cannot load such file -- base64
rubygem-net-sftp: cannot load such file -- base64
rubygem-net-ssh: cannot load such file -- base64
rubygem-rack: cannot load such file -- base64
rubygem-rack-protection: cannot load such file -- base64
rubygem-rack-test: cannot load such file -- base64
rubygem-rbvmomi: cannot load such file -- base64
rubygem-rmagick: cannot load such file -- observer
rubygem-rspec-core: cannot load such file -- drb/drb
rubygem-rttool: cannot load such file -- nkf
rubygem-selenium-webdriver: cannot load such file -- base64
rubygem-sinatra: cannot load such file -- base64
rubygem-sprockets: cannot load such file -- base64
rubygem-test-unit: cannot load such file -- csv
rubygem-text: cannot load such file -- csv
rubygem-typhoeus: cannot load such file -- base64
rubygem-webrobots: cannot load such file -- base64
rubygem-websocket: cannot load such file -- base64
rubygem-websocket-driver: cannot load such file -- base64
rubygems: cannot load such file -- base64

# 13: Hash#inspect
rubygem-activejob
rubygem-activemodel
rubygem-contracts
rubygem-asciidoctor
rubygem-formatador
rubygem-haml
rubygem-mocha
rubygem-mustermann (I am not sure)
rubygem-pg
rubygem-rdoc
rubygem-rspec-expectations (A bit difficult)
rubygem-rspec-mocks
rubygem-rspec-support

# 10: backquote->single quote in error message
rubygem-docile
rubygem-factory_bot
rubygem-flexmock
rubygem-minitest4
rubygem-msgpack
rubygem-pry
rubygem-puma
rubygem-require_all
rubygem-ruby-dbus
rubygem-thor

# 8: cucumber?
rubygem-aruba: unknown keywords: :strict, :proc (ArgumentError)
rubygem-cucumber: unknown keywords: :strict, :proc
rubygem-cucumber-wire: unknown keywords: :strict, :proc
rubygem-json_spec: unknown keywords: :strict, :proc (ArgumentError)
rubygem-minitest-around: unknown keywords: :strict, :proc (ArgumentError)
rubygem-nifti: unknown keywords: :strict, :proc (ArgumentError)
rubygem-rake-compiler: unknown keywords: :strict, :proc (ArgumentError)
rubygem-rspec-its: unknown keywords: :strict, :proc (ArgumentError)

# 2: uri
rubygem-cookiejar: uninitialized constant URI::REGEXP
rubygem-websocket: uninitialized constant URI::REGEXP

# 18: unknown
rubygem-actionview: undefined method 'series_url' for #<PolymorphicRoutesTest:0x00007f9e885a94d8>".
rubygem-asciidoctor-pdf: ??? 
rubygem-async_sinatra: uninitialized constant MiniTest (NameError)
rubygem-curb: ???
rubygem-ethon: uninitialized constant LocalhostServer::TimeoutError
rubygem-faraday: cannot load such file -- webmock/rspec
rubygem-globalid: ArgumentError expected but nothing was raised
rubygem-guard: Guard::Guardfile::Evaluator::NoPluginsError error!
rubygem-image_size: ??? Unexpected response
rubygem-memfs: ???
rubygem-puppet-lint: ??? rspec/json_expectations is not available
rubygem-redis-client: cannot load such file -- minitest/autorun (LoadError)
rubygem-ronn-ng: ??? NameError: uninitialized constant RonnTest::FileUtils
rubygem-sass: ???
rubygem-serialport: is deprecated: rb_io_descriptor
rubygem-webrick: uninitialized constant Remover (NameError)
rubygem-xmlrpc: ??? RuntimeError: HTTP-Error: 400 Bad Request
rubygem-method_source: Prism incompatibility https://github.com/banister/method_source/pull/84

[shyouhei (Shyouhei Urabe)]

• Subject changed from Don't change `Hash#inspect` formatting to Ruby 3.4 seems to have backwards compatibility issues more than its predecessors

(Let me change the subject of this thread. It is not only the matter of Hash#inspect anymore.)

[ioquatix (Samuel Williams)]

FWIW, I was fine with this change, but it broke some of my tests.

The solution was to not use string equality, but instead do this:

# Before
expect(thing).to be == '{"x"=>10}'

# After
expect(thing).to be == {"x"=>10}.to_s

It was an easy fix, and I like the new formatting. I think the tests are better too.

[vo.x (Vit Ondruch)]

mame (Yusuke Endoh) wrote in #note-12:

I took a quick look at all the failures of packages starting with "ruby"

Just FTR, all packages in the repository has dependency on Ruby, even those without ruby prefix. Some of them might use e.g. C-API for language bindings, others SWIG and some are using Ruby just for e.g. their test suites.

in https://copr.fedorainfracloud.org/coprs/vondruch/mpb/packages/, and categorized them. Please forgive me if this is not accurate.

• 38 of them seemed to be dependency issues that did not lead to Ruby test execution. I don't think this is Ruby's fault.

Some of these (especially the Gtk / Gnome related) are know for their circular dependencies. So they might need some special treatment.

• There were 38 failures like cannot load such file -- base64. I think is related to gemification.

Yes. I hope that majority will be fixed by updates of those packages, or adding appropriate dependencies.

• There were 13 failures that appeared to be due to Hash#inspect incompatibilities. Most of them seemed to be using inspect results for test expectations. rubygem-rspec-expectations seemed to be a bit of a pain, since it is really a gem about test expectations.
• There were 10 that appeared to be affected by the change from backquotes to single quotes in error messages.
• There were 8 that appeared to be due to cucumber incompatibilities and 2 that were due to a change in the default parser for URIs.
• The remaining 18 were not clear. It is possible that one of the above incompatibilities caused the code to behave differently than intended, resulting in an incomprehensible error.

Great analysis. Thanks for looking into those 🙏

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #20859: Make Base64 to core class added

[vo.x (Vit Ondruch)]

mame (Yusuke Endoh) wrote in #note-12:

8: cucumber?

rubygem-aruba: unknown keywords: :strict, :proc (ArgumentError)
rubygem-cucumber: unknown keywords: :strict, :proc

This seems to be fixed upstream:

https://github.com/cucumber/cucumber-ruby/pull/1757

But something is IMHO wrong if the change needs to explain the formatting.

rubygem-cucumber-wire: unknown keywords: :strict, :proc
rubygem-json_spec: unknown keywords: :strict, :proc (ArgumentError)
rubygem-minitest-around: unknown keywords: :strict, :proc (ArgumentError)
rubygem-nifti: unknown keywords: :strict, :proc (ArgumentError)
rubygem-rake-compiler: unknown keywords: :strict, :proc (ArgumentError)
rubygem-rspec-its: unknown keywords: :strict, :proc (ArgumentError)

[vo.x (Vit Ondruch)]

mame (Yusuke Endoh) wrote in #note-12:

8: cucumber?

rubygem-rspec-its: unknown keywords: :strict, :proc (ArgumentError)

Once I have fixed rubygem-cucumber, rpsec-its moved into backtick category:

https://github.com/rspec/rspec-its/blob/6dec295173dfd245fd844f76e3908fcc3f84df10/features/its.feature#L191

[mame (Yusuke Endoh)]

• Status changed from Assigned to Closed

At the dev meeting, I explained the findings of the incompatibility to matz. He stated that it was within the expected range of impacts and that no revert would be made.