Feature #19351: Promote bundled gems at Ruby 3.3

• Status: Assigned
• Priority: Normal
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -

[hsbt (Hiroshi SHIBATA)]

In Ruby 3.2, the default gems and bundled gems are changed only adding syntax_suggest. I and some committers are considering promote default gems to bundled gems again for Ruby 3.3+.

We hope to keep the current developer experience with dependency resolution and ignore the additional work like "Put gem "xxx" into your Gemfile" for developers.

Proposal

We propose the following libraries will promote default gems to bundled gems at Ruby 3.3. They are not the dependencies of Rails and RubyGems/Bundler.

abbrev
getoptlong
observable
resolv
resolv-replace
rinda
fcntl (C-ext)
nkf (C-ext)
syslog (C-ext)
win32ole (C-ext)

Update: I removed optparse and un from above list. Because they are used by Ruby build process.

optparse
un

Additional works

I also propose to promote rails dependencies without rubygems/bundler deps:

base64
benchmark
delegate
drb
forwardable
ipaddr
irb
mutex_m
ostruct
rdoc
singleton
tsort
weakref
bigdecimal (C-ext)
date(datetime) (C-ext)
racc (C-ext)

and gems maintained by @kou (Kouhei Sutou)

csv

Following gems also maintained by @kou (Kouhei Sutou), but they are used on RubyGems/Bundler or MJIT. Maybe, We couldn't promote them because RubyGems/Bundler couldn't bundle C-ext gems.

fiddle (C-ext): used by MJIT
stringio (C-ext) used by RubyGems/Bundler
strscan (C-ext) used by RubyGems/Bundler

But if we promote them to bundled gems, many of users need to add like gem "csv" into their Gemfile. I'm considering to avoid this situation.

Can we introduce the specific feature of bundled gems to RubyGems or Bundler? Example, bundler have allowed list for bundled gems. So, listed gems could be require without Gemfile under the bundle exec.

[vo.x (Vit Ondruch)]

But if we promote them to bundled gems, many of users need to add gem "csv" into their Gemfile. I'm considering to avoid this situation.

Please let gems to be gems. Please let everybody fix their Gemfiles. Don't make exceptions. There is nothing wrong with specifying dependencies.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-1:

But if we promote them to bundled gems, many of users need to add gem "csv" into their Gemfile. I'm considering to avoid this situation.

Please let gems to be gems. Please let everybody fix their Gemfiles. Don't make exceptions. There is nothing wrong with specifying dependencies.

BTW I don't think there would be that much fixing needed after all. E.g. if Rails console depends on IRB and Rails specified the dependency on IRB, Rails users would not need to do anything out of ordinary. But of course, the could be some exceptional handling for IRB. The only differences would be, that they would get faster IRB updates, which they won't notice ATM.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

I update rails dependencies.

@vo.x (Vit Ondruch) Thanks for your comment. Maybe we should remove irb from this list.

I welcome feedback for these list and additional gem proposal like cgi, erb and others.

[byroot (Jean Boussier)]

Yeah, a few of these will need to be handled by rubygems/bundler.

I believe that for instance strscan is used to parse the Gemfile.lock. Historically Bundler simply vendored the gems it needs, but not sure how it could do that since it has a C-ext part.

I assume you notified bundler people? What do they think?

[Eregon (Benoit Daloze)]

vo.x (Vit Ondruch) wrote in #note-3:

BTW I don't think there would be that much fixing needed after all.

As we have seen with the mail gem, it can be a huge pain to move from default gem to bundled gem (in that case for net-* gems).

What is the advantage to move from default gem to bundled gem?
In both cases it's shipped with Ruby and they can be updated.
So it seems the only difference is the disadvantage to break every script using one of these libraries and Bundler by forcing to add them in the Gemfile (which typically doesn't work for older Ruby versions so adds many complications).

Please also consider the concerns in #18567.
Quoting them here:

There are however multiple unwanted side effects of this:

1. Removing gems from stdlib (e.g., #17873) is a breaking change, which makes upgrading to a new Ruby version more difficult.
   I think this should only be done if there is a clear gain.
   Being a default gem is already enough to fix a security issue without a CRuby release.
2. When any gem depends on a default gem, it tends to break on all Ruby implementations except CRuby, and for older Ruby versions.

1. is the same as I just said, but makes it clear this change hurts adopting new Ruby versions.

2. is a very important requirement if we do this, to ensure it works on JRuby/TruffleRuby before publishing the gem.
   Replacing the implementation of a default gem (on JRuby/TruffleRuby) is only an issue if people depend on it explicitly. But for bundled gems it doesn't work at all.
   So we need to ensure those gems work on JRuby/TruffleRuby, or we need time so JRuby/TruffleRuby devs can fix that before the gem is published/people add a dependency on it.

[Eregon (Benoit Daloze)]

• Related to Bug #18567: Depending on default gems in stdlib gems when not needed considered harmful added

[kou (Kouhei Sutou)]

I'm OK with gems maintained by me.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned
• Assignee set to hsbt (Hiroshi SHIBATA)

I believe that for instance strscan is used to parse the Gemfile.lock. Historically Bundler simply vendored the gems it needs, but not sure how it could do that since it has a C-ext part.

Thanks. I also confirm that RubyGems/Bundler uses stringio that is C-ext. I may remove them from this proposal.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #16963: Remove English.rb from Ruby 2.8/3.0 added

[hsbt (Hiroshi SHIBATA)]

• Tracker changed from Bug to Feature
• Backport deleted (2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #18169: Local copies of gemified libraries are being released out of sync with their gems added

[hsbt (Hiroshi SHIBATA)]

What is the advantage to move from default gem to bundled gem?

It makes to leave out-of-sync status with published gem like #18169. I'm checking ALL diffs and commits of ruby/* repos every day for resolving this issue after #18169 requests. Example for https://github.com/ruby/ruby/pull/7025.

And we can add new maintainer easily different with ruby/ruby repo. In fact, net-smtp, net-imap, rexml and etc are maintained and released by maintainer's convenience.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[Eregon (Benoit Daloze)]

hsbt (Hiroshi SHIBATA) wrote in #note-16:

It makes to leave out-of-sync status with published gem like #18169. I'm checking ALL diffs and commits of ruby/* repos every day for resolving this issue after #18169 requests. Example for https://github.com/ruby/ruby/pull/7025.

My understanding is this is fully automated now, i.e., a commit to the ruby/some_default_gem repo is immediately applied to ruby/ruby and vice-versa.
So it seems imopossible to get out of sync, isn't it?

And we can add new maintainer easily different with ruby/ruby repo. In fact, net-smtp, net-imap, rexml and etc are maintained and released by maintainer's convenience.

The only constraint there is to have a new version number at ruby/ruby release time for those gems, right?
That can be done by just bumping the patch version or so of such gems in ruby/ruby, isn't it?

I think the pain of everyone using Ruby to change these gems from default to bundled gems is not worth those advantages.
At least things which don't use Bundler should not be impacted, e.g. ruby -rbenchmark -e 'p Benchmark.realtime { ... }'.

Also it seems the main motivation is #18169.
I think #18567 is far more problematic in practice for other Ruby implementations.
So like I said in https://bugs.ruby-lang.org/issues/18567#note-30, if we move anything to default/bundled gems, please make sure it works on JRuby/TruffleRuby by adding them in CI and defaulting to stdlib unless it works as-is (details described there).

[hsbt (Hiroshi SHIBATA)]

So it seems imopossible to get out of sync, isn't it?

No. some of commits conflicts usually. and some of people push ruby/ruby directly for fixing CI failure, not ruby/* repo. So, we are always out-of-sync status without my work.

I think the pain of everyone using Ruby to change these gems from default to bundled gems is not worth those advantages.

Because you didn't work release work of CRuby. Release team easily released new version of CGI without Ruby releases. Release team always spent over 6 hours with 5-6 people for security release of Ruby. We really hope to leave this hard work.

if we move anything to default/bundled gems, please make sure it works on JRuby/TruffleRuby by adding them in CI

I already work to merge JRuby code with JRuby team. I welcome your patch for TruffleRuby.

[Eregon (Benoit Daloze)]

hsbt (Hiroshi SHIBATA) wrote in #note-19:

No. some of commits conflicts usually. and some of people push ruby/ruby directly for fixing CI failure, not ruby/* repo. So, we are always out-of-sync status without my work.

I remember some commit I made, maybe in ruby/some_gem being immediately committed to ruby/ruby too.
So maybe that automatic cherry-picking is only in one direction, not the other, and maybe only for some gems and not all?

Because you didn't work release work of CRuby. Release team easily released new version of CGI without Ruby releases. Release team always spent over 6 hours with 5-6 people for security release of Ruby. We really hope to leave this hard work.

Correct me if I'm wrong, but to fix a security issue in a bundled or default gem it's the same, there is no need to release CRuby, only to release the gem, isn't it?

Maybe a CRuby release is considered for some default gems/big security issue because it's harder for users to find out about the security issue (e.g., dependabot won't tell them)?
But then they would also need to find about the new CRuby release, given it's the same channel (https://www.ruby-lang.org/ announcements) it doesn't seem to help much, except for ruby installed by package manager maybe (rare for Ruby applications).

[hsbt (Hiroshi SHIBATA)]

So maybe that automatic cherry-picking is only in one direction, not the other, and maybe only for some gems and not all?

I prepared auto-sync for all of default gems. But it's sometimes failed and ruby/ruby to ruby/* is not available for it. I always pick them and manually push all of repos.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

We discussed about this at #19357

@mame (Yusuke Endoh) suggested It's hard to research affected gems that are promoted bundled gems from default gems. Ex. resolve-replace, nkf are used by prod application in the real world. I'll work to introduce PoC for bypassing clean_env of bundler before triaging default gems.