Feature #20309: Bundled gems for Ruby 3.5

• Status: Assigned
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -

[hsbt (Hiroshi SHIBATA)]

I propose migrate the following default gems to bundled gems at Ruby 3.5. So, It means users will get warnings if users try to load them.

(Update with 2024/03/14)

• ostruct
  • I make ostruct as optional on json at https://github.com/flori/json/pull/565
• logger
  • activesupport needs to add logger to its dependency same as bigdecimal, drb or etc.
• fiddle
• pstore
• win32ole

I have a plan to migrate the following default gems too. But I need to more feedback from other committers about them.

• irb
  • We need to consider how works binding.irb after Ruby 3.5.
  • I consider to use irb without Gemfile.
• reline
• readline (wrapper file for readline-ext and reline)
• io-console
  • rubygems uses that. Should we make optional that?
• open-uri
• yaml (wrapper file for psych)
  • syck is retired today. I'm not sure what people uses psych directly, not yaml.
• rdoc
  • We need to change build task like download rdoc gem before document generation.
    • extract make doc from make all and invoke make doc before make install.
  • or We make document generation is optional from Ruby 3.5
    • We explicitly separate make install and make install-doc
• un
  • ruby -run is one of cool feature of Ruby. Should we avoid uninstalling un gem?
  • mkmf uses ruby -run for that. I need to investigate that.
• singleton
  • This is famous design pattern. Should we enforce users add them to their Gemfile?
• forwadable
  • reline needs to add forwardable their runtime_dependency after migration.
• weakref
  • I'm not sure how impact after migrating bundled gems.
• fcntl
  • Should we integrate these constants into ruby core?

I would like to migrate ipaddr and uri too. But these are used by webrick that is mock server for our test suite. We need to rewrite webrick with TCPSocker or extract ipaddr and uri dependency from webrick

Other default gems depend on our build process or other libraries deeply. I will update this proposal if I could extract them from default gems.

[rubyFeedback (robert heiler)]

ruby -run is one of cool feature of Ruby. Should we avoid uninstalling un gem?

I think -run is kind of neat; it's a bit like a mini-DSL for the commandline.

(Having said that, I actually do not use it myself; I instead use a custom executable that calls methods in other .rb files, so I kind of have the same or similar functionality available without needing to use special commandline flags; and lots of aliases.)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[retro (Josef Šimánek)]

I'm big fan (and user) of ruby -run -e httpd . -p 1234, this is also part of "local HTTP server oneliners" like https://gist.github.com/willurd/5720255 and it will break various tutorials online if un is not installed with ruby by default.

[Eregon (Benoit Daloze)]

hsbt (Hiroshi SHIBATA) wrote:

• ostruct
  • I make ostruct as optional on json at https://github.com/flori/json/pull/565

+1 since ostruct is already kind of deprecated.

• irb
  • We need to consider how works binding.irb after Ruby 3.5.

Will that break binding.irb under bundle exec? If so that seems a major problem.
I would think it would break it if binding.irb is implemented as e.g. gem "irb"; require "irb"; binding.irb.

• yaml (wrapper file for psych)
  • syck is retired today. I'm not sure what people uses psych directly, not yaml.

psych seems to be a default gem (and I guess will remain as so as a dependency of rubygems).
If so, what is the advantage to make yaml a bundled gem?
It's a trivial wrapper so I don't see advantage to make bundled instead of default gem, only overhead (adding gem "yaml" in Gemfiles which has no value as this code almost never changes) and troubles.

BTW I think most programs use require "yaml" instead of require "psych" and I think that's better as it is, psych is kind of an implementation detail, people want to load/dump YAML.
It seems busy unproductive work to replace require "yaml" by require "psych" or to add gem "yaml" in Gemfiles, so I am very much against making yaml a bundled gem.

• readline (wrapper file for readline-ext and reline)

Is the intention to have people do require "reline" instead of require "readline"?
It seems weird if people add gem "readline" to solve this, when gem "reline" would make much more sense.
It's a trivial wrapper, so I think making it a bundled gem creates more problems/overhead than it solves.

• un
  • ruby -run is one of cool feature of Ruby. Should we avoid uninstalling un gem?

ruby -run would still work if un is a bundled gem, so I think that's OK.
Maybe RubyGems should ask confirmation/extra warning when uninstalling a bundled gem.

• singleton
  • This is famous design pattern. Should we enforce users add them to their Gemfile?

IMO not worth it, it's pretty trivial code and too much overhead to ask users to add to Gemfile for something so basic.

• weakref
  • I'm not sure how impact after migrating bundled gems.

weakref is very tightly bound to implementation details, such as ::ObjectSpace::WeakMap.new on CRuby.
The implementation is completely different on JRuby and TruffleRuby.
It will cause problems to make this a bundled gem for alternative Ruby implementations.
So I think it is not worth to make this a bundled gem.
Also, that file is tiny and trivial, and had very few changes.

• fcntl
  • Should we integrate these constants into ruby core?

It seems low-level stuff so I think being behind a require is in general good.
But between core and bundled gem I would prefer core.

---

BTW the reasoning to change from default gem to bundled gem is at https://bugs.ruby-lang.org/issues/19351#note-16

[Eregon (Benoit Daloze)]

Eregon (Benoit Daloze) wrote in #note-6:

• fcntl
  • Should we integrate these constants into ruby core?

It seems low-level stuff so I think being behind a require is in general good.
But between core and bundled gem I would prefer core.

I misremembered, I thought there was something like Fcntl.fcntl, but it's IO#fcntl.
So fcntl is literally just a bunch of constants:
https://github.com/oracle/truffleruby/blob/master/lib/truffle/fcntl.rb
https://github.com/ruby/ruby/blob/master/ext/fcntl/fcntl.c

So I think this should be core then.

[Eregon (Benoit Daloze)]

• Related to Feature #20187: Bundled gems at Ruby 3.4 added

[Eregon (Benoit Daloze)]

• Related to Feature #19351: Promote bundled gems at Ruby 3.3 added

[jeremyevans0 (Jeremy Evans)]

Eregon (Benoit Daloze) wrote in #note-6:

hsbt (Hiroshi SHIBATA) wrote:

• singleton
  • This is famous design pattern. Should we enforce users add them to their Gemfile?

IMO not worth it, it's pretty trivial code and too much overhead to ask users to add to Gemfile for something so basic.

On the other hand, it's almost always better to use a normal constant with a singleton object than the singleton library. singleton's only advantage is you can delay initialization, and if you want that, you can use autoload for the file that defines the constant.

Instead of:

class Klass
  include Singleton

  # define methods
end

Klass.instance

Use:

CONSTANT = Object.new
CONSTANT.singleton_class.class_eval do
  # define methods
end

I think we should encourage users to use the singleton object support built into Ruby core, as opposed to a separate library that accomplishes mostly the same thing. I think the main advantage of the singleton library over standard Ruby is that it supports marshalling, but I'm not sure how common is the need to marshal singleton objects.

I'm in favor of moving the singleton library to bundled gems mainly to discourage users from using it.

[Eregon (Benoit Daloze)]

@jeremyevans0 (Jeremy Evans) One issue with that replacement is it does not name the class of the object, so it's quite unclear if there is an exception with it.
And also no easy way to define constants in that singleton class (and if one does A = ... there it will accidentally declare Object::A).

I think there is nothing wrong with the singleton stdlib, so I think we should not discourage using it.
There are cases where it's best to initialize a constant eagerly and that's fine, those usages do not need singleton.

[Eregon (Benoit Daloze)]

Also something I remembered now is YJIT does not compile singleton methods, except singleton methods of modules and classes.
So then that replacement can also be significantly slower.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

Thanks @Eregon (Benoit Daloze) and @jeremyevans0 (Jeremy Evans) .

I mostly agreed your comments. And I discussed this at DevMeeting 2024/03/14.

• No one against about ostruct. I will do that.
• We should consider to run irb without gem "irb" of Gemfile under the all of Bundler environment.
  • I will consider it with irb, reline and io-console.
• I try to run make doc before make install and use rdoc as bundled gems.
  • If I can do that, I will mark rdoc as bundled gems at Ruby 3.5.
• I'll extract another issue for singleton, un, fcntl.

I will update this issue until finalised target libraries.

[Eregon (Benoit Daloze)]

@hsbt (Hiroshi SHIBATA) What about yaml and readline?
I think these are not worth moving to bundled gems, i.e. the gains are (AFAIK) very small while the costs and confusion are high.
I think so because the code for these 2 trivial "loaders/wrappers" (see links above) almost never changes (so the overhead of keeping them in sync is basically 0, probably 0 security issues in that code, etc).
There seems to be no value to use track readline 0.0.4 vs readline 0.0.5 in a Gemfile.lock for instance, as the only likely changes would be to adapt to some incompatible change in Ruby, and then that's useless since the right version would be shipped with Ruby.

The main problem there/my main concern is the cost, many people would need to add gem "readline" or gem "yaml" to their Gemfile for Ruby 3.5, and gain basically nothing from it, so it would be almost pure overhead.
Also depending on gem "yaml" alone is not a good idea, because it doesn't mean any specific major version of psych. So one needs gem "psych" anyway, and the gem "yaml" is redundant.

To try to quantify that I looked at

• https://rubygems.org/gems/readline/reverse_dependencies: only 5 gems depend on readline right now. Yet there are 371+1158 matches for gem-codesearch 'require "readline"'+gem-codesearch "require 'readline'". All these gems would need to add gem "readline", for no gain.
• https://rubygems.org/gems/psych/reverse_dependencies vs https://rubygems.org/gems/yaml/reverse_dependencies. There are about 18x (54M/3M) as much downloads for psych than yaml. RubyGems.org doesn't show a total count for reverse dependencies but it seems clear few gems depend on yaml, much much more depend on psych. All the gems depending on psych and not yaml would need changes.

One more thing, as far as I can see, psych is a default gem. Having yaml as a bundled gem while psych is a default gem would lead to an awkward situation that when run under bundler and not adding either to the Gemfile, one can require "psych" but cannot require "yaml". So they could use Psych in code but not YAML which feels a wrong limitation (people might just do YAML=Psych themselves as a workaround, but then that could cause warnings).
So it seems clear yaml shouldn't become bundled gem before psych at least. And I think psych cannot become a bundled gem because RubyGems needs it.

[hsbt (Hiroshi SHIBATA)]

What about yaml and readline?

There is no conclusion yet. I understood your concern. Do not rush this.

[Eregon (Benoit Daloze)]

OK, thank you. I wanted to make sure my concern on that is clear.
At least it seems useful to have some data and extra details on it (the last paragraph).

[hsbt (Hiroshi SHIBATA)]

At least it seems useful to have some data and extra details on it (the last paragraph).

I see. Thanks for your investigation.

To be precise, there wasn't time to talk about wrapper files. We only discuss about https://bugs.ruby-lang.org/issues/20309#note-14 yesterday.

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #20347: Separate docs task from all added