Project

General

Profile

Actions

Feature #17873

closed

Update of default gems in Ruby 3.1

Added by hsbt (Hiroshi SHIBATA) almost 3 years ago. Updated about 2 years ago.

Status:
Closed
Target version:
-
[ruby-core:103912]

Description

I promoted many standard libraries to the default gems. You can see them at https://stdgems.org/#default-gems-ruby-301

I propose the following libraries to promote the bundled gems from the default gems.

matrix
prime
net/ftp
net/smtp
net/pop
net/imap

They have primary maintainer in the canonical repository in github. So, they can release the new version with their convenience.

@shugo (Shugo Maeda) @marcandre (Marc-Andre Lafortune) Any thought?

And I also propose to extract the following libraries from the default gems.

tracer
dbm
gdbm

They are no longer actively maintained and not widely used today.

Actions #1

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Tracker changed from Bug to Feature
  • Backport deleted (2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN)

Updated by Eregon (Benoit Daloze) almost 3 years ago

And I also propose to extract the following libraries from the default gems.

Those are already default gems.
So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Seems fine in general.

I'm hesitant about fcntl since it's so small (just a bunch of constants) and I think it doesn't make so much sense to have that as a separate gem (it's already a default gem, though I don't see the point of that).

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Yes. I meant to remove them ruby repository and release.

Updated by byroot (Jean Boussier) almost 3 years ago

They are no longer actively maintained and not widely used today.

mutex_m

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

Thanks for your information. I agreed your opinion and removed mutex_m from this proposal.

Updated by universato (Yoshimine Sato) almost 3 years ago

I would like prime library to remain in the default gems or the bundled gems.
I do competitive programming as a hobby, and I sometimes use it and find it useful.
Since external libraries are usually not available for competitive programming, I would like prime library to remain in internal gems.

Some Japanese competitors have also mentioned prime library as one of the advantages of Ruby.
Rubyで競プロするときのTips - ARMERIA
【AtCoder】Rubyで競プロに入門する - Qiita
【不安解消】Rubyでも競技プログラミングで問題が解ける→茶色に行けました! | RyuCoding
Rubyで競プロをしていた時に使っていたライブラリを供養する - Qiita

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)
  • Status changed from Open to Assigned

I would like prime library to remain in the default gems or the bundled gems.

Thanks. I moved prime to the bundled gems list.

And I removed fcntl from this proposal. It's also widely used in other gems.

Updated by marcandre (Marc-Andre Lafortune) almost 3 years ago

Thanks, I agree about prime.

I imagine matrix users are limited and will be ok. I'm not sure about ostruct.

Could we consider to bundle them in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

Updated by shugo (Shugo Maeda) almost 3 years ago

I agree about net/ftp and net/imap.

Actions #10

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)

I'm not sure about ostruct.

Thanks, I'm also aware this. So, I removed ostruct from this proposal.

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)

I added dbm and gdbm to removal list with other commiter's opinion.

Updated by knu (Akinori MUSHA) almost 3 years ago

I don't see a good reason to remove the syslog library from ruby.
It is so small, simple, and almost maintenance-free because the syslog facility is unlikely to change anymore.

Syslog has long been a standard logging facility in Unix/Linux and I believe it is good for ruby to have access to it out of the box.
I can imagine it is used by many admins in their system scripts.

Also, there seem to be many gems that load syslog.

knu@aluminium:~$ gem-codesearch "^ *require ['\"]syslog[/\"']"  | ruby -ne 'puts $_.split("/")[3]' | sort -u | wc -l
259

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

  • Description updated (diff)

@knu (Akinori MUSHA) I have no strong opinion to remove syslog in Ruby 3.1. I withdraws it from this proposal.

Updated by hsbt (Hiroshi SHIBATA) almost 3 years ago

I applied this proposal at https://github.com/ruby/ruby/pull/4530

But I keep to open this until releasing preview or rc version of Ruby 3.1.

Updated by byroot (Jean Boussier) almost 3 years ago

Just for the record:

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

Updated by byroot (Jean Boussier) almost 3 years ago

Actually, there's a bit of a problem. If you add a default gem to the gemspec it cause double loading issues, e.g. with mail, net-smtp pulls digest:

/opt/rubies/2.7.2/lib/ruby/2.7.0/digest.rb:6: warning: previous definition of REQUIRE_MUTEX was here
~/.gem/ruby/2.7.2/gems/digest-3.0.0/lib/digest.rb:8: warning: method redefined; discarding old const_missing

So I'm not sure what the path forward is for these gems.

Updated by deivid (David Rodríguez) almost 3 years ago

This sometimes happens when rubygems loads a default gem too early before bundler has been setup. Then when user code loads the actual gem it uses the $LOAD_PATH setup by bundler instead of the default one, causing the double load. Up until now we always found ways to fix these issues by making usages of default gems inside rubygems as lazy as possible, or by vendoring copies of the default gems that we need under our namespace so that we don't interfere with user's choice, but it seems like we're still loading digest too early.

I'll have a look at this.

Updated by byroot (Jean Boussier) almost 3 years ago

I'll have a look at this.

Thank you!

I used digest as an example, but there is also net/protocol: https://buildkite.com/rails/rails/builds/77709#8f4e417a-153d-403c-9ad0-6579b3c04adf/999-1010

Updated by marcandre (Marc-Andre Lafortune) almost 3 years ago

byroot (Jean Boussier) wrote in #note-16:

Just for the record:

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

Thanks.

I'll repeat: could we consider to still bundle these gems in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

Updated by Eregon (Benoit Daloze) over 2 years ago

@byroot (Jean Boussier) @deivid (David Rodríguez) I think it would best to track this double inclusion problem as an issue on https://github.com/rubygems/rubygems/issues, could you file one?

Updated by deivid (David Rodríguez) over 2 years ago

Thanks for the remainder, I forgot. I did try to repro this at the time by simulating locally what Rails CI does, but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

Updated by byroot (Jean Boussier) over 2 years ago

but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

So I was able to repro, but I can confirm that it is fixed on latest rubygems/bundler.

The problem is that lots of users don't keep both of these up to date, so that will cause lots of confusion.

Updated by deivid (David Rodríguez) over 2 years ago

Great to hear that it's fixed!

Regarding your concern, this is a change for ruby 3.1, so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

Updated by byroot (Jean Boussier) over 2 years ago

so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

No, the problem is that to be compatible with 3.1, these gems will have to specify these dependencies. But anyone using gems defining these dependencies but still using older rubygems/bundler will have the double loading issue.

So there's really nothing we can do :/, people will have to upgrade their rubygems/bundler.

Actions #27

Updated by deivid (David Rodríguez) over 2 years ago

I see. It doesn't seem like a big deal to me.

Updated by Eregon (Benoit Daloze) over 2 years ago

Doesn't it mean that in addition to the double-loading warnings (e.g., when using Rails) it might also break on current Ruby releases with the version of RubyGems they ship? (due to the two loaded versions creating some incompatibility)
It's also likely hard to guess that double loading means "update to latest RubyGems and Bundler" (they can be other causes for it).

In a Gemfile, it would be possible to specify those gems are only needed for Ruby 3.1+, but in a gemspec that does not seem possible unfortunately.

@byroot (Jean Boussier) Which Ruby versions did you see was affected? If it's only unsupported Ruby versions at least it'd be a smaller concern.

FWIW Rails main currently does this, the same can be done for generated applications's Gemfile.
https://github.com/rails/rails/blob/acee501edd0274acb1b073aac1ac41a3181aafdd/Gemfile#L172-L182
It is not pretty, but if it avoids double-loading maybe it is better than specifying the dependencies in the gem and potentially breaking running Rails on many Ruby releases.

Updated by deivid (David Rodríguez) over 2 years ago

Yeah, technically it could cause issues, that's why ruby prints a warning. But I've never seen an actual issue in real life, just users bothered by the warning. Concerned users usually look for the root cause, and eventually figure out they need to upgrade rubygems/bundler or report the issue to the affected repository. That's how I've been normally finding out about and fixing these issues in the past years since the gemification of the stdlib started.

Updated by byroot (Jean Boussier) over 2 years ago

Hum, actually I used a docker image to try to pinpoint when it was fixed, and I'm no longer sure it is fixed, or at least another issue cropped up:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "mail", github: "Shopify/mail", branch: "net-smtp-dependency"
end

require "mail"
$ docker run -v /tmp:/tmp -it ruby:3.0.1 bash -c 'gem update --system --no-document && gem install --no-document bundler && ruby /tmp/repro-double-loading.rb'
....

Installing net-imap 0.2.2
Using mail 2.8.0.edge from https://github.com/Shopify/mail.git (at net-smtp-dependency@58704be)
/usr/local/bundle/gems/bundler-2.2.22/lib/bundler/runtime.rb:302:in `check_for_activated_spec!': You have already activated net-protocol 0.1.0, but your Gemfile requires net-protocol 0.1.1. Since net-protocol is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports net-protocol as a default gem. (Gem::LoadError)

So now I get a hard failure. Not sure why it works on my host.

Updated by deivid (David Rodríguez) over 2 years ago

Alright, thanks. I'll have a look at that, then.

Updated by deivid (David Rodríguez) over 2 years ago

At first glance, it looks specific to bundler/inline, when installing gems and using gems happens in the same process. I'll fix it by vendoring net/http and its dependencies under the Bundler:: namespace like we do with the other pure ruby libraries that bundler depends on.

Updated by byroot (Jean Boussier) over 2 years ago

I figured the digest issue. It only happens if you have a git source gem as well in the Gemfile, because the git source does a MD5 of the repository URL.

I submitted a PR that could fix the problem: https://github.com/rubygems/rubygems/pull/4989

Updated by hsbt (Hiroshi SHIBATA) over 2 years ago

I prepared versions for Ruby 3.1.0-preview1. But I keep the version numbers of the following libraries.

Can maintainers update the version number of them before releasing Ruby 3.1.0preview1?

Updated by kou (Kouhei Sutou) over 2 years ago

OK. I'll do.

Updated by kou (Kouhei Sutou) over 2 years ago

Done: Fiddle, CSV, StringScanner

Updated by hsbt (Hiroshi SHIBATA) about 2 years ago

  • Status changed from Assigned to Closed

Ruby 3.1 released. Thanks to develop and release for this.

Updated by Eregon (Benoit Daloze) about 2 years ago

Note: Whenever a gem is extracted from stdlib, we should be careful the gem can be installed on older Ruby versions, otherwise it's not possible for a library to depend on this newer gem via RubyGems (and it has to, otherwise require fails when using Bundler).
See https://github.com/ruby/io-wait/pull/9#issuecomment-1026883889

Updated by retro (Josef Šimánek) about 2 years ago

Eregon (Benoit Daloze) wrote in #note-38:

Note: Whenever a gem is extracted from stdlib, we should be careful the gem can be installed on older Ruby versions, otherwise it's not possible for a library to depend on this newer gem via RubyGems (and it has to, otherwise require fails when using Bundler).
See https://github.com/ruby/io-wait/pull/9#issuecomment-1026883889

In your comment, you're specifically asking that gem can be installed on EOL Ruby. I think that's worth discussion.

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0