Project

General

Profile

Feature #5481

Gemifying Ruby standard library

Added by nahi (Hiroshi Nakamura) almost 6 years ago. Updated 26 days ago.

Status:
Assigned
Priority:
Normal
Target version:
[ruby-core:40316]

Description

=begin

Up-to-date summary of this proposal is at ((URL:https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem))

== Motivation

  • ruby's release cycle is slow for some standard libraries;
    • ex. security fix for WEBrick, xmlrpc and Zlib.
    • ex. API iteration for net/http, OpenSSL, json, psych, RDoc, minitest and rake.
  • There's already the feature called 'default gems' in ruby and some stdlibs are already using it:
    • rake, rdoc, minitest, json, io-console, bigdecimal
    • And some gems are already doing out-of-band releases.
  • When releasing we should give independence equally to all stdlibs, but in a consistent and controllable way.

== Proposal

  • Allow out-of-band stdlib releases.
    • We are not proposing changes to ruby's release management, the release manager would decide when they release ruby and stdlib.
  • Allow more stdlibs to be installed as a 'default gem'
  • Register these gems on RubyGems.org
    • Introduce a new mechanism: controlling supported ruby version so that we can avoid installing unexpected version of stdlib gems. For example, a WEBrick gem for ruby 2.0.1 (released from ruby_2_0_1 branch) should not be installed for ruby 2.0.0 (released from ruby_2_0_0 branch) unless we know it works for both 2.0.0 and 2.0.1.

Note:

  • Moving stdlibs repository location is not a target of this proposal. The implementation details of stdlib gems should hide this from ruby committers.
  • ruby_1_9_3 is not a target of this proposal. The change should be introduced from 2.0.0 release.

...Some more details of the proposal and discussion topics are going to follow as comments.
=end

5481.pdf (78.2 KB) 5481.pdf nahi (Hiroshi Nakamura), 07/01/2012 12:24 AM

Related issues

Related to Ruby trunk - Feature #9612: Gemify OpenSSLClosed
Related to Ruby trunk - Bug #10610: "make install" fails without zlibClosed2014-12-17
Related to Ruby trunk - Feature #11025: Gemify RakeClosed2015-04-02
Related to Ruby trunk - Feature #11057: Gemify JSONRejected
Related to Ruby trunk - Feature #12160: Extract XMLRPC library to bundled gemClosed
Related to Ruby trunk - Feature #8526: gemify tkClosed
Related to Ruby trunk - Misc #13072: Current state of date standard libraryOpen
Related to Ruby trunk - Feature #13173: Gemify webrickClosed
Related to Ruby trunk - Feature #13177: Gemify csvClosed
Related to Ruby trunk - Feature #13182: Gemify cmathClosed
Related to Ruby trunk - Feature #13183: Gemify dateClosed
Related to Ruby trunk - Feature #13197: Gemify fileutilsAssigned
Related to Ruby trunk - Feature #13201: Gemify dbmClosed
Related to Ruby trunk - Feature #13199: Gemify strscanAssigned
Related to Ruby trunk - Feature #13186: Gemify zlibClosed
Related to Ruby trunk - Feature #11083: Gemify net-telnetClosed
Related to Ruby trunk - Feature #8539: Unbundle ext/tkClosed
Related to Ruby trunk - Bug #6123: Properly gemify BigDecimalRejected2012-03-07
Related to Ruby trunk - Feature #2635: Unbundle rdocRejected2010-01-23
Related to Ruby trunk - Feature #13213: Gemify scanfClosed
Related to Ruby trunk - Feature #13248: Gemify gdbmAssigned
Related to Ruby trunk - Feature #13261: Gemify sdbmClosed
Related to Ruby trunk - Feature #9456: Include bin/racc with rubyAssigned
Related to Ruby trunk - Misc #13771: Digest, Ruby OpenSSL, OpenSSL v1.1.0Closed

History

#1 [ruby-core:40317] Updated by nahi (Hiroshi Nakamura) almost 6 years ago

=begin
== Implementation

=== Install stdlibs as 'default gems'

  • Newly created stdlib gems version scheme: ruby's version + '.n'(dot plus a number)
    • ex. WEBrick 'default gem': 2.0.0.0, 2.0.0.1, ...
    • Gems which supports multiple ruby versions (ex. rake, rdoc, etc.) keeps their version.
  • Controlling supported ruby version by specifying "platform". In RubyGems, currently the ruby platform only allows a major and minor version (1.8 or 1.9)
  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now. % gem uninstall webrick -v 2.0.0.0 #=> should raise an error.
  • An updated version of an stdlib gem should be treated as regular gems. With the current 'default gems' implementation, a newer gem will not automatically override the default gem with 'require': % ruby -rwebrick -e 'p WEBrick::VERSION' #=> 2.0.0.0 % gem update webrick Updating installed gems Updating webrick Fetching: webrick-2.0.1.5.gem (100%) Successfully installed webrick-2.0.1.5 Gems updated: webrick Installing ri documentation for webrick-2.0.1.5... Installing RDoc documentation for webrick-2.0.1.5... % ruby -rwebrick -e 'p WEBrick::VERSION' #=> 2.0.0.0 << should be 2.0.1.5
  • Make autoload for stdlib gems work as long as autoload feature exists in 2.0. % ruby -e 'autoload :JSON, "json"; p JSON' #=> "JSON"

For existing 'default gems' see tool/rbinstall.rb. It installs stdlibs as 'default gems'.
* It scans {lib,ext}/*/.gemspec and installs it as a spec-only gems via rubygems.
* Then installs gemdir/bin/* as executables (it would be better to use rubygems instead.)
* It also reads defs/default_gems, but those are obsolete.

=== Register 'default gems' on RubyGems.org

  • Improve 'default gems' for creating gem from stdlib file/directory layout.
  • stdlib gems maintainer registers the updated gems on RubyGems.org.
    • Maintainer must have an account at RubyGems.org

== ToDo

  • After installing updated stdlib gems, those should be treated as regular gems. How?
    (1) Installing 'default gems' as a real gem to /usr/local/lib/ruby/default_gems/, not to /usr/local/lib/ruby/site_ruby/
    * advantage: simple implementation
    * drawback: stdlib gems cannot be loaded with --disable-gems.
    (2) Implement it as a new feature of RubyGems.
    * advantage: stdlib can be loaded without --disable-gems
    * drawback: may penalize ruby startup time
    * drawback: may be complex to implement

    The RubyGems team tried to implement a feature similar to this for ruby 1.9.1 and ruby 1.9.2, but it did not work out very well...
    (3) ?

  • To avoid installing an incompatible version of stdlib gems, update RubyGems to allow three digits (1.8/1.9 -> 1.9.X) on a "ruby" platform in the gem spec.

  • Uninstalling 'default gems' should be blocked. Set default path to /usr/local/lib/ruby/default_gems/

  • Improve 'default gems' for creating gem from stdlib file/directory layout.

  • Decide which stdlibs should be gemified. Let's ask maintainers. We would need to find maintainers of some stdlibs if needed.

  • Decide tagging/branching policy for stdlib gems. Up to the release manager?

  • Find a way to allow autoloading stdlib gems.

    • It's because autoload does not respect require overwriting now. When we install 'default gems' as a real gem, it needs some care to work.

== What stdlibs should be gemified?

=== Principle

  • The lib must have a maintainer.
  • The lib should be highly independent from ruby's code base.
    • Feature dependencies are OK since RubyGems can resolve them. ex. net/http -> openssl

=== Stdlib status

  • lib/benchmark
  • lib/cgi
  • lib/csv
  • lib/drb
  • lib/erb
  • lib/irb
  • lib/logger: NaHi will maintain this as a 'default gem'
  • lib/optparse
  • lib/pstore
  • lib/racc
  • lib/rexml
  • lib/rinda
  • lib/rss
  • lib/webrick: NaHi will maintain this as a 'default gem'
  • lib/xmlrpc
  • lib/yaml
  • lib/rake
  • lib/net: Do we split this to core, ftp, http (and https), mail (imap, pop and smtp) and telnet?
  • ext/curses
  • ext/date
  • ext/digest
  • ext/iconv: Deprecated. New products and libraries should not use this
  • ext/openssl
  • ext/sych

Already a 'default gem':

  • lib/minitest
  • lib/rake
  • lib/rdoc
  • lib/rubygems
  • ext/bigdecimal: Not yet registered at RubyGems.org (Mrkn will register that to RubyGems.org after RubyConf 2011).
  • ext/io-console: Not yet registered at RubyGems.org
  • ext/json
  • ext/psych

== Call for discussion

  • Topics at ToDo above.
  • What stdlibs should be gemified?

Please post comments to this ticket. I'll update the Wiki page at https://redmine.ruby-lang.org/projects/ruby/wiki/StdlibGem as a summary.
=end

#2 [ruby-core:40318] Updated by nahi (Hiroshi Nakamura) almost 6 years ago

=begin
== Background

There was an awesome keynote by Aaron at RubyKaigi2011.

  • RubyKaigi 2011 keynote by Aaron Patterson (tenderlove) / AT&T Interactive: ((URL:http://vimeo.com/26507951))
  • Most relevant parts for this topic
    • 14'27" -- 25'55": Ruby's release cycle and gem inconsistencies
    • 24'05" -- 27'25": Converting stdlib into gems and release them along with the ruby tarball ("That's one thing I'd like to see changed")

At the next session, Matz and Japanese ruby-core committers discussed the next ruby release. Matz said 'there will be no 1.9.4 because it becomes 2.0'. (Note: See the discussion at ((URL:https://redmine.ruby-lang.org/issues/5056)).)

Any detailed changes for 2.0 were not known but I (NaHi) thought that gemifying stdlib should happen before the changes because it would include incompatible behaviors. The work for gemifying would be postponed after 2.0 release (we must be busy for stabilizing 2.0 release.) I was thinking that gemifying stdlib for faster iteration (shorter release cycle) which had been explained in Aaron's keynote was a must and I wanted to see that happen soon. I talked to Aaron(tenderlove) and Eric(drbrain, a maintainer of RubyGems) first about how we could make it happen and to make this proposal.

I also asked Matz about it.

Then I got the reponse 'Talk to Yuki Sonoda(yugui) and Aaron. I say yes'.

So I talked with many ruby committers including Yuki and Aaron at RubyKaigi2011. Lots of discussion about the benefits and what must not happen. Not a unanimous accord, but almost all committers there agreed/understood what we're planning to do by gemifying stdlib.

Now, at the time 1.9.3 is being released, I want to start the work.
=end

#3 [ruby-core:40328] Updated by nahi (Hiroshi Nakamura) almost 6 years ago

Hi,

On Tue, Oct 25, 2011 at 12:49, Yusuke Endoh mame@tsg.ne.jp wrote:

Could you summarize advantages (if any) and disadvantages
(or additional work) from POV of each person?

Sure, I will. But please don't wait to post your comment now. I should
be still missing some issues like 'autoload'. I'll try to summarize
per-role advantages/disadvantages once I collected initial responses.

// NaHi

#4 [ruby-core:40329] Updated by drbrain (Eric Hodel) almost 6 years ago

On Oct 24, 2011, at 8:49 PM, Yusuke Endoh wrote:

Hello,

Could you summarize advantages (if any) and disadvantages
(or additional work) from POV of each person?

I will describe my thoughts

  • user

Users can obtain improvements to parts of ruby without waiting for a new release.

No need to gem 'rdoc', gem 'psych', etc. to use the latest installed version of a gem.

With the current implementation default gems are special. This proposal will make them stop being special.

  • default gem developer

Default gem developers won't have to educate users to do something special to use the latest version of the library.

Default gem developers may need to support multiple versions of ruby in their gem or backport one fix to multiple releases (like backport requests to Ruby 1.9.3 and Ruby 1.9.2).

Default gem developers can release improvements between ruby release cycles.

  • rubygems developer

We may need to implement better RubyGems platform support, but the idea is applicable beyond just improving stdlib gems.

  • rubygems.org admin

The rubygems.org admin should not need to do anything special.

  • ruby committer <

#5 [ruby-core:40334] Updated by lucas (Lucas Nussbaum) almost 6 years ago

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

#6 [ruby-core:40343] Updated by trans (Thomas Sawyer) almost 6 years ago

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

#7 [ruby-core:40345] Updated by judofyr (Magnus Holm) almost 6 years ago

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

#8 [ruby-core:40348] Updated by mame (Yusuke Endoh) almost 6 years ago

Hello,

2011/10/25 Hiroshi Nakamura nakahiro@gmail.com:

Sure, I will. But please don't wait to post your comment now. I should
be still missing some issues like 'autoload'. I'll try to summarize
per-role advantages/disadvantages once I collected initial responses.

Who will upload stdlib gems to rubygems.org? The release manager?
Each stdlib gem developer?

When a security issue of stdlib gems is reported to the security
team (security@ruby-lang.org), how should it be handled?
If a new tarball of ruby distribution and a new gem file is both
released, the release (and announce) time lag must be minimized.

What is the version policy of stdlib gems? Any new feature or
even incompatibility may be included in the forth digit update?
I'm concerned about a relationship with the policy of patch
release.

Is no factor that prevent us from releasing Ruby increased, even
when stdlib gem developer is too busy to work for Ruby?

Small questions to your proposal follows:

2011/10/25 Hiroshi Nakamura nakahiro@gmail.com:

== Motivation
* When releasing we should give independence equally to all stdlibs, but in a consistent and controllable way.

I cannot understand this point. Could you erabolate?

== Proposal
Note:

  • Moving stdlibs repository location is not a target of this proposal. The implementation details of stdlib gems should hide this from ruby committers.

I cannot understand the context of the second sentence.
What do you mean?

== ToDo

NaHi-san will do (or coordinate) all the Todos, right? :-)

  • Find a way to allow autoloading stdlib gems.
    • It's because autoload does not respect require overwriting now. When we install 'default gems' as a real gem, it needs some care to work.

I recommend you create another ticket about autoload, with a
focus on what feature you need in the ruby core.

--
Yusuke Endoh mame@tsg.ne.jp

#9 [ruby-core:40349] Updated by mame (Yusuke Endoh) almost 6 years ago

Hello, Eric

Thank you. Your explanation helped me understand.

--
Yusuke Endoh mame@tsg.ne.jp

#10 [ruby-core:40353] Updated by mame (Yusuke Endoh) almost 6 years ago

Hello,

2011/10/25 Lucas Nussbaum lucas@lucas-nussbaum.net:

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

I cannot understand your issue.
Even if we release core+stdlib as a big package, you can split
the package to each deb package. Doesn't it work?

--
Yusuke Endoh mame@tsg.ne.jp

#11 [ruby-core:40358] Updated by trans (Thomas Sawyer) almost 6 years ago

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holm judofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

Moreover, it is possible to package up a set of gems into a ship-able
and install-able bundle for those who really need that.

#12 [ruby-core:40359] Updated by lucas (Lucas Nussbaum) almost 6 years ago

On 26/10/11 at 00:13 +0900, Trans wrote:

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holm judofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

What I don't like with the current situation is the fact that we
currently have some codebases that are forked between stdlib and gems.

In the 2.0 model, I would be completely fine if the stdlib was shipped
with the interpreter(s) provided that it is only provided as a
convenience snapshot. What we (Debian) would probably do is follow
stdlib releases independently of interpreter releases, and just ignore
the stdlib gem provided with the interpreter.

Moreover, it is possible to package up a set of gems into a ship-able
and install-able bundle for those who really need that.

Exactly.

Lucas

#13 [ruby-core:40360] Updated by lucas (Lucas Nussbaum) almost 6 years ago

On 25/10/11 at 22:43 +0900, Yusuke Endoh wrote:

Hello,

2011/10/25 Lucas Nussbaum lucas@lucas-nussbaum.net:

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

I cannot understand your issue.
Even if we release core+stdlib as a big package, you can split
the package to each deb package. Doesn't it work?

Let me try to rephrase more clearly.

There are several places different sets of software that are or could be
developed independently in the Ruby ecosystem.
- interpreters (+ core libs)
- standard library
- third-party gems

The problem is: how do we maintain and release interpreters and standard
library? There are two sides to this question:
- how do we avoid stdlib code duplication between interpreters?
- how do we encourage stdlib development?

I think that the cleanest solution is to consider MRI and the stdlib to
be two completely different projects, with their own release cycles,
their own security support, etc. That way:
- people working on the interpreters can focus on the interpreters
- people working on the stdlib can focus on the stdlib
- there's no code duplication

There are two problems with that solution:
- it requires each stdlib release to be well tested on the supported
interpreters, possibly taking into account past releases of the
interpreters.
- there could be a problem during the installation. We need to bootstrap
a ruby installation so that there's enough of stdlib installed to
run rubygems and install the remaining gems. That can easily be solved
by providing a full snapshot of stdlib. But then it needs to stay a
snapshot, and everybody should be expected to be able to use a newer
version of each lib.

With the current proposal, I see two ways for Debian to package Ruby
(and none of them are really good):
- provide a bundle of stdlib + mix of updated gems as a single package.
That means that Debian maintainers will have to choose which version
to include for each gem, and the user won't be able to do it.
- split the stdlib into a very large number of small packages for each
lib, so that each one can be updated separately. A prerequisite for
this solution is that we know beforehand how to split the stdlib.
(The root problem with Debian is that two packages cannot provide the
same file if they can be installed together. So we can't have a
package replacing/overriding only a part of another package.)

Lucas

#14 [ruby-core:40371] Updated by sdaubert (Sylvain Daubert) almost 6 years ago

Le 25/10/2011 17:13, Trans a écrit :

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holmjudofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransitiontransfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaumlu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

Hi,

Sometimes, ruby must be installed on systems who may not be connected to
internet (for security reasons for example).

Sylvain

#15 [ruby-core:40380] Updated by drbrain (Eric Hodel) almost 6 years ago

On Oct 25, 2011, at 5:56 AM, Magnus Holm wrote:

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

The current proposal does not envision installing gems from rubygems.org or any other non-local source when you run make install.

You won't need an internet connection to build and install ruby, just the tarball. The source layout is exactly as it is now and gems are created at install time on your machine from the tarball through a script in tool/.

(At release time these gems are also uploaded to rubygems.org.)

#16 [ruby-core:40381] Updated by drbrain (Eric Hodel) almost 6 years ago

On Oct 25, 2011, at 8:13 AM, Trans wrote:

I think it's time to accept that we are in an Internet world.

But we aren't.

There are many users of Ruby and RubyGems that can't access the internet from the computers they are using ruby on. The only way they can install ruby and gems on their machines is via a "sneaker net" of USB keys.

See:

http://help.rubygems.org/kb/rubygems/installing-gems-with-no-network

#17 [ruby-core:40389] Updated by MartinBosslet (Martin Bosslet) almost 6 years ago

I really like the proposal. Just wanted to add one thought - if we add yet more power to the gem infrastructure, we should probably revisit the idea of offering code-signed gems. The theory exists [1], we could build on that. This might be slightly off-topic, so I apologize in advance, but I still feel it is relevant, especially when gemifying stdlib, please let me explain why.

If the stdlib gets gemified we will have even more gem downloads than today. I'm not talking about the bundled gems that would be accessed locally when building Ruby for the first time (wrt to this proposal), rather about "updating" and adding gems to an existing installation. Updating will be a crucial task in production - people would certainly show interest in being able to stay up to date as fast as possible regarding security fixes, performance improvements etc.

Currently, these gem downloads are entirely unprotected, so just imagine a production deployment that connects to the outside world using a company proxy. The guy operating the company proxy could have a lot of fun serving "customized" gems that do all sorts of evil - nobody would notice. Serving the gems over an non-secure channel is definitely a risk. This might not seem such a big deal at first glance, but just think about discussing your architecture with a bank or other high security environments. Those people constantly give us the "<> <>" talk, so I could very well imagine that this is an issue when it comes to adopting Ruby in these environments. These people are often very conservative in their views, and even if they are still coding VB 6 GUIs and have no idea what they are talking about, still (in my own experience) they will readily accept signed Java applets... That's why I think offering signed gems would be helpful in sending a signal to those "enterprisy" clients, if not only to take away their arguments.

What we often find in these situations are SHA-1 checksums or similar features published on the download site. If the site is served over http this has effectively zero additional security. An attacker would simply man-in-the-middle the HTML of the site, too, handing you a different hash. In addition, even when sent over https, let's face it, nobody really compares these hashes manually.

I see two relatively simple options for mitigation: either secure the transport channel on the transport layer using TLS or secure the payload itself and therefore sign the gem. I prefer the second solution because it "persists" the security of the gem, the signature is tightly coupled with the gem itself and persists once downloaded, so that for example local gem cache servers would still be an option (they wouldn't be in the TLS case, unless effectively re-signing the gems).

Please note that a code signature will of course not guarantee that the code is free of bugs or not malicious. The only security it will provide is that one can be sure that the gem itself has not been modified after the holder of the code signing certificate applied the signature. Not more, not less. This narrows your trust decision down to asking yourself the question whether you want to trust the certificate holder to have validated the code to do what it is supposed to do. The person signing the gem is the last person who was able to alter the code. After applying the signature, the gem is effectively sealed. So you would have to base your trust on two things. Whether you trust this person to not have altered the code in malicious ways and whether you trust that person to have validated the code to ensure its proper functionality. Although far from perfect, in my eyes this is still a major improvement over having no guarantee at all.

A problem with code signing gems is that nobody is to keen on buying a "real" code signing certificate to sign their code, but on the other hand nobody would be keen on trusting a self-signed certificate either. This is probably one of the major issues why code signing is a rarely used feature. Now since gems are stored in a centralized repository, a solution could be to not require the authors to sign a gem but to sign a gem on the RubyGems server immediately after uploading, using a certificate that is exclusively issued to RubyGems. Solving the problem of trusting one dedicated certificate is easy, there are several options that would still need to be discussed, but it is certainly doable. However, this would put the burden of managing this certificate and securing access to it on the shoulders of the RubyGems maintainers.

I would also suggest to keep this entirely optional. To be honest, as with all things security, this will possibly affect usability, additional configuration will likely be necessary to use the feature. So you will have to assess your individual risks - you might very well come to the conclusion that a feature like this is overkill in your individual situation. You should of course still have the option to ignore this entirely, much like today.

I talked briefly about the idea with Eric Hodel and Hiroshi Nakamura. Eric told me that similar ideas in this direction exist. I would really appreciate to hear opinions on this. If you feel that this would be indeed a good option, I would suggest to open a new issue for further discussion.

[1] http://docs.rubygems.org/read/chapter/21

#18 [ruby-core:40582] Updated by vo.x (Vit Ondruch) almost 6 years ago

Hiroshi Nakamura wrote:

  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now. % gem uninstall webrick -v 2.0.0.0 #=> should raise an error.

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

#19 [ruby-core:40585] Updated by drbrain (Eric Hodel) almost 6 years ago

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

#20 [ruby-core:40650] Updated by vo.x (Vit Ondruch) almost 6 years ago

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.

Why do you want to make the gems special? What do you want the users to prevent from? Provide easy tool to fix the situation, not to prevent it. If somebody uninstalls default Rake for example, what is the problem to install it again?

#21 [ruby-core:40653] Updated by drbrain (Eric Hodel) almost 6 years ago

=begin
If net/http, fileutils, optparse, psych or zlib are released as gems uninstalling them breaks RubyGems and you will not be able to reinstall them without reinstalling ruby.

Removing such gems should be possible, I don't think it should be easy and I don't think RubyGems should go to extra effort to prevent it either.

Placing the gems in a separate path is sufficient as ((%gem uninstall%)) will only remove gems in ((|GEM_HOME|)) unless you use ((%-i%)).
=end

#22 [ruby-core:43129] Updated by nahi (Hiroshi Nakamura) over 5 years ago

Just let it follow URL change.

#23 [ruby-core:43131] Updated by vo.x (Vit Ondruch) over 5 years ago

Eric Hodel wrote:

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

Actually if the default gems will be in some directory which is accessible using GEM_PATH and it is not current GEM_HOME, the user will not be able to uninstall the default gems. Problem solved. No new features are necessary. The only think which have to be taken in account are a /usr/bin loaders.

#24 [ruby-core:43138] Updated by vo.x (Vit Ondruch) over 5 years ago

Vit Ondruch wrote:

Eric Hodel wrote:

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

Actually if the default gems will be in some directory which is accessible using GEM_PATH and it is not current GEM_HOME, the user will not be able to uninstall the default gems. Problem solved. No new features are necessary. The only think which have to be taken in account are a /usr/bin loaders.

And one more note. From Fedora's point of view, there is no difference between gems which originally comes with Ruby and the other Gems maintained by RPM. So it should be easy to merge the "default gems" with other "RPM managed gems". Our current operating_system.rb which sets-up the RubyGems directories can be found here [1]

[1] http://pkgs.fedoraproject.org/gitweb/?p=ruby.git;a=blob;f=operating_system.rb;hb=HEAD

#25 [ruby-core:45980] Updated by nahi (Hiroshi Nakamura) over 5 years ago

  • Assignee set to nahi (Hiroshi Nakamura)

#26 [ruby-core:45990] Updated by nahi (Hiroshi Nakamura) over 5 years ago

Endoh-san, here's "slide-show" of this proposal.

#28 [ruby-core:46070] Updated by mame (Yusuke Endoh) over 5 years ago

  • Status changed from Open to Assigned

Received, thank you.

I don't think this issue will be settled in a few minutes,
but it may be valuable enough to hear matz's view.

--
Yusuke Endoh mame@tsg.ne.jp

#29 [ruby-core:47568] Updated by kou (Kouhei Sutou) about 5 years ago

NaHi, could you tell us status of this work?
If you doesn't have a time to implement it, I'll implement it.

NOTE: I have another merit by this. If this feature is implemented, test-unit gem can be used without "gem 'test-unit'". So I'll implement it if nobody does it.

#30 [ruby-core:47631] Updated by nahi (Hiroshi Nakamura) about 5 years ago

Kou, thanks for the cooperation. We would need your help soon.

kou (Kouhei Sutou) wrote:

NaHi, could you tell us status of this work?

I did the presentation at the last dev meeting and I think I/we need to clarify a part of the proposal for blessing from Matz. I guess Endoh-san didn't update this ticket yet? (No intention to blame. My bad, I should have asked this earlier.)

#31 [ruby-core:47649] Updated by kou (Kouhei Sutou) about 5 years ago

Thanks for your reply.

I did the presentation at the last dev meeting and I think I/we need to clarify a part of the proposal for blessing from Matz.
OK. I'll wait for the answer of it.

There is another question. Could you show a list of unimplemented features. (In other words TODO list.)
Is https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem#ToDo still valid? I heard another requirement from Eric. It is that "gem contents" should work for default gems. It is used by RDoc. So the list may be outdated.

#32 [ruby-core:48224] Updated by vo.x (Vit Ondruch) almost 5 years ago

Is the plan to extract the gems out of StdLib and make from them proper gems again? I.e. there would be for example ruby/gems folder, with rake, rdoc, minitest and later the install script would install them into lets say ruby/default_gems? There seems to be some activity [1] which is not going in this way I am afraid.

[1] https://github.com/rubygems/rubygems/pull/377

#33 [ruby-core:48225] Updated by usa (Usaku NAKAMURA) almost 5 years ago

  • Target version changed from Next Major to 2.0.0

This ticket is the center of attention at ruby-dev ML :)
NaHi-san will take a certain action in a few days, I guess.

#34 [ruby-core:48228] Updated by vo.x (Vit Ondruch) almost 5 years ago

What actions exactly? Will the gems be gems or just another workaround? Can I see current state? How the result will look? Can I help somehow? Thank you.

#35 [ruby-core:48230] Updated by vo.x (Vit Ondruch) almost 5 years ago

Will it address concerns from #6124 as well?

#36 [ruby-core:48271] Updated by mame (Yusuke Endoh) almost 5 years ago

  • Target version changed from 2.0.0 to next minor

We discussed this request at the developers' meeting (Jul. 21),
and it was not accepted.

NaHi (the proposer) presented this request by himself. He said:

Because of "fake gems", the new files of a stdlib installed by
"gem update" are ignored unless a user writes gem "json"
explicitly. To address this issue, stdlibs must be provided
as normal gems, not "fake gems".

And the conclusion is as follows:

  • The current behavior of fake gems is considered "bug", so should be fixed. But it is not a good reason to make stdlib real gems.
  • If it is hard or even impossible to fix "the bug", NaHi should re-propose it.

After that, NaHi said, "fixing the behavior" effectively means
"making stdlib as normal gems", but I don't know the meaning.
Please wait for his words.

--
Yusuke Endoh mame@tsg.ne.jp

#37 [ruby-core:48288] Updated by vo.x (Vit Ondruch) almost 5 years ago

Thank you for clarification, waiting for NaHi and offering my help.

#38 [ruby-core:48598] Updated by vo.x (Vit Ondruch) almost 5 years ago

Hi, I'd like to question once more some parts of the proposal in wiki [1].

  • After installing updated stdlib gems, those should be treated as regular gems.

The only viable solution is proposal 1. It seems that there was some work started to implement 2, but that is really wrong approach. What would be the purpose of --disable-gems then? You disable gems but they will be loaded to load some default gems? That doesn't make sense to me.

Moreover, if you have reason to use --disable-gems option, you will be probably able to use -I to setup your load paths to load the gems manually.

  • Introduce a new mechanism: controlling supported ruby version

There is already mechanism in RubyGems, i.e. you can specify version of Ruby which is supported by the gem, not sure why there should be anything more. Since the StdLib gems would go into independent directory and they are uninstallable, there can't be installed any "unexpected version of stdlib gems". If this should solve that there might be installed to new gem, then I guess Ruby developers are already used to this. So not sure what would this improve.

  • Newly created stdlib gems version scheme: ruby's version + '.n'(dot plus a number)

Why WEBrick should have this kind of versioning? What if there was no change in WEBrick in between Ruby 2.0 and, 2.1? Why the version should be changed? Every gem should have independent versioning, as maintainer decides. Bind the gems versioning to Ruby versions makes no sense and it is not beneficial.

  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now.

Yes, since currently, the default gems are mixed with regular gems, they are uninstallable. This would be non issue ff they are in different path.

  • And one more note from Fedora's Ruby maintainer point of view

It has to be possible to unbundle the default gems out of Ruby and replace them with newer versions if needed. This is similar requirement as Lucas Nussbaum pointed our earlier in this discussion. There are also different reasons, such as for example, Ruby is dependency of VIM in Fedora and VIM users really don't want to install RDoc, Minitest, Rake etc, just because they want to use VIM.

Please note that we are already doing so in Fedora and there are no issues, except #6124

[1] https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem
[2] https://github.com/rubygems/rubygems/pull/377#issuecomment-9735417

#39 [ruby-core:48685] Updated by headius (Charles Nutter) almost 5 years ago

FWIW, here's how we're handling the gemification of OpenSSL...

Background: Previously, JRuby's 'openssl' support was always distributed as a gem, since we did not want to ship crypto (bouncycastle). Starting with JRuby 1.7, however, we ship 'openssl' in the standard JRuby distribution. We still want to support users on 1.6.x and want to be able to update 'openssl' independent of JRuby, so for Jruby 1.7.1 we will support using the gem if the gem is installed.

The file within JRuby is here: https://github.com/jruby/jruby/blob/master/lib/ruby/1.9/openssl.rb

This file attempts to load a file that's only in the gem, and the gem then adds its own dirs to LOAD_PATH so future requires will use the gem's versions of things instead of those built into JRuby.

This approach may advise a similar solution in MRI. At the very least, I'm interested in this bug so we use the same mechanism for loading the gem version of a stdlib instead of the shipped version.

I am disappointed this won't happen for 2.0.0.

#40 [ruby-core:50226] Updated by vo.x (Vit Ondruch) almost 5 years ago

=begin
Hi,

I played around a bit an put together a few patches. You can find it here: https://github.com/voxik/ruby/commits/gemified-stdlib . I took Rake as an example.

In a nutshell:

(1) I created the gems/ subdirectory in Ruby repository.
(2) I moved the Rake bits into gems/rake/
(3) I added .gemspec for Rake
(4) I modified rbinstall.rb to install each gem found in gems subdirectory into standard StdLib gem location
(5) I modified RubyGems to look into StdLib gem location for gems.

What is currently missing:

(1) The test suite is not passing currently, but I think that the Rake's test suite should be moved into the gems/rake/ subdirectory as well and there should be some mechanism, how to execute test suites of gems.
(2) If there would be some binary gem, its binary extension is not build ATM. However, I would say that similar approach used to build current exts could be used.

This approach has significant advantages IMO:

(1) Rake gem currently reflect the upstream repository and as soon as the tests are moved into the gems/rake/ subfolder, it will be almost exact match. It could be easily replaced by something like svn:externals or git submodule. This would greatly improve the maintainability of the code and reduced the possibility of forking.
(2) This would help alternative Ruby implementations to ship their additional libraries in consistent manner.
(3) Gems could be easily replaced by downstream distributions by different versions, if needed.

Any comments? Would this approach be acceptable? Should I try to spend more time polishing the test suite and binary extension's build?
=end

#41 [ruby-core:50229] Updated by trans (Thomas Sawyer) almost 5 years ago

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

#42 [ruby-core:50230] Updated by vo.x (Vit Ondruch) almost 5 years ago

trans (Thomas Sawyer) wrote:

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

Although I'd love to see Ruby shipping .gem files, the issue is that for example RubyGems needs Psych gem to work properly. So it is chicken/egg problem.

#43 [ruby-core:50263] Updated by sdaubert (Sylvain Daubert) almost 5 years ago

trans (Thomas Sawyer) wrote:

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

A full bundle may be easier installed when no internet connection is available (which often happens to me).

#44 [ruby-core:50270] Updated by trans (Thomas Sawyer) almost 5 years ago

sdaubert (Sylvain Daubert) That shouldn't be a problem in this case. All the necessary gems would be distributed with the Ruby source (it would sort of act like it's own miniature gem server, figuratively speaking).

#45 [ruby-core:50271] Updated by trans (Thomas Sawyer) almost 5 years ago

vo.x (Vit Ondruch) Psych would have to be an exception. And really, with the inclusion of libyaml, YAML is becoming more and more an integral aspect of Ruby.

#46 [ruby-core:50272] Updated by vo.x (Vit Ondruch) almost 5 years ago

trans (Thomas Sawyer) You know how it is with exceptions. They tend to become rules and justification for another exceptions. I believe that my proposal is the best solution, because it will be almost like shipping already preinstalled gems, which you would need to rebuild anyway. Packaged gems do not give you any additional benefit.

Anyway, I am not decision maker unfortunately. I just trying rectify current situation and prevent worser solutions by being proactive :)

#47 [ruby-core:50274] Updated by trans (Thomas Sawyer) almost 5 years ago

vo.x (Vit Ondruch) Don't count your eggs before they hatch ;) I think the exceptions will be pretty well defined, basically anything RubyGems requires --isn't that going to be the case regardless of which way it's done? I thought packaged gems would be a benefit b/c one would not have to keep source files in sync with library repos, nor make particularly exceptional code for installing "fake" gems vs. real ones.

Understood. I'm just trying to flesh out this approach so whomever makes the decision has it for consideration.

#48 [ruby-core:51256] Updated by trans (Thomas Sawyer) almost 5 years ago

What is the status of this? I had a couple of thoughts/questions:

  • I was wondering how standard lib like ostruct.rb fits into this. I did not notice it mentioned. Would it be a gem? Or are some standard libs staying "old-fashioned"?

  • RubyGems dependencies seem to create a cache-22, as quoted "net/http, fileutils, optparse, psych or zlib are released as gems uninstalling them breaks RubyGems and you will not be able to reinstall them without reinstalling ruby." This makes these libs kind of quasi-core. Maybe an effort should be made to try to reduce RubyGems dependencies on these in so far as it is possible, and whatever is not possible should just be accepted as core?

  • It still seems to me that maintenance of this would be immensely relieved if Ruby just bundled the set of standard gem packages, rather then redistribute copied source.

#49 [ruby-core:51781] Updated by vo.x (Vit Ondruch) over 4 years ago

BTW there should be process in place which ensures, that the libraries are properly versioned, i.e. if something is changed, it cannot have the same version as already released library (see #7761 and #7762).

#50 [ruby-core:62873] Updated by hsbt (Hiroshi SHIBATA) over 3 years ago

  • Assignee changed from nahi (Hiroshi Nakamura) to hsbt (Hiroshi SHIBATA)

#51 [ruby-core:65997] Updated by naruse (Yui NARUSE) almost 3 years ago

#52 [ruby-core:66910] Updated by vo.x (Vit Ondruch) almost 3 years ago

  • Related to Bug #10610: "make install" fails without zlib added

#53 Updated by vo.x (Vit Ondruch) over 2 years ago

#54 Updated by hsbt (Hiroshi SHIBATA) over 2 years ago

#55 Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

  • Related to Feature #12160: Extract XMLRPC library to bundled gem added

#56 Updated by hsbt (Hiroshi SHIBATA) over 1 year ago

#57 [ruby-core:78736] Updated by hsbt (Hiroshi SHIBATA) 10 months ago

After releasing Ruby 2.4.0 and Getting Matz approval, I will promote all of stdlibs to default gem.

I will ask this issue again at https://bugs.ruby-lang.org/projects/ruby/wiki/DevelopersMeeting20161221Japan

#58 [ruby-core:78737] Updated by duerst (Martin Dürst) 10 months ago

Hiroshi SHIBATA wrote:

After releasing Ruby 2.4.0 and Getting Matz approval, I will promote all of stdlibs written by pure ruby to default gem.

Is this going to be part of 2.4 or part of 2.5?

Please be careful about lib/unicode_normalize.rb and lib/unicode_normalize/*. These are located in lib/, but they are part of String and built in.

#59 [ruby-core:78740] Updated by hsbt (Hiroshi SHIBATA) 10 months ago

Is this going to be part of 2.4 or part of 2.5?

part of 2.5.

Please be careful about lib/unicode_normalize.rb and lib/unicode_normalize/*.

I see. Thank you.

#60 [ruby-core:78744] Updated by zverok (Victor Shepelev) 10 months ago

What would be the status of ext/date and related things?
As far as I can understand, it is considered unmaintained/outdated currently, while providing not only Date class (useful), and DateTime (moderately useful, duplicating internal Time being incompatible with it), but also Date._parse, which is used by lib/time and entire Ruby world...

#61 Updated by hsbt (Hiroshi SHIBATA) 10 months ago

  • Related to Misc #13072: Current state of date standard library added

#62 [ruby-core:79135] Updated by mrkn (Kenta Murata) 9 months ago

  • Description updated (diff)

#63 [ruby-core:79136] Updated by matz (Yukihiro Matsumoto) 9 months ago

  • Description updated (diff)

Go ahead and keep gemifying libraries discussing with their maintainer.

Matz.

#64 Updated by hsbt (Hiroshi SHIBATA) 9 months ago

#65 Updated by hsbt (Hiroshi SHIBATA) 9 months ago

#66 Updated by hsbt (Hiroshi SHIBATA) 9 months ago

#67 Updated by hsbt (Hiroshi SHIBATA) 9 months ago

#68 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#69 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#70 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#71 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#72 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#73 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#74 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

  • Related to Bug #6123: Properly gemify BigDecimal added

#75 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#76 [ruby-core:79500] Updated by shevegen (Robert A. Heiler) 8 months ago

Looks as if this ticket may soon be resolved/finished after 5 years. :)

The only one on the todo list above is "Related to Ruby trunk - Misc #13072:
Current state of date standard library", or is there something else missing
that has not been listed/updated yet? (5 years is a long time to keep track of)

#77 [ruby-core:79522] Updated by headius (Charles Nutter) 8 months ago

Please coordinate with us on the JRuby team. The released gems will need to be loadable in JRuby, since people will start depending on specific versions, upgrading them independently, etc.

MOST of stdlib in JRuby is identical to that in CRuby, but we do have a small set of patches. The diff can be seen by comparing the jruby-ruby-2_4 branch on our fork at http://github.com/jruby/ruby with the release version of Ruby 2.4, or similar branch for Ruby 2.3.

Of special concern are any stdlib that have C extensions. For those we can support, we have Java versions, and those will need to be released as gems as well.

We don't want another bigdecimal issue where there's a core gem that can't load on JRuby.

#78 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#79 [ruby-core:79528] Updated by hsbt (Hiroshi SHIBATA) 8 months ago

headius (Charles Nutter)

Thank you for your information. I have never seen jruby patches in jruby/ruby repository.

https://github.com/jruby/ruby/commit/b34aa3d97872a70ca91d8e2dd481628444f33d5e

I think that the above patch is a good way to merge and maintain together after becoming a gem.

Of special concern are any stdlib that have C extensions. For those we can support, we have Java versions, and those will need to be released as gems as well.

It's good to move same situation as psych gem.

#80 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#81 Updated by hsbt (Hiroshi SHIBATA) 8 months ago

#82 [ruby-core:80160] Updated by marcandre (Marc-Andre Lafortune) 7 months ago

I'd like to gemify matrix & ostruct, but it's not clear to me what needs to be done besides creating the gemspec and how future versions are meant to be released.

#83 [ruby-core:80163] Updated by headius (Charles Nutter) 7 months ago

hsbt (Hiroshi SHIBATA) wrote:

Thank you for your information. I have never seen jruby patches in jruby/ruby repository.

https://github.com/jruby/ruby/commit/b34aa3d97872a70ca91d8e2dd481628444f33d5e

I think that the above patch is a good way to merge and maintain together after becoming a gem.

Many of these changes could be merged back into MRI. I can start this process if you think it would be a good idea. We would like to reduce the size of this patch as much as possible, to ease maintenance.

It's good to move same situation as psych gem.

I will look over the other gemify issues and update with JRuby details.

#84 [ruby-core:80171] Updated by hsbt (Hiroshi SHIBATA) 7 months ago

I'd like to gemify matrix & ostruct

marcandre, Thank you for your attention.

I try to clarify to gemification process for you. I will create issue of matrix and ostruct gemification.

#85 Updated by hsbt (Hiroshi SHIBATA) 5 months ago

#86 Updated by hsbt (Hiroshi SHIBATA) 3 months ago

  • Related to Misc #13771: Digest, Ruby OpenSSL, OpenSSL v1.1.0 added

#87 [ruby-core:82394] Updated by zenspider (Ryan Davis) 2 months ago

Hi hi... if you all want a meta-gem that you can version along with your releases and have it point to all the originally bundled gems, I am happy to transfer the ruby gem over to you. I was trying to get this ball rolling a long time ago and I camped it for just this reason...

That way if an install gets messed up it should be as simple as gem install ruby -v 2.5.3.

#88 [ruby-core:82396] Updated by hsbt (Hiroshi SHIBATA) 2 months ago

zenspider

Thanks for your offer. meta-gem idea is useful for update default gems.

Could you transfer ownership of ruby namespace to me?

#89 [ruby-core:82397] Updated by rbjl (Jan Lelis) 2 months ago

I like the idea that you could "update your standard library", for example, you are using Ruby 2.5.2, but want the standard library of 2.6.1. (as long as every gem in 2.6.1 is compatible with your old version of Ruby), which would be possible then.

It needs some thoughts though:

  • More maintenance required: The ruby gem must be created and released together with Ruby. As long their is no automated process for importing upstream gem repos (or vice versa, for gems where development happens in core), there is always the danger that the gems on rubygems.org and the actually included default gems might get out of sync - which would cause a lot of confusion
  • Should the ruby gem include bundled gems or not?

#90 [ruby-core:82946] Updated by akihikodaki (Akihiko Odaki) 26 days ago

Hi, I have some feedback implying a possibility of unexpected dependency on features packaged in Ruby installation when their gem versions are intended to be used.

I use Bundler v1.13.7 provided by CentOS SCL. The version always uses openssl packaged in Ruby installation even if there is a bundled one.
That is caused by an dependency of Bundler on openssl. The dependency is implicitly made by securerandom. It is not documented in openssl documentation, and the documentation securerandom just says it may use openssl as random generator.

Though the dependency is removed by https://github.com/bundler/bundler/commit/6167ec85e9ef4e9146fad305864b8359d4a9b021, the commit message describes the change as performance improvement, showing the developers were not aware of the dependency on openssl and it was locked-in. I searched Bundler's issue tracker for related issues but had no luck.

This case reveals a fact that features packaged in Ruby installation can unexpectedly be used, and users may not be aware of that.
It is problematic especially for such a security-related one. The issue points out security fixes as motivation, but it is hardly secure when it can be implicitly disabled.

Here I suggest two possible solutions for the problem:
* Document dependencies of standard libraries on gemified features
* Let RubyGems and Bundler warn if a gemified feature is already loaded

The later solution should actually be done in RubyGems and Bundler, but I decided to share it in this issue since it is the only reason such a warning should be implemented. I have not posted the feedback on the issue tracker of RubyGems nor one of Bundler, so they should be notified later if you think the problem should be handled by them.

Also available in: Atom PDF