Feature #5481

Gemifying Ruby standard library

Added by Hiroshi Nakamura over 2 years ago. Updated about 1 year ago.

[ruby-core:40316]
Status:Assigned
Priority:Normal
Assignee:Hiroshi Nakamura
Category:lib
Target version:next minor

Description

=begin

Up-to-date summary of this proposal is at ((URL:https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem))

== Motivation

  • ruby's release cycle is slow for some standard libraries;
    • ex. security fix for WEBrick, xmlrpc and Zlib.
    • ex. API iteration for net/http, OpenSSL, json, psych, RDoc, minitest and rake.
  • There's already the feature called 'default gems' in ruby and some stdlibs are already using it:
    • rake, rdoc, minitest, json, io-console, bigdecimal
    • And some gems are already doing out-of-band releases.
  • When releasing we should give independence equally to all stdlibs, but in a consistent and controllable way.

== Proposal

  • Allow out-of-band stdlib releases.
    • We are not proposing changes to ruby's release management, the release manager would decide when they release ruby and stdlib.
  • Allow more stdlibs to be installed as a 'default gem'
  • Register these gems on RubyGems.org
    • Introduce a new mechanism: controlling supported ruby version so that we can avoid installing unexpected version of stdlib gems. For example, a WEBrick gem for ruby 2.0.1 (released from ruby201 branch) should not be installed for ruby 2.0.0 (released from ruby200 branch) unless we know it works for both 2.0.0 and 2.0.1.

Note:

  • Moving stdlibs repository location is not a target of this proposal. The implementation details of stdlib gems should hide this from ruby committers.
  • ruby19_3 is not a target of this proposal. The change should be introduced from 2.0.0 release.

...Some more details of the proposal and discussion topics are going to follow as comments.
=end

5481.pdf (78.2 KB) Hiroshi Nakamura, 07/01/2012 12:24 AM

History

#1 Updated by Hiroshi Nakamura over 2 years ago

=begin
== Implementation

=== Install stdlibs as 'default gems'

  • Newly created stdlib gems version scheme: ruby's version + '.n'(dot plus a number)
    • ex. WEBrick 'default gem': 2.0.0.0, 2.0.0.1, ...
    • Gems which supports multiple ruby versions (ex. rake, rdoc, etc.) keeps their version.
  • Controlling supported ruby version by specifying "platform". In RubyGems, currently the ruby platform only allows a major and minor version (1.8 or 1.9)
  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now. % gem uninstall webrick -v 2.0.0.0 #=> should raise an error.
  • An updated version of an stdlib gem should be treated as regular gems. With the current 'default gems' implementation, a newer gem will not automatically override the default gem with 'require': % ruby -rwebrick -e 'p WEBrick::VERSION' #=> 2.0.0.0 % gem update webrick Updating installed gems Updating webrick Fetching: webrick-2.0.1.5.gem (100%) Successfully installed webrick-2.0.1.5 Gems updated: webrick Installing ri documentation for webrick-2.0.1.5... Installing RDoc documentation for webrick-2.0.1.5... % ruby -rwebrick -e 'p WEBrick::VERSION' #=> 2.0.0.0 << should be 2.0.1.5
  • Make autoload for stdlib gems work as long as autoload feature exists in 2.0. % ruby -e 'autoload :JSON, "json"; p JSON' #=> "JSON"

For existing 'default gems' see tool/rbinstall.rb. It installs stdlibs as 'default gems'.
* It scans {lib,ext}/*/.gemspec and installs it as a spec-only gems via rubygems.
* Then installs gemdir/bin/* as executables (it would be better to use rubygems instead.)
* It also reads defs/default_gems, but those are obsolete.

=== Register 'default gems' on RubyGems.org

  • Improve 'default gems' for creating gem from stdlib file/directory layout.
  • stdlib gems maintainer registers the updated gems on RubyGems.org.
    • Maintainer must have an account at RubyGems.org

== ToDo

  • After installing updated stdlib gems, those should be treated as regular gems. How?
    (1) Installing 'default gems' as a real gem to /usr/local/lib/ruby/defaultgems/, not to /usr/local/lib/ruby/siteruby/
    * advantage: simple implementation
    * drawback: stdlib gems cannot be loaded with --disable-gems.
    (2) Implement it as a new feature of RubyGems.
    * advantage: stdlib can be loaded without --disable-gems
    * drawback: may penalize ruby startup time
    * drawback: may be complex to implement

    The RubyGems team tried to implement a feature similar to this for ruby 1.9.1 and ruby 1.9.2, but it did not work out very well...
    (3) ?

  • To avoid installing an incompatible version of stdlib gems, update RubyGems to allow three digits (1.8/1.9 -> 1.9.X) on a "ruby" platform in the gem spec.

  • Uninstalling 'default gems' should be blocked. Set default path to /usr/local/lib/ruby/default_gems/

  • Improve 'default gems' for creating gem from stdlib file/directory layout.

  • Decide which stdlibs should be gemified. Let's ask maintainers. We would need to find maintainers of some stdlibs if needed.

  • Decide tagging/branching policy for stdlib gems. Up to the release manager?

  • Find a way to allow autoloading stdlib gems.

    • It's because autoload does not respect require overwriting now. When we install 'default gems' as a real gem, it needs some care to work.

== What stdlibs should be gemified?

=== Principle

  • The lib must have a maintainer.
  • The lib should be highly independent from ruby's code base.
    • Feature dependencies are OK since RubyGems can resolve them. ex. net/http -> openssl

=== Stdlib status

  • lib/benchmark
  • lib/cgi
  • lib/csv
  • lib/drb
  • lib/erb
  • lib/irb
  • lib/logger: NaHi will maintain this as a 'default gem'
  • lib/optparse
  • lib/pstore
  • lib/racc
  • lib/rexml
  • lib/rinda
  • lib/rss
  • lib/webrick: NaHi will maintain this as a 'default gem'
  • lib/xmlrpc
  • lib/yaml
  • lib/rake
  • lib/net: Do we split this to core, ftp, http (and https), mail (imap, pop and smtp) and telnet?
  • ext/curses
  • ext/date
  • ext/digest
  • ext/iconv: Deprecated. New products and libraries should not use this
  • ext/openssl
  • ext/sych

Already a 'default gem':

  • lib/minitest
  • lib/rake
  • lib/rdoc
  • lib/rubygems
  • ext/bigdecimal: Not yet registered at RubyGems.org (Mrkn will register that to RubyGems.org after RubyConf 2011).
  • ext/io-console: Not yet registered at RubyGems.org
  • ext/json
  • ext/psych

== Call for discussion

  • Topics at ToDo above.
  • What stdlibs should be gemified?

Please post comments to this ticket. I'll update the Wiki page at https://redmine.ruby-lang.org/projects/ruby/wiki/StdlibGem as a summary.
=end

#2 Updated by Hiroshi Nakamura over 2 years ago

=begin
== Background

There was an awesome keynote by Aaron at RubyKaigi2011.

  • RubyKaigi 2011 keynote by Aaron Patterson (tenderlove) / AT&T Interactive: ((URL:http://vimeo.com/26507951))
  • Most relevant parts for this topic
    • 14'27" -- 25'55": Ruby's release cycle and gem inconsistencies
    • 24'05" -- 27'25": Converting stdlib into gems and release them along with the ruby tarball ("That's one thing I'd like to see changed")

At the next session, Matz and Japanese ruby-core committers discussed the next ruby release. Matz said 'there will be no 1.9.4 because it becomes 2.0'. (Note: See the discussion at ((URL:https://redmine.ruby-lang.org/issues/5056)).)

Any detailed changes for 2.0 were not known but I (NaHi) thought that gemifying stdlib should happen before the changes because it would include incompatible behaviors. The work for gemifying would be postponed after 2.0 release (we must be busy for stabilizing 2.0 release.) I was thinking that gemifying stdlib for faster iteration (shorter release cycle) which had been explained in Aaron's keynote was a must and I wanted to see that happen soon. I talked to Aaron(tenderlove) and Eric(drbrain, a maintainer of RubyGems) first about how we could make it happen and to make this proposal.

I also asked Matz about it.

Then I got the reponse 'Talk to Yuki Sonoda(yugui) and Aaron. I say yes'.

So I talked with many ruby committers including Yuki and Aaron at RubyKaigi2011. Lots of discussion about the benefits and what must not happen. Not a unanimous accord, but almost all committers there agreed/understood what we're planning to do by gemifying stdlib.

Now, at the time 1.9.3 is being released, I want to start the work.
=end

#3 Updated by Hiroshi Nakamura over 2 years ago

Hi,

On Tue, Oct 25, 2011 at 12:49, Yusuke Endoh mame@tsg.ne.jp wrote:

Could you summarize advantages (if any) and disadvantages
(or additional work) from POV of each person?

Sure, I will. But please don't wait to post your comment now. I should
be still missing some issues like 'autoload'. I'll try to summarize
per-role advantages/disadvantages once I collected initial responses.

// NaHi

#4 Updated by Eric Hodel over 2 years ago

On Oct 24, 2011, at 8:49 PM, Yusuke Endoh wrote:

Hello,

Could you summarize advantages (if any) and disadvantages
(or additional work) from POV of each person?

I will describe my thoughts

  • user

Users can obtain improvements to parts of ruby without waiting for a new release.

No need to gem 'rdoc', gem 'psych', etc. to use the latest installed version of a gem.

With the current implementation default gems are special. This proposal will make them stop being special.

  • default gem developer

Default gem developers won't have to educate users to do something special to use the latest version of the library.

Default gem developers may need to support multiple versions of ruby in their gem or backport one fix to multiple releases (like backport requests to Ruby 1.9.3 and Ruby 1.9.2).

Default gem developers can release improvements between ruby release cycles.

  • rubygems developer

We may need to implement better RubyGems platform support, but the idea is applicable beyond just improving stdlib gems.

  • rubygems.org admin

The rubygems.org admin should not need to do anything special.

  • ruby committer <

#5 Updated by Lucas Nussbaum over 2 years ago

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

#6 Updated by Thomas Sawyer over 2 years ago

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

#7 Updated by Magnus Holm over 2 years ago

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

#8 Updated by Yusuke Endoh over 2 years ago

Hello,

2011/10/25 Hiroshi Nakamura nakahiro@gmail.com:

Sure, I will. But please don't wait to post your comment now. I should
be still missing some issues like 'autoload'. I'll try to summarize
per-role advantages/disadvantages once I collected initial responses.

Who will upload stdlib gems to rubygems.org? The release manager?
Each stdlib gem developer?

When a security issue of stdlib gems is reported to the security
team (security@ruby-lang.org), how should it be handled?
If a new tarball of ruby distribution and a new gem file is both
released, the release (and announce) time lag must be minimized.

What is the version policy of stdlib gems? Any new feature or
even incompatibility may be included in the forth digit update?
I'm concerned about a relationship with the policy of patch
release.

Is no factor that prevent us from releasing Ruby increased, even
when stdlib gem developer is too busy to work for Ruby?

Small questions to your proposal follows:

2011/10/25 Hiroshi Nakamura nakahiro@gmail.com:

== Motivation
* When releasing we should give independence equally to all stdlibs, but in a consistent and controllable way.

I cannot understand this point. Could you erabolate?

== Proposal
Note:

  • Moving stdlibs repository location is not a target of this proposal. The implementation details of stdlib gems should hide this from ruby committers.

I cannot understand the context of the second sentence.
What do you mean?

== ToDo

NaHi-san will do (or coordinate) all the Todos, right? :-)

  • Find a way to allow autoloading stdlib gems.
    • It's because autoload does not respect require overwriting now. When we install 'default gems' as a real gem, it needs some care to work.

I recommend you create another ticket about autoload, with a
focus on what feature you need in the ruby core.

--
Yusuke Endoh mame@tsg.ne.jp

#9 Updated by Yusuke Endoh over 2 years ago

Hello, Eric

Thank you. Your explanation helped me understand.

--
Yusuke Endoh mame@tsg.ne.jp

#10 Updated by Yusuke Endoh over 2 years ago

Hello,

2011/10/25 Lucas Nussbaum lucas@lucas-nussbaum.net:

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

I cannot understand your issue.
Even if we release core+stdlib as a big package, you can split
the package to each deb package. Doesn't it work?

--
Yusuke Endoh mame@tsg.ne.jp

#11 Updated by Thomas Sawyer over 2 years ago

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holm judofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

Moreover, it is possible to package up a set of gems into a ship-able
and install-able bundle for those who really need that.

#12 Updated by Lucas Nussbaum over 2 years ago

On 26/10/11 at 00:13 +0900, Trans wrote:

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holm judofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransition transfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaum lu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

So we still have to ship the gems with the release, which is what
Lucas doesn't like…

What I don't like with the current situation is the fact that we
currently have some codebases that are forked between stdlib and gems.

In the 2.0 model, I would be completely fine if the stdlib was shipped
with the interpreter(s) provided that it is only provided as a
convenience snapshot. What we (Debian) would probably do is follow
stdlib releases independently of interpreter releases, and just ignore
the stdlib gem provided with the interpreter.

Moreover, it is possible to package up a set of gems into a ship-able
and install-able bundle for those who really need that.

Exactly.

Lucas

#13 Updated by Lucas Nussbaum over 2 years ago

On 25/10/11 at 22:43 +0900, Yusuke Endoh wrote:

Hello,

2011/10/25 Lucas Nussbaum lucas@lucas-nussbaum.net:

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

I cannot understand your issue.
Even if we release core+stdlib as a big package, you can split
the package to each deb package. Doesn't it work?

Let me try to rephrase more clearly.

There are several places different sets of software that are or could be
developed independently in the Ruby ecosystem.
- interpreters (+ core libs)
- standard library
- third-party gems

The problem is: how do we maintain and release interpreters and standard
library? There are two sides to this question:
- how do we avoid stdlib code duplication between interpreters?
- how do we encourage stdlib development?

I think that the cleanest solution is to consider MRI and the stdlib to
be two completely different projects, with their own release cycles,
their own security support, etc. That way:
- people working on the interpreters can focus on the interpreters
- people working on the stdlib can focus on the stdlib
- there's no code duplication

There are two problems with that solution:
- it requires each stdlib release to be well tested on the supported
interpreters, possibly taking into account past releases of the
interpreters.
- there could be a problem during the installation. We need to bootstrap
a ruby installation so that there's enough of stdlib installed to
run rubygems and install the remaining gems. That can easily be solved
by providing a full snapshot of stdlib. But then it needs to stay a
snapshot, and everybody should be expected to be able to use a newer
version of each lib.

With the current proposal, I see two ways for Debian to package Ruby
(and none of them are really good):
- provide a bundle of stdlib + mix of updated gems as a single package.
That means that Debian maintainers will have to choose which version
to include for each gem, and the user won't be able to do it.
- split the stdlib into a very large number of small packages for each
lib, so that each one can be updated separately. A prerequisite for
this solution is that we know beforehand how to split the stdlib.
(The root problem with Debian is that two packages cannot provide the
same file if they can be installed together. So we can't have a
package replacing/overriding only a part of another package.)

Lucas

#14 Updated by Sylvain Daubert over 2 years ago

Le 25/10/2011 17:13, Trans a écrit :

On Tue, Oct 25, 2011 at 8:56 AM, Magnus Holmjudofyr@gmail.com wrote:

On Tue, Oct 25, 2011 at 14:45, Intransitiontransfire@gmail.com wrote:

On Oct 25, 1:58 am, Lucas Nussbaumlu...@lucas-nussbaum.net wrote:

Issue #5481 has been updated by Lucas Nussbaum.

(With my Debian hat)
I don't really like the plan of having releases of libraries both with the interpreter and as separate gems. Duplicating distribution paths for libraries will make things very complex since there would be no single "latest version" of a given library. We already have that problem with rubygems due to its bundling in 1.9.

Instead, I would prefer a plan where the interpreter no longer releases with the stdlib, but just releases with a "core lib". Then the user would install the stdlib as a set of gems.

One benefit that this would bring is that interpreter implementations could all use the same stdlib. Currently jruby and rubinius are both lagging behind in that regard. This would reinforce an healthy competition between implementations, and make shipping several implems easier to support.

+1.

I mean, how hard is it to add gem install stdlib to end of make
install
routine? Where stdlib would be an (otherwise empty) meta-
package with dependencies to all standard gems.

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

I think it's time to accept that we are in an Internet world.

Hi,

Sometimes, ruby must be installed on systems who may not be connected to
internet (for security reasons for example).

Sylvain

#15 Updated by Eric Hodel over 2 years ago

On Oct 25, 2011, at 5:56 AM, Magnus Holm wrote:

That would require internet connection. I think you should be able to
install a complete version of Ruby without an internet connection.

The current proposal does not envision installing gems from rubygems.org or any other non-local source when you run make install.

You won't need an internet connection to build and install ruby, just the tarball. The source layout is exactly as it is now and gems are created at install time on your machine from the tarball through a script in tool/.

(At release time these gems are also uploaded to rubygems.org.)

#16 Updated by Eric Hodel over 2 years ago

On Oct 25, 2011, at 8:13 AM, Trans wrote:

I think it's time to accept that we are in an Internet world.

But we aren't.

There are many users of Ruby and RubyGems that can't access the internet from the computers they are using ruby on. The only way they can install ruby and gems on their machines is via a "sneaker net" of USB keys.

See:

http://help.rubygems.org/kb/rubygems/installing-gems-with-no-network

#17 Updated by Martin Bosslet over 2 years ago

I really like the proposal. Just wanted to add one thought - if we add yet more power to the gem infrastructure, we should probably revisit the idea of offering code-signed gems. The theory exists [1], we could build on that. This might be slightly off-topic, so I apologize in advance, but I still feel it is relevant, especially when gemifying stdlib, please let me explain why.

If the stdlib gets gemified we will have even more gem downloads than today. I'm not talking about the bundled gems that would be accessed locally when building Ruby for the first time (wrt to this proposal), rather about "updating" and adding gems to an existing installation. Updating will be a crucial task in production - people would certainly show interest in being able to stay up to date as fast as possible regarding security fixes, performance improvements etc.

Currently, these gem downloads are entirely unprotected, so just imagine a production deployment that connects to the outside world using a company proxy. The guy operating the company proxy could have a lot of fun serving "customized" gems that do all sorts of evil - nobody would notice. Serving the gems over an non-secure channel is definitely a risk. This might not seem such a big deal at first glance, but just think about discussing your architecture with a bank or other high security environments. Those people constantly give us the "<> <>" talk, so I could very well imagine that this is an issue when it comes to adopting Ruby in these environments. These people are often very conservative in their views, and even if they are still coding VB 6 GUIs and have no idea what they are talking about, still (in my own experience) they will readily accept signed Java applets... That's why I think offering signed gems would be helpful in sending a signal to those "enterprisy" clients, if not only to take away their arguments.

What we often find in these situations are SHA-1 checksums or similar features published on the download site. If the site is served over http this has effectively zero additional security. An attacker would simply man-in-the-middle the HTML of the site, too, handing you a different hash. In addition, even when sent over https, let's face it, nobody really compares these hashes manually.

I see two relatively simple options for mitigation: either secure the transport channel on the transport layer using TLS or secure the payload itself and therefore sign the gem. I prefer the second solution because it "persists" the security of the gem, the signature is tightly coupled with the gem itself and persists once downloaded, so that for example local gem cache servers would still be an option (they wouldn't be in the TLS case, unless effectively re-signing the gems).

Please note that a code signature will of course not guarantee that the code is free of bugs or not malicious. The only security it will provide is that one can be sure that the gem itself has not been modified after the holder of the code signing certificate applied the signature. Not more, not less. This narrows your trust decision down to asking yourself the question whether you want to trust the certificate holder to have validated the code to do what it is supposed to do. The person signing the gem is the last person who was able to alter the code. After applying the signature, the gem is effectively sealed. So you would have to base your trust on two things. Whether you trust this person to not have altered the code in malicious ways and whether you trust that person to have validated the code to ensure its proper functionality. Although far from perfect, in my eyes this is still a major improvement over having no guarantee at all.

A problem with code signing gems is that nobody is to keen on buying a "real" code signing certificate to sign their code, but on the other hand nobody would be keen on trusting a self-signed certificate either. This is probably one of the major issues why code signing is a rarely used feature. Now since gems are stored in a centralized repository, a solution could be to not require the authors to sign a gem but to sign a gem on the RubyGems server immediately after uploading, using a certificate that is exclusively issued to RubyGems. Solving the problem of trusting one dedicated certificate is easy, there are several options that would still need to be discussed, but it is certainly doable. However, this would put the burden of managing this certificate and securing access to it on the shoulders of the RubyGems maintainers.

I would also suggest to keep this entirely optional. To be honest, as with all things security, this will possibly affect usability, additional configuration will likely be necessary to use the feature. So you will have to assess your individual risks - you might very well come to the conclusion that a feature like this is overkill in your individual situation. You should of course still have the option to ignore this entirely, much like today.

I talked briefly about the idea with Eric Hodel and Hiroshi Nakamura. Eric told me that similar ideas in this direction exist. I would really appreciate to hear opinions on this. If you feel that this would be indeed a good option, I would suggest to open a new issue for further discussion.

[1] http://docs.rubygems.org/read/chapter/21

#18 Updated by Vit Ondruch over 2 years ago

Hiroshi Nakamura wrote:

  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now. % gem uninstall webrick -v 2.0.0.0 #=> should raise an error.

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

#19 Updated by Eric Hodel over 2 years ago

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

#20 Updated by Vit Ondruch over 2 years ago

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.

Why do you want to make the gems special? What do you want the users to prevent from? Provide easy tool to fix the situation, not to prevent it. If somebody uninstalls default Rake for example, what is the problem to install it again?

#21 Updated by Eric Hodel over 2 years ago

=begin
If net/http, fileutils, optparse, psych or zlib are released as gems uninstalling them breaks RubyGems and you will not be able to reinstall them without reinstalling ruby.

Removing such gems should be possible, I don't think it should be easy and I don't think RubyGems should go to extra effort to prevent it either.

Placing the gems in a separate path is sufficient as ((%gem uninstall%)) will only remove gems in ((|GEM_HOME|)) unless you use ((%-i%)).
=end

#22 Updated by Hiroshi Nakamura about 2 years ago

Just let it follow URL change.

#23 Updated by Vit Ondruch about 2 years ago

Eric Hodel wrote:

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

Actually if the default gems will be in some directory which is accessible using GEMPATH and it is not current GEMHOME, the user will not be able to uninstall the default gems. Problem solved. No new features are necessary. The only think which have to be taken in account are a /usr/bin loaders.

#24 Updated by Vit Ondruch about 2 years ago

Vit Ondruch wrote:

Eric Hodel wrote:

=begin

While I like your proposal, I really don't see point, why it should be disabled to uninstall the "default" gem. There might be "gem install default" command which restores the original gem version for example, but disabling uninstallation really doesn't make sense to me.

I don't think it needs to be absolutely disabled, but I do want to prevent foot-shooting.

If you want to do extra work to uninstall default gems that should be allowed, but it should not happen without extra effort by the user.
=end

Actually if the default gems will be in some directory which is accessible using GEMPATH and it is not current GEMHOME, the user will not be able to uninstall the default gems. Problem solved. No new features are necessary. The only think which have to be taken in account are a /usr/bin loaders.

And one more note. From Fedora's point of view, there is no difference between gems which originally comes with Ruby and the other Gems maintained by RPM. So it should be easy to merge the "default gems" with other "RPM managed gems". Our current operating_system.rb which sets-up the RubyGems directories can be found here [1]

[1] http://pkgs.fedoraproject.org/gitweb/?p=ruby.git;a=blob;f=operating_system.rb;hb=HEAD

#25 Updated by Hiroshi Nakamura almost 2 years ago

  • Assignee set to Hiroshi Nakamura

#26 Updated by Hiroshi Nakamura almost 2 years ago

Endoh-san, here's "slide-show" of this proposal.

#28 Updated by Yusuke Endoh almost 2 years ago

  • Status changed from Open to Assigned

Received, thank you.

I don't think this issue will be settled in a few minutes,
but it may be valuable enough to hear matz's view.

Yusuke Endoh mame@tsg.ne.jp

#29 Updated by Kouhei Sutou over 1 year ago

NaHi, could you tell us status of this work?
If you doesn't have a time to implement it, I'll implement it.

NOTE: I have another merit by this. If this feature is implemented, test-unit gem can be used without "gem 'test-unit'". So I'll implement it if nobody does it.

#30 Updated by Hiroshi Nakamura over 1 year ago

Kou, thanks for the cooperation. We would need your help soon.

kou (Kouhei Sutou) wrote:

NaHi, could you tell us status of this work?

I did the presentation at the last dev meeting and I think I/we need to clarify a part of the proposal for blessing from Matz. I guess Endoh-san didn't update this ticket yet? (No intention to blame. My bad, I should have asked this earlier.)

#31 Updated by Kouhei Sutou over 1 year ago

Thanks for your reply.

I did the presentation at the last dev meeting and I think I/we need to clarify a part of the proposal for blessing from Matz.
OK. I'll wait for the answer of it.

There is another question. Could you show a list of unimplemented features. (In other words TODO list.)
Is https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem#ToDo still valid? I heard another requirement from Eric. It is that "gem contents" should work for default gems. It is used by RDoc. So the list may be outdated.

#32 Updated by Vit Ondruch over 1 year ago

Is the plan to extract the gems out of StdLib and make from them proper gems again? I.e. there would be for example ruby/gems folder, with rake, rdoc, minitest and later the install script would install them into lets say ruby/default_gems? There seems to be some activity [1] which is not going in this way I am afraid.

[1] https://github.com/rubygems/rubygems/pull/377

#33 Updated by Usaku NAKAMURA over 1 year ago

  • Target version changed from Next Major to 2.0.0

This ticket is the center of attention at ruby-dev ML :)
NaHi-san will take a certain action in a few days, I guess.

#34 Updated by Vit Ondruch over 1 year ago

What actions exactly? Will the gems be gems or just another workaround? Can I see current state? How the result will look? Can I help somehow? Thank you.

#35 Updated by Vit Ondruch over 1 year ago

Will it address concerns from #6124 as well?

#36 Updated by Yusuke Endoh over 1 year ago

  • Target version changed from 2.0.0 to next minor

We discussed this request at the developers' meeting (Jul. 21),
and it was not accepted.

NaHi (the proposer) presented this request by himself. He said:

Because of "fake gems", the new files of a stdlib installed by
"gem update" are ignored unless a user writes gem "json"
explicitly. To address this issue, stdlibs must be provided
as normal gems, not "fake gems".

And the conclusion is as follows:

  • The current behavior of fake gems is considered "bug", so should be fixed. But it is not a good reason to make stdlib real gems.
  • If it is hard or even impossible to fix "the bug", NaHi should re-propose it.

After that, NaHi said, "fixing the behavior" effectively means
"making stdlib as normal gems", but I don't know the meaning.
Please wait for his words.

Yusuke Endoh mame@tsg.ne.jp

#37 Updated by Vit Ondruch over 1 year ago

Thank you for clarification, waiting for NaHi and offering my help.

#38 Updated by Vit Ondruch over 1 year ago

Hi, I'd like to question once more some parts of the proposal in wiki [1].

  • After installing updated stdlib gems, those should be treated as regular gems.

The only viable solution is proposal 1. It seems that there was some work started to implement 2, but that is really wrong approach. What would be the purpose of --disable-gems then? You disable gems but they will be loaded to load some default gems? That doesn't make sense to me.

Moreover, if you have reason to use --disable-gems option, you will be probably able to use -I to setup your load paths to load the gems manually.

  • Introduce a new mechanism: controlling supported ruby version

There is already mechanism in RubyGems, i.e. you can specify version of Ruby which is supported by the gem, not sure why there should be anything more. Since the StdLib gems would go into independent directory and they are uninstallable, there can't be installed any "unexpected version of stdlib gems". If this should solve that there might be installed to new gem, then I guess Ruby developers are already used to this. So not sure what would this improve.

  • Newly created stdlib gems version scheme: ruby's version + '.n'(dot plus a number)

Why WEBrick should have this kind of versioning? What if there was no change in WEBrick in between Ruby 2.0 and, 2.1? Why the version should be changed? Every gem should have independent versioning, as maintainer decides. Bind the gems versioning to Ruby versions makes no sense and it is not beneficial.

  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now.

Yes, since currently, the default gems are mixed with regular gems, they are uninstallable. This would be non issue ff they are in different path.

  • And one more note from Fedora's Ruby maintainer point of view

It has to be possible to unbundle the default gems out of Ruby and replace them with newer versions if needed. This is similar requirement as Lucas Nussbaum pointed our earlier in this discussion. There are also different reasons, such as for example, Ruby is dependency of VIM in Fedora and VIM users really don't want to install RDoc, Minitest, Rake etc, just because they want to use VIM.

Please note that we are already doing so in Fedora and there are no issues, except #6124

[1] https://bugs.ruby-lang.org/projects/ruby/wiki/StdlibGem
[2] https://github.com/rubygems/rubygems/pull/377#issuecomment-9735417

#39 Updated by Charles Nutter over 1 year ago

FWIW, here's how we're handling the gemification of OpenSSL...

Background: Previously, JRuby's 'openssl' support was always distributed as a gem, since we did not want to ship crypto (bouncycastle). Starting with JRuby 1.7, however, we ship 'openssl' in the standard JRuby distribution. We still want to support users on 1.6.x and want to be able to update 'openssl' independent of JRuby, so for Jruby 1.7.1 we will support using the gem if the gem is installed.

The file within JRuby is here: https://github.com/jruby/jruby/blob/master/lib/ruby/1.9/openssl.rb

This file attempts to load a file that's only in the gem, and the gem then adds its own dirs to LOAD_PATH so future requires will use the gem's versions of things instead of those built into JRuby.

This approach may advise a similar solution in MRI. At the very least, I'm interested in this bug so we use the same mechanism for loading the gem version of a stdlib instead of the shipped version.

I am disappointed this won't happen for 2.0.0.

#40 Updated by Vit Ondruch over 1 year ago

=begin
Hi,

I played around a bit an put together a few patches. You can find it here: https://github.com/voxik/ruby/commits/gemified-stdlib . I took Rake as an example.

In a nutshell:

(1) I created the gems/ subdirectory in Ruby repository.
(2) I moved the Rake bits into gems/rake/
(3) I added .gemspec for Rake
(4) I modified rbinstall.rb to install each gem found in gems subdirectory into standard StdLib gem location
(5) I modified RubyGems to look into StdLib gem location for gems.

What is currently missing:

(1) The test suite is not passing currently, but I think that the Rake's test suite should be moved into the gems/rake/ subdirectory as well and there should be some mechanism, how to execute test suites of gems.
(2) If there would be some binary gem, its binary extension is not build ATM. However, I would say that similar approach used to build current exts could be used.

This approach has significant advantages IMO:

(1) Rake gem currently reflect the upstream repository and as soon as the tests are moved into the gems/rake/ subfolder, it will be almost exact match. It could be easily replaced by something like svn:externals or git submodule. This would greatly improve the maintainability of the code and reduced the possibility of forking.
(2) This would help alternative Ruby implementations to ship their additional libraries in consistent manner.
(3) Gems could be easily replaced by downstream distributions by different versions, if needed.

Any comments? Would this approach be acceptable? Should I try to spend more time polishing the test suite and binary extension's build?
=end

#41 Updated by Thomas Sawyer over 1 year ago

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

#42 Updated by Vit Ondruch over 1 year ago

trans (Thomas Sawyer) wrote:

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

Although I'd love to see Ruby shipping .gem files, the issue is that for example RubyGems needs Psych gem to work properly. So it is chicken/egg problem.

#43 Updated by Sylvain Daubert over 1 year ago

trans (Thomas Sawyer) wrote:

Why keep the source code in the Ruby repository at all? Development occurs at the library's repository anyway, so why waste time with copying the files into Ruby repository for each release. Instead, just ship the .gem package with Ruby which can be installed via RubyGems (via rbinstall.rb) per the normal means of installing a gem.

A full bundle may be easier installed when no internet connection is available (which often happens to me).

#44 Updated by Thomas Sawyer over 1 year ago

@sdaubert That shouldn't be a problem in this case. All the necessary gems would be distributed with the Ruby source (it would sort of act like it's own miniature gem server, figuratively speaking).

#45 Updated by Thomas Sawyer over 1 year ago

@vo.x Psych would have to be an exception. And really, with the inclusion of libyaml, YAML is becoming more and more an integral aspect of Ruby.

#46 Updated by Vit Ondruch over 1 year ago

@trans You know how it is with exceptions. They tend to become rules and justification for another exceptions. I believe that my proposal is the best solution, because it will be almost like shipping already preinstalled gems, which you would need to rebuild anyway. Packaged gems do not give you any additional benefit.

Anyway, I am not decision maker unfortunately. I just trying rectify current situation and prevent worser solutions by being proactive :)

#47 Updated by Thomas Sawyer over 1 year ago

@vo.x Don't count your eggs before they hatch ;) I think the exceptions will be pretty well defined, basically anything RubyGems requires --isn't that going to be the case regardless of which way it's done? I thought packaged gems would be a benefit b/c one would not have to keep source files in sync with library repos, nor make particularly exceptional code for installing "fake" gems vs. real ones.

Understood. I'm just trying to flesh out this approach so whomever makes the decision has it for consideration.

#48 Updated by Thomas Sawyer over 1 year ago

What is the status of this? I had a couple of thoughts/questions:

  • I was wondering how standard lib like ostruct.rb fits into this. I did not notice it mentioned. Would it be a gem? Or are some standard libs staying "old-fashioned"?

  • RubyGems dependencies seem to create a cache-22, as quoted "net/http, fileutils, optparse, psych or zlib are released as gems uninstalling them breaks RubyGems and you will not be able to reinstall them without reinstalling ruby." This makes these libs kind of quasi-core. Maybe an effort should be made to try to reduce RubyGems dependencies on these in so far as it is possible, and whatever is not possible should just be accepted as core?

  • It still seems to me that maintenance of this would be immensely relieved if Ruby just bundled the set of standard gem packages, rather then redistribute copied source.

#49 Updated by Vit Ondruch about 1 year ago

BTW there should be process in place which ensures, that the libraries are properly versioned, i.e. if something is changed, it cannot have the same version as already released library (see #7761 and #7762).

Also available in: Atom PDF