Bug #9106: 'gem install' doesn't copy .so files of ext libs

• Status: Closed
• Assignee: drbrain (Eric Hodel)
• Target version: 2.1.0
• ruby -v: ruby 2.1.0dev (2013-11-12 trunk 43653) [x86_64-linux]
• Backport: 1.9.3: DONTNEED, 2.0.0: DONTNEED

[tagomoris (Satoshi Tagomori)]

'gem install' should copy files of 'ext/**/*.so' of each installed gems, but doesn't.

$ ruby -v
ruby 2.1.0dev (2013-11-12 trunk 43653) [x86_64-linux]
$ gem install msgpack
Fetching: msgpack-0.5.7.gem (100%)
Building native extensions. This could take a while...
Successfully installed msgpack-0.5.7

[]

Parsing documentation for msgpack-0.5.7
Installing ri documentation for msgpack-0.5.7
Done installing documentation for msgpack after 0 seconds
1 gem installed
$ ruby -rmsgpack -e ''
/home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in require': cannot load such file -- /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack/msgpack (LoadError) from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in require'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack.rb:8:in rescue in <top (required)>' from /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack.rb:3:in <top (required)>'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in require' from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in rescue in require'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:144:in `require'
$ ls -l local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack
total 4
-rw-r--r-- 1 edge-dev edge 42 Nov 13 15:43 version.rb
$ ls -l local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/ext/msgpack/msgpack.so
-rwxr-xr-x 1 edge-dev edge 1390955 Nov 13 15:43 local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/ext/msgpack/msgpack.so

[sorah (Sorah Fukumori)]

• Status changed from Open to Assigned
• Assignee set to drbrain (Eric Hodel)

Eric, could you handle this?

[drbrain (Eric Hodel)]

• Priority changed from 7 to Normal

Yes, I will investigate.

[drbrain (Eric Hodel)]

=begin
Ugh. This is a misfeature of msgpack, it should trust that RubyGems will do the right thing:

here = File.expand_path(File.dirname(FILE))
require File.join(here, 'msgpack', 'version')
begin
m = /(\d+.\d+)/.match(RUBY_VERSION)
ver = m[1]
require File.join(here, 'msgpack', ver, 'msgpack')
rescue LoadError
require File.join(here, 'msgpack', 'msgpack')
end

=end

[hsbt (Hiroshi SHIBATA)]

• Status changed from Assigned to Feedback

tagomoris: Thank you for your report. Can you investigate to msgpack gem?

[drbrain (Eric Hodel)]

• Status changed from Feedback to Assigned
• Backport changed from 1.9.3: UNKNOWN, 2.0.0: UNKNOWN to 1.9.3: DONTNEED, 2.0.0: DONTNEED

For backwards compatibility I will need to alter RubyGems before Ruby 2.1 (I haven't had time yet).

Future versions of RubyGems will not support this lack of trust, though.

[tagomoris (Satoshi Tagomori)]

hsbt (Hiroshi SHIBATA) wrote:

tagomoris: Thank you for your report. Can you investigate to msgpack gem?

I don't understand why msgpack requires with such code. So I'll tell it to @frsyuki (Sadayuki Furuhashi), original developer of msgpack.

[frsyuki (Sadayuki Furuhashi)]

Hi I'm @frsyuki (Sadayuki Furuhashi).

1. I don't understand which behavior of RubyGems should the code trust. It assumes that msgpack.so is placed into lib/msgpack/msgpack.so because ext/msgpack/extconf.rb includes this line:
   create_makefile('msgpack/msgpack')

2. Which fix is appropriate?:
   a) require File.join(here, '..', 'ext', 'msgpack', 'msgpack')
   b) add "ext" directory to require_paths in gemspec
   c) require "msgpack/msgpack"
   d) require "msgpack#{RbConfig::CONFIG["DLEXT"]}"

[drbrain (Eric Hodel)]

=begin
@frsyuki (Sadayuki Furuhashi): you should trust rubygems to make msgpack/msgpack.so accessible somewhere in the $LOAD_PATH so:

require 'msgpack/msgpack'

so option c is the best fix.

With RubyGems 2.2 the directory you gem install to can be shared across multiple ruby versions because the extensions end up in $GEM_HOME/extensions///. RubyGems will add that directory to $LOAD_PATH so require can find files in it.

For msgpack, on OS X with ruby 2.1.0 this would be:

$GEM_HOME/extensions/x86_64-darwin-12/2.0.0/msgpack-0.5.7/msgpack/msgpack.bundle

I will figure out a backward compatible solution so older msgpack will still work for RubyGems 2.x. There is no reason for you to rush out a new release of msgpack.
=end

[frsyuki (Sadayuki Furuhashi)]

It sounds good improvement of RubyGems to manage binaries for each ruby API versions.
Your comment totally makes sense. I'll fix msgpack next version. Thank you!

[luislavena (Luis Lavena)]

Hello,

Here is a 2009 discussion about gems and multiple binaries (targeting different versions of Ruby):

http://rubyforge.org/pipermail/rubygems-developers/2009-April/004522.html

My alternative to that issue is what you can see in projects like sqlite3-ruby:

https://github.com/sparklemotion/sqlite3-ruby/blob/master/lib/sqlite3.rb#L1-L7

And others that follow the guidelines of rake-compiler to deal with these multiple binaries in a single gem.

Of course, this is to deal with gems that package binaries, gems that are compiled on installation, we let RubyGems do its job.

[drbrain (Eric Hodel)]

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r43714.
Satoshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• lib/rubygems: Update to RubyGems master 6a3d9f9. Changes include:

  Compatibly renamed Gem::DependencyResolver to Gem::Resolver.

  Added support for git gems in gem.deps.rb and Gemfile.

  Fixed resolver bugs.

• test/rubygems: ditto.

• lib/rubygems/LICENSE.txt: Updated to license from RubyGems trunk.
  [ruby-trunk - Bug #9086]

• lib/rubygems/commands/which_command.rb: RubyGems now indicates
  failure when any file is missing. [ruby-trunk - Bug #9004]

• lib/rubygems/ext/builder: Extensions are now installed into the
  extension install directory and the first directory in the require
  path from the gem. This allows backwards compatibility with msgpack
  and other gems that calculate full require paths.
  [ruby-trunk - Bug #9106]

[tagomoris (Satoshi Tagomori)]

This problem is reproduced on ruby 2.1.0-preview2 on Linux/OSX.
'msgpack' gem cannot be loaded (LoadError) and 'ext/msgpack/msgpack.so' (or .bundle) is not copied to 'lib/msgpack/msgpack.so'.
About other library (ex: cool.io), binary files are copied to 'lib' directory.

P.S. I cannot reopen this ticket.

[nagachika (Tomoyuki Chikanaga)]

• Status changed from Closed to Assigned

[drbrain (Eric Hodel)]

Hrm, I will check it.

[drbrain (Eric Hodel)]

I found my bug!

I didn't recursively copy the files.

Fix soon!

[drbrain (Eric Hodel)]

• Status changed from Assigned to Closed

This issue was solved with changeset r43953.
Satoshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• lib/rubygems: Update to RubyGems master baa965b. Notable changes:

  Copy directories to lib/ when installing extensions. This completes
  the fix for [ruby-trunk - Bug #9106]

• test/rubygems: ditto.