Bug #9106

'gem install' doesn't copy .so files of ext libs

Added by Satoshi TAGOMORI 5 months ago. Updated 5 months ago.

[ruby-core:58306]
Status:Closed
Priority:Normal
Assignee:Eric Hodel
Category:-
Target version:2.1.0
ruby -v:ruby 2.1.0dev (2013-11-12 trunk 43653) [x86_64-linux] Backport:1.9.3: DONTNEED, 2.0.0: DONTNEED

Description

'gem install' should copy files of 'ext/*/.so' of each installed gems, but doesn't.

$ ruby -v
ruby 2.1.0dev (2013-11-12 trunk 43653) [x86_64-linux]
$ gem install msgpack
Fetching: msgpack-0.5.7.gem (100%)
Building native extensions. This could take a while...
Successfully installed msgpack-0.5.7

[]

Parsing documentation for msgpack-0.5.7
Installing ri documentation for msgpack-0.5.7
Done installing documentation for msgpack after 0 seconds
1 gem installed
$ ruby -rmsgpack -e ''
/home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/coreext/kernelrequire.rb:55:in require': cannot load such file -- /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack/msgpack (LoadError)
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in
require'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack.rb:8:in rescue in <top (required)>'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack.rb:3:in
'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/coreext/kernelrequire.rb:135:in require'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in
rescue in require'
from /home/user/local/ruby-2.1.0-trunk/lib/ruby/2.1.0/rubygems/coreext/kernelrequire.rb:144:in `require'
$ ls -l local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/lib/msgpack
total 4
-rw-r--r-- 1 edge-dev edge 42 Nov 13 15:43 version.rb
$ ls -l local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/ext/msgpack/msgpack.so
-rwxr-xr-x 1 edge-dev edge 1390955 Nov 13 15:43 local/ruby-2.1.0-trunk/lib/ruby/gems/2.1.0/gems/msgpack-0.5.7/ext/msgpack/msgpack.so

Associated revisions

Revision 43714
Added by Eric Hodel 5 months ago

  • lib/rubygems: Update to RubyGems master 6a3d9f9. Changes include:

    Compatibly renamed Gem::DependencyResolver to Gem::Resolver.

    Added support for git gems in gem.deps.rb and Gemfile.

    Fixed resolver bugs.

  • test/rubygems: ditto.

  • lib/rubygems/LICENSE.txt: Updated to license from RubyGems trunk.
    [ruby-trunk - Bug #9086]

  • lib/rubygems/commands/which_command.rb: RubyGems now indicates
    failure when any file is missing. [ruby-trunk - Bug #9004]

  • lib/rubygems/ext/builder: Extensions are now installed into the
    extension install directory and the first directory in the require
    path from the gem. This allows backwards compatibility with msgpack
    and other gems that calculate full require paths.
    [ruby-trunk - Bug #9106]

Revision 43953
Added by Eric Hodel 5 months ago

  • lib/rubygems: Update to RubyGems master baa965b. Notable changes:

    Copy directories to lib/ when installing extensions. This completes
    the fix for [ruby-trunk - Bug #9106]

  • test/rubygems: ditto.

History

#1 Updated by Shota Fukumori 5 months ago

  • Status changed from Open to Assigned
  • Assignee set to Eric Hodel

Eric, could you handle this?

#2 Updated by Eric Hodel 5 months ago

  • Priority changed from Immediate to Normal

Yes, I will investigate.

#3 Updated by Eric Hodel 5 months ago

=begin
Ugh. This is a misfeature of msgpack, it should trust that RubyGems will do the right thing:

here = File.expandpath(File.dirname(FILE))
require File.join(here, 'msgpack', 'version')
begin
m = /(\d+.\d+)/.match(RUBY
VERSION)
ver = m[1]
require File.join(here, 'msgpack', ver, 'msgpack')
rescue LoadError
require File.join(here, 'msgpack', 'msgpack')
end

=end

#4 Updated by Hiroshi SHIBATA 5 months ago

  • Status changed from Assigned to Feedback

tagomoris: Thank you for your report. Can you investigate to msgpack gem?

#5 Updated by Eric Hodel 5 months ago

  • Status changed from Feedback to Assigned
  • Backport changed from 1.9.3: UNKNOWN, 2.0.0: UNKNOWN to 1.9.3: DONTNEED, 2.0.0: DONTNEED

For backwards compatibility I will need to alter RubyGems before Ruby 2.1 (I haven't had time yet).

Future versions of RubyGems will not support this lack of trust, though.

#6 Updated by Satoshi TAGOMORI 5 months ago

hsbt (Hiroshi SHIBATA) wrote:

tagomoris: Thank you for your report. Can you investigate to msgpack gem?

I don't understand why msgpack requires with such code. So I'll tell it to @frsyuki, original developer of msgpack.

#7 Updated by Sadayuki Furuhashi 5 months ago

Hi I'm @frsyuki.
1) I don't understand which behavior of RubyGems should the code trust. It assumes that msgpack.so is placed into lib/msgpack/msgpack.so because ext/msgpack/extconf.rb includes this line:
create_makefile('msgpack/msgpack')

2) Which fix is appropriate?:
a) require File.join(here, '..', 'ext', 'msgpack', 'msgpack')
b) add "ext" directory to require_paths in gemspec
c) require "msgpack/msgpack"
d) require "msgpack#{RbConfig::CONFIG["DLEXT"]}"

#8 Updated by Eric Hodel 5 months ago

=begin
@frsyuki: you should trust rubygems to make msgpack/msgpack.so accessible somewhere in the $LOAD_PATH so:

require 'msgpack/msgpack'

so option c is the best fix.

With RubyGems 2.2 the directory you gem install to can be shared across multiple ruby versions because the extensions end up in $GEMHOME/extensions///. RubyGems will add that directory to $LOADPATH so require can find files in it.

For msgpack, on OS X with ruby 2.1.0 this would be:

$GEMHOME/extensions/x8664-darwin-12/2.0.0/msgpack-0.5.7/msgpack/msgpack.bundle

I will figure out a backward compatible solution so older msgpack will still work for RubyGems 2.x. There is no reason for you to rush out a new release of msgpack.
=end

#9 Updated by Sadayuki Furuhashi 5 months ago

It sounds good improvement of RubyGems to manage binaries for each ruby API versions.
Your comment totally makes sense. I'll fix msgpack next version. Thank you!

#10 Updated by Luis Lavena 5 months ago

Hello,

Here is a 2009 discussion about gems and multiple binaries (targeting different versions of Ruby):

http://rubyforge.org/pipermail/rubygems-developers/2009-April/004522.html

My alternative to that issue is what you can see in projects like sqlite3-ruby:

https://github.com/sparklemotion/sqlite3-ruby/blob/master/lib/sqlite3.rb#L1-L7

And others that follow the guidelines of rake-compiler to deal with these multiple binaries in a single gem.

Of course, this is to deal with gems that package binaries, gems that are compiled on installation, we let RubyGems do its job.

#11 Updated by Eric Hodel 5 months ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r43714.
Satoshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • lib/rubygems: Update to RubyGems master 6a3d9f9. Changes include:

    Compatibly renamed Gem::DependencyResolver to Gem::Resolver.

    Added support for git gems in gem.deps.rb and Gemfile.

    Fixed resolver bugs.

  • test/rubygems: ditto.

  • lib/rubygems/LICENSE.txt: Updated to license from RubyGems trunk.
    [ruby-trunk - Bug #9086]

  • lib/rubygems/commands/which_command.rb: RubyGems now indicates
    failure when any file is missing. [ruby-trunk - Bug #9004]

  • lib/rubygems/ext/builder: Extensions are now installed into the
    extension install directory and the first directory in the require
    path from the gem. This allows backwards compatibility with msgpack
    and other gems that calculate full require paths.
    [ruby-trunk - Bug #9106]

#12 Updated by Satoshi TAGOMORI 5 months ago

This problem is reproduced on ruby 2.1.0-preview2 on Linux/OSX.
'msgpack' gem cannot be loaded (LoadError) and 'ext/msgpack/msgpack.so' (or .bundle) is not copied to 'lib/msgpack/msgpack.so'.
About other library (ex: cool.io), binary files are copied to 'lib' directory.

P.S. I cannot reopen this ticket.

#13 Updated by Tomoyuki Chikanaga 5 months ago

  • Status changed from Closed to Assigned

#14 Updated by Eric Hodel 5 months ago

Hrm, I will check it.

#15 Updated by Eric Hodel 5 months ago

I found my bug!

I didn't recursively copy the files.

Fix soon!

#16 Updated by Eric Hodel 5 months ago

  • Status changed from Assigned to Closed

This issue was solved with changeset r43953.
Satoshi, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • lib/rubygems: Update to RubyGems master baa965b. Notable changes:

    Copy directories to lib/ when installing extensions. This completes
    the fix for [ruby-trunk - Bug #9106]

  • test/rubygems: ditto.

Also available in: Atom PDF