Bug #7698

RubyGems 2.0 has an incompatibility about installation of extension libraries

Added by Kenta Murata over 1 year ago. Updated 10 months ago.

[ruby-core:51437]
Status:Closed
Priority:High
Assignee:Nobuyoshi Nakada
Category:lib
Target version:2.1.0
ruby -v:ruby 2.0.0dev (2013-01-14 trunk 38812) [x86_64-linux] Backport:

Description

The current rubygem included in ruby-head has an incompatibility about installation of extension libraries.
This incompatibility makes make install to be failed because the current gem doesn't separate build, source and install directories.

This is reported as a bug of bigdecimal's gemspec in #7344 firstly, but Tadashi Saito pointed out it also can be reproduced in other gems including extension libraries, such as decimal.gem.
I think this incompatibility will make gem-developers to be in trouble after ruby-2.0 is released.
So this should be resolved before ruby-2.0 is released.

0001-ext_conf_builder.rb-build-in-separate-dir.patch Magnifier (4.03 KB) Nobuyoshi Nakada, 02/27/2013 06:08 PM


Related issues

Related to ruby-trunk - Bug #7344: gem pristine bigdecimal が失敗してしまう Closed 11/13/2012
Related to ruby-trunk - Bug #6904: make -j all fails (sometimes) Closed 08/22/2012
Related to Backport200 - Backport #7991: backport r39542 Closed 03/01/2013
Duplicated by ruby-trunk - Bug #7971: bson_ext doesn't install correctly in Ruby 2.0.0-p0 Closed 02/26/2013
Duplicated by ruby-trunk - Bug #7949: Gem::Ext::Builder removes dependency to directory creatio... Closed 02/25/2013

Associated revisions

Revision 38864
Added by Kouhei Sutou over 1 year ago

  • lib/rubygems/ext/builder.rb (Gem::Ext::Builder.make): Remove .time dependency from *.rb install target. It causes needless *.rb install. [Bug #7698] Reported by Tadashi Saito. Thanks!!!
  • test/rubygems/testgeminstaller.rb (TestGemInstaller#testinstallextensionandscript): Add a test for the above change.

Revision 39558
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: build in the source directory

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): revert use of temporary directory for build, to work some buggy extconf.rb which cannot build outside the source directory. [Bug #7698]

Revision 39559
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: use RUBYOPT

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): use RUBYOPT instead of -r option, and revert some tests. [Bug #7698]

Revision 39560
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: hack for obsolete sytle gems

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.hackforobsoletesytlegems): remove circular dependencies for old style gems which locate extconf.rb on the toplevel. [ruby-trunk - Bug #7698]

Revision 39572
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: remove circular dependency in install-so

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.hackforobsoletestylegems): remove circular dependencies in install-so too. [Bug #7698]

Revision 39579
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: install via temporary directory

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): fix for unusal cases again. install to a temporary directory once and move instaled files to the destination directory, if it is same as the current directory. [Bug #7698]

Revision 39590
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: clear DESTDIR

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): clear DESTDIR so RUBYARCHDIR and RUBYLIBDIR are not be overrdden. [Bug #7698]

Revision 39592
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: clear DESTDIR

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): clear DESTDIR so RUBYARCHDIR and RUBYLIBDIR are not be overrdden. [Bug #7698]

Revision 39696
Added by Nobuyoshi Nakada about 1 year ago

extconfbuilder.rb: use intermediate destdir always

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): it is impossible to predict which file will be installed to where, by the arguments, so use intermediate destination directory always. [Bug #7698]

History

#1 Updated by Kenta Murata over 1 year ago

Sorry, I made misspelling.

This incompatibility makes make install to be failed because the current gem doesn't separate build, source and install directories.

This incompatibility makes make install to be failed because the current rubygem doesn't separate build, source and install directories.

#2 Updated by Shota Fukumori over 1 year ago

Translating from , Tadashi said:

We can avoid this bug by changing directory tree in gems.
But, Current status means any gems may be not installable after 2.0.0, by incompatibility in current rubygem bundled in 2.0.0.

So, I propose the following solutions:

  1. Revert that incompatibility before 2.0.0 release or 2.0.0's first patch level release to make such gems installable.
  2. If you cannot, please review all gems that installable in 2.0.0 and notify to authors if a gem can't be installed in 2.0.0. (This may be rubygems.org's task)
  3. Announce that like "Your gem may be not installable in 2.0.0" to all gem authors.

I think solution (1) is the best. (2) seems high-cost.
(3) is the worst way for me, because I've not seen an announcement like such,
and seems too late to announce.

#3 Updated by Kenta Murata over 1 year ago

Thank you translation, sorah!

#4 Updated by Kouhei Sutou over 1 year ago

This can be fixed by the following patch. This is caused by #6904 change. Should this problem be fixed in RubyGems?

Index: lib/rubygems/ext/builder.rb

--- lib/rubygems/ext/builder.rb (revision 38833)
+++ lib/rubygems/ext/builder.rb (working copy)
@@ -19,6 +19,7 @@
mf = Gem.readbinary 'Makefile'
mf = mf.gsub(/RUBYARCHDIR\s=\s\$[$]*/, "RUBYARCHDIR = #{dest
path}")
mf = mf.gsub(/RUBYLIBDIR\s=\s\$[$]*/, "RUBYLIBDIR = #{dest_path}")
+ mf = mf.gsub(/\s*\S+.time$/, "")

 File.open('Makefile', 'wb') {|f| f.print mf}

#5 Updated by Eric Hodel over 1 year ago

I am unfamiliar with Makefiles, so what does this patch change?

Also, .time is new in ruby 2.0, what effect does this have on ruby 1.9 and ruby 1.8?

I trust your judgement in adding this to RubyGems. If you commit it to Ruby please commit it to RubyGems and let me know!

#6 Updated by Kouhei Sutou over 1 year ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

This issue was solved with changeset r38864.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


  • lib/rubygems/ext/builder.rb (Gem::Ext::Builder.make): Remove .time dependency from *.rb install target. It causes needless *.rb install. [Bug #7698] Reported by Tadashi Saito. Thanks!!!
  • test/rubygems/testgeminstaller.rb (TestGemInstaller#testinstallextensionandscript): Add a test for the above change.

#7 Updated by Kouhei Sutou over 1 year ago

drbrain (Eric Hodel) wrote:

Also, .time is new in ruby 2.0, what effect does this have on ruby 1.9 and ruby 1.8?

This change doesn't have effect on 1.9 and 1.8 because Makefile on 1.9 and 1.8 doesn't have .time. So the gsub is just ignored.

I trust your judgement in adding this to RubyGems. If you commit it to Ruby please commit it to RubyGems and let me know!

Thanks. :-)
I've committed this change to Ruby and RubyGems with a test case.
Detail is written in commit message of the commit at the RubyGems repository: https://github.com/rubygems/rubygems/commit/a7cd1be541f2194fc8c1436d86bc05eb3a94500d

#8 Updated by Yusuke Endoh about 1 year ago

  • Status changed from Closed to Assigned
  • Assignee changed from Kouhei Sutou to Eric Hodel

Hello,

This issue seems not to be fixed completely.
"gem install rbtree" still hit this issue.
It reproduces on both trunk and 2.0.0-p0.

$ gem install rbtree
Building native extensions. This could take a while...
ERROR: Error installing rbtree:
ERROR: Failed to build gem native extension.

/home/mame/local/bin/ruby extconf.rb

checking for ruby/st.h... yes
checking for rbexecrecursive() in ruby.h... yes
creating Makefile

make
compiling rbtree.c
compiling dict.c
linking shared-object rbtree.so

make install
/usr/bin/install -c -m 0755 rbtree.so /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/.
/usr/bin/install: rbtree.so' and/home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/./rbtree.so' are the same file
make: *** [install-so] Error 1

Gem files will remain installed in /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0 for inspection.
Results logged to /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/./gem_make.out

Yusuke Endoh mame@tsg.ne.jp

#9 Updated by Nobuyoshi Nakada about 1 year ago

  • File 0001-ext_conf_builder.rb-build-in-separate-dir.patch added

It's not good idea to mess up the source directory.

#10 Updated by Nobuyoshi Nakada about 1 year ago

I've forgot to fix a few tests.

#11 Updated by Nobuyoshi Nakada about 1 year ago

  • File deleted (0001-ext_conf_builder.rb-build-in-separate-dir.patch)

#13 Updated by Eric Hodel about 1 year ago

Committed r39542. Thank you nobu for the RubyGems patch!

#14 Updated by Eric Hodel about 1 year ago

  • Assignee changed from Eric Hodel to Nobuyoshi Nakada
  • % Done changed from 100 to 50

=begin
This patch causes the rdiscount extension to fail to install since it reads a local file:

https://github.com/rtomayko/rdiscount/blob/master/ext/extconf.rb#L18

The exception:

Building native extensions. This could take a while...
/usr/local/bin/ruby -r./siteconf /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb
checking for random()... yes
checking for srandom()... yes
checking for rand()... yes
checking for srand()... yes
checking size of unsigned long... 8
checking size of unsigned int... 4
checking size of unsigned int... 4
*** /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=/Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext
--curdir
--ruby=/usr/local/bin/ruby
--with-rdiscount-dir
--without-rdiscount-dir
--with-rdiscount-include
--without-rdiscount-include=${rdiscount-dir}/include
--with-rdiscount-lib
--without-rdiscount-lib=${rdiscount-dir}/
/Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb:18:in read': No such file or directory - VERSION (Errno::ENOENT)
from /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb:18:in
'
ERROR: Error installing rdiscount:
ERROR: Failed to build gem native extension.

  Building has failed. See above output for more information on the failure.

Gem files will remain installed in /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1 for inspection.

=end

#15 Updated by Yui NARUSE about 1 year ago

It also breaks a test.
http://u64.rubyci.org/~chkbuild/ruby-trunk/log/20130301T010202Z.log.html.gz

8) Error:
testinstallextensionandscript(TestGemInstaller):
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/home/chkbuild/build/20130301T010202Z/ruby/ruby -r./siteconf /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/extconf.rb

creating Makefile

make
make[1]: Entering directory /home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'
make[1]: Nothing to be done for
all'.
make[1]: Leaving directory `/home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'

make install
make[1]: Entering directory /home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'
installing default libraries
make[1]: Circular /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb <- /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb dependency dropped.
/usr/bin/install:
/home/chkbuild/build/20130301T010202Z/tmp/testrubygems11578/gemhome/gems/a-2/lib/a.rb' and /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb' are the same file
make[1]: *** [/home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb] Error 1
make[1]: Leaving directory
/home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'

Gem files will remain installed in /home/chkbuild/build/20130301T010202Z/tmp/testrubygems11578/gemhome/gems/a-2 for inspection.
Results logged to /home/chkbuild/build/20130301T010202Z/tmp/testrubygems11578/gemhome/gems/a-2/./gem_make.out

/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:55:in `run'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:28:in `block in make'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:26:in `each'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:26:in `make'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:33:in `block (2 levels) in build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:20:in `chdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:20:in `block in build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/tmpdir.rb:88:in `mktmpdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:19:in `build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:678:in `block (2 levels) in build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:677:in `chdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:677:in `block in build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:652:in `each'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:652:in `build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:218:in `install'
/home/chkbuild/build/20130301T010202Z/ruby/test/rubygems/test_gem_installer.rb:1046:in `block in test_install_extension_and_script'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/user_interaction.rb:40:in `use_ui'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/user_interaction.rb:63:in `use_ui'
/home/chkbuild/build/20130301T010202Z/ruby/test/rubygems/test_gem_installer.rb:1042:in `test_install_extension_and_script'

#16 Updated by Eric Hodel about 1 year ago

What make do you have?

I have GNU Make 3.81 and cannot reproduce.

#17 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

This issue was solved with changeset r39558.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


extconfbuilder.rb: build in the source directory

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): revert use of temporary directory for build, to work some buggy extconf.rb which cannot build outside the source directory. [Bug #7698]

#18 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Closed to Assigned
  • Target version changed from 2.0.0 to 2.1.0
  • % Done changed from 100 to 50

#19 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

This issue was solved with changeset r39559.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


extconfbuilder.rb: use RUBYOPT

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.build): use RUBYOPT instead of -r option, and revert some tests. [Bug #7698]

#20 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Closed to Assigned
  • % Done changed from 100 to 50

#21 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

This issue was solved with changeset r39560.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


extconfbuilder.rb: hack for obsolete sytle gems

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.hackforobsoletesytlegems): remove circular dependencies for old style gems which locate extconf.rb on the toplevel. [ruby-trunk - Bug #7698]

#22 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Closed to Assigned
  • % Done changed from 100 to 50

#23 Updated by Nobuyoshi Nakada about 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

This issue was solved with changeset r39572.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.


extconfbuilder.rb: remove circular dependency in install-so

  • lib/rubygems/ext/extconfbuilder.rb (Gem::Ext::ExtConfBuilder.hackforobsoletestylegems): remove circular dependencies in install-so too. [Bug #7698]

#24 Updated by Zachary Scott 10 months ago

I can still reproduce this on RubyGems 2.0.3 with trunk:
ruby 2.1.0dev (2013-06-16 trunk 41328) [x86_64-darwin12.3.0]

To repro: gem install rbtree

Is this a bug in rbtree?

#25 Updated by Tomoyuki Chikanaga 10 months ago

Hello,

The following is the output of gem install rbtree on my environment (ruby 2.1.0dev (2013-06-16 trunk 41337) [x86_64-darwin10.8.0])
It seems that rbtree need to catch up with constify of RBasic::klass and RHash::ifnone for RGenGC.
ko1, what do you think? Is it intended incompatibility?

% gem install rbtree
Fetching: rbtree-0.4.1.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing rbtree:
ERROR: Failed to build gem native extension.

/Users/nagachika/opt/ruby-trunk/bin/ruby-trunk extconf.rb

checking for ruby/st.h... yes
checking for rbexecrecursive() in ruby.h... yes
checking for rbexecrecursivepaired() in ruby.h... yes
checking for rb
proclambdap() in ruby.h... yes
creating Makefile

make
compiling dict.c
compiling rbtree.c
rbtree.c: In function ‘copydict’:
rbtree.c:755: error: assignment of read-only member ‘klass’
rbtree.c: In function ‘rbtree
to_hash’:
rbtree.c:1292: error: assignment of read-only member ‘ifnone’
make: *** [rbtree.o] Error 1

#26 Updated by Nobuyoshi Nakada 10 months ago

It's a completely different issue, and intended incompatibility AFAIK.

Also available in: Atom PDF