Bug #7698: RubyGems 2.0 has an incompatibility about installation of extension libraries

• Status: Closed
• Assignee: nobu (Nobuyoshi Nakada)
• Target version: 2.1.0
• ruby -v: ruby 2.0.0dev (2013-01-14 trunk 38812) [x86_64-linux]

[mrkn (Kenta Murata)]

The current rubygem included in ruby-head has an incompatibility about installation of extension libraries.
This incompatibility makes make install to be failed because the current gem doesn't separate build, source and install directories.

This is reported as a bug of bigdecimal's gemspec in #7344 firstly, but Tadashi Saito pointed out it also can be reproduced in other gems including extension libraries, such as decimal.gem.
I think this incompatibility will make gem-developers to be in trouble after ruby-2.0 is released.
So this should be resolved before ruby-2.0 is released.

[mrkn (Kenta Murata)]

Sorry, I made misspelling.

This incompatibility makes make install to be failed because the current gem doesn't separate build, source and install directories.

This incompatibility makes make install to be failed because the current rubygem doesn't separate build, source and install directories.

[sorah (Sorah Fukumori)]

Translating from [ruby-dev:46864], Tadashi said:

We can avoid this bug by changing directory tree in gems.
But, Current status means any gems may be not installable after 2.0.0, by incompatibility in current rubygem bundled in 2.0.0.

So, I propose the following solutions:

1. Revert that incompatibility before 2.0.0 release or 2.0.0's first patch level release to make such gems installable.
2. If you cannot, please review all gems that installable in 2.0.0 and notify to authors if a gem can't be installed in 2.0.0.
   (This may be rubygems.org's task)
3. Announce that like "Your gem may be not installable in 2.0.0" to all gem authors.

I think solution (1) is the best. (2) seems high-cost.
(3) is the worst way for me, because I've not seen an announcement like such,
and seems too late to announce.

[mrkn (Kenta Murata)]

Thank you translation, sorah!

[kou (Kouhei Sutou)]

This can be fixed by the following patch. This is caused by #6904 change. Should this problem be fixed in RubyGems?

Index: lib/rubygems/ext/builder.rb

--- lib/rubygems/ext/builder.rb (revision 38833)
+++ lib/rubygems/ext/builder.rb (working copy)
@@ -19,6 +19,7 @@
mf = Gem.read_binary 'Makefile'
mf = mf.gsub(/^RUBYARCHDIR\s*=\s*$[^$]/, "RUBYARCHDIR = #{dest_path}")
mf = mf.gsub(/^RUBYLIBDIR\s=\s*$[^$]*/, "RUBYLIBDIR = #{dest_path}")

• mf = mf.gsub(/\s*\S+.time$/, "")

  File.open('Makefile', 'wb') {|f| f.print mf}

[drbrain (Eric Hodel)]

I am unfamiliar with Makefiles, so what does this patch change?

Also, .time is new in ruby 2.0, what effect does this have on ruby 1.9 and ruby 1.8?

I trust your judgement in adding this to RubyGems. If you commit it to Ruby please commit it to RubyGems and let me know!

[kou (Kouhei Sutou)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r38864.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• lib/rubygems/ext/builder.rb (Gem::Ext::Builder.make): Remove
  .time dependency from *.rb install target. It causes needless
  *.rb install. [Bug #7698] [ruby-core:51437]
  Reported by Tadashi Saito. Thanks!!!
• test/rubygems/test_gem_installer.rb
  (TestGemInstaller#test_install_extension_and_script): Add a test
  for the above change.

[kou (Kouhei Sutou)]

drbrain (Eric Hodel) wrote:

Also, .time is new in ruby 2.0, what effect does this have on ruby 1.9 and ruby 1.8?

This change doesn't have effect on 1.9 and 1.8 because Makefile on 1.9 and 1.8 doesn't have .time. So the gsub is just ignored.

I trust your judgement in adding this to RubyGems. If you commit it to Ruby please commit it to RubyGems and let me know!

Thanks. :-)
I've committed this change to Ruby and RubyGems with a test case.
Detail is written in commit message of the commit at the RubyGems repository: https://github.com/rubygems/rubygems/commit/a7cd1be541f2194fc8c1436d86bc05eb3a94500d

[mame (Yusuke Endoh)]

• Status changed from Closed to Assigned
• Assignee changed from kou (Kouhei Sutou) to drbrain (Eric Hodel)

Hello,

This issue seems not to be fixed completely.
"gem install rbtree" still hit this issue.
It reproduces on both trunk and 2.0.0-p0.

$ gem install rbtree
Building native extensions. This could take a while...
ERROR: Error installing rbtree:
ERROR: Failed to build gem native extension.

/home/mame/local/bin/ruby extconf.rb

checking for ruby/st.h... yes
checking for rb_exec_recursive() in ruby.h... yes
creating Makefile

make
compiling rbtree.c
compiling dict.c
linking shared-object rbtree.so

make install
/usr/bin/install -c -m 0755 rbtree.so /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/.
/usr/bin/install: rbtree.so' and /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/./rbtree.so' are the same file
make: *** [install-so] Error 1

Gem files will remain installed in /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0 for inspection.
Results logged to /home/mame/local/lib/ruby/gems/2.0.0/gems/rbtree-0.3.0/./gem_make.out

--
Yusuke Endoh [e-mail]

[nobu (Nobuyoshi Nakada)]

• File 0001-ext_conf_builder.rb-build-in-separate-dir.patch added

It's not good idea to mess up the source directory.

[nobu (Nobuyoshi Nakada)]

• File 0001-ext_conf_builder.rb-build-in-separate-dir.patch 0001-ext_conf_builder.rb-build-in-separate-dir.patch added

I've forgot to fix a few tests.

[nobu (Nobuyoshi Nakada)]

• File deleted (0001-ext_conf_builder.rb-build-in-separate-dir.patch)

[nobu (Nobuyoshi Nakada)]

Made a branch https://github.com/ruby/ruby/pull/252.

[drbrain (Eric Hodel)]

Committed r39542. Thank you nobu for the RubyGems patch!

[drbrain (Eric Hodel)]

• Assignee changed from drbrain (Eric Hodel) to nobu (Nobuyoshi Nakada)
• % Done changed from 100 to 50

=begin
This patch causes the rdiscount extension to fail to install since it reads a local file:

https://github.com/rtomayko/rdiscount/blob/master/ext/extconf.rb#L18

The exception:

Building native extensions. This could take a while...
/usr/local/bin/ruby -r./siteconf /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb
checking for random()... yes
checking for srandom()... yes
checking for rand()... yes
checking for srand()... yes
checking size of unsigned long... 8
checking size of unsigned int... 4
checking size of unsigned int... 4
*** /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=/Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext
--curdir
--ruby=/usr/local/bin/ruby
--with-rdiscount-dir
--without-rdiscount-dir
--with-rdiscount-include
--without-rdiscount-include=${rdiscount-dir}/include
--with-rdiscount-lib
--without-rdiscount-lib=${rdiscount-dir}/
/Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb:18:in read': No such file or directory - VERSION (Errno::ENOENT) from /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1/ext/extconf.rb:18:in '
ERROR: Error installing rdiscount:
ERROR: Failed to build gem native extension.

  Building has failed. See above output for more information on the failure.

Gem files will remain installed in /Users/drbrain/tmp/gems/gems/rdiscount-2.0.7.1 for inspection.

=end

[naruse (Yui NARUSE)]

It also breaks a test.
http://u64.rubyci.org/~chkbuild/ruby-trunk/log/20130301T010202Z.log.html.gz

8. Error:
   test_install_extension_and_script(TestGemInstaller):
   Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

/home/chkbuild/build/20130301T010202Z/ruby/ruby -r./siteconf /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/extconf.rb

creating Makefile

make
make[1]: Entering directory /home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn' make[1]: Nothing to be done for all'.
make[1]: Leaving directory `/home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'

make install
make[1]: Entering directory /home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn' installing default libraries make[1]: Circular /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb <- /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb dependency dropped. /usr/bin/install: /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb' and /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb' are the same file make[1]: *** [/home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/lib/a.rb] Error 1 make[1]: Leaving directory /home/chkbuild/build/20130301T010202Z/tmp/gem-install.20130301-11578-1y4vzyn'

Gem files will remain installed in /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2 for inspection.
Results logged to /home/chkbuild/build/20130301T010202Z/tmp/test_rubygems_11578/gemhome/gems/a-2/./gem_make.out

/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:55:in `run'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:28:in `block in make'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:26:in `each'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/builder.rb:26:in `make'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:33:in `block (2 levels) in build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:20:in `chdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:20:in `block in build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/tmpdir.rb:88:in `mktmpdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/ext/ext_conf_builder.rb:19:in `build'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:678:in `block (2 levels) in build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:677:in `chdir'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:677:in `block in build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:652:in `each'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:652:in `build_extensions'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/installer.rb:218:in `install'
/home/chkbuild/build/20130301T010202Z/ruby/test/rubygems/test_gem_installer.rb:1046:in `block in test_install_extension_and_script'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/user_interaction.rb:40:in `use_ui'
/home/chkbuild/build/20130301T010202Z/ruby/lib/rubygems/user_interaction.rb:63:in `use_ui'
/home/chkbuild/build/20130301T010202Z/ruby/test/rubygems/test_gem_installer.rb:1042:in `test_install_extension_and_script'

[drbrain (Eric Hodel)]

What make do you have?

I have GNU Make 3.81 and cannot reproduce.

[nobu (Nobuyoshi Nakada)]

• Status changed from Assigned to Closed
• % Done changed from 50 to 100

This issue was solved with changeset r39558.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

ext_conf_builder.rb: build in the source directory

• lib/rubygems/ext/ext_conf_builder.rb (Gem::Ext::ExtConfBuilder.build):
  revert use of temporary directory for build, to work some buggy
  extconf.rb which cannot build outside the source directory.
  [ruby-core:53056] [Bug #7698]

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Assigned
• Target version changed from 2.0.0 to 2.1.0
• % Done changed from 100 to 50

[nobu (Nobuyoshi Nakada)]

• Status changed from Assigned to Closed
• % Done changed from 50 to 100

This issue was solved with changeset r39559.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

ext_conf_builder.rb: use RUBYOPT

• lib/rubygems/ext/ext_conf_builder.rb (Gem::Ext::ExtConfBuilder.build):
  use RUBYOPT instead of -r option, and revert some tests. [Bug #7698]

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Assigned
• % Done changed from 100 to 50

[nobu (Nobuyoshi Nakada)]

• Status changed from Assigned to Closed
• % Done changed from 50 to 100

This issue was solved with changeset r39560.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

ext_conf_builder.rb: hack for obsolete sytle gems

• lib/rubygems/ext/ext_conf_builder.rb
  (Gem::Ext::ExtConfBuilder.hack_for_obsolete_sytle_gems): remove
  circular dependencies for old style gems which locate extconf.rb on
  the toplevel. [ruby-core:53059] [ruby-trunk - Bug #7698]

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Assigned
• % Done changed from 100 to 50

[nobu (Nobuyoshi Nakada)]

• Status changed from Assigned to Closed
• % Done changed from 50 to 100

This issue was solved with changeset r39572.
Kenta, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

ext_conf_builder.rb: remove circular dependency in install-so

• lib/rubygems/ext/ext_conf_builder.rb
  (Gem::Ext::ExtConfBuilder.hack_for_obsolete_style_gems): remove
  circular dependencies in install-so too. [ruby-core:52882]
  [Bug #7698]

[zzak (zzak _)]

I can still reproduce this on RubyGems 2.0.3 with trunk:
ruby 2.1.0dev (2013-06-16 trunk 41328) [x86_64-darwin12.3.0]

To repro: gem install rbtree

Is this a bug in rbtree?

[nagachika (Tomoyuki Chikanaga)]

Hello,

The following is the output of gem install rbtree on my environment (ruby 2.1.0dev (2013-06-16 trunk 41337) [x86_64-darwin10.8.0])
It seems that rbtree need to catch up with constify of RBasic::klass and RHash::ifnone for RGenGC.
ko1, what do you think? Is it intended incompatibility?

% gem install rbtree
Fetching: rbtree-0.4.1.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing rbtree:
ERROR: Failed to build gem native extension.

/Users/nagachika/opt/ruby-trunk/bin/ruby-trunk extconf.rb

checking for ruby/st.h... yes
checking for rb_exec_recursive() in ruby.h... yes
checking for rb_exec_recursive_paired() in ruby.h... yes
checking for rb_proc_lambda_p() in ruby.h... yes
creating Makefile

make
compiling dict.c
compiling rbtree.c
rbtree.c: In function ‘copy_dict’:
rbtree.c:755: error: assignment of read-only member ‘klass’
rbtree.c: In function ‘rbtree_to_hash’:
rbtree.c:1292: error: assignment of read-only member ‘ifnone’
make: *** [rbtree.o] Error 1

[nobu (Nobuyoshi Nakada)]

It's a completely different issue, and intended incompatibility AFAIK.