https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112020-11-21T05:01:19ZRuby Issue Tracking SystemRuby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886642020-11-21T05:01:19Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>This is a tough problem.</p>
<p>Ruby ships several hundreds of C/C++ header files (those under <code>include/</code> in our distribution). 3rd party extension libraries are expected to include them. The problem is, there are header files which depend on <em>runtime</em> API/ABI detected by our configure script. For an example we choose one of our 3 distinct <code>select(2)</code> wrapper depending on how many file descriptor the system can take at once. If the header file does not agree with generated ruby binary whatever bad thing can happen. As for C++ compilers, we want to know the availability of <code>std::nullptr_t</code>. If there is that type we provide extra function overloads that prevent warnings around casts.</p>
<p>The particular C++ feature detection business, I guess, could be possibly delayed until people configure their extension library. C++ nowadays are largely header-only. But if you want to separate the entire environment between the core and the extensions, I honestly have no idea if that is even possible.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886662020-11-21T05:57:40Zsawa (Tsuyoshi Sawada)
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/88666/diff?detail_id=58354">diff</a>)</li></ul> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886672020-11-21T06:01:17Zsawa (Tsuyoshi Sawada)
<ul><li><strong>Subject</strong> changed from <i>Don't embed Ruby build time configuration into Ruby</i> to <i>Don't embed Ruby build-time configuration in Ruby</i></li></ul> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886822020-11-22T09:37:40Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>Should we move such detections (for other than C, not only C++) to mkmf.rb?</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886872020-11-22T13:19:16Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>nobu (Nobuyoshi Nakada) wrote in <a href="#note-4">#note-4</a>:</p>
<blockquote>
<p>Should we move such detections (for other than C, not only C++) to mkmf.rb?</p>
</blockquote>
<p>JFYI because compilers tend to LTO these days, libruby can be a compiler-specific internal representation of the entire source code, not an archive of object files. If that is the case, people cannot use arbitrary C++ compiler. It must exactly match the C compiler used to generate libruby. Detection of such things could be done in mkmf.rb I guess, but not as easy as to “move” the current logic.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886892020-11-22T13:55:38Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-5">#note-5</a>:</p>
<blockquote>
<p>nobu (Nobuyoshi Nakada) wrote in <a href="#note-4">#note-4</a>:</p>
<blockquote>
<p>Should we move such detections (for other than C, not only C++) to mkmf.rb?</p>
</blockquote>
<p>JFYI because compilers tend to LTO these days, libruby can be a compiler-specific internal representation of the entire source code, not an archive of object files. If that is the case, people cannot use arbitrary C++ compiler. It must exactly match the C compiler used to generate libruby. Detection of such things could be done in mkmf.rb I guess, but not as easy as to “move” the current logic.</p>
</blockquote>
<p>IOW, mkmf rather should fail in such situation?<br>
(I'm not sure how to check it.)</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886962020-11-23T14:13:58Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-1">#note-1</a>:<br>
Thx for elaborating some of the reasons behind.</p>
<p>However, I don't think that embedding of the compile time options addresses any of these concerns. If they should be addressed, that would mean to do the same configuration checks once done for Ruby also for build of extensions and fail the build if they differs.</p>
<p>Also, I don't really know, what component is responsible for number of file descriptors, but on Linux, I'd assume this is Kernel. But if the value is changed in between kernel versions, then the compile time choice of <code>select(2)</code> might not be optimal.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=886972020-11-23T14:23:41Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-5">#note-5</a>:</p>
<blockquote>
<p>JFYI because compilers tend to LTO these days, libruby can be a compiler-specific internal representation of the entire source code, not an archive of object files. If that is the case, people cannot use arbitrary C++ compiler. It must exactly match the C compiler used to generate libruby. Detection of such things could be done in mkmf.rb I guess, but not as easy as to “move” the current logic.</p>
</blockquote>
<p>This sounds that this applies to statically linked Ruby, or does it apply to shared libruby as well? I care just about the later if that makes any difference.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=887042020-11-24T01:08:57Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>vo.x (Vit Ondruch) wrote in <a href="#note-7">#note-7</a>:</p>
<blockquote>
<p>However, I don't think that embedding of the compile time options addresses any of these concerns. If they should be addressed, that would mean to do the same configuration checks once done for Ruby also for build of extensions and fail the build if they differs.</p>
</blockquote>
<p>Right. I'm not saying we are doing well. Rather I say there is a headache. Ruby had already assumed the exact same runtime environment for both core & extensions. You want to separate them. I understand your motivation. But... that is harder than it sounds.</p>
<blockquote>
<p>Also, I don't really know, what component is responsible for number of file descriptors, but on Linux, I'd assume this is Kernel. But if the value is changed in between kernel versions, then the compile time choice of <code>select(2)</code> might not be optimal.</p>
</blockquote>
<p>That would happen. We have experienced situations like for instance getrandom(2) system call already implemented in the Kernel, but not yet available in glibc. Luckily that did not affect extension libraries though.</p>
<p>vo.x (Vit Ondruch) wrote in <a href="#note-8">#note-8</a>:</p>
<blockquote>
<p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-5">#note-5</a>:</p>
<blockquote>
<p>JFYI because compilers tend to LTO these days, libruby can be a compiler-specific internal representation of the entire source code, not an archive of object files. If that is the case, people cannot use arbitrary C++ compiler. It must exactly match the C compiler used to generate libruby. Detection of such things could be done in mkmf.rb I guess, but not as easy as to “move” the current logic.</p>
</blockquote>
<p>This sounds that this applies to statically linked Ruby, or does it apply to shared libruby as well? I care just about the later if that makes any difference.</p>
</blockquote>
<p>AFAIK no dynamic linker interfaces with compilers' LTO facilities.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=887152020-11-24T09:20:50Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-9">#note-9</a>:</p>
<blockquote>
<p>Ruby had already assumed the exact same runtime environment for both core & extensions.</p>
</blockquote>
<p>While considering everything what was said here, this is not surprise. Nevertheless, I am not sure this is wildly known and documented enough.</p>
<p>However, in Red Hat ecosystem, this was never true. Build of Ruby RPM always happens on different system then build of RubyGems RPM packages. The configuration of the runtime system is certainly different from the builders. The underlying Kernel can change including changing important defaults. Using Ruby in containers, there is not enough information about undrlying Kernel, one cannot even rely on kernel-headers, because they very likely don't match. Discussing <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: JIT vs hardened GCC with PCH (Closed)" href="https://bugs.ruby-lang.org/issues/16694">#16694</a> with RH toolchain folks, I was even suggested to use CLang to build Ruby itself, while keep using GCC to build extensions.</p>
<p>Therefore, given the real world use cases, isn't it time to reconsider this? I understand I'm leaving a lot of technical "details" aside, but if I could come in this case to Eventmachine upstream and say "you should not rely on Ruby detected configuration, do your own detection for best results, because Ruby upstream acknowledges the build time and runtime environment might differ", I'd be certainly in better place.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=887272020-11-24T21:08:30ZEregon (Benoit Daloze)
<ul></ul><p>vo.x (Vit Ondruch) wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<p>if I could come in this case to Eventmachine upstream and say "you should not rely on Ruby detected configuration, do your own detection for best results, because Ruby upstream acknowledges the build time and runtime environment might differ", I'd be certainly in better place.</p>
</blockquote>
<p>I disagree with that, specifically, C/C++/native extensions should rely on configuration from Ruby libraries such as <code>mkmf</code> and <code>RbConfig</code>.<br>
Now, those values could reflect the runtime system and not the build system, I would not mind that and it sounds better (but it might be quite difficult to achieve for CRuby).<br>
But I really don't want any native extension to detect its own C/C++ compiler, flags, etc, that leads to many problems and technical debt, notably when trying to compile on TruffleRuby which actually ships its own LLVM toolchain and where <code>RbConfig::CONFIG["CC/CXX"]</code> should be used to compile (which just works fine when using <code>mkmf</code> and not trying to hack around it).</p>
<p>FWIW TruffleRuby detects most things at runtime, and having a given fixed toolchain avoids so many of these headaches, which e.g. makes it possible to have a single prebuilt Linux binary running on all distributions, with the exception that the openssl C extension is recompiled on the runtime system at installation time because libssl varies too much per platform.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=887312020-11-24T22:32:34Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>Eregon (Benoit Daloze) wrote in <a href="#note-11">#note-11</a>:</p>
<blockquote>
<p>vo.x (Vit Ondruch) wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<p>if I could come in this case to Eventmachine upstream and say "you should not rely on Ruby detected configuration, do your own detection for best results, because Ruby upstream acknowledges the build time and runtime environment might differ", I'd be certainly in better place.</p>
</blockquote>
<p>I disagree with that, specifically, C/C++/native extensions should rely on configuration from Ruby libraries such as <code>mkmf</code> and <code>RbConfig</code>.</p>
</blockquote>
<p>I don't think I have anything specifically against <code>mkmf</code>. It is fine by me if it can be utilized. However, reusing embedded values from RbConfig is not always the best idea.</p>
<blockquote>
<p>FWIW TruffleRuby detects most things at runtime, and having a given fixed toolchain avoids so many of these headaches, which e.g. makes it possible to have a single prebuilt Linux binary running on all distributions, with the exception that the openssl C extension is recompiled on the runtime system at installation time because libssl varies too much per platform.</p>
</blockquote>
<p>Given the above mentioned relation between Kernel and Ruby build configurations, that leads me to believe that TruffleRuby is doing fine for most of the cases, but there are probably some corner cases. And similar situation is probably on Red Hat platforms.</p>
<p>Also, believe me that if you are using Fedora/RHEL provided libraries, you are basically in the same situation as you described the situation of TruffleRuby. But since we still want our users allow to use <code>gem install</code>, then I have to care also about these situations.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=888242020-11-29T20:37:12ZRinkana (Max Berends)
<ul><li><strong>File</strong> <a href="/attachments/8617">Dockerfile</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/8617/Dockerfile">Dockerfile</a> added</li></ul><p>It seems that this issue also breaks building the sassc gem from version 2.1 onward.<br>
Building it on the ruby 3.0-rc docker image (ruby:3.0-rc-slim-buster) gives me:</p>
<pre><code class="shell syntaxhl" data-language="shell"> current directory: /usr/local/bundle/gems/sassc-2.4.0/ext
/usr/local/bin/ruby <span class="nt">-I</span> /usr/local/lib/ruby/site_ruby/3.0.0 <span class="nt">-r</span> ./siteconf20201129-11-yfpi8m.rb extconf.rb
creating Makefile
current directory: /usr/local/bundle/gems/sassc-2.4.0/ext
make <span class="s2">"DESTDIR="</span> clean
current directory: /usr/local/bundle/gems/sassc-2.4.0/ext
make <span class="s2">"DESTDIR="</span>
compiling ./libsass/src/ast.cpp
make: I.: Command not found
make: <span class="o">[</span>Makefile:237: ast.o] Error 127 <span class="o">(</span>ignored<span class="o">)</span>
<span class="o">[</span>....]
compiling ./libsass/src/values.cpp
make: I.: Command not found
make: <span class="o">[</span>Makefile:237: values.o] Error 127 <span class="o">(</span>ignored<span class="o">)</span>
linking shared-object sassc/libsass.so
make: shared: Command not found
make: <span class="o">[</span>Makefile:262: libsass.so] Error 127 <span class="o">(</span>ignored<span class="o">)</span>
strip: <span class="s1">'libsass.so'</span>: No such file
make: <span class="k">***</span> <span class="o">[</span>Makefile:263: libsass.so] Error 1
make failed, <span class="nb">exit </span>code 2
Gem files will remain installed <span class="k">in</span> /usr/local/bundle/gems/sassc-2.4.0 <span class="k">for </span>inspection.
Results logged to /usr/local/bundle/extensions/x86_64-linux/3.0.0/sassc-2.4.0/gem_make.out
</code></pre>
<p>This is with just a clean Rails install (6.1.0-rc1) and no additional gems.<br>
sassc version 2.0.1 builds fine. As does sassc 2.4.0 on the ruby 2.7.2 image (ruby:2.7-slim-buster).</p>
<p>Attached is my Dockerfile. It requires an entrypoint.sh that contains just:</p>
<pre><code class="shell syntaxhl" data-language="shell"><span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-e</span>
<span class="nb">exec</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre>
<p>So it seems to me that this change can have a big impact as my setup is not too uncommon (correct me if i'm wrong).</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=892122020-12-14T02:02:11Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>We need to discuss the "whole principle" part. But in the meamtine we could revert this particular C++ compiler detection business to help eventmachine & sassc. Thoughts?</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=892182020-12-14T14:41:21Zvo.x (Vit Ondruch)v.ondruch@tiscali.cz
<ul></ul><p>shyouhei (Shyouhei Urabe) wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>But in the meamtine we could revert this particular C++ compiler detection business to help eventmachine & sassc. Thoughts?</p>
</blockquote>
<p>+1</p>
<p>Should I open separate ticket for the revert and keep this ticket open for continuing the "whole principle" discussion?</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=892372020-12-15T23:48:06Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="configure.ac: avoid squashing CXX=g++ We are discussing this issue at [Bug #17337] but in the me..." href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/ccc828f49956424b2548e32cb3bc3a78e16e207b">git|ccc828f49956424b2548e32cb3bc3a78e16e207b</a>.</p>
<hr>
<p>configure.ac: avoid squashing CXX=g++</p>
<p>We are discussing this issue at [Bug <a class="issue tracker-5 status-1 priority-4 priority-default" title="Misc: Don't embed Ruby build-time configuration in Ruby (Open)" href="https://bugs.ruby-lang.org/issues/17337">#17337</a>] but in the meantime, leave<br>
this questionable autoconf glitch as-is to save sassc and eventmachine.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=892602020-12-17T00:30:11Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>Open</i></li></ul><p>vo.x (Vit Ondruch) wrote in <a href="#note-15">#note-15</a>:</p>
<blockquote>
<p>Should I open separate ticket for the revert and keep this ticket open for continuing the "whole principle" discussion?</p>
</blockquote>
<p>Don’t worry, I have reverted that part already. Use this ticket for the discussion.</p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=893792020-12-21T16:17:00Zwebzorg (Lasha Abulashvili)
<ul></ul><p>I was told about this bug on freenode#ruby</p>
<p>It maybe related to my recent problem with installing pg gem on ruby 2.7.2</p>
<p>gem_make.out - <a href="https://paste.ubuntu.com/p/s6YD8C5yqS/" class="external">https://paste.ubuntu.com/p/s6YD8C5yqS/</a><br>
mkmf.log - <a href="https://paste.ubuntu.com/p/BKTVvdkJt6/" class="external">https://paste.ubuntu.com/p/BKTVvdkJt6/</a></p>
<p>according to rvm logs the binary was simply downloaded precompiled.<br>
<a href="https://paste.ubuntu.com/p/H7r4zjmRTN/" class="external">https://paste.ubuntu.com/p/H7r4zjmRTN/</a></p> Ruby master - Misc #17337: Don't embed Ruby build-time configuration in Rubyhttps://bugs.ruby-lang.org/issues/17337?journal_id=1042892023-08-24T18:11:39Zjeremyevans0 (Jeremy Evans)merch-redmine@jeremyevans.net
<ul><li><strong>Tracker</strong> changed from <i>Bug</i> to <i>Misc</i></li><li><strong>ruby -v</strong> deleted (<del><i>ruby 3.0.0dev (2020-11-04 master 704fb0b815) [x86_64-linux]</i></del>)</li><li><strong>Backport</strong> deleted (<del><i>2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN</i></del>)</li></ul>