Misc #22275
Updated by hsbt (Hiroshi SHIBATA) 23 days ago
This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place. ## Goal The goal is to realize [Feature #13847](https://bugs.ruby-lang.org/issues/13847) with Ruby::Box: loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems/Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years; Box removes that limitation. As a further step, we want RubyGems/Bundler to activate different gem versions per box, which enables the use cases already proposed around Box: gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process. ## First milestone Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under `RUBY_BOX=1`, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress. ## Current status Both suites now run under `RUBY_BOX=1` with the pull requests below applied. The rubygems suite is at 0 failures ([ruby/rubygems#9826](https://github.com/ruby/rubygems/pull/9826), open). The bundler suite runs 3957 examples with box and non-box at parity; the single remaining failure is an unrelated upstream problem in a Rust extension test. Pending marks reference the unfixed core bugs below. ## Fixed ### ruby/ruby * [ruby/ruby#18218](https://github.com/ruby/ruby/pull/18218) / [ruby/ruby#18219](https://github.com/ruby/ruby/pull/18219) — `BUNDLER_SETUP` was consumed outside the main box ([Bug #22123](https://bugs.ruby-lang.org/issues/22123)). Shipped in 4.0.7. * [ruby/ruby#18509](https://github.com/ruby/ruby/pull/18509) — box-local extension DLLs on Windows were unloaded too early; now deferred. * [ruby/ruby#18534](https://github.com/ruby/ruby/pull/18534) — autoload-triggered require bypassed the box's `Kernel#require` ([Bug #21830](https://bugs.ruby-lang.org/issues/21830)). This alone took Rails from 500 on every request to fully working. * [ruby/ruby#18535](https://github.com/ruby/ruby/pull/18535) — with `--disable=gems`, modules prepended to `Kernel` in a user box ended up behind `Kernel` in the ancestry ([Bug #22270](https://bugs.ruby-lang.org/issues/22270)). * [ruby/ruby#18536](https://github.com/ruby/ruby/pull/18536) — the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable ([Bug #22110](https://bugs.ruby-lang.org/issues/22110), [Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18578](https://github.com/ruby/ruby/pull/18578) — the process-private directory added by the previous fix did not survive `fork`; a forked child removed it at exit ([Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575) — box resolution crashed on an IFUNC frame ([Bug #21977](https://bugs.ruby-lang.org/issues/21977)). Without this the bundler suite died with `[BUG]` and the dead workers cascaded into unrelated failures. * [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) — `Symbol#to_proc` ignored box-local method definitions ([Bug #22015](https://bugs.ruby-lang.org/issues/22015), revived [ruby/ruby#16865](https://github.com/ruby/ruby/pull/16865)). Without this Bundler's spec harness died in rspec-core's `&:shellsplit` before running a single spec. ### ruby/rubygems * [ruby/rubygems#9809](https://github.com/ruby/rubygems/pull/9809) — Bundler evaluated gemspecs through `TOPLEVEL_BINDING`, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the right `Gem::Specification`. * [ruby/rubygems#9810](https://github.com/ruby/rubygems/pull/9810) — the `gem` CLI died under `RUBY_BOX=1`. `Marshal`-based deep copies could not resolve `Gem::` constants across boxes and were replaced with a plain deep dup, and `RUBY_BOX` is now stripped from extension build subprocesses, where mkmf `have_devel?` recurses until `SystemStackError` ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). Includes a CLI canary test that runs under `RUBY_BOX=1`. ## Open pull requests All review-requested to @tagomoris: * [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) — `Symbol#to_proc` ignores box-local method definitions ([Bug #22015](https://bugs.ruby-lang.org/issues/22015), revives [ruby/ruby#16865](https://github.com/ruby/ruby/pull/16865)). This alone unblocks Bundler's spec harness, which dies in rspec-core's `&:shellsplit` before running a single spec. Approved, not merged yet. * [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) — `Marshal.load` resolves classes in the root box ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)); an in-process round-trip of `Gem::Version` fails today. Under review. * [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574) — reassigning `$stdout`/`$stderr` is invisible to builtin writers ([Bug #21867](https://bugs.ruby-lang.org/issues/21867)), which breaks output-capturing test helpers everywhere. * [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577) — `$?` is uninitialized after `Kernel#system` and `IO.popen` ([Bug #22280](https://bugs.ruby-lang.org/issues/22280)). * [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579) — assignments to `$VERBOSE` and `$DEBUG` have no effect ([Bug #22282](https://bugs.ruby-lang.org/issues/22282)). * [ruby/ruby#18586](https://github.com/ruby/ruby/pull/18586) — `defined?` does not see global variables assigned in a box ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). This is why mkmf `have_devel?` never memoizes and recurses until the stack is exhausted, which is the "stack level too deep in extconf.rb" entry under Known issues in `doc/language/box.md`. ## Remaining problems without a fix * [Bug #22295](https://bugs.ruby-lang.org/issues/22295) — `ruby -r<gem>` and `RUBYOPT=-r<gem>` do not activate gems under Box. `require_libraries_in_main_box()` calls `rb_require_string()` directly, bypassing the RubyGems `Kernel#require` override. Item 3 of [Bug #21760](https://bugs.ruby-lang.org/issues/21760) reports the same gap in `Ruby::Box#require`. * [Misc #22296](https://bugs.ruby-lang.org/issues/22296) — stubbing a core class does not reach code already loaded in the root box, so existing test suites change behaviour under Box without failing loudly. `allow(File).to receive(:expand_path)` in the main box leaves `Pathname#expand_path` calling the real method. We hit this in the Bundler suite and scoped the stub as a workaround, but existing test code cannot be expected to know the rule. If this is intended box semantics, it should be documented as a known limitation, because rspec-mocks style stubbing of core classes is everywhere. ## Critical path [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) and [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) are what is the last item left on the critical path, because without it path. The first gates Bundler's entire spec suite and the second makes the artifice-based install specs fail wholesale. [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) and [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575) were was the other two third and are is now merged. [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574) and [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577) then let us drop most of the pending marks.