Misc #22275
Updated by hsbt (Hiroshi SHIBATA) 19 days ago
This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place. ## Goal The goal is to realize [Feature #13847](https://bugs.ruby-lang.org/issues/13847) with Ruby::Box, which means Ruby::Box: loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems and Bundler RubyGems/Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years. years; Box removes that limitation. As a further step, we want RubyGems and Bundler RubyGems/Bundler to activate different gem versions per box. That box, which enables the use cases already proposed around Box, such as Box: gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process. ## First milestone Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under `RUBY_BOX=1`, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress. --- ## Current status Both suites now run under `RUBY_BOX=1` with the pull requests below applied. The rubygems suite is at 0 failures under `RUBY_BOX=1` with [ruby/rubygems#9826](https://github.com/ruby/rubygems/pull/9826), which is still open. ([ruby/rubygems#9826](https://github.com/ruby/rubygems/pull/9826), open). The bundler suite runs 3957 examples with box and non-box at parity, and parity; the single remaining failure is an unrelated upstream problem in a Rust extension test. ruby/ruby's own `make check` now passes under `RUBY_BOX=1` on [ruby/ruby#18705](https://github.com/ruby/ruby/pull/18705), which collects every open fix below and adds a CI lane for it. The branch is for measuring Pending marks reference the integrated state, not for merging as it is. Decisions 3 to 5 came out of that work. ## Critical problems * [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)) is the last item on the Bundler critical path. Without it the artifice-based install specs fail wholesale. It waits on decision 1 unfixed core bugs below. [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) and [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575), the other two, are merged. * [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574) ([Bug #21867](https://bugs.ruby-lang.org/issues/21867)) and [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577) ([Bug #22280](https://bugs.ruby-lang.org/issues/22280)) then let us drop most of the pending marks in the Bundler suite. ## Decisions needed These block the rest and need a direction rather than more review. 1. Which box a builtin written in Ruby resolves its calls in. Today it is the master box. That matches not doing local rebinding ([Bug #21362#note-3](https://bugs.ruby-lang.org/issues/21362#note-3)), but it makes `Marshal.load` resolve classes outside the caller's box ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)), hides core-class stubs from builtins ([Misc #22296](https://bugs.ruby-lang.org/issues/22296)), and affects `Ractor.new`, `Kernel#clone` and `Kernel#warn` the same way. [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) opts `Marshal.load` into the caller's box with `Primitive.attr! :caller_user_box`. Opting every builtin in instead brings the inline cache leak of [Bug #21362](https://bugs.ruby-lang.org/issues/21362) back, because all boxes share a builtin's call sites. On the [ruby/ruby#18705](https://github.com/ruby/ruby/pull/18705) branch, which tries that, redefining `Integer#succ` in one box changes `10.times.to_a` in the main box. 2. Whether command-line `-r` activates gems, and which boxes it reaches ([Bug #22295](https://bugs.ruby-lang.org/issues/22295)). Under Box, `-r` and `RUBYOPT=-r` go through `Ruby::Box#require` without the RubyGems override, so a regular gem cannot be loaded that way, and a test harness that injects a gem with `RUBYOPT=-r` fails. The feature also stays in the main box, so a box created later does not see it. Item 3 of [Bug #21760](https://bugs.ruby-lang.org/issues/21760) reports the same gap in `Ruby::Box#require` itself. 3. What C-backed global variables mean in an optional box ([Bug #22307](https://bugs.ruby-lang.org/issues/22307)). Assignments to `$/`, `$stdout` and the like stay in a box-local table that C never reads. [ruby/ruby#18711](https://github.com/ruby/ruby/pull/18711) makes the main box behave like Ruby without boxes and leaves optional boxes open. 4. How prism ships ([Bug #22305](https://bugs.ruby-lang.org/issues/22305)). The statically linked prism loads in only one box, so a gem that requires it, such as error_highlight, fails in every box but the first. 5. How JIT hooks reach boxes ([Bug #22306](https://bugs.ruby-lang.org/issues/22306)). Enabling YJIT switches `Array#each` and a few other methods to Ruby only in the master box, so the root and main boxes never get them. --- ## Open pull requests * [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544): `Marshal.load` resolves classes outside the caller's box ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)). Waiting on decision 1. * [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574): reassigning `$stdout` and `$stderr` is invisible to builtin writers ([Bug #21867](https://bugs.ruby-lang.org/issues/21867)), which breaks output-capturing test helpers everywhere. * [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577): `$?` is uninitialized after `Kernel#system` and `IO.popen` ([Bug #22280](https://bugs.ruby-lang.org/issues/22280)). * [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579): assignments to `$VERBOSE` and `$DEBUG` have no effect ([Bug #22282](https://bugs.ruby-lang.org/issues/22282)). * [ruby/ruby#18586](https://github.com/ruby/ruby/pull/18586): `defined?` does not see global variables assigned in a box ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). This is why mkmf `have_devel?` never memoizes and recurses until the stack is exhausted. * [ruby/ruby#18711](https://github.com/ruby/ruby/pull/18711): C-backed global variables in the main box ([Bug #22307](https://bugs.ruby-lang.org/issues/22307)). It depends on [ruby/ruby#18704](https://github.com/ruby/ruby/pull/18704), which fixes `alias $new $old` in a box. * [ruby/ruby#18708](https://github.com/ruby/ruby/pull/18708): `trace_var` hooks do not run for assignments in a box. * [ruby/ruby#18707](https://github.com/ruby/ruby/pull/18707): the top self of a box lacks `include`, `using`, `private` and the other top-level definition methods, so `load(file, true)` fails. * [ruby/ruby#18713](https://github.com/ruby/ruby/pull/18713): `pp` and `binding.irb` require into the master box. [Feature #21881](https://bugs.ruby-lang.org/issues/21881) proposed loading `prelude.rb` per box, and this does it. * [ruby/ruby#18710](https://github.com/ruby/ruby/pull/18710): `RUBY_FREE_AT_EXIT=1` hangs at exit on macOS and crashes on Linux. * [ruby/ruby#18701](https://github.com/ruby/ruby/pull/18701): a frozen `$LOADED_FEATURES` raises `FrozenError` while the feature index is rebuilt. This is not specific to Box, but every box hits it on its first `require`. ## Fixed ### ruby/ruby * [ruby/ruby#18218](https://github.com/ruby/ruby/pull/18218) / [ruby/ruby#18219](https://github.com/ruby/ruby/pull/18219): [ruby/ruby#18219](https://github.com/ruby/ruby/pull/18219) — `BUNDLER_SETUP` was consumed outside the main box ([Bug #22123](https://bugs.ruby-lang.org/issues/22123)). Shipped in 4.0.7. * [ruby/ruby#18509](https://github.com/ruby/ruby/pull/18509): [ruby/ruby#18509](https://github.com/ruby/ruby/pull/18509) — box-local extension DLLs on Windows were unloaded too early. They are early; now deferred. * [ruby/ruby#18534](https://github.com/ruby/ruby/pull/18534): [ruby/ruby#18534](https://github.com/ruby/ruby/pull/18534) — autoload-triggered require bypassed the box's `Kernel#require` ([Bug #21830](https://bugs.ruby-lang.org/issues/21830)). This alone took Rails from 500 on every request to fully working. * [ruby/ruby#18535](https://github.com/ruby/ruby/pull/18535): [ruby/ruby#18535](https://github.com/ruby/ruby/pull/18535) — with `--disable=gems`, modules prepended to `Kernel` in a user box ended up behind `Kernel` in the ancestry ([Bug #22270](https://bugs.ruby-lang.org/issues/22270)). * [ruby/ruby#18536](https://github.com/ruby/ruby/pull/18536): [ruby/ruby#18536](https://github.com/ruby/ruby/pull/18536) — the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable ([Bug #22110](https://bugs.ruby-lang.org/issues/22110), [Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18578](https://github.com/ruby/ruby/pull/18578): [ruby/ruby#18578](https://github.com/ruby/ruby/pull/18578) — the process-private directory added by the previous fix did not survive `fork`. A `fork`; a forked child removed it at exit ([Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575): [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575) — box resolution crashed on an IFUNC frame ([Bug #21977](https://bugs.ruby-lang.org/issues/21977)). Without this the bundler suite died with `[BUG]` and the dead workers cascaded into unrelated failures. * [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546): [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) — `Symbol#to_proc` ignored box-local method definitions ([Bug #22015](https://bugs.ruby-lang.org/issues/22015), revived [ruby/ruby#16865](https://github.com/ruby/ruby/pull/16865)). Without this Bundler's spec harness died in rspec-core's `&:shellsplit` before running a single spec. * [ruby/ruby#18475](https://github.com/ruby/ruby/pull/18475): an isolated Proc made in a class or module body lost its box and crashed ([Bug #22260](https://bugs.ruby-lang.org/issues/22260)). * [ruby/ruby#18698](https://github.com/ruby/ruby/pull/18698), [ruby/ruby#18702](https://github.com/ruby/ruby/pull/18702), [ruby/ruby#18703](https://github.com/ruby/ruby/pull/18703), [ruby/ruby#18706](https://github.com/ruby/ruby/pull/18706) and [ruby/ruby#18712](https://github.com/ruby/ruby/pull/18712): test and harness fixes so that `make check` can run under `RUBY_BOX=1`. ### ruby/rubygems * [ruby/rubygems#9809](https://github.com/ruby/rubygems/pull/9809): [ruby/rubygems#9809](https://github.com/ruby/rubygems/pull/9809) — Bundler evaluated gemspecs through `TOPLEVEL_BINDING`, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the right `Gem::Specification`. * [ruby/rubygems#9810](https://github.com/ruby/rubygems/pull/9810): [ruby/rubygems#9810](https://github.com/ruby/rubygems/pull/9810) — the `gem` CLI died under `RUBY_BOX=1`. `Marshal`-based deep copies could not resolve `Gem::` constants across boxes and were replaced with a plain deep dup, and `RUBY_BOX` is now stripped from extension build subprocesses, where mkmf `have_devel?` recurses until `SystemStackError` ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). It includes Includes a CLI canary test that runs under `RUBY_BOX=1`. ## Open pull requests All review-requested to @tagomoris: * [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) — `Marshal.load` resolves classes in the root box ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)); an in-process round-trip of `Gem::Version` fails today. Under review. * [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574) — reassigning `$stdout`/`$stderr` is invisible to builtin writers ([Bug #21867](https://bugs.ruby-lang.org/issues/21867)), which breaks output-capturing test helpers everywhere. * [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577) — `$?` is uninitialized after `Kernel#system` and `IO.popen` ([Bug #22280](https://bugs.ruby-lang.org/issues/22280)). * [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579) — assignments to `$VERBOSE` and `$DEBUG` have no effect ([Bug #22282](https://bugs.ruby-lang.org/issues/22282)). * [ruby/ruby#18586](https://github.com/ruby/ruby/pull/18586) — `defined?` does not see global variables assigned in a box ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). This is why mkmf `have_devel?` never memoizes and recurses until the stack is exhausted, which is the "stack level too deep in extconf.rb" entry under Known issues in `doc/language/box.md`. ## Remaining problems without a fix * [Bug #22295](https://bugs.ruby-lang.org/issues/22295) — `ruby -r<gem>` and `RUBYOPT=-r<gem>` do not activate gems under Box. `require_libraries_in_main_box()` calls `rb_require_string()` directly, bypassing the RubyGems `Kernel#require` override. Item 3 of [Bug #21760](https://bugs.ruby-lang.org/issues/21760) reports the same gap in `Ruby::Box#require`. * [Misc #22296](https://bugs.ruby-lang.org/issues/22296) — stubbing a core class does not reach code already loaded in the root box, so existing test suites change behaviour under Box without failing loudly. `allow(File).to receive(:expand_path)` in the main box leaves `Pathname#expand_path` calling the real method. We hit this in the Bundler suite and scoped the stub as a workaround, but existing test code cannot be expected to know the rule. If this is intended box semantics, it should be documented as a known limitation, because rspec-mocks style stubbing of core classes is everywhere. ## Critical path [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) is the last item on the critical path, because without it the artifice-based install specs fail wholesale. [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546) and [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575) were the other two and are now merged. [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574) and [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577) then let us drop most of the pending marks.