Misc #22275
open
Ruby::Box support plan for RubyGems and Bundler
Description
This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place.
Goal¶
The goal is to realize Feature #13847 with Ruby::Box, which means loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems and Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years. Box removes that limitation.
As a further step, we want RubyGems and Bundler to activate different gem versions per box. That enables the use cases already proposed around Box, such as gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process.
First milestone¶
Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under RUBY_BOX=1, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress.
Current status¶
ruby/rubygems now runs both suites against a ruby-core master build with and without RUBY_BOX=1 on every pull request and push (ruby/rubygems#9826), and all four jobs pass. That covers the continuous part of the first milestone for ruby/rubygems.
With the stdio, $? and defined? fixes merged, ruby/rubygems#9892 removed 28 pending marks from the rubygems suite and 3 skips from the bundler suite. The rubygems suite now runs 3619 tests with 0 failures in both modes, with 10 pendings under Box and 8 without. The two box-only pendings wait on $VERBOSE, and the remaining bundler skip waits on -r, both below.
ruby/ruby's own make check passes under RUBY_BOX=1 on ruby/ruby#18705, which collects every open fix below and adds a CI lane for it. The branch is for measuring the integrated state, not for merging as it is. Decisions 3 to 7 came out of that work.
Critical problems¶
- ruby/ruby#18579 (Bug #22282) is the last fix the rubygems suite waits on: under Box,
$VERBOSE = nildoes not reach the interpreter. @tagomoris (Satoshi Tagomori) asked on the pull request whether$VERBOSEcan stay per box instead, which is decision 3. - Bug #22295 (decision 2) has no fix yet. The bundler install specs skip it, and the cargo builder test works around it by requiring inside
-e.
Decisions needed¶
These block the rest and need a direction rather than more review.
- Which box a builtin written in Ruby resolves its calls in. It is the master box by default, which matches not doing local rebinding (Bug #21362#note-3). ruby/ruby#18544 settled
Marshal.loadby opting it into the caller's box withPrimitive.attr! :caller_user_box. What remains is which other builtins opt in the same way, such asRactor.new(ruby/ruby#18821),Kernel#clone,Kernel#warnand prelude (Bug #22332), and whether core-class stubs should reach builtins (Misc #22296). Opting every builtin in by default brings the inline cache leak of Bug #21362 back, because all boxes share a builtin's call sites. On the ruby/ruby#18705 branch, which tries that, redefiningInteger#succin one box changes10.times.to_ain the main box. - Whether command-line
-ractivates gems, and which boxes it reaches (Bug #22295). Under Box,-randRUBYOPT=-rgo throughRuby::Box#requirewithout the RubyGems override, so a regular gem cannot be loaded that way, and a test harness that injects a gem withRUBYOPT=-rfails. The feature also stays in the main box, so a box created later does not see it. Item 3 of Bug #21760 reports the same gap inRuby::Box#requireitself. - What C-backed global variables mean in an optional box (Bug #22307). Assignments to
$/,$stdoutand the like stay in a box-local table that C never reads. ruby/ruby#18711 makes the main box behave like Ruby without boxes and leaves optional boxes open.$VERBOSEin ruby/ruby#18579 is the same question for one variable. - How prism ships (Bug #22305). The statically linked prism loads in only one box, so a gem that requires it, such as error_highlight, fails in every box but the first.
- How JIT hooks reach boxes (Bug #22306). Enabling YJIT switches
Array#eachand a few other methods to Ruby only in the master box, so the root and main boxes never get them. - What a box's top self should be (Bug #22331). Only the master box's top self has
include,using,privateand the other top-level definition methods, soRuby::Box#loadandload(file, true)fail. - Where prelude is evaluated (Bug #22332). It runs once in the master box, so
ppandbinding.irbrequire into it.
Open pull requests¶
ruby/ruby¶
- ruby/ruby#18579: assignments to
$VERBOSEand$DEBUGhave no effect (Bug #22282). Waiting on decision 3. - ruby/ruby#18711: C-backed global variables in the main box (Bug #22307). It depends on ruby/ruby#18704, which fixes
alias $new $oldin a box. - ruby/ruby#18708:
trace_varhooks do not run for assignments in a box. - ruby/ruby#18707: the top self of a box lacks the top-level definition methods (Bug #22331).
- ruby/ruby#18713:
ppandbinding.irbrequire into the master box (Bug #22332). Feature #21881 proposed loadingprelude.rbper box, and this does it. - ruby/ruby#18821:
Ractor.newresolves the classes of its arguments in the master box, so passing an instance of a class required in the main box raisesundefined class/module. It touches the Ractor and Box integration that Feature #22226#note-6 left to be decided with the Box author. - ruby/ruby#18842: a box classext is attached to its class only after it is filled in, so a GC in between frees what it holds.
- ruby/ruby#18710:
RUBY_FREE_AT_EXIT=1hangs at exit on macOS and crashes on Linux. - ruby/ruby#18701: a frozen
$LOADED_FEATURESraisesFrozenErrorwhile the feature index is rebuilt. This is not specific to Box, but every box hits it on its firstrequire.
Fixed¶
ruby/ruby¶
- ruby/ruby#18544:
Marshal.loadresolved classes outside the caller's box (Bug #22090). This was the last item on the Bundler critical path. - ruby/ruby#18574: reassigning
$stdoutand$stderrwas invisible to builtin writers (Bug #21867), which broke output-capturing test helpers everywhere. - ruby/ruby#18577:
$?was uninitialized afterKernel#systemandIO.popen(Bug #22280). - ruby/ruby#18586:
defined?did not see global variables assigned in a box (Bug #22283), so mkmfhave_devel?recursed until the stack was exhausted. - ruby/ruby#18218 / ruby/ruby#18219:
BUNDLER_SETUPwas consumed outside the main box (Bug #22123). Shipped in 4.0.7. - ruby/ruby#18509: box-local extension DLLs on Windows were unloaded too early. They are now deferred.
- ruby/ruby#18534: autoload-triggered require bypassed the box's
Kernel#require(Bug #21830). This alone took Rails from 500 on every request to fully working. - ruby/ruby#18535: with
--disable=gems, modules prepended toKernelin a user box ended up behindKernelin the ancestry (Bug #22270). - ruby/ruby#18536: the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable (Bug #22110, Bug #22271).
- ruby/ruby#18578: the process-private directory added by the previous fix did not survive
fork. A forked child removed it at exit (Bug #22271). - ruby/ruby#18575: box resolution crashed on an IFUNC frame (Bug #21977). Without this the bundler suite died with
[BUG]and the dead workers cascaded into unrelated failures. - ruby/ruby#18546:
Symbol#to_procignored box-local method definitions (Bug #22015, revived ruby/ruby#16865). Without this Bundler's spec harness died in rspec-core's&:shellsplitbefore running a single spec. - ruby/ruby#18475: an isolated Proc made in a class or module body lost its box and crashed (Bug #22260).
- ruby/ruby#18698, ruby/ruby#18702, ruby/ruby#18703, ruby/ruby#18706 and ruby/ruby#18712: test and harness fixes so that
make checkcan run underRUBY_BOX=1.
ruby/rubygems¶
- ruby/rubygems#9892: re-enabled the stdio capture tests under Box once Bug #21867 was fixed.
- ruby/rubygems#9826: made the RubyGems and Bundler suites green under
RUBY_BOX=1and added the Box lanes to CI. - ruby/rubygems#9809: Bundler evaluated gemspecs through
TOPLEVEL_BINDING, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the rightGem::Specification. - ruby/rubygems#9810: the
gemCLI died underRUBY_BOX=1.Marshal-based deep copies could not resolveGem::constants across boxes and were replaced with a plain deep dup, andRUBY_BOXis now stripped from extension build subprocesses, where mkmfhave_devel?recurses untilSystemStackError(Bug #22283). It includes a CLI canary test that runs underRUBY_BOX=1.
Updated by 
Updated by