Project

General

Profile

Actions

Bug #22305

closed

Ruby::Box: statically linked prism loads in only one box

Bug #22305: Ruby::Box: statically linked prism loads in only one box

Added by hsbt (Hiroshi SHIBATA) 2 months ago. Updated about 1 month ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27]
[ruby-core:126653]

Description

Under RUBY_BOX=1, the statically linked prism can be required in only one box. The second box raises LoadError.

$ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})'
lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError)

Swapping the order fails the same way. --disable-gems matters here. With a prism gem installed, each box dlopens its own copy of the gem's prism.bundle and the problem disappears, which makes it easy to miss.

There is no prism.so file. prism_init.c:8 registers the name prism/prism.so with ruby_init_ext, pointing at the Init_prism already linked into the interpreter, so require "prism/prism" behaves as if it had loaded an extension. That entry can be used only once (load.c:1280). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first.

The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail is_a?(Prism::ProgramNode). That trades the LoadError for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the require "prism/prism" inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The ruby-lsp benchmark fails on exactly that.

I see three designs and would like one chosen.

  1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under --with-static-linked-ext. It changes the build and needs symbol exports on Windows.
  2. Treat it as part of the interpreter, like String. It is initialized once before the boxes and every box shares its classes. defined?(Prism) becomes true without a require, it adds 339 classes at boot, and a prism gem can no longer take over.
  3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading.

How prism came to ship this way

#19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with ruby_init_ext. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it.

The second design has a history of its own. Matz first approved a built-in Ruby::Parser that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent Ruby::Node for Proc#syntax_tree (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser.


Related issues 6 (1 open — 5 closed)

Related to Ruby - Misc #22275: Ruby::Box support plan for RubyGems and BundlerOpenActions
Related to Ruby - Feature #19741: Mirror YARP to RubyClosedActions
Related to Ruby - Misc #19772: API naming for YARP compilerClosedActions
Related to Ruby - Feature #19900: Rename YARP to PrismClosedActions
Related to Ruby - Bug #21618: Allow to use the build-in prism version to parse codeClosedActions
Related to Ruby - Feature #21795: Methods for retrieving ASTsClosedActions

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #1

  • Related to Misc #22275: Ruby::Box support plan for RubyGems and Bundler added

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #2

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #3

  • Related to Misc #19772: API naming for YARP compiler added

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #4

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #5

  • Related to Bug #21618: Allow to use the build-in prism version to parse code added

Updated by hsbt (Hiroshi SHIBATA) 2 months ago Actions #6

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago Actions #7 [ruby-core:126888]

I pended the TestProcSyntaxTree tests under RUBY_BOX=1 until a design is chosen here. https://github.com/ruby/ruby/pull/19139

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago Actions #8

  • Description updated (diff)

Updated by matz (Yukihiro Matsumoto) about 2 months ago Actions #9 [ruby-core:126957]

I prefer option 1.

In Box, things defined by the interpreter are shared, and things loaded by require belong to each box. Option 1 keeps that model as it is, and it is also what #19900 originally described. Option 2 makes every process pay for 339 classes at boot and stops a prism gem from taking over, which I don't want. Option 3 adds a third kind of thing to Box, and I'd like to avoid that while we are still working out the Box semantics.

@tagomoris (Satoshi Tagomori), please tell us if you see it differently from the Box side.

One question: how would option 1 work on platforms without dynamic loading, where every extension is statically linked?

Matz.

Updated by tagomoris (Satoshi Tagomori) about 2 months ago Actions #10 [ruby-core:126971]

I also prefer option 1. That makes box possible to load different Prism versions in boxes (if users want).

Updated by mame (Yusuke Endoh) about 2 months ago Actions #11 [ruby-core:126986]

I'm also fine with option 1.

My concern about option 1 was that Proc#syntax_tree loads the prism Ruby API via require "prism" and relies on it being the same prism that is built into the interpreter, which a per-box prism gem cannot guarantee. With Ruby::Prism proposed in #22379, Proc#syntax_tree no longer needs the prism gem, so require "prism" can simply load a separate prism in each box.

What remains is Ruby::Prism itself. Since it is statically linked into the interpreter, it cannot be loaded separately in each box. I think it should be shared among all boxes, like other builtins. This needs a bit of hassle, since Ruby::Prism has to be autoloaded in the master box, but I have a prototype and will share it once #22379 is settled.

Updated by kddnewton (Kevin Newton) about 2 months ago Actions #12 [ruby-core:126987]

I agree with @mame (Yusuke Endoh) here, I think option 1 is best if we can get it to work, which it seems like after #22379 it will be.

Updated by hsbt (Hiroshi SHIBATA) about 2 months ago Actions #13 [ruby-core:127002]

This is a note to sort out the discussion. The direction in #22379 is a hybrid of options 1 and 2. In mame's PoC (https://github.com/ruby/ruby/pull/19248), the prism gem becomes an ordinary bundled gem that each box dlopens on its own, and the parser built into the interpreter becomes Ruby::Prism, so the gem no longer collides with it.

What remains here is sharing Ruby::Prism across boxes, which the PoC does not do yet. As mame noted, loading the autoload registered in the root box into the root box and sharing it with every box would solve it. I'll follow #22379 and keep this ticket for that part.

Updated by mame (Yusuke Endoh) about 1 month ago Actions #14 [ruby-core:127068]

I've opened a pull request to share Ruby::Prism among boxes: https://github.com/ruby/ruby/pull/19296

Ruby::Prism is autoloaded in the master box, so that every box sees the same definition, like other builtins. This did not work because of two problems in autoload under Box:

  1. When an autoload is triggered, the constant is defined only in the box that triggered it, not in the one that registered it. So after one box resolves the autoload, other boxes get NameError.
  2. The autoload table is shared by all boxes. Once one box resolves the autoload, the entry is removed from the shared table, and other boxes can no longer see either the constant or the autoload.

The PR has two commits:

  1. Set the constants in the box that registered the autoload, and copy them to the triggering box.
  2. Remove the autoload table and keep the autoload information in each constant entry instead.

With 2, the autoload information is copied per box along with the constant table.

I also tried copying the autoload table per box for 2, but the change was bigger and more complex because it needs a new mechanism to copy the autoload table. The PR is much simpler than that, and it even has fewer lines than the current master.

Without Box, the behavior does not change.

Note that require "ruby/prism" in a box loads it in that box, which leads to strange behavior. @tagomoris (Satoshi Tagomori) has decided that users must not explicitly require "ruby/prism", and the PR also documents it.

Updated by mame (Yusuke Endoh) about 1 month ago Actions #15

  • Status changed from Open to Closed

Applied in changeset git|dfbcdda93952f7ed18131a14e1010d403e7f5c02.


Set autoloaded constants in the box that registered the autoload

The constants of an autoload were set in the box that triggered it. For
an autoload registered in the master box (e.g., Ruby::Prism), this
defined the constant only in the first box that accessed it, and the
other boxes got NameError.

Set the constants in the registering box instead. For the master box,
they are set in the prime classext, which is shared by all boxes; a box
that has its own copy of the class (e.g., the one that triggered the
autoload) takes the value from the prime one.

[Bug #22305]

Co-Authored-By: Claude Opus 5.5

Actions

Also available in: PDF Atom