Bug #21618: Allow to use the build-in prism version to parse code

• Status: Open
• Assignee: -
• Target version: -
• Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN

[Earlopain (Earlopain _)]

Prism is a multi-version parser, which is a great feature.

If one calls Prism.parse("foo"), it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would.

Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: Prism.parse("foo", version: RUBY_VERSION). But doing so is not feasable because prism is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences.

ripper does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself.

Libraries like rails have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new prism versions that know about more recent syntax. error_highlight for example also suffers from the same defect.

It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like Prism.parse_success? (does the parse result have any errors?). I would be happy with supporting only the minimal functions like Prism.parse and Prism.parse_file. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror RubyVM::InstructionSequence.compile_file and RubyVM::InstructionSequence.compile.

It would be optimal if I would be able to transparently call Prism.parse("foo", version: "current") (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself.

@kddnewton (Kevin Newton) does this feature request make sense? Do you have any other ideas/inputs?

[tenderlovemaking (Aaron Patterson)]

I think this is a good idea. I recall discussing this type of feature with @kddnewton (Kevin Newton) at one point. He had some ideas about implementing it, but I don't recall the exact ideas. The use case seems very clear though.

[kddnewton (Kevin Newton)]

Yes, this was brought up by @k0kubun (Takashi Kokubun) originally as one of the requirements. At the moment the official answer is Prism.parse_file(filepath, version: RUBY_VERSION), because that will work for all Ruby versions in normal maintenance right now.

That being said, I do think it would make sense to support a feature like this going forward, it just becomes difficult for other implementations, depending on how they ship Prism. It might make sense for it to be RubyVM.parse to make it clear it's just for CRuby. I'm not sure if @Eregon (Benoit Daloze) or @enebo would have thoughts on this.

[Earlopain (Earlopain _)]

RubyVM.parse doesn't sound particularly nice to me. If you put it there then there have to be defined? checks and fallback implementations for non-CRuby which to me kinda defeats the point of using prism in the first place. I would massively prefer for it to be part of the stable/shared api, if at all feasable.

Of course, I don't have any better suggestions at the moment.

[kddnewton (Kevin Newton)]

So the issue is that there is the C library and then there is the Ruby library. The C library can parse and provide an AST in C structures, but to reify it into Ruby objects requires a specific Ruby library. This is possible, but only if the runtime ships the same matching version for both. I don't think any other implementation does this, because they only need the C library + FFI (JRuby/TruffleRuby/Garnet) or the C library + native bindings (Natalie/Opal).

Of course it is always possible to include methods that are just unimplemented for other runtimes, it just becomes less portable.

[Eregon (Benoit Daloze)]

Why is Prism.parse_file(filepath, version: RUBY_VERSION) not good enough?
In which concrete scenarios does it not work?

The prism default gem shipped with Ruby 3.3+ will always support version: RUBY_VERSION.

Another prism version might be used when using bundler or installing a newer prism (e.g. gem install prism).
If that prism version is newer, version: RUBY_VERSION will work.

If that prism version is older, it's possible it doesn't recognize the RUBY_VERSION.
That case though could be seen as "that old prism doesn't support a recent Ruby", i.e. the prism dependency should be updated to be less old.
In that sense it's not different than many other gems with older versions which might not work on the newest Ruby version.

Another POV here is: either prism is not specified in the Gemfile, and version: RUBY_VERSION will always work,
or it's specified in the Gemfile and then it's the responsibility of the maintainers of that Gemfile to keep it recent enough, e.g. by not using a Gemfile.lock or by regularly updating the Gemfile.lcok, at least when supported a new Ruby version (which is the same workflow as for other gem dependencies).

---

I'm guessing what you are asking is to get the Prism Ruby API for the prism C parser used by the Ruby interpreter to parse Ruby code?
I think that cannot be provided easily, because it would conflict with the prism gem, of which another version might be loaded.
I suppose one could try to nest that "current Ruby" Prism Ruby API under a different module than ::Prism, but it feels suboptimal and complicated, I think there is not a strong enough need to justify such complexity.

[Eregon (Benoit Daloze)]

From the description it seems a (the?) main concern is that Prism.parse(code) etc use the latest version by default and not version: RUBY_VERSION by default.
I share that concern, that feels error-prone.

Is there a use case for the default being latest?
I think not, usages would either want the current Ruby syntax or a specific one and then they would specify it explicitly (like in RuboCop).

So how about changing the default parse/syntax version of Prism then to be RUBY_VERSION?

[Earlopain (Earlopain _)]

I'm guessing what you are asking is to get the Prism Ruby API for the prism C parser used by the Ruby interpreter to parse Ruby code?

Ripper for all its faults was able to do just that. I understand that there are implementation concerns, I am just asking for something to make that possible again. I don't think parsing exactly (not just almost the same) as the ruby I am running as is an unreasonable request.

You should always bump all gems on ruby version updates. But prism is not patch level aware (that is fine) so even if you do RUBY_VERSION it is not perfect. 3.2.0 I think had a bug with nested anonymous block parameters that had a pretty big impact. Now I am required to also bump prism with every patch version. If I run older patch versions, I would be have to run older prism versions from around the time my patch version was released.

From the description it seems a (the?) main concern is that Prism.parse(code) etc use the latest version by default

Yeah, I am not a fan of that behaviour as well. I guess this is intential, similar to how the unversioned parser translator class also uses the latest version. But changing that was not really what I was asking for.

---

To be clear, patch versions of ruby should never contain any big syntax changes. The impact of using any prism version that understands my major/minor version should be minimal in the majority of cases. But, I would prefer to just have something where I don't even have to worry about such things.

Prism.parse("foo", version: RUBY_VERSION) works good enough but differently to the real intention to the point where it makes me uncomfortable.