Misc #21825: Status of the universal parser implementing the Prism API

• Status: Open
• Assignee: -

[Eregon (Benoit Daloze)]

I'm opening this issue to discuss and ask the status of the universal parser implementing the Prism API.

Matz has agreed that going forward the official parser API for Ruby will be the Prism API,
therefore if parse.y wants to remain a longer-term possibility it will have to somehow implement the Prism API.
(this is not new, I recall discussing this at RubyKaigi 2024 with matz and @yui-knk (Kaneko Yuichiro))

The Prism API covers clearly at least the Ruby API Prism::Node.
But also if the universal parser is a complete alternative for Prism (if not it wouldn't really be universal and lots of existing Prism usages couldn't switch to it),
it would need to also support the Java, JavaScript and Rust APIs of Prism.
And also the AST part of the C API of Prism too.
I think the most sensible and maybe the only realistic way is for parse.y to generate a Prism C AST using the AST structs from Prism:
https://github.com/eregon/prism-generated/blob/generated/include/prism/ast.h such as struct pm_call_node.
That way, all the higher-level APIs and code in the prism gem can be reused as-is and the only real difference is whether using the prism.c parser or the parse.y parser, but the output would be the exact same.

When that happens it would actually become possible to just switch the parser with --parser=, vs currently switching the CRuby source-to-bytecode compiler as well.
IOW, there would be no need for the massive duplication in compile.c and prism_compile.c, we'd keep only prism_compile.c (and probably rename it), since parse.y would also generated a Prism C AST.