Feature #18481: Porting YJIT to Rust (request for feedback)

• Status: Closed
• Assignee: maximecb (Maxime Chevalier-Boisvert)
• Target version: -

[maximecb (Maxime Chevalier-Boisvert)]

TL;DR: The YJIT team wants to explore using Rust to help develop YJIT. The rest of CRuby will continue to build without Rust tools and building YJIT will remain optional.

We’re currently exploring the possibility of porting YJIT to Rust and working on a small proof of concept that should be ready next month. The motivation behind this is that we are facing challenges in terms of code maintainability. As you know, JIT compilers can get very complex, and C99 doesn't offer many tools to manage this complexity. There are no classes and methods, limited type checking, and it's hard to fully separate code into modules, for instance.

We believe that having access to object oriented programming and a more expressive type system would help us manage growing complexity better and also improve the safety/robustness of YJIT. For instance we would like to add Windows support and a new backend to YJIT. That means we’ll have two separate backends (x86, arm64) and we’ll need to support two different calling conventions (Microsoft, SystemV), but currently, we have limited tools to build the abstractions needed, such as preprocessor macros and if-statements.

We’ve discussed the idea of porting YJIT to Rust with some of the Ruby core developers (@ko1 (Koichi Sasada), @k0kubun (Takashi Kokubun), @mame (Yusuke Endoh)), and it seems they would be open to merging something like this if it works well. I’m opening this ticket so that everyone can have a chance to provide feedback and participate in the discussion. We realize that adding Rust code to the CRuby codebase could be challenging and that there are open questions.

We are planning to make it so that YJIT will only need the Rust compiler and cargo to build. Building YJIT would then require the Rust compiler to be installed, but CRuby could build without YJIT and without the Rust compiler. There would be no new dependencies for the compiled binary. Rust is supported on Mac, Windows, BSDs, and Linux, which covers all the platforms on which we plan to support YJIT. Since Rust is based on LLVM, it has good support for cross-compilation.

We would like to solicit input from Ruby distributors who create .deb and .rpm packages. We will likely remain conservative when updating Rust versions to make OS packaging easier. We believe that in the general, the resulting code should be easier to maintain because it will be better organized, but the YJIT team will help out with YJIT-related backports and will be available to help if needed.

Value proposition:

• Rust type systems will catch more bugs early, help prevent new bugs
• Easier to manage growing complexity of YJIT
• Easier to maintain codebase, fewer “footguns”
• Easier for newcomers because the compiler catches more bugs
• Better performance because we can implement more sophisticated optimizations
• Easier to add support for new platforms (which adds complexity)
• Rust has mature and easy-to-install tools such as source code formatter and editor plugins
• Rust as a programming language community has a great deal of enthusiasm behind it. This could translate to more enthusiasm for YJIT and for Ruby as well.

Integration:

• YJIT will only depend on the Rust language and the standard library, no other dependencies
• YJIT will be able to build without an internet connection
• Rust has good support for cross-compilation
• Rust is supported on all platforms on which we plan to support with YJIT (Mac, Linux, Windows)
• The compiled CRuby binary won’t have any new dependencies on shared libraries
• CRuby will still be able to build without rustc, with YJIT disabled

[Eregon (Benoit Daloze)]

Would it make sense to automatically download a Rust toolchain when building with YJIT?

I assume the Rust code would need a specific version of Rust (newer versions might work but not always, backward compat is not eternal) and so it'd probably need a fixed version to be sure it builds.
Package managers often only provide a single version of Rust and so would be of little help.

In other words, I think it would be nice when building CRuby if it could just download the Rust toolchain it needs, and not require everyone building CRuby to have the right version of Rust, which is likely non-trivial especially for people not familiar at all with Rust.
This would also avoid Ruby installers like ruby-install/ruby-build/RVM/etc to have to add new dependencies, which sounds difficult for e.g. older OS and their package managers which might not have a recent Rust available.

There could still be a configure option to use an existing Rust toolchain, or automatically detect if the right version is installed.

As an alternative, building CRuby (without disabling YJIT) and without the right Rust installed could output instructions for how to install Rust. But this would be much less convenient.

[jeremyevans0 (Jeremy Evans)]

Eregon (Benoit Daloze) wrote in #note-1:

Would it make sense to automatically download a Rust toolchain when building with YJIT?

I don't think so. Trying to download during a build will result in problems for many packagers. Ruby doesn't try to download during build for any of the other dependencies it has, and it makes no sense to start doing so now, IMO.

I assume the Rust code would need a specific version of Rust (newer versions might work but not always, backward compat is not eternal) and so it'd probably need a fixed version to be sure it builds.
Package managers often only provide a single version of Rust and so would be of little help.

Rust is supposed to be fairly backwards compatible for the last few years. As long as the YJIT team commits to supporting a Rust version that was released more than 12 months ago (say Rust 1.50), and continues to support that version going forward (as Ruby continues to support C99), I don't expect this to be a significant issue. However, that should be a hard requirement, we would want to avoid version creep where each new version of Ruby requires a newer version of Rust to build YJIT. If the YJIT team does not think that is feasible, to me that would be a sign that Rust is not yet mature enough for Ruby to rely on.

In other words, I think it would be nice when building CRuby if it could just download the Rust toolchain it needs, and not require everyone building CRuby to have the right version of Rust, which is likely non-trivial especially for people not familiar at all with Rust.

If Ruby does this, it should only even attempt to do if it is sure in advance that such a toolchain exists for the platform (binary downloads only). It doesn't makes sense to download and compile Rust itself, considering Rust takes much much longer to build than Ruby. I still think that is a bad idea, but at least it doesn't hurt platforms that don't have a such a toolchain built for them.

This would also avoid Ruby installers like ruby-install/ruby-build/RVM/etc to have to add new dependencies, which sounds difficult for e.g. older OS and their package managers which might not have a recent Rust available.

There could still be a configure option to use an existing Rust toolchain, or automatically detect if the right version is installed.

Or a configure option to opt-in to such toolchain downloading, which IMO would be the best way to support such automated downloading (if we want to support it at all).

As an alternative, building CRuby (without disabling YJIT) and without the right Rust installed could output instructions for how to install Rust. But this would be much less convenient.

I agree that outputing instructions and stopping the build is a bad idea.

I think Rust should be treated like OpenSSL. If you build Ruby without OpenSSL, Ruby doesn't download OpenSSL for you, it just doesn't build the openssl extension. Similarly, if you build Ruby without Rust, Ruby just won't build yjit. That seems like the simplest and best way to handle things, IMO. After all, yjit is a purely optional part of Ruby, and use of yjit should be transparent, modulo performance.

[Eregon (Benoit Daloze)]

jeremyevans0 (Jeremy Evans) wrote in #note-2:

I don't think so. Trying to download during a build will result in problems for many packagers.

Yes, for packagers I think they would typically prefer to use an existing Rust package (and that much should work out for a recent enough Rust as it would be for new Ruby packages on new OS version).

Ruby doesn't try to download during build for any of the other dependencies it has, and it makes no sense to start doing so now, IMO.

It already does e.g., for bundled gems and config.guess/config.sub, and maybe more.

If Ruby does this, it should only even attempt to do if it is sure in advance that such a toolchain exists for the platform (binary downloads only).

I agree it wouldn't make sense to try to build Rust from source (fail clearly or don't build YJIT seems fine then).

Or a configure option to opt-in to such toolchain downloading, which IMO would be the best way to support such automated downloading (if we want to support it at all).

Would also work, but that means we'd need to change how many people build Ruby to include this extra flag.
I think there are rather few package managers, and far more people using ruby builders or building Ruby from source themselves.

I think Rust should be treated like OpenSSL. If you build Ruby without OpenSSL, Ruby doesn't download OpenSSL for you, it just doesn't build the openssl extension. Similarly, if you build Ruby without Rust, Ruby just won't build yjit. That seems like the simplest and best way to handle things, IMO. After all, yjit is a purely optional part of Ruby, and use of yjit should be transparent, modulo performance.

YJIT not being included just because there is no Rust toolchain feels pretty unexpected.
I think many people building Ruby wouldn't realize this consequence.
Of course a message should be shown when building if skipped, but it's also easy to miss (from my own experience).

[jeremyevans0 (Jeremy Evans)]

Eregon (Benoit Daloze) wrote in #note-3:

jeremyevans0 (Jeremy Evans) wrote in #note-2:

Ruby doesn't try to download during build for any of the other dependencies it has, and it makes no sense to start doing so now, IMO.

It already does e.g., for bundled gems and config.guess/config.sub, and maybe more.

I don't think that is true, at least not for releases. ruby-3.1.0.tar.gz includes all bundled gems in a .bundle directory. tool/config.guess and tool/config.sub are also included in the release tarball.

Or a configure option to opt-in to such toolchain downloading, which IMO would be the best way to support such automated downloading (if we want to support it at all).

Would also work, but that means we'd need to change how many people build Ruby to include this extra flag.

Only if they want to force the building of YJIT by default. It's clear to me that a switch from C to Rust moves YJIT from "always built on supported platforms (x86_64)", to "optional, built only if dependencies are available", similar to the openssl extension.

I think there are rather few package managers, and far more people using ruby builders or building Ruby from source themselves.

There may be many people building Ruby that don't want to download Rust in order to do so, and forcing a Rust download by default seems worse, IMO.

YJIT not being included just because there is no Rust toolchain feels pretty unexpected.

Not to me, but expectations are subjective.

[austin (Austin Ziegler)]

jeremyevans0 (Jeremy Evans) wrote in #note-2:

Rust is supposed to be fairly backwards compatible for the last few years. As long as the YJIT team commits to supporting a Rust version that was released more than 12 months ago (say Rust 1.50), and continues to support that version going forward (as Ruby continues to support C99), I don't expect this to be a significant issue. However, that should be a hard requirement, we would want to avoid version creep where each new version of Ruby requires a newer version of Rust to build YJIT. If the YJIT team does not think that is feasible, to me that would be a sign that Rust is not yet mature enough for Ruby to rely on.

I am not a YJIT maintainer, but I believe that this is trivially supportable with the use of Rust editions, and if they are able to target the 2018 edition, it would ensure maximum compatibility. A schedule could be figured out for when it would be appropriate to switch to the 2021 edition (which might be the better target of such a migration).

[maximecb (Maxime Chevalier-Boisvert)]

I'd like to get input from people who are packaging Ruby, and those maintaining Ruby installation scripts.

I don't think CRuby should download anything during build. IMO rustc should be treated like gcc, it's something you should install before running the ./configure script.

It seems like rustc is easy to install on supported platforms, and so should not be difficult to use to prebuild Ruby binary for packages, which covers most Ruby users. We would of course prefer to use more recent versions of Rust if possible, but we are flexible. If using an older version of Rust is necessary to make packaging easier, we can accomodate that requirement.

believe that this is trivially supportable with the use of Rust editions, and if they are able to target the 2018 edition, it would ensure maximum compatibility. A schedule could be figured out for when it would be appropriate to switch to the 2021 edition (which might be the better target of such a migration).

2018 was quite a while back and there's been a decent amount of change to the language since then. The question I'd like to ask here is "is this necessary?" Would standardizing on the latest stable version or rust installable through the rustup tool not be enough? Would 6 or 12 months back be enough?

In terms of scripts that automatically download and build CRuby from source, maybe those scripts could auto-download the rust toolchain, or ask the user if they want to download the toolchain to build YJIT?

[jeremyevans0 (Jeremy Evans)]

maximecb (Maxime Chevalier-Boisvert) wrote in #note-6:

believe that this is trivially supportable with the use of Rust editions, and if they are able to target the 2018 edition, it would ensure maximum compatibility. A schedule could be figured out for when it would be appropriate to switch to the 2021 edition (which might be the better target of such a migration).

2018 was quite a while back and there's been a decent amount of change to the language since then. The question I'd like to ask here is "is this necessary?" Would standardizing on the latest stable version or rust installable through the rustup tool not be enough? Would 6 or 12 months back be enough?

I think 12 months back is fine. However, whatever Rust version is initially agreed upon should be supported going forward. Bumping the version of Rust necessary to build YJIT should be considered with the same gravity as the decision to move from C89 to C99. If that seems impractical or too restrictive, that would signal to me that Rust is not yet mature enough.

[Eregon (Benoit Daloze)]

maximecb (Maxime Chevalier-Boisvert) wrote in #note-6:

I'd like to get input from people who are packaging Ruby, and those maintaining Ruby installation scripts.

The current active maintainers of e.g. ruby-build would be @hsbt (Hiroshi SHIBATA) and myself, I tried to base my replies on that, as well on my ruby/ruby-builder work.
It feels suboptimal to me that each ruby installation script/command-line would need to be adapted for this (or it'd disable YJIT) and e.g., add some way to install rust (OTOH it may be unavoidable).
If it's as simple as an extra package to install then it's probably fine.

As a concrete example, ruby-build currently only requires to install system packages: https://github.com/rbenv/ruby-build/wiki#suggested-build-environment
If we can just add rust/rust-devel or so to those lists then it seems fine (although it will be unnecessary for older Ruby versions).
That would mean it needs to work at least e.g. on latest Debian and RHEL releases (at the time of Ruby 3.2 release).
If we'd need some curl + rustup or so then I'd think it'd be quickly a lot more involved.

Also because many users already have ruby-build installed they likely wouldn't notice the dependencies changed and so would build without YJIT.
A good error message when using --yjit and it was not built which mentions why (due to not having rust X.Y available when building Ruby) seems helpful for that.

Also what should be the behavior if the rust version is too old? As if it was missing?

ruby-install similarly keeps a list of packages to install: https://github.com/postmodern/ruby-install/blob/master/share/ruby-install/ruby/dependencies.txt
RVM as well.

Basically it will be N places to update vs 1 which is why I wonder if the solution to automatically download Rust when building CRuby (potentially opt-in) isn't better.
Also currently ruby-build and RVM don't show the build output of CRuby by default, hence they would completely miss that YJIT was not build.

to prebuild Ruby binary for packages, which covers most Ruby users.

I would not be sure about that. Most developers using Ruby install it from source AFAIK (because OS tend to only package a single Ruby version but apps typically need a specific version), there are no official binary builds of CRuby, and only RVM has some binaries for some OS versions.

[vo.x (Vit Ondruch)]

maximecb (Maxime Chevalier-Boisvert) wrote in #note-6:

I'd like to get input from people who are packaging Ruby, and those maintaining Ruby installation scripts.

I don't think CRuby should download anything during build. IMO rustc should be treated like gcc, it's something you should install before running the ./configure script.

Downloading anything (especially binary blobs) or even bundling into release tarball would be big no-go for Fedora/CentOS Stream/RHEL. The current state of the Ruby tarball is already problematic enough.

It seems like rustc is easy to install on supported platforms, and so should not be difficult to use to prebuild Ruby binary for packages, which covers most Ruby users. We would of course prefer to use more recent versions of Rust if possible, but we are flexible. If using an older version of Rust is necessary to make packaging easier, we can accomodate that requirement.

TBH, with my Fedora/CS/RHEL maintainer hat on, this proposal scares me.

1. If nothing else, it adds one additional and not exactly small dependency (which is immediately followed by bundling proposals). But I am not sure if it will stop there. Because you might start to require some additional libraries, not just Rust itself.
2. It makes bootstrapping of distribution harder. And if will be even harder once YJIT becomes default. It won't be today, tomorrow or maybe even years, but should YJIT be successful, it will eventually become default.
3. Last but not least, we would have to deal with new language (which can be seen as an opportunity, of course ;) )

I might be pessimistic, but it won't make life of package maintainer easier.

BTW should this be start of Ruby migration/rewrite in Rust, that would be different discussion ;-P

[maximecb (Maxime Chevalier-Boisvert)]

TBH, with my Fedora/CS/RHEL maintainer hat on, this proposal scares me.

I completely understand. This is a departure from the Ruby tradition so to speak and as such, has to be considered carefully.

I am not sure if it will stop there. Because you might start to require some additional libraries, not just Rust itself.

If it makes you feel any better, we are very disciplined programmers. As it is, YJIT has no external run-time dependencies besides what comes with the C compiler, and an optional disassembler which we built without by default. We're very careful about bloat and dependency creep. We don't want to build a fragile software product.

It won't be today, tomorrow or maybe even years, but should YJIT be successful, it will eventually become default.

That's true, but in order to get to that point, YJIT has to succeed, and we feel that we are running in to real maintainability challenges with C99. JIT compilers have to maintain a lot of auxiliary data structures that are associated with generated code. Intermediate representations, control flow graphs, etc. All of these things have to be allocated, initialized and freed correctly as they're being transformed. We run into situations where it's very easy to forget calling a function at the right place, etc. We also have more pedestrian issues such as, we have an x86 assembler, and soon we'll have an arm64 assembler. A lot of the instructions have the same name. You can add prefixes to your C function names to avoid clashes, but soon you end up with very long function names.

We want to implement things like lazily marking a page of code as writable/executable to reduce the number of system calls that we make. This is also more challenging to implement in C, because again, you need auxiliary data structures to track this sort of thing, which you're going to have to manually track memory allocation for.

Last but not least, we would have to deal with new language (which can be seen as an opportunity, of course ;) )
I might be pessimistic, but it won't make life of package maintainer easier.

I understand. I know that there are challenges. We want to be accommodating. Would you be provide some specifics of how difficult it would be to compile rust code while packaging Ruby for Fedora/CS/RHEL? Is there an existing rustc package? Would you be able to use rustup during bundling?

BTW should this be start of Ruby migration/rewrite in Rust, that would be different discussion ;-P

Probably not happening but one can dream ;)

[vo.x (Vit Ondruch)]

maximecb (Maxime Chevalier-Boisvert) wrote in #note-10:

Would you be provide some specifics of how difficult it would be to compile rust code while packaging Ruby for Fedora/CS/RHEL? Is there an existing rustc package? Would you be able to use rustup during bundling?

Just quickly checking, there is Rust available in Fedora/CS/RHEL as far as I can tell. I am not sure about level of support and expertise on the compiler itself. Nevertheless, I'd be optimistic that as long as Firefox is using Rust, it should be in good enough shape. OTOH, on RHEL, we might struggle with older versions, although it does not appear to be the case ATM. I'd need to poke some people to get more insights, because I am not very familiar with Rust TBH.

[naruse (Yui NARUSE)]

As a release manager, adding rust to the tool chain is big change. But your proposal is well considered about the problem. While the YJIT support and Rust dependency is optional and it doesn't affect the user experience on the YJIT-disabled environment, it's open to merge.

Personally it's interesting experiment. As vit mentioned "BTW should this be start of Ruby migration/rewrite in Rust, that would be different discussion ;-P", the rewiring in Rust is different discussion and a long long journey, but I expect the experiment will provide how it looks like in Ruby development.

[postmodern (Hal Brodigan)]

Maintainer of ruby-install here along with @havenwood. By design ruby-install will install all build and runtime dependencies from the system's package manager needed to compile/run a ruby with default configuration options; which is how one would normally install dependencies when compiling software manually. The currently supported package managers are:

• apt (Debian/Ubuntu)
• yum (CentOS)
• dnf (RHEL/Fedora, is replacing yum)
• zypper (OpenSUSE)
• pacman (Arch Linux)
• brew (aka Homebrew)
• port (aka MacPorts)
• pkg (aka FreeBSD Ports)

Note: OpenBSD/NetBSD's pkg_add is currently not supported.
Note: ruby-install currently does not support tracking additional or optional dependencies, but this could be added; it has been requested in the past for compiling CRuby with jemalloc support.

If CRuby does add rust (more specifically rustc and cargo) as build dependencies, I would only ask that a survey of the above mentioned package managers be done to verify that:

1. they provide packages for rustc and cargo.
2. those packages are relatively up-to-date.

To make things easier, I would only worry about the currently maintained and stable releases of OSes/Linux Distros. Users still on older unmaintained OS/Distro versions should upgrade in order to take advantage of security updates.

• Debian (stable):
  • https://packages.debian.org/bullseye/rustc
  • https://packages.debian.org/bullseye/cargo
• Ubuntu (20.04 LTS) rust packages:
  • https://packages.ubuntu.com/focal/rustc
  • https://packages.ubuntu.com/focal/cargo
• Fedora:
  • https://src.fedoraproject.org/rpms/rust
  • https://src.fedoraproject.org/rpms/cargo or https://src.fedoraproject.org/rpms/rust-cargo ?
• OpenSUSE:
  • https://software.opensuse.org/package/rustc-1_9
  • https://software.opensuse.org/package/cargo
• Arch Linux: https://archlinux.org/packages/extra/x86_64/rust/
• Homebrew: https://formulae.brew.sh/formula/rust
• MacPorts: https://ports.macports.org/port/rust/
• FreeBSD Ports: https://www.freshports.org/lang/rust

A problem I have seen in the past with LLVM-based languages, is that they either target an extremely recent version of LLVM not yet provided by the various package managers, or they target a much older version of LLVM which contains known bugs/issues they must workaround, or they give up and simply download and build their own copy of LLVM.

Targeting a very recent version of rust could make it difficult to build on systems who's package manager still has an older version of rust. Targeting a much older "stable" version of rust might also cause issues when the YJIT team runs into known bugs/issues. Downloading your own copy of rust (precompiled or building from source) may seem like the path of least resistance, however this can consume additional disk space, bandwidth, CPU, and time; and can lead to friction and grumbling among developers (ex: nokogiri compiles it's own copy of libxml2 each time you update it). While rust does recommend using their rustup script, which can be installed via some (but not all) package managers, I would defer to using the system's approved package that has been QAed by the package maintainers and is assured to work with the rest of the system; and make life easier for the ruby package maintainers.

[matz (Yukihiro Matsumoto)]

Accepted.

To be clear, it's OK to use Rust to implement YJIT (and other optional features in the future), but mainline CRuby will not be implemented in Rust. The Rust implemented Ruby should be an alternative implementation just like JRuby, TruffleRuby, or Artichoke.

If there's some porting problem caused by Rust on some platforms, as @postmodern (Hal Brodigan) mentioned, for example, the feature should be dropped on the platform.

Matz.

[maximecb (Maxime Chevalier-Boisvert)]

Thank you for chiming in @matz (Yukihiro Matsumoto) :)

We will try to work with the people doing the packages/ports to help them resolve problems if there are any.

We are currently still early in our investigation of the Rust port. I am curious to know if you would also be open to YJIT being written in a language like Zig, or is that too new/experimental? The tradeoff there is that Zig is not as mature as Rust, but it is easier to interface with C code. Since YJIT has very few dependencies, both options seem very feasible, though I suspect the community would much prefer Rust, even though it is farther removed from C.

[ianks (Ian Ker-Seymer)]

Exciting stuff! This doesn't need to turn into an "all things Rust" thread, but I wanted to highlight some Rust related stuff I'm working on bringing to the Ruby community. Hopefully it's helpful to some.

• Cargo support in Rubygems for native Rust extensions
• rb-sys - Auto-generated Rust bindings based on ruby.h

Given that Cargo’d gems will be built as cdylibs, there shouldn’t be any conflicts in Rust runtimes (rust_eh_personality et. al)

Please feel free to reach out!

Ian

[matz (Yukihiro Matsumoto)]

@maximecb (Maxime Chevalier-Boisvert), it's OK for me to use Zig to implement YJIT, as long as foreseeable portability problems (mentioned above) are addressed.
Besides that, I prefer Zig to Rust, as a long-term C programmer, but all up to you.

Matz.

[mwlang (Michael Lang)]

Suggestion: Consider Crystal language for this effort. It's Ruby-like syntax would make the YJIT more accessible to Rubyists getting involved and contributing and we would gain all the benefits of a modern language, which is the stated reasons for considering Rust.

[duerst (Martin Dürst)]

maximecb (Maxime Chevalier-Boisvert) wrote in #note-15:

We are currently still early in our investigation of the Rust port. I am curious to know if you would also be open to YJIT being written in a language like Zig, or is that too new/experimental? The tradeoff there is that Zig is not as mature as Rust, but it is easier to interface with C code. Since YJIT has very few dependencies, both options seem very feasible, though I suspect the community would much prefer Rust, even though it is farther removed from C.

I'm a bit confused. You mentioned that C99 doesn't have classes or methods, but I haven't found classes in Zig either. What language features are you actually looking for?

[postmodern (Hal Brodigan)]

@mwlang (Michael Lang) I'm a big fan of Crystal, but unfortunately it contains a GC and cannot compile down to a shared or static library that can be embedded. This disqualifies it, along with Nim and Go. However, it would definitely be possible to create an alternative Ruby built upon Crystal, similar to how PyPy is built on top of RPython.

@maximecb (Maxime Chevalier-Boisvert) I have been watching zig and definitely think it occupies an optimal middle ground between C and Rust. Unfortunately, zig is not yet available in most package managers. Zig is also currently working on support for compiling down to C (see https://github.com/ziglang/zig/issues/3772 and zig build-lib -ofmt=c ...), but it is currently very buggy. This would allow projects to use Zig along with C, but still release pure-C source archives that would not require the end-user to install the zig compiler in order to build from source.

[maximecb (Maxime Chevalier-Boisvert)]

After more discussions with the team, we've made the decision to go forward with Rust for the time being. The fact that Rust has such a large community of enthusiasts is an important factor. This makes it easier to find documentation, help and also to recruit talent. The maturity of the language and ecosystem is also a big plus.

I'm quite partial to Zig myself as it has some very powerful features (e.g. compile-time code execution) and interop with C is somewhat easier than with Rust. However, the fact that Zig hasn't reached version 1.0 yet is somewhat of a problem for us. We think the language needs a bit more time to mature before it's truly ready for prime time, but it's very promising, and I wouldn't be surprised if it tremendously grows in popularity in the future, maybe even passing Rust.

[bbrklm (Benson Muite)]

Numba uses llvmlite https://github.com/numba/llvmlite for its JIT, but seems to need many patches. They also primarily use the stable C LLVM API rather than the C++ API, though LLVM would have slow JIT startup time compared to the YJIT approach. Drawing from this experience, is there a stable Rust core that can be used? Probably a naive question, but what are the drawbacks in building scaffolding in Ruby or a Ruby DSL that will generate C code?

[zw963 (Wei Zheng)]

I think Rust should be treated like OpenSSL. If you build Ruby without OpenSSL, Ruby doesn't download OpenSSL for you, it just doesn't build the openssl extension. Similarly, if you build Ruby without Rust, Ruby just won't build yjit. That seems like the simplest and best way to handle things, IMO. After all, yjit is a purely optional part of Ruby, and use of yjit should be transparent, modulo performance.

I consider this solution is more better.

[zw963 (Wei Zheng)]

jeremyevans0 (Jeremy Evans) wrote in #note-7:

maximecb (Maxime Chevalier-Boisvert) wrote in #note-6:

believe that this is trivially supportable with the use of Rust editions, and if they are able to target the 2018 edition, it would ensure maximum compatibility. A schedule could be figured out for when it would be appropriate to switch to the 2021 edition (which might be the better target of such a migration).

2018 was quite a while back and there's been a decent amount of change to the language since then. The question I'd like to ask here is "is this necessary?" Would standardizing on the latest stable version or rust installable through the rustup tool not be enough? Would 6 or 12 months back be enough?

I think 12 months back is fine. However, whatever Rust version is initially agreed upon should be supported going forward. Bumping the version of Rust necessary to build YJIT should be considered with the same gravity as the decision to move from C89 to C99. If that seems impractical or too restrictive, that would signal to me that Rust is not yet mature enough.

Just want to mention, When build emacs editor from source, it will check the available feature when do ./configure, if one feature not available, will auto skip it, and show some message like this to user.

Does Emacs use -lxft? no
Does Emacs use -lsystemd? yes
Does Emacs use -ljansson? yes
Does Emacs use the GMP library? yes
Does Emacs directly use zlib? yes
Does Emacs have dynamic modules support? yes
Does Emacs use toolkit scroll bars? yes
Does Emacs support Xwidgets? no
Does Emacs have threading support in lisp? yes
Does Emacs support the portable dumper? yes
Does Emacs support legacy unexec dumping? no
Which dumping strategy does Emacs use? pdumper
Does Emacs have native lisp compiler? no

Emacs can still continue run make successful, for optional feature, i think this solution can be refer to.

[maximecb (Maxime Chevalier-Boisvert)]

• Status changed from Open to Closed
• Assignee set to maximecb (Maxime Chevalier-Boisvert)

Updating this ticket to say that Rust YJIT has been upstreamed as part of this pull request:
https://github.com/ruby/ruby/pull/5826

Rust YJIT is now at feature parity with the original C YJIT.

To enable it, a --enable-yjit must be set when configuring, e.g.:
./configure --enable-yjit

More details can be found here:
https://github.com/ruby/ruby/blob/master/doc/yjit/yjit.md#installation