Feature #16517: mkmf.rb - changes for Windows ?

• Status: Open
• Assignee: -
• Target version: -

[MSP-Greg (Greg L)]

I propose two changes to mkmf.rb to make it more Windows friendly.

1. mingw - devkit has been the standard for 'enabling' compile tools in publicly available MinGW builds for quite a while. Could something like the following be added? Not sure whether to rescue on LoadError or not, or whether to output anything if it doesn't load.

if $mingw
  begin
    require 'devkit'
  rescue
  end
end

2. mswin - most compile tools other than msvc will find libraries without a lib prefix. Note the following code in extconf.rb for OpenSSL:

ret = have_library("crypto", "CRYPTO_malloc") &&
  have_library("ssl", "SSL_new")
return ret if ret

if $mswin
  # OpenSSL >= 1.1.0: libcrypto.lib and libssl.lib.
  if have_library("libcrypto", "CRYPTO_malloc") &&
      have_library("libssl", "SSL_new")
    return true
  end

If something like the following was added, the above wouldn't be needed:

if $mswin
  alias_method :orig_find_library, :find_library
  
  def find_library(lib, func, *paths, &b)
    orig_find_library(lib, func, *paths, b) || orig_find_library("lib#{lib}", func, *paths, b)
  end

  alias_method :orig_have_library, :have_library

  def have_library(lib, func = nil, headers = nil, opt = "", &b)
    orig_have_library(lib, func, headers, opt, b) || orig_have_library("lib#{lib}", func, headers, opt, b)
  end
end

Adding something similar to above two items would remove the need for Windows specific build code in many extension gems.

[nobu (Nobuyoshi Nakada)]

MSP-Greg (Greg L) wrote:

I propose two changes to mkmf.rb to make it more Windows friendly.

1. mingw

What is devkit?
Is it https://rubygems.org/gems/devkit ?
It doesn't look special for MinGW.

2. mswin - most compile tools other than msvc will find libraries without a lib prefix.
   Adding something similar to above two items would remove the need for Windows specific build code in many extension gems.

Until such gems remove the code, it won't repeat same compilations 4 times?

[duerst (Martin Dürst)]

nobu (Nobuyoshi Nakada) wrote:

What is devkit?
Is it https://rubygems.org/gems/devkit ?

I don't think so. Please have a look at https://github.com/oneclick/rubyinstaller/wiki/Development-Kit.

[nobu (Nobuyoshi Nakada)]

duerst (Martin Dürst) wrote:

I don't think so. Please have a look at https://github.com/oneclick/rubyinstaller/wiki/Development-Kit.

Thank you, I thought it might be one of RubyInstaller but couldn't find out.
If so, it sounds a RubyInstaller specific file.
If it is required always, non-RubyInstaller mingw build can't compile extensions anymore.
And, as the name conflicts with the above gem, it is unable to install with rubygems.
I think this part is unacceptable.

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada)

MinGW devkit issue

devkit.rb is a file included in RubyInstaller and RubyInstaller2 builds. These are the standard for Windows Rubies, and all CI that I have ever worked with uses them, and devkit.rb is included (on current master, located in lib/ruby/site_ruby/2.8.0). 'RubyInstaller' Rubies are compiled with MSYS2 tools, and hence are 'mingw'.

On non Windows OS's, c build tools and many packages are installed and available. On Windows, MSYS2 tools are needed to build mingw Ruby, and also to compile extension gems. MSYS2 tools are not normally in one's PATH, so devkit.rb has code to add them so make, gcc, etc are available. Many CI scripts (rake, yml, etc) account for this by requiring devkit in some way or other.

As to installing extension gems, 'RubyInstaller' Rubies set PATH to include MSYS2 tools via Gem.pre_install located in rubygems/defaults/operating_system.rb.

So, installing gems is being accounted for, but directly using mkmf.rb doesn't use those hooks, so CI won't working unless specific code is added to require devkit.

Note that I loaded it with rescue, as (for instance) it obviously won't load when building ruby itself. Also, the devkit gem you mentioned is rather old, and I really doubt any windows systems will have it installed.

mswin library issue

I think with the OpenSSL code listed, compile will be called three times:

have_library("crypto", "CRYPTO_malloc")     #=> fail
have_library("libcrypto", "CRYPTO_malloc")  #=> pass
have_library("libssl", "SSL_new")           #=> pass

The code I suggested will call have_library four times, which is an increase. But, the vast majority of extconf.rb files do not have mswin specific code, and will fail if a needed library starts with lib.

mswin builds are not publicly available, but they are supported here in CI. When working on https://github.com/ruby/ruby/pull/2838, I saved the build as an artifact and installed it locally. I then tried to build puma, which compiles with OpenSSL, and noticed the issue. I then looked at ruby/openssl, and noticed the mswin specific code.

Many packages/libraries required by popular extension gems are available from https://github.com/Microsoft/vcpkg, so there is a chance that many extension may compile on mswin.

One issues with MinGW builds is that MSYS2 packages are compiled against msvcrt.dll, while many Windows apps are compiled against vcruntime140.dll. This runtime mismatch can cause issues, especially with 'embedded Ruby'. Hence, I think there are reasons to have 'real world' support of mswin, which uses vcruntime140.dll.

General reasons for this request

1. With the arrival of Actions, having one CI script/workflow run with a minimal amount of platform specific code is possible with extension gems on MinGW & mswin.

2. The two suggested changes are real world issues that I believe fall within the definition of 'supporting a platform'. They could be backported to currently supported Ruby versions.

[larskanis (Lars Kanis)]

I think mkmf.rb is a good place to add require 'devkit'. It doesn't catch the cases where the Rakefile makes use of development or shell tools (which etc.), but it would enable the typical extension build in CI without further measurements. require 'devkit' will fail in cross build environment aka. rake-compiler-dock, but since it's rescued, it doesn't matter much.

If this is not acceptable to be added to ruby, I'm positive to add this to RubyInstaller including backports. That's maybe the better place anyway, since the corresponding gem install hook is added in RubyInstaller as well (not in ruby or rubygems).

Although I don't case much about mswin alias Visual-C platform, require 'devkit' could be useful there as well. They use a similar development environment which is enabled by various environment variables.

[MSP-Greg (Greg L)]

Re the mswin change, the cost is low, and the benefit is high.

Re requiring devkit, my request is a specific solution to a specific problem.

RubyGems has hooks, and will also load/require specific files, like operating_system.rb. *.gemspec files also allow for 'free-form' metadata.

Rather than the specific solution, should something like the following be added as a hook for packagers in mkmf.rb?

begin
  require "#{RbConfig::CONFIG['sitelibdir']}/mkmf_pre.rb"
rescue
end

MinGW builds using RubyInstaller2/RIDK use gemspec metadata to install packages/libraries needed to compile/install an extension gem.

Something similar could be used in mkmf_pre.rb to allow packagers to do the same to 'enable the typical extension build in CI'.

Taking it further, *.gemspec metadata could include the names of packages required for Ubuntu, macOS, mingw, and maybe mswin. This could be used for both extension CI builds and gem installation. Certainly would make cross platform building/installing/compiling easier on MRI builds, and minimize the platform specific code needed.

@larskanis (Lars Kanis) - Thanks for the post. I apologize for not pinging you here. I'm not quite all there (flu).

[shevegen (Robert A. Heiler)]

Just a comment or two - please don't get distracted from the main issue greg referred
to, it is just my opinion:

Taking it further, *.gemspec metadata could include the names of packages required
for Ubuntu, macOS, mingw, and maybe mswin. T

I think it would be better to not include specific distributions per se, as people
may want to add their own distributions to that list ("why is distribution A a
first-class citizen but not distribution B?"). IMO it would be best if rubygems

• mkmf are just as agnostic as possible, and as flexible/adaptable as possible.
  That way distributions can adjust the ruby ecosystem to their needs, without
  needing special per-distribution hooks as such.

As for where to include the code as such, I would actually recommend to have mkmf
support the functionality if that functionality is currently missing, rather than
put it into the one-click installer or elsewhere, IF this is in general
windows-platform specific (been some time since I last used windows + msys/mingw).
So from that point of view, if there are no significant trade-offs otherwise, I
would agree with lars in regards to his statement "[...]mkmf.rb is a good place
to add require 'devkit' [...]", with which I would like to agree with.

On a not-so-related note, the meson/ninja team once wondered whether, perhaps
two years ago, to include code for general enhancement of functionality that
only one or two distributions did want or added (fedora was one), or whether to
let the distributions handle such add-on case(s) on their own.

Back then I reasoned that it would be better to include general-purpose functionality
that could benefit distributions right into meson directly, and bundle it there, since
that would benefit more people than special per-distribution modifications only. It
would also help against dilution of enhancements to meson, which I think is better
in the long run to avoid - sort of like having a "standard", "generic" specification
for meson in one place primarily.

Following the same reasoning here for ruby and the ruby-ecosystem, IMO it would make
sense to have mkmf support it too (perhaps structure that code as a gem-plugin for
mkmf or so; that way you can always decide to not include it lateron, but I would also
recommend to add support for it into mkmf; after all one reason for gemifying
the ruby stdlib was to make it easier to add/remove gems/code by default too,
like what Hiroshi Shibata has been doing a lot in the last years :)).

[dsisnero (Dominic Sisneros)]

As we are adding the ability to make gems from rust added in rubygems, can we come back to this and enable this or add some kind of hook to set shell environment - if a hook is added it would work for rubyinstaller and also Ruby MSVC/ by running vcvars64.bat or doing the equivalent in ruby.