Bug #18970: CRuby adds an invalid header to bin/bundle (and others) which makes it unusable in Bash on Windows

• Status: Third Party's Issue
• Assignee: -
• Target version: -
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[Eregon (Benoit Daloze)]

Same as https://github.com/oneclick/rubyinstaller2/issues/299, but I figured it's extremely likely to be a bug in CRuby and not in RubyInstaller2.

The original user issue is: https://github.com/ruby/setup-ruby/issues/371.
bundle does not work in a Bash shell on Windows -- without an extra gem install bundler --, and the reason is building CRuby on Windows either does not produce a bin/bundle or it has the wrong permissions and the wrong start.

I downloaded all latest releases from https://github.com/ruby/setup-ruby/blob/master/windows-versions.json and extracted them (I'm on Linux FWIW).

$ ls
rubyinstaller-2.4.10-1-x64     rubyinstaller-2.6.10-1-x64     rubyinstaller-3.0.4-1-x64     rubyinstaller-head-x64     ruby-mswin
rubyinstaller-2.4.10-1-x64.7z  rubyinstaller-2.6.10-1-x64.7z  rubyinstaller-3.0.4-1-x64.7z  rubyinstaller-head-x64.7z  ruby-mswin.7z
rubyinstaller-2.5.9-1-x64      rubyinstaller-2.7.6-1-x64      rubyinstaller-3.1.2-1-x64     ruby-mingw                 ruby-ucrt
rubyinstaller-2.5.9-1-x64.7z   rubyinstaller-2.7.6-1-x64.7z   rubyinstaller-3.1.2-1-x64.7z  ruby-mingw.7z              ruby-ucrt.7z

Of course only Ruby 2.7+ ships with Bundler, so for <=2.6 it's expected to be missing.

$ ls -l */bin/bundle
-rw-r--r--. 1 eregon eregon 707 Apr 19 22:22 rubyinstaller-3.1.2-1-x64/bin/bundle
-rw-r--r--. 1 eregon eregon 707 Aug 19 22:40 rubyinstaller-head-x64/bin/bundle
-rw-rw-rw-. 1 eregon eregon 564 Aug 20 11:15 ruby-mingw/bin/bundle
-rw-rw-rw-. 1 eregon eregon 829 Aug 20 11:09 ruby-mswin/bin/bundle
-rw-rw-rw-. 1 eregon eregon 564 Aug 20 11:20 ruby-ucrt/bin/bundle

So only 3.1 and head have bin/bundle.
But those 2 bin/bundle do not have the executable bit set.

They also start like this which sounds invalid for Bash:

$ cat rubyinstaller-3.1.2-1-x64/bin/bundle
:""||{ ""=> %q<-*- ruby -*-
@"%~dp0ruby" -x "%~f0" %*
@exit /b %ERRORLEVEL%
};{ #
bindir="${0%/*}" #
exec "$bindir/ruby" "-x" "$0" "$@" #
>,
}
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
...

On https://github.com/eregon/setup-ruby/runs/7843304711?check_suite_focus=true we can see 3.0, 3.1 and head fail for echo ~ && which -a bundle in bash.
2.7 avoids the issue in that CI run because the Bundler version is considered too old by setup-ruby and so gem install bundler is done there.

Needed fix

In general, I think it is very important that CRuby does NOT modify files in bin/, and so that they are exactly the same as when RubyGems would write them when installing the corresponding gem.
This has been a problem not only here but also in these two other issues:

• gem install bundler fails on Windows, needs --force to workaround: https://github.com/rubygems/rubygems/issues/5245
• gem install bundler fails on --enable-load-relative Rubies: https://github.com/ruby/setup-ruby/issues/98#issuecomment-719950719

tool/rbinstall.rb seems to be responsible for changing the bin/ files and therefore causing those bugs:
https://github.com/ruby/ruby/blob/209631a45f9682dedf718f4b4a140efe7d21a6fc/tool/rbinstall.rb#L487
Can we remove that?

Not that this issue cannot be solved in RubyGems, it's CRuby breaking bin/bundle (and others) in Bash on Windows.

[Eregon (Benoit Daloze)]

Ah, maybe this cryptic header actually works for Bash too?

Then the problem is that CRuby adds it instead of RubyGems.
It should be RubyGems adding it (if it's necessary at all), so that bin/bundle is the same before and after gem install bundler and gem install bundler doesn't fail as mentioned above.

The missing executable permission is likely still an issue in CRuby.

[nobu (Nobuyoshi Nakada)]

Eregon (Benoit Daloze) wrote in #note-1:

Ah, maybe this cryptic header actually works for Bash too?

The header is polyglot (triglot?) code.

For /bin/sh, the first :"" successes and the block after || is not executed, then in the next block, starting with { #, execs the ruby in the same directory with -x option, which direct to search #!ruby shebang line.

:""||{ ""=> %q<-*- ruby -*-
@"%~dp0ruby" -x "%~f0" %*
@exit /b %ERRORLEVEL%
};{ #
bindir="${0%/*}" #
exec "$bindir/ruby" "-x" "$0" "$@" #
>,
}

For cmd.exe, the first line is a comment (precisely a label which cannot use), then executes the ruby and exits.

:""||{ ""=> %q<-*- ruby -*-
@"%~dp0ruby" -x "%~f0" %*
@exit /b %ERRORLEVEL%
};{ #
bindir="${0%/*}" #
exec "$bindir/ruby" "-x" "$0" "$@" #
>,
}

For ruby, the first symbol literal is truthy, then the next hash literal is ignored till the end.

:""||{ ""=> %q<-*- ruby -*-
@"%~dp0ruby" -x "%~f0" %*
@exit /b %ERRORLEVEL%
};{ #
bindir="${0%/*}" #
exec "$bindir/ruby" "-x" "$0" "$@" #
>,
}

Then the problem is that CRuby adds it instead of RubyGems.
It should be RubyGems adding it (if it's necessary at all), so that bin/bundle is the same before and after gem install bundler and gem install bundler doesn't fail as mentioned above.

The rest should be the content generated by RubyGems.

The missing executable permission is likely still an issue in CRuby.

The rubyinstaller is for Windows, there is no concept of the executable permission.
On Windows, bundle.bat should be executed.

[Eregon (Benoit Daloze)]

@nobu (Nobuyoshi Nakada) Thank you for the explanation, quite clever code.

I think it is still an issue due to being fragile/inconsistent that this header only exists for bin/* files shipped with Ruby but not for bin/* files installed by RubyGems later.

The rubyinstaller is for Windows, there is no concept of the executable permission.
On Windows, bundle.bat should be executed.

Some people/users do use bash on Windows.
Does bash look at bundle.bat?
$PATHEXT is .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC so I guess maybe yes?

I'm not quite sure what's wrong, I'm no Windows expert, but somehow bundle install/which bundle doesn't work on Windows without gem install bundler.

2.7 with gem install bundler:
https://github.com/eregon/setup-ruby/runs/7938304106?check_suite_focus=true
That works fine, because of the gem install bundler.

-rwxr-xr-x 1 runneradmin None     567 Aug 21 10:56 bundle
-rw-r--r-- 1 runneradmin None      41 Aug 21 10:56 bundle.bat
-rw-r--r-- 1 runneradmin None     672 Apr 19 20:22 bundle.cmd

ls -l in bash reports bundle as executable, and there is all 3 files.

bundle install in bash shell works: https://github.com/eregon/setup-ruby/runs/7938351861?check_suite_focus=true#step:26:1

3.0:
https://github.com/eregon/setup-ruby/runs/7938304140?check_suite_focus=true

-rw-r--r-- 1 runneradmin None     672 Apr 19 20:22 bundle.cmd

There is no bin/bundle file (and also no bundle.bat)!
And bundle does not work, when used from a bash shell, I guess as a result of that.
And indeed in https://github.com/oneclick/rubyinstaller2/releases/download/RubyInstaller-3.0.4-1/rubyinstaller-3.0.4-1-x64.7z I also see no bin/bundle or bundle.bat.

Is it an issue of RubyInstaller2 or maybe an issue of the build process on Windows or of tool/rbinstall.rb maybe?

bundle install in bash shell fails: https://github.com/eregon/setup-ruby/runs/7938351901?check_suite_focus=true#step:26:24
D:\a\_temp\99c37f86-615f-4118-ab92-9280a5b4f61f.sh: line 1: bundle: command not found

3.1:
https://github.com/eregon/setup-ruby/runs/7938304170?check_suite_focus=true

-rw-r--r-- 1 runneradmin None     707 Apr 19 20:22 bundle
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 bundle.bat

There is bundle and bundle.bat but it does not work for which bundle. Maybe bundle.cmd is needed for which bundle?
Note also how ls -l reports bundle is not executable here. Maybe because of the missing bundle.cmd?

bundle install in bash shell works: https://github.com/eregon/setup-ruby/runs/7938351950?check_suite_focus=true#step:26:24
But which bundle fails: https://github.com/eregon/setup-ruby/runs/7938351950?check_suite_focus=true#step:30:25

[Eregon (Benoit Daloze)]

Another thought, maybe -ls -l in bash shows the x bit if the file starts with a shebang or is a native executable, but due to this header it's neither?

[Eregon (Benoit Daloze)]

Here is another run, where none of the Rubies does a gem install Bundler.
https://github.com/eregon/setup-ruby/runs/7938431059?check_suite_focus=true

2.6, 2.7 and 3.0 all fail bundle install in bash with bundle: command not found.
All these only have bundle.cmd and no bundle/bundle.bat file.

For 3.1 and master, bundle install in bash works, but which bundle fails.
These 2 have bundle (with the polyglot header) and bundle.bat.

As a side note, bundle install and where bundle in PowerShell does work for all of them.
So it's an issue when using bundle from the bash shell on Windows.

[Eregon (Benoit Daloze)]

I also noted from the run above, on 2.6-3.0:

# Output from 2.6
$ ls -l $(dirname $(which ruby))
total 3627
-rw-r--r-- 1 runneradmin None     672 Apr 19 20:27 bundle.cmd
-rw-r--r-- 1 runneradmin None     674 Apr 19 20:27 bundler.cmd
-rwxr-xr-x 1 runneradmin None    5086 Apr 19 20:27 erb
-rw-r--r-- 1 runneradmin None    5228 Apr 19 20:27 erb.cmd
-rwxr-xr-x 1 runneradmin None     546 Apr 19 20:27 gem
-rw-r--r-- 1 runneradmin None     688 Apr 19 20:27 gem.cmd
-rwxr-xr-x 1 runneradmin None     508 Apr 19 20:27 irb
-rw-r--r-- 1 runneradmin None     685 Apr 19 20:27 irb.cmd
-rwxr-xr-x 1 runneradmin None     502 Aug 21 11:20 rake
-rw-r--r-- 1 runneradmin None      41 Aug 21 11:20 rake.bat
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:27 rake.cmd
-rwxr-xr-x 1 runneradmin None     514 Apr 19 20:27 rdoc
-rw-r--r-- 1 runneradmin None     656 Apr 19 20:27 rdoc.cmd
-rwxr-xr-x 1 runneradmin None     510 Apr 19 20:27 ri
-rw-r--r-- 1 runneradmin None     652 Apr 19 20:27 ri.cmd
-rw-r--r-- 1 runneradmin None     694 Apr 19 20:27 ridk.cmd
-rw-r--r-- 1 runneradmin None     876 Apr 19 20:27 ridk.ps1
-rwxr-xr-x 1 runneradmin None   34304 Apr 19 20:27 ruby.exe
drwxr-xr-x 1 runneradmin None       0 Apr 19 20:27 ruby_builtin_dlls
-rwxr-xr-x 1 runneradmin None   34304 Apr 19 20:27 rubyw.exe
-rw-r--r-- 1 runneradmin None     312 Apr 19 20:27 setrbvars.cmd
-rwxr-xr-x 1 runneradmin None 3597824 Apr 19 20:27 x64-msvcrt-ruby260.dll

vs on 3.1-master:

$ ls -l $(dirname $(which ruby))
total 4148
-rw-r--r-- 1 runneradmin None     707 Apr 19 20:22 bundle
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 bundle.bat
-rw-r--r-- 1 runneradmin None     709 Apr 19 20:22 bundler
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 bundler.bat
-rw-r--r-- 1 runneradmin None     668 Apr 19 20:22 erb
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 erb.bat
-rwxr-xr-x 1 runneradmin None     546 Apr 19 20:22 gem
-rw-r--r-- 1 runneradmin None     689 Apr 19 20:22 gem.cmd
-rw-r--r-- 1 runneradmin None     668 Apr 19 20:22 irb
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 irb.bat
-rw-r--r-- 1 runneradmin None     674 Apr 19 20:22 racc
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 racc.bat
-rw-r--r-- 1 runneradmin None     674 Apr 19 20:22 rake
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 rake.bat
-rw-r--r-- 1 runneradmin None     668 Apr 19 20:22 rbs
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 rbs.bat
-rw-r--r-- 1 runneradmin None     674 Apr 19 20:22 rdoc
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 rdoc.bat
-rw-r--r-- 1 runneradmin None     670 Apr 19 20:22 ri
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 ri.bat
-rw-r--r-- 1 runneradmin None     694 Apr 19 20:22 ridk.cmd
-rw-r--r-- 1 runneradmin None     876 Apr 19 20:22 ridk.ps1
-rwxr-xr-x 1 runneradmin None   35840 Apr 19 20:22 ruby.exe
drwxr-xr-x 1 runneradmin None       0 Apr 19 20:22 ruby_builtin_dlls
-rwxr-xr-x 1 runneradmin None   35840 Apr 19 20:22 rubyw.exe
-rw-r--r-- 1 runneradmin None     312 Apr 19 20:22 setrbvars.cmd
-rw-r--r-- 1 runneradmin None     698 Apr 19 20:22 typeprof
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 typeprof.bat
-rwxr-xr-x 1 runneradmin None 4140544 Apr 19 20:22 x64-ucrt-ruby310.dll

So erb/gem/irb/rdoc/ri are all seen as executables on 2.6-3.0, and they all start with #!/usr/bin/env ruby (so no header).

rake has the header on 2.6-3.0 though and is seen as executable:

-rw-r--r-- 1 runneradmin None     672 Apr 19 20:22 bundle.cmd
...
-rwxr-xr-x 1 runneradmin None     519 Aug 21 11:19 rake
-rw-r--r-- 1 runneradmin None      41 Aug 21 11:19 rake.bat
-rw-r--r-- 1 runneradmin None      41 Apr 19 20:22 rake.cmd

So maybe the .cmd is needed to be seen as executable?

On 3.1-master, none of the bin/* are seen as executable and all have the header.
Except for gem which is seen as executable and has no header.

[nobu (Nobuyoshi Nakada)]

Does this work?
https://github.com/nobu/ruby/tree/prolog_script

[Eregon (Benoit Daloze)]

Unfortunately I can't build ruby from source on Windows myself to test, maybe @MSP-Greg can test that?

From above:

rake has the header on 2.6-3.0 though and is seen as executable:

Specifically:

$ cat rubyinstaller-3.0.4-1-x64/bin/rake
:""||{ ""=> %q<-*- ruby -*-
@"%~dp0ruby" -x "%~f0" %*
@exit /b %ERRORLEVEL%
};{#
bindir="${0%/*}" #
exec "$bindir/ruby" "-x" "$0" "$@" #
>,
}
#!/usr/bin/env ruby
...

So that seems to disprove my guess that the shebang makes the script seen as executable.

I think best would be:

• no triglot header or other header (since RubyGems doesn't generate it, and it causes various issues when RubyGems installs over such a file with header)
• For every bin/file: file, file.bat and file.cmd (on Windows only of course for the last two)

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada) - I will try that build, but I can't until later.

I don't know what the best approach is for Windows binstubs. Ruby-loco and RubyInstaller2 have always supplied 'portable' builds, not sure if that should be the case for a standard ruby/ruby build.

1. Not sure how far to go with getting things to run 'perfectly' when binstubs are ran in an install folder that is not the ENV Ruby.

2. Not sure about modifying ENV['Path'] in binstub scripts.

Re the current scripts, when run from a Windows cmd shell, I believe they worked. But, when run from a PowerShell window, they opened a separate cmd window...

[austin (Austin Ziegler)]

MSP-Greg (Greg L) wrote in #note-9:

@nobu (Nobuyoshi Nakada) - I will try that build, but I can't until later.

I don't know what the best approach is for Windows binstubs. Ruby-loco and RubyInstaller2 have always supplied 'portable' builds, not sure if that should be the case for a standard ruby/ruby build.

1. Not sure how far to go with getting things to run 'perfectly' when binstubs are ran in an install folder that is not the ENV Ruby.

2. Not sure about modifying ENV['Path'] in binstub scripts.

Re the current scripts, when run from a Windows cmd shell, I believe they worked. But, when run from a PowerShell window, they opened a separate cmd window...

Do we need the multi-language header anymore with ruby -S binary? It seems to me that bin/bundle could be what it is now with only the Unix headers, and bin/bundle.bat and bin/bundle.cmd just call ruby -S bin/bundle or something like that?

I haven’t used Windows for a very long time, but it feels like this could be something that is part of the Ruby installation and possibly glommed onto binstub generation for bundler and/or Rubygems so that on systems that need .bat or .cmd files, those are added regardless because it ends up calling ruby -s bin/bundle. More or less.

[Eregon (Benoit Daloze)]

I also noticed which rake does not work on Ruby 2.5+, but it does work on Ruby < 2.5:
https://github.com/ruby/setup-ruby/runs/7938245817?check_suite_focus=true

austin (Austin Ziegler) wrote in #note-10:

Do we need the multi-language header anymore with ruby -S binary? It seems to me that bin/bundle could be what it is now with only the Unix headers, and bin/bundle.bat and bin/bundle.cmd just call ruby -S bin/bundle or something like that?

ruby -S on CRuby doesn't actually look in that CRuby's bin/ dir first, so that's not helpful or better than just binary:

(on Linux, with `puts "3.0.3 bundle"` in ~/.rubies/ruby-3.0.3/bin/bundle)
$ which bundle
~/.rubies/ruby-3.0.3/bin/bundle
$ ~/.rubies/ruby-master/bin/ruby -S bundle --version
3.0.3 bundle
Bundler version 2.4.0.dev

On TruffleRuby and JRuby it does look in bin/ first and it's helpful for such cases.

Agreed we shouldn't need the multi-language header.
When using Bash on Windows, it's IMHO fine to expect the ruby used is first in PATH.

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada) & @Eregon (Benoit Daloze)

I just tried the prolog_script script branch. Wondering if the following example would be fine with everyone? It seems to work for me locally. If they're run as 'in path' commands, they work, if they're run from the folder of a 'non-system' Ruby, they run, although, since they don't set Path, any Ruby sub-processes hopefully will use Gem.ruby or something similar...

I tested the below with MSYS2 bash and Git bash.

Windows bash file 'top'

{
bindir=$(dirname "$0")
exec "$bindir/ruby" "-x" "$0" "$@"
}
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.

-- remaining code --

Windows *.cmd or *.bat binstub

@ECHO OFF
@"%~dp0ruby.exe" -x "%~dpn0" %*

I built a test, link is https://github.com/MSP-Greg/ruby-loco-test/releases/download/ruby-master/ruby-mswin.7z. It's the same as the builds used in CI.

Note that the bash scripts work the same as the Windows (*.bat, *.cmd) scripts. They launch with the ruby.exe in the same folder, not the ENV Ruby.

Finally, the ruby/ruby code isn't generating a 'gem' bash file.

[MSP-Greg (Greg L)]

The Ruby build linked to above had incorrect bash scripts. Fixed as of now. Sorry, bad day yesterday, too many interruptions...

[nobu (Nobuyoshi Nakada)]

MSP-Greg (Greg L) wrote in #note-12:

Windows bash file 'top'

{
bindir=$(dirname "$0")
exec "$bindir/ruby" "-x" "$0" "$@"
}
#!/usr/bin/env ruby

I'm not sure where the header came from.
rbinstall.rb doesn't use dirname command.
Maybe here?

Windows *.cmd or *.bat binstub

@ECHO OFF
@"%~dp0ruby.exe" -x "%~dpn0" %*

Nor @ECHO OFF.

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada)

Sorry the above are what I thought would work best with the ruby-loco builds. All three builds (mingw, mswin, & ucrt) are now done that way. Interesting thing that I found is that MSYS2's which command considers a file to executable if it is an exe file or if it starts with a shebang (#!). The bash file 'top' is now:

#!
{
bindir=$(dirname "$0")
exec "$bindir/ruby" "-x" "$0" "$@"
}
#!/usr/bin/env ruby

The above allows the correct Ruby to be used if a bash binstub is called with a path, which is similar to the way the bat/cmd files work.

[nobu (Nobuyoshi Nakada)]

MSP-Greg (Greg L) wrote in #note-15:

Interesting thing that I found is that MSYS2's which command considers a file to executable if it is an exe file or if it starts with a shebang (#!).

That behavior should be inherited from Cygwin.

The above allows the correct Ruby to be used if a bash binstub is called with a path, which is similar to the way the bat/cmd files work.

Then this issue can be closed as rubyinstaller2 is not using rbinstall.rb?

[MSP-Greg (Greg L)]

That behavior should be inherited from Cygwin.

Thanks, never saw a specific reference to that.

I'm ok to close, not sure about RubyInstaller2, probably better to have an issue there if there are issues with Bash bin files...

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Third Party's Issue