Project

General

Profile

Bug #14964

[MJIT] missing builtin methods

Added by ahorek (Pavel Rosický) 10 days ago. Updated 8 days ago.

Status:
Open
Priority:
Normal
Target version:
-
[ruby-core:88292]

Description

I'm having trouble building Ruby 2.6 on Windows.

this is the first error:

building rb_mjit_header.h
rb_mjit_header.h updated
building .ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h
./miniruby.exe -I../ruby_2_6/lib -I. -I.ext/common  ../ruby_2_6/tool/transform_mjit_header.rb "x86_64-w64-mingw32-gcc " rb_mjit_header.h .ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h
error in initial header file:
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:1: warning: "__STDC__" redefined
 #define __STDC__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:2: warning: "__STDC_VERSION__" redefined
 #define __STDC_VERSION__ 201710L

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:3: warning: "__STDC_UTF_16__" redefined
 #define __STDC_UTF_16__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:4: warning: "__STDC_UTF_32__" redefined
 #define __STDC_UTF_32__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:5: warning: "__STDC_HOSTED__" redefined
 #define __STDC_HOSTED__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:29: warning: "__BIGGEST_ALIGNMENT__" redefined
 #define __BIGGEST_ALIGNMENT__ 32

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c: In function '__crc32b':
C:/Users/pdaho/AppData/Local/Temp/20180805-10076-12b1zho.c:6216:10: error: implicit declaration of function '__builtin_ia32_crc32qi'; did you mean '__builtin_ia32_rolqi'? [-Werror=implicit-function-declaration]
   return __builtin_ia32_crc32qi (__C, __V);
          ^~~~~~~~~~~~~~~~~~~~~~
          __builtin_ia32_rolqi
compilation terminated due to -Wfatal-errors.
cc1.exe: some warnings being treated as errors
make: *** [.ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h] Error 1
rm x64-msvcrt-ruby260.rc
rake aborted!

I tried various CFLAGS, it looks like -mcrc32 helped

building rb_mjit_header.h
rb_mjit_header.h updated
building .ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h
./miniruby.exe -I../ruby_2_6/lib -I. -I.ext/common  ../ruby_2_6/tool/transform_mjit_header.rb "x86_64-w64-mingw32-gcc " rb_mjit_header.h .ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h
error in initial header file:
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:1: warning: "__STDC__" redefined
 #define __STDC__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:2: warning: "__STDC_VERSION__" redefined
 #define __STDC_VERSION__ 201710L

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:3: warning: "__STDC_UTF_16__" redefined
 #define __STDC_UTF_16__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:4: warning: "__STDC_UTF_32__" redefined
 #define __STDC_UTF_32__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:5: warning: "__STDC_HOSTED__" redefined
 #define __STDC_HOSTED__ 1

<built-in>: note: this is the location of the previous definition
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c: In function '_mm_stream_sd':
C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:9317:3: error: implicit declaration of function '__builtin_ia32_movntsd'; did you mean '__builtin_ia32_movntpd'? [-Werror=implicit-function-declaration]
   __builtin_ia32_movntsd (__P, (__v2df) __Y);
   ^~~~~~~~~~~~~~~~~~~~~~
   __builtin_ia32_movntpd
compilation terminated due to -Wfatal-errors.
cc1.exe: some warnings being treated as errors
make: *** [.ext/include/x64-mingw32/rb_mjit_min_header-2.6.0.h] Error 1
rm x64-msvcrt-ruby260.rc
rake aborted!

I tried to add -march=barcelona -mtune=barcelona -msse4a as the gcc documentation says, but it didn't help. It looks like these instruction are SSE4a specific and some headers are missing.

rev. https://github.com/ruby/ruby/commit/88975c821c99b852959958e2934de68a45be15e6
Windows 10 64bit, x86_64-w64-mingw32
GCC 7.2.0 - http://downloads.sourceforge.net/project/mingw-w64/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/7.2.0/threads-win32/seh
AMD FX-8300 8C

I can compile the lastest Ruby 2.5.1 with the same options.

Is it a known problem? Maybe GCC isn't supported on Windows yet?

mjitheaders.zip (1.31 MB) mjitheaders.zip ahorek (Pavel Rosický), 08/05/2018 03:51 PM
diff.diff (3.65 KB) diff.diff ahorek (Pavel Rosický), 08/05/2018 04:04 PM
rb_mjit_header_msys2.zip (673 KB) rb_mjit_header_msys2.zip ahorek (Pavel Rosický), 08/06/2018 10:35 PM
diff_msys2.diff (1.75 MB) diff_msys2.diff ahorek (Pavel Rosický), 08/06/2018 10:35 PM
env.txt (2.58 KB) env.txt ahorek (Pavel Rosický), 08/07/2018 12:19 PM
output_archflag_72.txt (109 KB) output_archflag_72.txt ahorek (Pavel Rosický), 08/07/2018 12:19 PM
output_optflag_81.txt (83.3 KB) output_optflag_81.txt ahorek (Pavel Rosický), 08/07/2018 12:19 PM
output_witharch_81.txt (31 KB) output_witharch_81.txt ahorek (Pavel Rosický), 08/07/2018 12:19 PM

History

#1 [ruby-core:88293] Updated by shevegen (Robert A. Heiler) 10 days ago

[...] I can compile the lastest Ruby 2.5.1 with the same options.
Maybe GCC isn't supported on Windows yet? [...]

I assume that k0kubun is working on the mjit-component right now (possibly
excluding Sundays :D) - he is like super-busy changing mjit code almost
every day. :)

The changelog that, I think, Hiroshi Shibata made available through git
via a web interface here, shows that:

https://git.ruby-lang.org/ruby.git/

There were some commits in regards to reshuffling code in/for headers
such as:

https://git.ruby-lang.org/ruby.git/commit/?id=d7786b1e7814a776b7c0307e64fbbf13324131a9

MSP-Greg ( https://bugs.ruby-lang.org/users/11129 ) is also quite active in
regards to windows-related (mswin I think?) parts and providing feedback, so
my recommendation would be to try again in a few days; 2.6 may compile just
fine again.

#2 [ruby-core:88294] Updated by MSP-Greg (Greg L) 10 days ago

ahorek (Pavel Rosický)

The most recent ruby-loco build (MinGW) passed all tests and had the following output when running test_jit.rb locally:

Finished tests in 279.718157s, 0.2789 tests/s, 1.7696 assertions/s.
78 tests, 495 assertions, 0 failures, 0 errors, 6 skips

ruby -v: ruby 2.6.0dev (2018-08-05 trunk 64193) [x64-mingw32]
untested insns are found!: putspecialobject putiseq

It's using the MSYS2 build tools, they very recently updated to gcc 8.3.0, but previously were using 7.3.0.

The repo is at https://github.com/MSP-Greg/ruby-loco, it can be run locally. The most recent build is at https://ci.appveyor.com/project/MSP-Greg/ruby-loco/build/artifacts (ruby_trunk.7z).

shevegen (Robert A. Heiler) Thanks for the mention. I have worked with mswin (vc12) and vc14 builds, but ruby-loco and most of my work is with MinGW builds. The new RubyInstall builds are MSYS2/MinGW, and most of the Windows pre-compiled ruby gems are also MSYS2/MinGW...

#3 [ruby-core:88295] Updated by ahorek (Pavel Rosický) 10 days ago

I'm using MSYS1 that could be a problem, but I don't see any difference or a reason why it shouldn't work. You don't pass any additional flags in ruby loco, also you have some patches, but none of them seem to be related to this error. Maybe the environment is different somehow even if the gcc version is the same (I tried multiple versions).
So if MSYS2 works, it's ok I think. Unless somebody knows where the problem is and it's simple to fix.

My only concern is, if mjit requires sse4 specific instructions, it could be a problem on platforms that do not support them? What about ARM?

#4 [ruby-core:88296] Updated by k0kubun (Takashi Kokubun) 10 days ago

  • Assignee set to k0kubun (Takashi Kokubun)

Is it a known problem?

No, thanks to report.

Maybe GCC isn't supported on Windows yet?

I intend to support at least gcc on msys2/MinGW in MJIT and sometimes check if it works.

Windows 10 64bit, x86_64-w64-mingw32
GCC 7.2.0 - http://downloads.sourceforge.net/project/mingw-w64/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/7.2.0/threads-win32/seh
AMD FX-8300 8C

I'm using Windows 10 64bit too, and:

$ uname -a
MINGW64_NT-10.0 DESKTOP-BI5TMNA 2.9.0(0.318/5/3) 2017-09-13 23:16 x86_64 Msys
/home/k0kubun/src/github.com/ruby/ruby status: 0                                                                                          (git)-[trunk]-
$ gcc -v
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/7.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-7.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 7.2.0 (Rev1, Built by MSYS2 project)

Some differences: You're using AMD and I'm using Intel processor. You're using msys1 and I'm using msys2. Not sure which is related.

C:/Users/pdaho/AppData/Local/Temp/20180805-1196-12y4d1j.c:1: warning: "STDC" redefined
#define STDC 1

Anyway, this code is expected to be removed in this line: https://github.com/ruby/ruby/blob/trunk/tool/transform_mjit_header.rb#L209. Otherwise the check will be broken like you shared.
So the header before transformation is something not expected. Could you upload your rb_mjit_header.h in your build directory?

#5 [ruby-core:88297] Updated by ahorek (Pavel Rosický) 10 days ago

after some investigation, I found the problem is my build options

this builds successfully

rb_mjit_header_ok.h
"CXXFLAGS='-O3'",
"CFLAGS='-O3'"

but if I add march/mtune it fails

rb_mjit_header_fail.h
"CXXFLAGS='-march=barcelona -mtune=barcelona -O3'",
"CFLAGS='-march=barcelona -mtune=barcelona -O3'"

I attached both header files

#6 Updated by ahorek (Pavel Rosický) 10 days ago

  • File diff.diff added

#7 Updated by ahorek (Pavel Rosický) 10 days ago

  • File deleted (diff.diff)

#9 [ruby-core:88300] Updated by k0kubun (Takashi Kokubun) 9 days ago

Could you test configuring ARCH_FLAG instead of CFLAGS/CXXFLAGS? I'm not sure which variable should be used for -mtune, but it seems that at least -march=barcelona is expected to be set as ARCH_FLAG like ARCH_FRAG="-march=barcelona". I expect ARCH_FLAG="-march=barcelona -mtune" to work, but otherwise please try ARCH_FLAG"-march=barcelona" OPTFLAGS="-mtune=barcelona -O3" as well.

#10 [ruby-core:88316] Updated by ahorek (Pavel Rosický) 8 days ago

ARCH_FLAG="-march=barcelona" works, but it isn't propagated to CFLAGS, so it doesn't have the desired effect. I think setting CFLAGS directly shouln't be a problem.

It looks like it doesn't work only on MSYS1. I did the same on Linux and MSYS2 (gcc 8.2.0) without any problem.
An interesting part is that on MSYS2, mjit header doesn't contain #define __SSE4A__ 1

irb(main):001:0> RbConfig::CONFIG['CFLAGS']
=> "-march=barcelona -mtune=barcelona -O3 -pipe"

maybe march option should be ignored when building the mjit header and for some reason MSYS1 used it?

MSYS1 isn't supported anymore and if it works properly on MSYS2, this can be closed.

thanks k0kubun (Takashi Kokubun)

#11 [ruby-core:88318] Updated by k0kubun (Takashi Kokubun) 8 days ago

ARCH_FLAG="-march=barcelona" works, but it isn't propagated to CFLAGS, so it doesn't have the desired effect. I think setting CFLAGS directly shouln't be a problem.
maybe march option should be ignored when building the mjit header and for some reason MSYS1 used it?

See also: https://github.com/ruby/ruby/blob/24daf1400b5336eb22c50f4e4b4915345b6b4283/Makefile.in#L67-L68
While unfortunately I actually don't know well about Ruby's build system and I may be giving wrong instructions to you, but ARCH_FLAG should be propagated to CFLAGS according to that part.
And, since current MJIT's build pipeline deals with arch flags specially, using it in a wrong way may result in a build failure.

Could you try ./configure --with-arch="barcelona" as well? As far as I can see from ./configure --help, "--with-arch" looks the correct way to specify architecture. Also, could you share the output of env command and the full exact commands you're using for ./configure ... and mingw32-make ...?

MSYS1 isn't supported anymore and if it works properly on MSYS2, this can be closed.

I'm trying to support platforms in https://github.com/ruby/ruby/blob/trunk/doc/contributing.rdoc#platform-maintainers, and I believe your environment is included in mingw32.

Since I have only x86_64 Linux, x86_64 macOS and x86_64 Windows (msys2/MinGW and VC++) machines, only these platforms have a direct support by me. I debug other platforms indirectly by mailing list like this. So whether msys1 is supported or not will be effectively decided by this thread.

#12 [ruby-core:88325] Updated by ahorek (Pavel Rosický) 8 days ago

ok, I attached full build logs and my env vars.

output_witharch_81.txt - --with-arch-64='barcelona' or ARCH_FLAG='barcelona'
(gcc bug?)

during RTL pass: final
../ruby_2_6/array.c: In function 'sort_by_i':
../ruby_2_6/array.c:2680:1: internal compiler error: in based_loc_descr, at dwarf2out.c:14264
}

output_optflag_81.txt - --optflags='-march=barcelona' or CFLAGS='-march=barcelona'
(same result with gcc 7.2)

C:/Users/pdaho/AppData/Local/Temp/20180807-8108-1a1so3h.c: In function '_mm_stream_sd':
C:/Users/pdaho/AppData/Local/Temp/20180807-8108-1a1so3h.c:9001:3: error: implicit declaration of function '__builtin_ia32_movntsd'; did you mean '__builtin_ia32_movntpd'? [-Werror=implicit-function-declaration]

output_archflag_72.txt - --with-arch-64='barcelona' or ARCH_FLAG='barcelona' or without flags

successful

Also available in: Atom PDF