Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found

• Status: Closed
• Assignee: kjtsanaktsidis (KJ Tsanaktsidis)
• Target version: -
• Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED

[jaruga (Jun Aruga)]

I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2].

Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found

Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb,

$ autoconf
$ ./configure --enable-shared
$ make

$ ls libruby.so.3.1.0
libruby.so.3.1.0*

If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this.

$ sudo dnf -y install annobin-annocheck

$ sudo yum -y install annobin-annocheck

Then

$ annocheck libruby.so.3.1.0

If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared.

Prepare the following Dockerfile.

$ cat Dockerfile 
FROM docker.io/fedora:34

RUN cat /etc/fedora-release
RUN dnf -y install annobin-annocheck
WORKDIR /work

Then build the container image with the Dockerfile and run the annocheck command for the libruby.so.3.1.0 on your host environment. The -v is an option for bind mount between host and container environment.

$ docker build --rm -t fedora-annocheck .

$ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

The message Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build.

Asking a colleague, I was told that the coroutine/*/Context.S files such as coroutine/x86/Context.S cause the failure. Do you have any idea how to fix this? Thanks.

• [1] https://sourceware.org/annobin/
• [2] You can see man annocheck or https://www.mankier.com/1/annocheck .

[nobu (Nobuyoshi Nakada)]

• File 0001-Add-.note.gnu.property-sections.patch 0001-Add-.note.gnu.property-sections.patch added

Does it work just by adding an empty section?

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

[jaruga (Jun Aruga)]

I applied the patch for the Ruby 3.0.2 in RHEL 8, built. But I still see the failed message.

annocheck: Version 9.65.
Hardened: libruby.so.3.0.2: FAIL: property-note test because no .note.gnu.property section found
Hardened: Rerun annocheck with --verbose to see more information on the tests.

I also applied the patch to the Ruby on master branch 1cb5a669d39fe741388be7ac45fc88c336b097cf . But the message is same.

$ autoconf
$ ./configure --enable-shared
$ make

$ annocheck libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

So, I don't think that adding the empty section works.

[jaruga (Jun Aruga)]

By the way, I found a case generating ".note.gnu.property section" in the OpenSSL project.
https://github.com/openssl/openssl/commit/51994e505dbb1cd0dd76869ec962e2948b77b585

The commit message explains how they verified the change. So in the case of Ruby, here is a way to verify?

I also applied the patch to the Ruby on master branch 1cb5a669d39fe741388be7ac45fc88c336b097cf .

On the above commit, I see an error because there is no .note.gnu.property non-empty section?

$ autoconf

$ CC="gcc -Wl,-z,cet-report=error -fcf-protection" ./configure --enable-shared

$ make
...
/bin/ld: coroutine/amd64/Context.o: error: missing IBT and SHSTK properties
/bin/ld: probes.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
make: *** [Makefile:277: miniruby] Error 1

$ echo $?
2

[jaruga (Jun Aruga)]

It seems we can check if the .note.gnu.property exists in a binary file by the command readelf -S --wide <file>. The binary file ruby has it. But the file libruby.so.3.1.0 doesn't it.

$ readelf -S --wide ruby
There are 40 section headers, starting at offset 0x36038:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        0000000000400318 000318 00001c 00   A  0   0  1
  [ 2] .note.gnu.property NOTE            0000000000400338 000338 000020 00   A  0   0  8
...

$ readelf -S --wide libruby.so.3.1.0
There are 42 section headers, starting at offset 0xc15a38:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .note.gnu.build-id NOTE            0000000000000270 000270 000024 00   A  0   0  4
  [ 2] .gnu.hash         GNU_HASH        0000000000000298 000298 0033e4 00   A  3   0  8
...

[nobu (Nobuyoshi Nakada)]

• File 0001-Add-.note.gnu.property-sections.patch 0001-Add-.note.gnu.property-sections.patch added

That seems the same as a section generated by GCC.

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

[ioquatix (Samuel Williams)]

On another side, we have an Intel CET [2], [3]:
"Control-flow Enforcement Technology (CET) provides the following capabilities to defend
against ROP/JOP style control-flow subversion attacks:
- Shadow Stack - return address protection to defend against Return Oriented Programming,
- Indirect branch tracking - free branch protection to defend against Jump/Call Oriented Programming.​"

Intel CET uses following processor-specific program property types in .note.gnu.property [3, p85, p87]:

* GNU_PROPERTY_X86_FEATURE_1_IBT This indicates that all executable sections are compatible
with IBT (see Section 13.1.1) when endbr64 instruction starts each valid target where an indirect
branch instruction can land. 8
* GNU_PROPERTY_X86_FEATURE_1_SHSTK This indicates that all executable sections are compatible
with SHSTK (see Section 13.1.2) where return address popped from shadow stack always matches
return address popped from normal stack.

We should confirm that coroutine stack swapping is compatible with GNU_PROPERTY_X86_FEATURE_1_SHSTK.

[jaruga (Jun Aruga)]

Thanks for the patch! Let me check it.

0001-Add-.note.gnu.property-sections.patch (3.69 KB)

I tried the 2nd patch on an older master commit 1cb5a669d39fe741388be7ac45fc88c336b097cf , as I couldn't build on the latest master commit due to #18072 . But I still see similar or same result.

$ gcc --version
gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

With -Wl,-z,cet-report=error

$ autoconf
$ CC="gcc -Wl,-z,cet-report=error -fcf-protection" ./configure --enable-shared
$ make
...
vm_call_iseq_optimized.inc updated
generating vmtc.inc
compiling vm.c
compiling vm_backtrace.c
compiling vm_dump.c
compiling vm_sync.c
making trace_point.rbinc
compiling vm_trace.c
assembling coroutine/amd64/Context.S
processing probes in object files
compiling enc/ascii.c
compiling enc/us_ascii.c
compiling enc/unicode.c
compiling enc/utf_8.c
generating enc/trans/newline.c ...
compiling enc/trans/newline.c
compiling ./missing/setproctitle.c
compiling ./missing/strlcat.c
compiling ./missing/strlcpy.c
compiling addr2line.c
compiling dmyenc.c
linking miniruby
/bin/ld: probes.o: error: missing IBT and SHSTK properties
collect2: error: ld returned 1 exit status
make: *** [Makefile:277: miniruby] Error 1

Check by annocheck and readelf.

$ autoconf
$ ./configure --enable-shared
$ make

$ annocheck libruby.so.3.1.0
annocheck: Version 9.79.
Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage 
Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found 
Hardened: Rerun annocheck with --verbose to see more information on the tests.

$ readelf -S --wide libruby.so.3.1.0
  => No .note.gnu.property section.

[jaruga (Jun Aruga)]

• Backport changed from 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED

[ioquatix (Samuel Williams)]

I investigated this issue today.

It doesn't appear to just be a matter of adding a few properties, this actually involves correctly implementing an intel-specific shadow stack.

Assembly coroutine backend and x86 CET support (in QEMU): https://lists.sr.ht/~philmd/qemu/patches/4691

My understanding is we'd need to implement the shadow stack handling in the coroutine code. Or maybe we can just specify that it's not supported and that's also okay?

I think in the future, it's more likely that C compilers will provide native coroutine functions - in this situation maybe we can just use those instead of our native implementations and this problem will go away.

(1) Can we fix this issue without introducing shadow stacks?
(2) Is that sufficient to pass the tests above?

By the way, the error shows an issue with probes.o and while I see the issue also exists for the coroutine implementation, the fix provided by @nobu (Nobuyoshi Nakada) at least hides the error - whether the actual shadow stack is working is anyone's guess - actual programs may just crash on coroutine swap.

See this branch with Nobu's patch: https://github.com/ruby/ruby/pull/5895

From https://www.phoronix.com/scan.php?page=news_item&px=Intel-CET-IBT-SHSTK-glibc:

Landing yesterday in Glibc for Intel's Control-flow Enforcement Technology (CET) were the instructions for Indirect Branch Tracking (IBT) and Shadow Stack (SHSTK).

These Intel CET bits for the GNU C Library amount to a fair amount of code being added. The commit message explains some of the CET steps taken. The Control-flow Enforcement Technology behavior can be changed for SHSTK/IBT at run-time through the "GLIBC_TUNABLES" environment variable.

CET aims to fend off return-oriented programming (ROP) and call-jump-oriented programming (COP/JOP) attacks via a shadow stack to keep track of expected return addresses and to raise faults when the return addresses don't match up with what's found in the shadow stack. The Indirect branch tracking is for the stopping of jump/call oriented attacks. CET has yet to appear in released Intel processors while this security feature has already been squared away in the GCC compiler and other areas of the software stack and is now getting setup within Glibc.

which links to https://sourceware.org/git/?p=glibc.git;a=commit;h=f753fa7dea3367bd3eb7b543103ff8cda182a3fa which has a lot more details.

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) Thank you for your investigation! I prepared make test-annocheck command for your convenient, and for us to test this case on CI. You can pick up my commit from https://github.com/ruby/ruby/pull/5900 .

[jaruga (Jun Aruga)]

• File config-pie.log config-pie.log added

I am testing Samuel's PR: https://github.com/ruby/ruby/pull/5895.

To pass the security tool, annocheck with the PR, I am trying to build with minimal compiler flags.

I am using the following gcc on my local.

$ gcc --version
gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The following command works. The number of failures by annobin is reduced from 5 to 2 by the flags.

$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"

But the following command doesn't work with -Wl,-pie.

$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now -Wl,-pie"
checking for ruby... /bin/ruby
tool/config.guess already exists
tool/config.sub already exists
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for ld... ld
checking for gcc-ar... gcc-ar
checking for g++... g++
checking for gcc-nm... gcc-nm
checking for gcc-ranlib... gcc-ranlib
checking for gcc... gcc -fcf-protection -Wl,-z,now -Wl,-pie
checking whether the C compiler works... no
configure: error: in `/home/jaruga/git/ruby/ruby':
configure: error: C compiler cannot create executables
See `config.log' for more details

I would upload the config.log (config-pie.log). Do you know what's wrong?

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) My PR above was merged. Now after you rebase your PR on the latest master branch, you can test your PR with make test-annocheck on your local or test with CI with the following modification. Thanks.

$ git diff
diff --git a/.github/workflows/compilers.yml b/.github/workflows/compilers.yml
index ddb53cadb8..8ae6e40b9e 100644
--- a/.github/workflows/compilers.yml
+++ b/.github/workflows/compilers.yml
@@ -87,7 +87,7 @@ jobs:
               # https://bugs.ruby-lang.org/issues/18061
               # https://sourceware.org/annobin/annobin.html/Test-pie.html
               # https://sourceware.org/annobin/annobin.html/Test-notes.html
-              TEST_ANNOCHECK_OPTS: "--skip-pie --skip-notes"
+              TEST_ANNOCHECK_OPTS: "--skip-pie"
             check: true
           - { key: default_cc, name: clang-15,  value: clang-15,  container: clang-15 }
           - { key: default_cc, name: clang-14,  value: clang-14,  container: clang-14 }

[jaruga (Jun Aruga)]

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

Before the commit, you see the annocheck was executed in make test-annocheck using Fedora container in Ubuntu focal container on gcc-11 annocheck case on Ubuntu host in .github/workflows/compilers.yml. The log is here. But after the commit, the annocheck is not executed any more in the make test-annocheck in the gcc-11 annocheck case on the CI any more. The log is here.

[jaruga (Jun Aruga)]

On the current latest master f2dc97294086c31604a45ee350c4fa5d51762f45, here is the result on my Fedora 35. The Fedora container and annocheck are not executed.

$ git clean -fdx
$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"
$ make
$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
make: Nothing to be done for 'test-annocheck'.

Here is the test on the previous commit of the commit a7577dbfd3ea53cccf7aaf94208069784ad17791 above. The Fedora container and annocheck can be executed multiple times after make.

$ git clean -fdx
$ git checkout 641c3830df8177b38fcfca33d3a0ece8a2c1e7d2
$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now"
$ make

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
+ /bin/docker run --rm -t -v /home/jaruga/git/ruby/ruby:/work ruby-fedora-annocheck annocheck --verbose --skip-pie --skip-notes ruby
...
Hardened: ruby: Overall: PASS.

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
+ /bin/docker run --rm -t -v /home/jaruga/git/ruby/ruby:/work ruby-fedora-annocheck annocheck --verbose --skip-pie --skip-notes ruby
...
Hardened: ruby: Overall: PASS.

[nobu (Nobuyoshi Nakada)]

jaruga (Jun Aruga) wrote in #note-15:

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

TEST_RUNNABLE is a flag for native/cross compiling.
Since annocheck works on Fedora Docker image, it is unrelated to the host OS, and it may work even when cross compiling if Docker works.

[jaruga (Jun Aruga)]

nobu (Nobuyoshi Nakada) wrote in #note-17:

jaruga (Jun Aruga) wrote in #note-15:

@nobu (Nobuyoshi Nakada) what is your intent for this commit, https://github.com/ruby/ruby/commit/a7577dbfd3ea53cccf7aaf94208069784ad17791 ? What issue did you see?

TEST_RUNNABLE is a flag for native/cross compiling.
Since annocheck works on Fedora Docker image, it is unrelated to the host OS, and it may work even when cross compiling if Docker works.

I see. Thanks for explaining it, and thanks for fixing the issue on the commit https://github.com/ruby/ruby/commit/2411f0ad8cf608d6aae221ce4d803b5c5271795a . I confirmed it works after the commit on the CI (the log is here) and my local Fedora. But as you said, it may work on Windows, Mac, *BSD too where Docker is installed. If it works there, maybe the current target linux-test-annocheck is not enough.

[nobu (Nobuyoshi Nakada)]

jaruga (Jun Aruga) wrote in #note-18:

But as you said, it may work on Windows, Mac, *BSD too where Docker is installed. If it works there, maybe the current target linux-test-annocheck is not enough.

Isn't Docker always Linux, regardless hosts?

[jaruga (Jun Aruga)]

Isn't Docker always Linux, regardless hosts?

Yes, Docker where annocheck is running in is always Linux (= docker.io/fedora:latest container). But the target_os (= "linux" on host os: Fedora and Ubuntu) in common.mk is maybe not "linux" on host: Windows, Mac and *BSD, right? I will try to test make annocheck on Mac.

https://github.com/ruby/ruby/blob/11af23ee923858fbcec18f2d1365296fdfe24dc2/common.mk#L1450-L1453

test-annocheck: $(target_os)-test-annocheck
linux-test-annocheck: $(PROGRAM)
	$(tooldir)/test-annocheck.sh $(PROGRAM)
$(target_os)-test-annocheck: PHONY

[jaruga (Jun Aruga)]

I will try to test make annocheck on Mac.

I sent the PR here. It's working in progress.
https://github.com/ruby/ruby/pull/5916

[jaruga (Jun Aruga)]

Here is a minimal reproducer to see the property-note test failure. The ./configure --enable-shared --with-gcc="gcc -fcf-protection -Wl,-z,now" above was wrong. The -Wl,-z,now should not be in `--with-gcc="..". I checked the reproducer on my Fedora 36.

$ ./autogen.sh
$ ./configure --enable-shared --with-gcc="gcc -fcf-protection" LDFLAGS=-Wl,-z,now
$ make
$ make test-annocheck
...
Hardened: ruby: MAYB: test: notes because not all of the .text section is covered by notes
...
Hardened: ruby: FAIL: pie test because not built with '-Wl,-pie'
...
Hardened: ruby: Overall: FAIL.

$ TEST_ANNOCHECK_OPTS="--skip-pie --skip-notes" make test-annocheck
...
Hardened: ruby: Overall: PASS.

[jaruga (Jun Aruga)]

@ioquatix (Samuel Williams) (Samuel Williams) @nobu (Nobuyoshi Nakada), possibly Samuel already fixed the property-note issue failed by annocheck correctly on the master branch. Sorry for my mistake.

I tested Ruby on the (relatively) latest master branch 78425d7e74887b57ee15e6b8933bd3878db6a888. And when I built with all the build flags used to build Fedora's Ruby RPM package, the annocheck passed for the ruby binary.

This repository is my experiment, and the details: https://github.com/junaruga/ruby-annocheck-test/
I built with the flags below in the build_with_fedora_build_flags.sh.

https://github.com/junaruga/ruby-annocheck-test/blob/main/build_with_fedora_build_flags.sh

CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' \
CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' \
LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 ' \
./configure \
    --enable-shared \
    --enable-mkmf-verbose 2>&1 | tee configure.log

For the gcc -specs=file options I used, I put the files in the https://github.com/junaruga/ruby-annocheck-test/tree/main/gcc_specs.
I will try to understand what gcc flags fixed the 2 failures (https://bugs.ruby-lang.org/issues/18061#note-22), then update the CI .github/workflows/compilers.yml annocheck case with the minimal flags.

[ioquatix (Samuel Williams)]

While it was passing, it's now failing:

Previously Passing

https://github.com/ruby/ruby/runs/6956677552?check_suite_focus=true#step:20:54

================================================================================
 Package                 Architecture Version               Repository     Size
================================================================================
Installing:
 annobin-annocheck       x86_64       10.73-1.fc36          updates       165 k
Installing dependencies:
 annobin-docs            noarch       10.73-1.fc36          updates        90 k
 cpio                    x86_64       2.13-12.fc36          fedora        270 k
Transaction Summary
================================================================================

... snip ...

annocheck: Version 10.73.
Hardened: ruby: PASS: optimization test 
Hardened: ruby: PASS: pic test 
Hardened: ruby: PASS: stack-prot test 
Hardened: ruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: ruby: PASS: writable-got test 
Hardened: ruby: PASS: dynamic-segment test 
Hardened: ruby: PASS: bind-now test 
Hardened: ruby: PASS: run-path test because the DT_RUNPATH dynamic tag is present and correct 
Hardened: ruby: PASS: entry test 
Hardened: ruby: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: ruby: PASS: gnu-relro test 
Hardened: ruby: skip: branch-protection test because not an AArch64 binary 
Hardened: ruby: skip: dynamic-tags test because AArch64 specific 
Hardened: ruby: skip: fortify test because no C/C++ compiled code found 
Hardened: ruby: skip: glibcxx-assertions test because source language not C++ 
Hardened: ruby: skip: go-revision test because no GO compiled code found 
Hardened: ruby: PASS: instrumentation test 
Hardened: ruby: skip: lto test because not compiled C/C++ code 
Hardened: ruby: skip: only-go test because no GO compiled code found 
Hardened: ruby: PASS: production test 
Hardened: ruby: PASS: property-note test because CET enabled property note found 
Hardened: ruby: PASS: rwx-seg test 
Hardened: ruby: PASS: short-enums test 
Hardened: ruby: skip: stack-clash test because no C/C++ compiled code found 
Hardened: ruby: skip: stack-realign test because not a 32-bit i686 executable 
Hardened: ruby: PASS: textrel test 
Hardened: ruby: PASS: threads test 
Hardened: ruby: PASS: unicode test 
Hardened: ruby: skip: warnings test because no C/C++ compiled code found 
Hardened: ruby: Overall: PASS.

Now Failing

https://github.com/ruby/ruby/runs/6959694909?check_suite_focus=true#step:20:53

================================================================================
 Package                 Architecture Version               Repository     Size
================================================================================
Installing:
 annobin-annocheck       x86_64       10.76-1.fc36          updates       167 k
Installing dependencies:
 annobin-docs            noarch       10.76-1.fc36          updates        91 k
 cpio                    x86_64       2.13-12.fc36          fedora        270 k
Transaction Summary
================================================================================

... snip ...

annocheck: Version 10.76.
Hardened: ruby: PASS: optimization test 
Hardened: ruby: PASS: pic test 
Hardened: ruby: PASS: stack-prot test 
Hardened: ruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: ruby: PASS: writable-got test 
Hardened: ruby: PASS: dynamic-segment test 
Hardened: ruby: PASS: bind-now test 
Hardened: ruby: PASS: run-path test because the DT_RUNPATH dynamic tag is present and correct 
Hardened: ruby: PASS: entry test 
Hardened: ruby: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: ruby: PASS: gnu-relro test 
Hardened: ruby: MAYB: test: gaps because no notes found 
Hardened: ruby: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-gaps.html
Hardened: ruby: skip: branch-protection test because not an AArch64 binary 
Hardened: ruby: skip: dynamic-tags test because AArch64 specific 
Hardened: ruby: skip: fortify test because no C/C++ compiled code found 
Hardened: ruby: skip: glibcxx-assertions test because source language not C++ 
Hardened: ruby: skip: go-revision test because no GO compiled code found 
Hardened: ruby: PASS: instrumentation test 
Hardened: ruby: skip: lto test because not compiled C/C++ code 
Hardened: ruby: skip: only-go test because no GO compiled code found 
Hardened: ruby: PASS: production test 
Hardened: ruby: PASS: property-note test because CET enabled property note found 
Hardened: ruby: PASS: rwx-seg test 
Hardened: ruby: PASS: short-enums test 
Hardened: ruby: skip: stack-clash test because no C/C++ compiled code found 
Hardened: ruby: skip: stack-realign test because not a 32-bit i686 executable 
Hardened: ruby: PASS: textrel test 
Hardened: ruby: PASS: threads test 
Hardened: ruby: PASS: unicode test 
Hardened: ruby: skip: warnings test because no C/C++ compiled code found 
Hardened: ruby: Overall: FAIL (due to MAYB results).

I haven't touched the implementation except for my PR. I don't think this is a matter of just adding notes, I think the coroutine implementation needs to maintain a shadow stack for the purpose of return value tracking etc.

I don't know enough about the relevant technologies to implement this right now. We can't just enable the flags without the corresponding implementation - with CET enabled, I'd imagine it would crash.

I did implement ASAN which has a similar shadow stack. With that in mind, it might be possible to follow a similar implementation. But this is not something I'm very familiar with.

[ioquatix (Samuel Williams)]

I started messing around with the implementation again.

But my computer is too old, it looks like I need 11th gen Intel CPU to test CET :(

Is anyone able to help test the PR if I update it?

[ioquatix (Samuel Williams)]

In my latest PR, I try to detect control-flow protections:

   * with thread:         pthread
   * with coroutine:      amd64
   * with c-f protection: no

(not supported on my CPU).

Then, I've eye-balled an implementation. Not sure if it works, maybe it will in GH Actions?

[jaruga (Jun Aruga)]

ioquatix (Samuel Williams) wrote in #note-24:

While it was passing, it's now failing:

Thanks for the report. I will take a look and fix the CI. Maybe it comes from the annocheck version update from 10.73 to 10.76. My guess is that the annocheck changed something between the versions. Maybe the current workaround is to add another skip option to TEST_ANNOCHECK_OPTS: "--skip-pie --skip-notes" in the .github/workflows/compilers.yml.

https://github.com/ruby/ruby/runs/6956677552?check_suite_focus=true#step:20:108

annocheck: Version 10.73.

https://github.com/ruby/ruby/runs/6959694909?check_suite_focus=true#step:20:109

annocheck: Version 10.76.

Comparing the 2 commits, I don't see any issues causing this failure.

$ git diff aeab4058784c86df47a455ffdb08714b337d0209..150bd6e2427a8a6178aa5f8ebc39ce6e59601e1f

[ioquatix (Samuel Williams)]

https://lwn.net/Articles/885220/

I was studying this and the related LKML discussion. I don't think this problem is solved any time soon. I don't think coroutine support using ret is compatible yet, unless some how we can acess wrss instruction. I need to do some more investigation and probably get hold of a compatible CPU.

[jaruga (Jun Aruga)]

We opened the 2 PRs for that now :) @mame (Yusuke Endoh) You can merge your PR. I will add my PR later with some comments.
https://github.com/ruby/ruby/pull/6043
https://github.com/ruby/ruby/pull/6044

[jaruga (Jun Aruga)]

jaruga (Jun Aruga) wrote in #note-29:

We opened the 2 PRs for that now :) @mame (Yusuke Endoh) You can merge your PR. I will add my PR later with some comments.
https://github.com/ruby/ruby/pull/6043
https://github.com/ruby/ruby/pull/6044

The PR was merged to the master branch.
https://github.com/ruby/ruby/commit/f8b6d4f4d1b0eb0b3220fc83f157bd33ddcc5d4e

[jaruga (Jun Aruga)]

Maybe I found a way to fix annocheck gaps and notes tests on Ubuntu focal on CI.
Here is the PR: https://github.com/ruby/ruby/pull/6045.

[jaruga (Jun Aruga)]

ioquatix (Samuel Williams) wrote in #note-25:

I started messing around with the implementation again.

But my computer is too old, it looks like I need 11th gen Intel CPU to test CET :(

Is anyone able to help test the PR if I update it?

I have my own Framework Laptop first generation with the 11th gen Intel CPU (11th Gen Intel i7-1165G7) with Fedora 36 and gcc 12. I can help to test it. :)

[jaruga (Jun Aruga)]

I got useful info from the helpful annocheck upstream maintainer to fix the annocheck gaps and pie tests.

Bug 29269 - annocheck: gaps test: question for the flag: -Wa,--generate-missing-build-notes=yes .
https://sourceware.org/bugzilla/show_bug.cgi?id=29269#c2

As a reference to understand the content, here are the specs files used by the gcc -specs=file options. I want to see that we add minimal flags without using specs files into our annocheck CI case.

$ cat redhat-annobin-cc1 
*cc1_options:
+ %{!-fno-use-annobin:%{!iplugindir*:%:find-plugindir()} -fplugin=annobin}

$ cat redhat-hardened-cc1 
*cc1_options:
+ %{!r:%{!fpie:%{!fPIE:%{!fpic:%{!fPIC:%{!fno-pic:-fPIE}}}}}}

$ cat redhat-hardened-ld 
*self_spec:
+ %{!static:%{!shared:%{!r:-pie}}}

[ioquatix (Samuel Williams)]

• Related to Bug #20029: coroutine/arm64/Context.S does not support PAC/BTI added

[vo.x (Vit Ondruch)]

Just FTR, the annocheck test is incomplete in case --enable-shared is used, because the test is checking just ruby executable, while it should also check libruby.so.

I should also mention that apart of the coroutines, the YJIT compiled by Rust is also problematic ATM. Therefore as long as Rust does not properly support CF protection, the test should use --disable-yjit.

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-35:

Just FTR, the annocheck test is incomplete in case --enable-shared is used, because the test is checking just ruby executable, while it should also check libruby.so.

The test should be likely executed above all executable ELF files and object files. I think that something like annocheck . 2>/dev/null | grep FAIL could do the job, because annocheck scans the whole directory structure. Here is the result on my system:

$ annocheck . 2>/dev/null | grep FAIL
Hardened: Context.o: Overall: FAIL (due to MAYB results).
Hardened: miniruby: FAIL: cf-protection test because .note.gnu.property section did not contain the necessary flags 
Hardened: miniruby: FAIL: property-note test because a property note was found but it shows that cf-protection is not enabled 
Hardened: miniruby: Overall: FAIL.
Hardened: libruby-static.a:Context.o: Overall: FAIL (due to MAYB results).
Hardened: libruby.so.3.3.1: FAIL: cf-protection test because .note.gnu.property section did not contain the necessary flags 
Hardened: libruby.so.3.3.1: FAIL: property-note test because a property note was found but it shows that cf-protection is not enabled 
Hardened: libruby.so.3.3.1: Overall: FAIL.

[alanwu (Alan Wu)]

• Has duplicate Bug #20527: Control-Flow protection cannot be enabled for Ruby due to ASM bits added

[kjtsanaktsidis (KJ Tsanaktsidis)]

• Assignee set to kjtsanaktsidis (KJ Tsanaktsidis)

[vo.x (Vit Ondruch)]

So I am experimenting with this simple patch:

diff --git a/coroutine/amd64/Context.S b/coroutine/amd64/Context.S
index fcc4b67bdc..b364fc06a1 100644
--- a/coroutine/amd64/Context.S
+++ b/coroutine/amd64/Context.S
@@ -4,6 +4,8 @@
 ##  Created by Samuel Williams on 10/5/2018.
 ##  Copyright, 2018, by Samuel Williams.
 ##
+
+#include <cet.h>
 
 #define TOKEN_PASTE(x,y) x##y
 #define PREFIXED_SYMBOL(prefix,name) TOKEN_PASTE(prefix,name)
@@ -12,6 +14,8 @@
 
 .globl PREFIXED_SYMBOL(SYMBOL_PREFIX,coroutine_transfer)
 PREFIXED_SYMBOL(SYMBOL_PREFIX,coroutine_transfer):
+
+ 	_CET_ENDBR
 
 	# Make space on the stack for 6 registers:
 	subq $48, %rsp

However, the patch alone is not enough, because there needs to be also the -fcf-protection propagated into the compiler. That can be workarounded by export ASFLAGS='-fcf-protection'. With these bits in place, I can pass the annocheck:

<mock-chroot> sh-5.2$ annocheck -v ./coroutine/amd64/Context.o
annocheck: Version 12.59.
Hardened: ./coroutine/amd64/Context.o: info: No matching profile found.
Hardened: ./coroutine/amd64/Context.o: PASS: pie test because the ELF file header has the correct type 
Hardened: ./coroutine/amd64/Context.o: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: ./coroutine/amd64/Context.o: PASS: gnu-stack test because non-executable .note.GNU-stack section found 
Hardened: ./coroutine/amd64/Context.o: skip: gaps test because no notes found - therefore there are no gaps! 
Hardened: ./coroutine/amd64/Context.o: MAYB: test: notes, reason: annobin notes not found
Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-notes.html
Hardened: ./coroutine/amd64/Context.o: skip: bind-now test because only needed for executables 
Hardened: ./coroutine/amd64/Context.o: skip: branch-protection test because not an AArch64 binary 
Hardened: ./coroutine/amd64/Context.o: PASS: dynamic-segment test 
Hardened: ./coroutine/amd64/Context.o: skip: dynamic-tags test because AArch64 specific 
Hardened: ./coroutine/amd64/Context.o: PASS: entry test 
Hardened: ./coroutine/amd64/Context.o: PASS: fast test 
Hardened: ./coroutine/amd64/Context.o: skip: fips test because not a GO binary 
Hardened: ./coroutine/amd64/Context.o: skip: fortify test because no compiled C/C++ code found 
Hardened: ./coroutine/amd64/Context.o: skip: glibcxx-assertions test because no compiled C/C++ code found 
Hardened: ./coroutine/amd64/Context.o: skip: gnu-relro test because not needed in object files 
Hardened: ./coroutine/amd64/Context.o: skip: go-revision test because no GO compiled code found 
Hardened: ./coroutine/amd64/Context.o: skip: implicit-values test because  These tests are only relevent to C source code 
Hardened: ./coroutine/amd64/Context.o: PASS: instrumentation test 
Hardened: ./coroutine/amd64/Context.o: skip: lto test because not compiled from C/C++ code 
Hardened: ./coroutine/amd64/Context.o: PASS: openssl-engine test 
Hardened: ./coroutine/amd64/Context.o: MAYB: test: optimization, reason: could not determine how the code was created
Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-optimization.html
Hardened: ./coroutine/amd64/Context.o: WARN: This can happen if the program is compiled from a language unknown to annocheck
Hardened: ./coroutine/amd64/Context.o: WARN:  or because there are no annobin build notes (could they be in a separate file ?)
Hardened: ./coroutine/amd64/Context.o: WARN: For more details see https://sourceware.org/annobin/annobin.html/Absence-of-compiled-code.html
Hardened: ./coroutine/amd64/Context.o: MAYB: test: pic, reason: no valid notes found regarding this test
Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-pic.html
Hardened: ./coroutine/amd64/Context.o: PASS: production test 
Hardened: ./coroutine/amd64/Context.o: skip: property-note test because property notes not needed in object files 
Hardened: ./coroutine/amd64/Context.o: PASS: run-path test 
Hardened: ./coroutine/amd64/Context.o: PASS: rwx-seg test 
Hardened: ./coroutine/amd64/Context.o: PASS: short-enums test 
Hardened: ./coroutine/amd64/Context.o: MAYB: test: stack-clash, reason: could not determine how the code was created
Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-stack-clash.html
Hardened: ./coroutine/amd64/Context.o: WARN: This can happen if the program is compiled from a language unknown to annocheck
Hardened: ./coroutine/amd64/Context.o: WARN:  or because there are no annobin build notes (could they be in a separate file ?)
Hardened: ./coroutine/amd64/Context.o: WARN: For more details see https://sourceware.org/annobin/annobin.html/Absence-of-compiled-code.html
Hardened: ./coroutine/amd64/Context.o: MAYB: test: stack-prot, reason: could not determine how the code was created
Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-stack-prot.html
Hardened: ./coroutine/amd64/Context.o: WARN: This can happen if the program is compiled from a language unknown to annocheck
Hardened: ./coroutine/amd64/Context.o: WARN:  or because there are no annobin build notes (could they be in a separate file ?)
Hardened: ./coroutine/amd64/Context.o: WARN: For more details see https://sourceware.org/annobin/annobin.html/Absence-of-compiled-code.html
Hardened: ./coroutine/amd64/Context.o: skip: stack-realign test because not an i686 executable 
Hardened: ./coroutine/amd64/Context.o: PASS: textrel test 
Hardened: ./coroutine/amd64/Context.o: PASS: threads test 
Hardened: ./coroutine/amd64/Context.o: PASS: unicode test 
Hardened: ./coroutine/amd64/Context.o: skip: warnings test because no compiled C/C++ code found 
Hardened: ./coroutine/amd64/Context.o: PASS: writable-got test 
Hardened: ./coroutine/amd64/Context.o: Overall: FAIL (due to MAYB results).
<mock-chroot> sh-5.2$ annocheck -v mini
miniinit.o  miniruby    
<mock-chroot> sh-5.2$ annocheck -v miniruby 
annocheck: Version 12.59.
Hardened: miniruby: info: No matching profile found.
Hardened: miniruby: PASS: pie test because the ELF file header has the correct type 
Hardened: miniruby: info: written in C (source: DW_AT_language string).
Hardened: miniruby: PASS: pic test because option found in DW_AT_producer string 
Hardened: miniruby: PASS: stack-prot test because option found in DW_AT_producer string 
Hardened: miniruby: PASS: optimization test because option found in DW_AT_producer string 
Hardened: miniruby: PASS: lto test because detected in DW_AT_producer string 
Hardened: miniruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: miniruby: PASS: writable-got test 
Hardened: miniruby: PASS: dynamic-segment test 
Hardened: miniruby: PASS: bind-now test 
Hardened: miniruby: PASS: notes test because annobin notes found in the .annobin.notes section 
Hardened: miniruby: skip: fast test because function ../sysdeps/x86/abi-note.c is part of the C library's static code and does use math functions 
Hardened: miniruby: skip: lto test because function ../sysdeps/x86/abi-note.c is part of the C library which is deliberately built without LTO 
Hardened: miniruby: PASS: implicit-values test because -Wimplicit-int and -Wimplicit-function-decalration enabled 
Hardened: miniruby: PASS: stack-clash test because compiled with -fstack-clash-protection 
Hardened: miniruby: skip: fortify test because function ../sysdeps/x86/abi-note.c is part of the C library, and as such it does not need fortification 
Hardened: miniruby: PASS: glibcxx-assertions test 
Hardened: miniruby: skip: warnings test because LTO compilation discards preprocessor options 
Hardened: miniruby: skip: implicit-values test because -Wimplicit-int setting is hidden by LTO 
Hardened: miniruby: PASS: entry test 
Hardened: miniruby: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: miniruby: PASS: gnu-relro test 
Hardened: miniruby: skip: gaps test because string notes imply full coverage 
Hardened: miniruby: skip: branch-protection test because not an AArch64 binary 
Hardened: miniruby: skip: dynamic-tags test because AArch64 specific 
Hardened: miniruby: skip: fips test because not a GO binary 
Hardened: miniruby: skip: go-revision test because no GO compiled code found 
Hardened: miniruby: PASS: instrumentation test 
Hardened: miniruby: PASS: openssl-engine test 
Hardened: miniruby: PASS: production test 
Hardened: miniruby: PASS: property-note test because CET enabled property note found 
Hardened: miniruby: PASS: run-path test 
Hardened: miniruby: PASS: rwx-seg test 
Hardened: miniruby: PASS: short-enums test 
Hardened: miniruby: skip: stack-realign test because not an i686 executable 
Hardened: miniruby: PASS: textrel test 
Hardened: miniruby: PASS: threads test 
Hardened: miniruby: PASS: unicode test 
Hardened: miniruby: Overall: PASS.
<mock-chroot> sh-5.2$ annocheck -v libruby.so.3.3.2 
annocheck: Version 12.59.
Hardened: libruby.so.3.3.2: info: No matching profile found.
Hardened: libruby.so.3.3.2: PASS: pie test because the ELF file header has the correct type 
Hardened: libruby.so.3.3.2: info: written in C (source: DW_AT_language string).
Hardened: libruby.so.3.3.2: PASS: pic test because option found in DW_AT_producer string 
Hardened: libruby.so.3.3.2: PASS: stack-prot test because option found in DW_AT_producer string 
Hardened: libruby.so.3.3.2: PASS: optimization test because option found in DW_AT_producer string 
Hardened: libruby.so.3.3.2: PASS: lto test because detected in DW_AT_producer string 
Hardened: libruby.so.3.3.2: PASS: cf-protection test because correct flags found in .note.gnu.property note 
Hardened: libruby.so.3.3.2: PASS: writable-got test 
Hardened: libruby.so.3.3.2: PASS: dynamic-segment test 
Hardened: libruby.so.3.3.2: PASS: bind-now test 
Hardened: libruby.so.3.3.2: PASS: notes test because annobin notes found in the .annobin.notes section 
Hardened: libruby.so.3.3.2: skip: warnings test because LTO compilation discards preprocessor options 
Hardened: libruby.so.3.3.2: skip: implicit-values test because -Wimplicit-int setting is hidden by LTO 
Hardened: libruby.so.3.3.2: skip: fortify test because LTO compilation discards preprocessor options 
Hardened: libruby.so.3.3.2: PASS: glibcxx-assertions test 
Hardened: libruby.so.3.3.2: PASS: stack-clash test because compiled with -fstack-clash-protection 
Hardened: libruby.so.3.3.2: skip: entry test because shared libraries do not use entry points 
Hardened: libruby.so.3.3.2: PASS: gnu-stack test because stack segment exists with the correct permissions 
Hardened: libruby.so.3.3.2: PASS: gnu-relro test 
Hardened: libruby.so.3.3.2: skip: gaps test because string notes imply full coverage 
Hardened: libruby.so.3.3.2: skip: branch-protection test because not an AArch64 binary 
Hardened: libruby.so.3.3.2: skip: dynamic-tags test because AArch64 specific 
Hardened: libruby.so.3.3.2: PASS: fast test 
Hardened: libruby.so.3.3.2: skip: fips test because not a GO binary 
Hardened: libruby.so.3.3.2: skip: go-revision test because no GO compiled code found 
Hardened: libruby.so.3.3.2: PASS: instrumentation test 
Hardened: libruby.so.3.3.2: PASS: openssl-engine test 
Hardened: libruby.so.3.3.2: PASS: production test 
Hardened: libruby.so.3.3.2: PASS: property-note test because CET enabled property note found 
Hardened: libruby.so.3.3.2: PASS: run-path test 
Hardened: libruby.so.3.3.2: PASS: rwx-seg test 
Hardened: libruby.so.3.3.2: PASS: short-enums test 
Hardened: libruby.so.3.3.2: skip: stack-realign test because not an i686 executable 
Hardened: libruby.so.3.3.2: PASS: textrel test 
Hardened: libruby.so.3.3.2: PASS: threads test 
Hardened: libruby.so.3.3.2: PASS: unicode test 
Hardened: libruby.so.3.3.2: Overall: PASS.

and this is the diff to version without the steps above:

--- <bez názvu>
+++ <bez názvu>
@@ -2,14 +2,13 @@
 annocheck: Version 12.59.
 Hardened: ./coroutine/amd64/Context.o: info: No matching profile found.
 Hardened: ./coroutine/amd64/Context.o: PASS: pie test because the ELF file header has the correct type 
+Hardened: ./coroutine/amd64/Context.o: PASS: cf-protection test because correct flags found in .note.gnu.property note 
 Hardened: ./coroutine/amd64/Context.o: PASS: gnu-stack test because non-executable .note.GNU-stack section found 
 Hardened: ./coroutine/amd64/Context.o: skip: gaps test because no notes found - therefore there are no gaps! 
 Hardened: ./coroutine/amd64/Context.o: MAYB: test: notes, reason: annobin notes not found
 Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-notes.html
 Hardened: ./coroutine/amd64/Context.o: skip: bind-now test because only needed for executables 
 Hardened: ./coroutine/amd64/Context.o: skip: branch-protection test because not an AArch64 binary 
-Hardened: ./coroutine/amd64/Context.o: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
-Hardened: ./coroutine/amd64/Context.o: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
 Hardened: ./coroutine/amd64/Context.o: PASS: dynamic-segment test 
 Hardened: ./coroutine/amd64/Context.o: skip: dynamic-tags test because AArch64 specific 
 Hardened: ./coroutine/amd64/Context.o: PASS: entry test 
@@ -51,7 +50,9 @@
 Hardened: ./coroutine/amd64/Context.o: PASS: unicode test 
 Hardened: ./coroutine/amd64/Context.o: skip: warnings test because no compiled C/C++ code found 
 Hardened: ./coroutine/amd64/Context.o: PASS: writable-got test 
-Hardened: ./coroutine/amd64/Context.o: Overall: FAIL.
+Hardened: ./coroutine/amd64/Context.o: Overall: FAIL (due to MAYB results).
+<mock-chroot> sh-5.2$ annocheck -v mini
+miniinit.o  miniruby    
 <mock-chroot> sh-5.2$ annocheck -v miniruby 
 annocheck: Version 12.59.
 Hardened: miniruby: info: No matching profile found.
@@ -61,6 +62,7 @@
 Hardened: miniruby: PASS: stack-prot test because option found in DW_AT_producer string 
 Hardened: miniruby: PASS: optimization test because option found in DW_AT_producer string 
 Hardened: miniruby: PASS: lto test because detected in DW_AT_producer string 
+Hardened: miniruby: PASS: cf-protection test because correct flags found in .note.gnu.property note 
 Hardened: miniruby: PASS: writable-got test 
 Hardened: miniruby: PASS: dynamic-segment test 
 Hardened: miniruby: PASS: bind-now test 
@@ -78,16 +80,13 @@
 Hardened: miniruby: PASS: gnu-relro test 
 Hardened: miniruby: skip: gaps test because string notes imply full coverage 
 Hardened: miniruby: skip: branch-protection test because not an AArch64 binary 
-Hardened: miniruby: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
-Hardened: miniruby: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
 Hardened: miniruby: skip: dynamic-tags test because AArch64 specific 
 Hardened: miniruby: skip: fips test because not a GO binary 
 Hardened: miniruby: skip: go-revision test because no GO compiled code found 
 Hardened: miniruby: PASS: instrumentation test 
 Hardened: miniruby: PASS: openssl-engine test 
 Hardened: miniruby: PASS: production test 
-Hardened: miniruby: FAIL: property-note test because a property note was found but it shows that cf-protection is not enabled 
-Hardened: miniruby: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-property-note.html
+Hardened: miniruby: PASS: property-note test because CET enabled property note found 
 Hardened: miniruby: PASS: run-path test 
 Hardened: miniruby: PASS: rwx-seg test 
 Hardened: miniruby: PASS: short-enums test 
@@ -95,7 +94,7 @@
 Hardened: miniruby: PASS: textrel test 
 Hardened: miniruby: PASS: threads test 
 Hardened: miniruby: PASS: unicode test 
-Hardened: miniruby: Overall: FAIL.
+Hardened: miniruby: Overall: PASS.
 <mock-chroot> sh-5.2$ annocheck -v libruby.so.3.3.2 
 annocheck: Version 12.59.
 Hardened: libruby.so.3.3.2: info: No matching profile found.
@@ -105,6 +104,7 @@
 Hardened: libruby.so.3.3.2: PASS: stack-prot test because option found in DW_AT_producer string 
 Hardened: libruby.so.3.3.2: PASS: optimization test because option found in DW_AT_producer string 
 Hardened: libruby.so.3.3.2: PASS: lto test because detected in DW_AT_producer string 
+Hardened: libruby.so.3.3.2: PASS: cf-protection test because correct flags found in .note.gnu.property note 
 Hardened: libruby.so.3.3.2: PASS: writable-got test 
 Hardened: libruby.so.3.3.2: PASS: dynamic-segment test 
 Hardened: libruby.so.3.3.2: PASS: bind-now test 
@@ -119,8 +119,6 @@
 Hardened: libruby.so.3.3.2: PASS: gnu-relro test 
 Hardened: libruby.so.3.3.2: skip: gaps test because string notes imply full coverage 
 Hardened: libruby.so.3.3.2: skip: branch-protection test because not an AArch64 binary 
-Hardened: libruby.so.3.3.2: FAIL: cf-protection test because no .note.gnu.property section = no control flow information 
-Hardened: libruby.so.3.3.2: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-cf-protection.html
 Hardened: libruby.so.3.3.2: skip: dynamic-tags test because AArch64 specific 
 Hardened: libruby.so.3.3.2: PASS: fast test 
 Hardened: libruby.so.3.3.2: skip: fips test because not a GO binary 
@@ -128,8 +126,7 @@
 Hardened: libruby.so.3.3.2: PASS: instrumentation test 
 Hardened: libruby.so.3.3.2: PASS: openssl-engine test 
 Hardened: libruby.so.3.3.2: PASS: production test 
-Hardened: libruby.so.3.3.2: FAIL: property-note test because a property note was found but it shows that cf-protection is not enabled 
-Hardened: libruby.so.3.3.2: info: For more information visit: https://sourceware.org/annobin/annobin.html/Test-property-note.html
+Hardened: libruby.so.3.3.2: PASS: property-note test because CET enabled property note found 
 Hardened: libruby.so.3.3.2: PASS: run-path test 
 Hardened: libruby.so.3.3.2: PASS: rwx-seg test 
 Hardened: libruby.so.3.3.2: PASS: short-enums test 
@@ -137,4 +134,4 @@
 Hardened: libruby.so.3.3.2: PASS: textrel test 
 Hardened: libruby.so.3.3.2: PASS: threads test 
 Hardened: libruby.so.3.3.2: PASS: unicode test 
-Hardened: libruby.so.3.3.2: Overall: FAIL.
+Hardened: libruby.so.3.3.2: Overall: PASS.

[vo.x (Vit Ondruch)]

Forgot to mention that Ruby needs to be built with --disable-yjit, because Rust is another problem:

https://bugzilla.redhat.com/show_bug.cgi?id=2284605#c1

[vo.x (Vit Ondruch)]

As can be seen, I am testing this with Ruby 3.3.2. The upstream situation is, however, already a bit different since #10987 (I actually took the inspiration from that PR). Given that GCC documentation mentions that specifying the -fcf-protection compiler option defines the CET macro, I think that we should build from here and condition the #10987 based on the __CET__ macro instead of specializing it to __OpenBSD__. And build system should somehow propagate the flag to the ASFLAGS.

BTW given what the annocheck referenced documentation says, I think that the patch above should be enough and the #5895 likely address SHSTK, but certainly not the IBT what is the main point of this ticket.

[vo.x (Vit Ondruch)]

And FTR, this is the cet.h content

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-41:

Given that GCC documentation mentions that specifying the -fcf-protection compiler option defines the CET macro, I think that we should build from here and condition the #10987 based on the __CET__ macro instead of specializing it to __OpenBSD__.

PR implementing the first part: https://github.com/ruby/ruby/pull/11081

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned

[ioquatix (Samuel Williams)]

Thanks, I agree with your proposed change.

[ioquatix (Samuel Williams)]

BTW, IIUC, if we can use CET instead of the shadow stack, it's a far simpler and more efficient solution, so I'm strongly in favour of that direction (assuming it's viable).

[vo.x (Vit Ondruch)]

ioquatix (Samuel Williams) wrote in #note-46:

BTW, IIUC, if we can use CET instead of the shadow stack, it's a far simpler and more efficient solution, so I'm strongly in favour of that direction (assuming it's viable).

If I understand correctly, CET are two things: shadow stack and indirect branch tracking. My proposal is to improve the IBT implementation, which seems to be enough to make the annocheck happy.

I still think that exploring the shadow stack would be worth of the effort. If nothing else, the shadow stack was quite often mentioned during the frame pointer discussions 1. Therefore it would probably make sense to extract this part into separate ticket and try to finalize your PR in the future.

@fweimer (Florian Weimer) do you have by a chance any thoughts on this topic?

[fweimer (Florian Weimer)]

The change in https://github.com/ruby/ruby/pull/11081/files does not look correct to me because it enables shadow stack and indirect branch tracking in the ELF markup. But the changes to Context.S merely enable indirect branch tracking markup. The shadow stack is not context-switched, so this will crash on the ret instruction.

[vo.x (Vit Ondruch)]

fweimer (Florian Weimer) wrote in #note-48:

The change in https://github.com/ruby/ruby/pull/11081/files does not look correct to me because it enables shadow stack and indirect branch tracking in the ELF markup. But the changes to Context.S merely enable indirect branch tracking markup. The shadow stack is not context-switched, so this will crash on the ret instruction.

So either the metadata should signal just IBT or the shadow stack needs to be implemented, right?

[kjtsanaktsidis (KJ Tsanaktsidis)]

I wonder if distributors who want shadow-stack support should just compile Ruby with --with-coroutine=ucontext to use the swapcontext(3) based implementation of fibers instead of the assembly language one. I've spent the morning looking at the glibc sources trying to figure out how CET shadow stacks are supposed to work, but... we also can just use the glibc implementation instead of copying it :)

[kjtsanaktsidis (KJ Tsanaktsidis)]

So either the metadata should signal just IBT

in the meanwhile I've done this in https://github.com/ruby/ruby/pull/11112

• Status changed from Assigned to Closed

Applied in changeset git|3381c60836ae499899179f454c6bb1dd3b0891ba.

---

Ensure CFLAGS are passed to probes.o generation

The dtrace python script from systemtap on Linux actually looks at the
CFLAGS environment variable when invoking gcc to make the probes.o file.
If we don't pass the CFLAGS we're using, this probes.o file can wind up
without the required annotations indicating that it supports e.g. Intel
CET.

Fix this by explicitly exporting our build flags to the environment for
this script.

[Bug #18061]