Bug #19909: s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures - Ruby master - Ruby Issue Tracking System

Actions

Copy link

Bug #19909

closed

s390x zlib different deflate algorithm producing a different compressed byte stream causing test failures

Added by jaruga (Jun Aruga) about 1 year ago. Updated about 1 year ago.

Status:

Closed

Assignee:

Target version:

ruby -v:

Backport:

3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[ruby-core:114942]

Description

I faced the test failures ruby/zlib (zlib gem) in make test-all and ruby/spec (make test-spec) in Ubuntu version jammy (22.04) s390x. I intend to manage this issue on this ticket, and giving the space for users to comment about the issue.

The test failures¶

First, below are the test failures by make test-all and make test-spec on Ubuntu jammy s390x.

Here is the used zlib version.

$ dpkg -s "$(dpkg -S /usr/include/zlib.h | cut -d ":" -f 1)" | grep ^Version
Version: 1:1.2.11.dfsg-2ubuntu9.2

+ make -s test-all -o exts TESTOPTS=-j3 -q --tty=no  RUBYOPT=-w
generating enc.mk
making enc
making srcs under enc
generating transdb.h
transdb.h unchanged
making trans
making encs
Run options: 
  --seed=42873
  "--ruby=./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems"
  --excludes-dir=../test/.excludes
  --name=!/memory_leak/
  -j3
  -q
  --tty=no

# Running tests:


Retrying...

  1) Failure:
TestZlibDeflate#test_deflate_chunked [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:66]:
<7253> expected but was
<8325>.

  2) Failure:
TestZlibDeflate#test_deflate_chunked_break [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:92]:
<3632> expected but was
<4702>.

  3) Failure:
TestZlibGzipReader#test_unused2 [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:968]:
<24> expected but was
<23>.

  4) Failure:
TestZlib#test_gzip [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1419]:
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFK\xCB\xCF\a\x00!es\x8C\x03\x00\x00\x00"> expected but was
<"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\xFFJ\xCB\xCF\a\f\x00!es\x8C\x03\x00\x00\x00">.

  5) Failure:
TestZlib#test_deflate_stream [/home/travis/build/junaruga/ruby/test/zlib/test_zlib.rb:1411]:
<20016> expected but was
<21085>.

Finished tests in 290.303048s, 88.4145 tests/s, 21049.4999 assertions/s.
25667 tests, 6110734 assertions, 5 failures, 0 errors, 172 skips

$SETARCH make -s test-spec
...
1)
Zlib::Deflate.deflate deflates some data FAILED
Expected 
"x\x9Cc\x80\x03\x00\x00
\x00\x01" == 
"x\x9Cc`\x80\x01\x00\x00
\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:10:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

2)
Zlib::Deflate.deflate deflates lots of data FAILED
Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0Eh\rF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x91\x0E\x06;\x00\x00\x80\x00\x00\x01" == 
"x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:18:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

3)
Zlib::Deflate.deflate deflates chunked data FAILED
Expected 21085 == 20016
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:32:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:4:in `<top (required)>'

4)
Zlib::Deflate#deflate deflates some data FAILED
Expected 
"x\x9Cc\x80\x03\x00\x00
\x00\x01" == 
"x\x9Cc`\x80\x01\x00\x00
\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:47:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>'

5)
Zlib::Deflate#deflate deflates lots of data FAILED
Expected "x\x9Cc\x18\xE1`\xA4\x83\x91\x0EF:\x18\xE9`\xA4\x83\x81\x06#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\x8Ct0\xD2\xC1H\a#\x1D\f4\x00\x00\x80\x00\x00\x01" == 
"x\x9C\xED\xC1\x01\x01\x00\x00\x00\x80\x90\xFE\xAF\xEE\b
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x18\x80\x00\x00\x01"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:56:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:36:in `<top (required)>'

6)
Zlib::Deflate#deflate without break deflates chunked data with final chunk FAILED
Expected 8325 == 7253
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:96:in `block (3 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>'

7)
Zlib::Deflate#deflate with break deflates chunked data with final chunk FAILED
Expected 4702 == 3632
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:123:in `block (3 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/deflate_spec.rb:68:in `<top (required)>'

8)
Zlib::Deflate#set_dictionary sets the dictionary FAILED
Expected "x\xBB\x14\xE1\x03\xCBJLJNIMK\xCF\xC8\xCC\x02\f\x00\x15\x86\x03\xF8" == "x\xBB\x14\xE1\x03\xCBKLJNIMK\xCF\xC8\xCC\x02\x00\x15\x86\x03\xF8"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:10:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate/set_dictionary_spec.rb:4:in `<top (required)>'

9)
Zlib.deflate deflates some data FAILED
Expected 
"x\x9C3\x84\x03\x00
\x91\x01\xEB" == 
"x\x9C34\x84\x01\x00
\x91\x01\xEB"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:6:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/deflate_spec.rb:4:in `<top (required)>'

10)
Zlib.gzip gzips the given string FAILED
Expected 
"24261MLJNI\x05\f\x00\x9D\x05\x00$
\x00\x00\x00" == 
"34261MLJNI\x05\x00\x9D\x05\x00$
\x00\x00\x00"
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:13:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzip_spec.rb:4:in `<top (required)>'

11)
Zlib::GzipWriter#write writes some compressed data FAILED
Expected [50, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 12, 0, 157, 5, 0, 36, 10, 0, 0, 0] == [51, 52, 50, 54, 49, 77, 76, 74, 78, 73, 5, 0, 157, 5, 0, 36, 10, 0, 0, 0]
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:19:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>'

12)
Zlib::GzipWriter#write handles inputs of 2^23 bytes FAILED
Expected 34263 == 8176
to be truthy but was false
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:34:in `block (2 levels) in <top (required)>'
/home/travis/build/junaruga/ruby/spec/ruby/library/zlib/gzipwriter/write_spec.rb:5:in `<top (required)>'

Finished in 114.477929 seconds

3715 files, 32609 examples, 190709 expectations, 12 failures, 0 errors, 0 tagged

The root cause¶

This issue can happen with zlib library applying the patch zlib/pull#410 that is to enable a different deflate algorithm producing a different compressed byte stream causing the test failures. I am not sure how a Ubuntu zlib deb package is developed in Ubuntu. However I was able to download and check the source file zlib_1.2.11.dfsg-2ubuntu9.2.debian.tar.xz on the Ubuntu jammy-updates zlib deb package page.

debian/rules

...
 49 # s390x fails at compatibility.
 50 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x))
 51 m32=-m31
 52 CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e
 53 CONFIGURE_COMMON += --dfltcc
 54 CONFIGURE_HOST += --crc32-vx
 55 else
 56 m32=-m32
 57 endif
...

debian/patches/series

...
  4 410.patch
...

debian/patches/410.patch

From 992a7afc3edfa511dff0650d1c545b11bf64e655 Mon Sep 17 00:00:00 2001
From: Ilya Leoshkevich <iii@linux.ibm.com>
Date: Wed, 18 Jul 2018 13:14:07 +0200
Subject: [PATCH] Add support for IBM Z hardware-accelerated deflate
...

And the 410.patch is the the patch zlib/pull#410 above.

A reproducer¶

For convenience, here is a relatively small reproducer.

$ cat test/zlib/_test_zlib_test_deflate_chunked.rb
require 'zlib'

z = Zlib::Deflate.new
input = "\x01"
z.deflate(input) do |chunk|
end
final = z.finish
puts "final.length: #{final.length}"

Below is ok case.

$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 9

Below is the case with the zlib where the tests are failing in s390x.

$ ruby -I./lib test/zlib/_test_zlib_test_deflate_chunked.rb
final.length: 10

Affected distributions and versions¶

Affected: at least Ubuntu jammy 22.04

I didn't see this issue in Ubuntu focal (20.04). The same 410.patch is applied in both Ubuntu focal and jammy. But how to build is different. In Ubuntu jammy, the zlib is configured by ./configure --dfltcc.

$ diff -u ubuntu_focal/zlib1g/debian/rules ubuntu_jammy/zlib1g/debian/rules
--- ubuntu_focal/zlib1g/debian/rules	2020-08-20 01:52:59.000000000 +0200
+++ ubuntu_jammy/zlib1g/debian/rules	2021-08-12 05:28:03.000000000 +0200
@@ -21,6 +21,9 @@
 LDFLAGS = `dpkg-buildflags --get LDFLAGS`
 EXTRA_MAKE =

+CONFIGURE_COMMON=--shared --prefix=/usr
+CONFIGURE_HOST=--libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+
 # binutils doesn't supply the prefixed version normally like GCC does so
 # we can't just unconditionally use DEB_HOST_GNU_TYPE-ar
 ifeq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))
@@ -46,8 +49,9 @@
 # s390x fails at compatibility.
 ifneq (,$(findstring $(DEB_HOST_ARCH), s390x))
 m32=-m31
-CFLAGS += -DDFLTCC
-EXTRA_MAKE += OBJA=dfltcc.o PIC_OBJA=dfltcc.lo
+CFLAGS += -DDFLTCC_LEVEL_MASK=0x7e
+CONFIGURE_COMMON += --dfltcc
+CONFIGURE_HOST += --crc32-vx
 else
 m32=-m32
 endif
@@ -95,7 +99,7 @@

 	if [ ! -f Makefile.stash ]; then cp Makefile Makefile.stash ; fi

-	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/lib/$(DEB_HOST_MULTIARCH)
+	AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc" CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" uname=GNU ./configure $(CONFIGURE_COMMON) $(CONFIGURE_HOST)

 	touch $@

@@ -106,7 +110,7 @@
 	cp -r $(COPYLIST) debian/64
 	cd debian/64 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m64)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr  --libdir=\$${prefix}/usr/lib64
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib64
 	touch $@

 configure32-stamp: configure
@@ -116,7 +120,7 @@
 	cp -r $(COPYLIST) debian/32
 	cd debian/32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(m32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configuren32-stamp: configure
@@ -126,7 +130,7 @@
 	cp -r $(COPYLIST) debian/n32
 	cd debian/n32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mn32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/lib32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/lib32
 	touch $@

 configurex32-stamp: configure
@@ -136,7 +140,7 @@
 	cp -r $(COPYLIST) debian/x32
 	cd debian/x32 && AR=$(AR) CC="$(DEB_HOST_GNU_TYPE)-gcc $(mx32)" \
 	    CFLAGS="$(CFLAGS)" LDFLAGS="$(LDFLAGS)" \
-	    uname=GNU ./configure --shared --prefix=/usr --libdir=\$${prefix}/usr/libx32
+	    uname=GNU ./configure $(CONFIGURE_COMMON) --libdir=\$${prefix}/usr/libx32
 	touch $@

 build: build-stamp $(EXTRA_BUILD)

I also didn't see this issue with the zlib RPM package in Fedora rawhide (Fedora 40), though I see the zlib is configured by ./configure --dfltcc applying the patch zlib-*-IBM-Z-hw-accelerated-deflate.patch. But when comparing the zlib-1.2.13-IBM-Z-hw-accelerated-deflate.patch in the zlib in Fedora rawhide (Fedora 40), and 410.patch in Ubuntu jammy-updates. It is quite different.

Workaround¶

The patch author commented that a workaround is to set environment variable DFLTCC=0 to disable the different deflate algorithm at zlib/issues#60.

How I fixed the issue with the workaround¶

ruby/zlib: https://github.com/ruby/zlib/pull/65
ruby/spec: https://github.com/ruby/spec/pull/1088
ruby/ruby: https://github.com/ruby/ruby/pull/8401
RubyCI - s390x (Ubuntu), the server is actually Ubuntu jammy, and the cron job is running with the DFLTCC=0 in the crontab.

How can we do?¶

The upstream patch zlib/pull#410 is not merged to the zlib repository yet. And I am not sure if we want to do like this, modifying the common.mk (and configure.ac).

diff --git a/common.mk b/common.mk
index b8ae911ef2..2605569443 100644
--- a/common.mk
+++ b/common.mk
@@ -982,6 +982,7 @@ test-spec: $(TEST_RUNNABLE)-test-spec
 yes-test-spec: yes-test-spec-precheck
        $(ACTIONS_GROUP)
        $(gnumake_recursive)$(Q) \
+       DFLTCC=0 \
        $(RUNRUBY) -r./$(arch)-fake -r$(tooldir)/rubyspec_temp \
                $(srcdir)/spec/mspec/bin/mspec run -B $(srcdir)/spec/default.mspec $(MSPECOPT) $(SPECOPTS)
        $(ACTIONS_ENDGROUP)

So, it seems to me that just documenting this issue and workaround somewhere in ruby/ruby is a good fix for this issue.