Bug #18058: 3.1.0-dev with MJIT enabled Zlib::BufError during `gem install`

• Status: Closed
• Assignee: k0kubun (Takashi Kokubun)
• Target version: -
• ruby -v: ruby 3.1.0dev (2021-08-01T00:54:36Z master 605421f4eb) +JIT [x86_64-linux]
• Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN

[byroot (Jean Boussier)]

It doesn't always happen (maybe 5-10% of the cases), but since I added MJIT to our CI builds I've been seeing this error a lot.

I was able to reproduce it in docker run -it rubylang/ruby:master-nightly-bionic after 15 or so tries:

# ruby -v
ruby 3.1.0dev (2021-08-01T00:54:36Z master 605421f4eb) [x86_64-linux]
# gem uninstall bundler -v 2.2.21; env RUBYOPT="--enable-jit --jit-max-cache=10000 --jit-min-calls=10 $RUBYOPT" gem install --no-document --user-install bundler -v 2.2.21
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Exception while verifying /usr/local/lib/ruby/gems/3.1.0/cache/bundler-2.2.21.gem
ERROR:  While executing gem ... (Zlib::BufError)
    buffer error

e.g.

$ docker run -it rubylang/ruby:master-nightly-bionic bash -c 'while env RUBYOPT="--enable-jit --jit-max-cache=10000 --jit-min-calls=10 $RUBYOPT" gem install --no-document --user-install bundler -v 2.2.21; do gem uninstall bundler -v 2.2.21; done'
Fetching bundler-2.2.21.gem
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Successfully installed bundler-2.2.21
1 gem installed
Successfully uninstalled bundler-2.2.21
WARNING:  You don't have /root/.local/share/gem/ruby/3.1.0/bin in your PATH,
	  gem executables will not run.
Exception while verifying /usr/local/lib/ruby/gems/3.1.0/cache/bundler-2.2.21.gem
ERROR:  While executing gem ... (Zlib::BufError)
    buffer error

[k0kubun (Takashi Kokubun)]

• Assignee set to k0kubun (Takashi Kokubun)

[k0kubun (Takashi Kokubun)]

Do you know if we can let Bundler rubygems (you're right, it wasn't Bundler) show the backtrace? The first step to fix this is to increase reproduction probability, and the backtrace information might be helpful for reproducing the issue with a smaller script for it.

[byroot (Jean Boussier)]

Do you know if we can let Bundler show the backtrace?

It's rubygems not bundler. I don't know how make it print the backtrace, but I can take a look. Worst case I can probably directly patch it to let the error bubble up.

[byroot (Jean Boussier)]

Here you go:

/usr/local/lib/ruby/3.1.0/rubygems/package.rb:698:in `read': buffer error (Zlib::BufError)
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:698:in `block in verify_gz'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:697:in `wrap'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:697:in `verify_gz'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:664:in `verify_entry'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:676:in `block in verify_files'
	from /usr/local/lib/ruby/3.1.0/rubygems/package/tar_reader.rb:65:in `each'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:675:in `verify_files'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:606:in `block (2 levels) in verify'
	from /usr/local/lib/ruby/3.1.0/rubygems/package/tar_reader.rb:27:in `new'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:603:in `block in verify'
	from /usr/local/lib/ruby/3.1.0/rubygems/package/file_source.rb:29:in `open'
	from /usr/local/lib/ruby/3.1.0/rubygems/package/file_source.rb:29:in `with_read_io'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:602:in `verify'
	from /usr/local/lib/ruby/3.1.0/rubygems/package.rb:583:in `spec'
	from /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:269:in `spec'
	from /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:719:in `verify_spec'
	from /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:891:in `pre_install_checks'
	from /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:286:in `install'
	from /usr/local/lib/ruby/3.1.0/rubygems/resolver/specification.rb:104:in `install'
	from /usr/local/lib/ruby/3.1.0/rubygems/request_set.rb:194:in `block in install'
	from /usr/local/lib/ruby/3.1.0/rubygems/request_set.rb:182:in `each'
	from /usr/local/lib/ruby/3.1.0/rubygems/request_set.rb:182:in `install'
	from /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:210:in `install_gem'
	from /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:226:in `block in install_gems'
	from /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:219:in `each'
	from /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:219:in `install_gems'
	from /usr/local/lib/ruby/3.1.0/rubygems/commands/install_command.rb:167:in `execute'
	from /usr/local/lib/ruby/3.1.0/rubygems/command.rb:323:in `invoke_with_build_args'
	from /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:178:in `process_args'
	from /usr/local/lib/ruby/3.1.0/rubygems/command_manager.rb:147:in `run'
	from /usr/local/lib/ruby/3.1.0/rubygems/gem_runner.rb:53:in `run'
	from /usr/local/bin/gem:21:in `<main>'

[byroot (Jean Boussier)]

• Related to Bug #18277: buffer error (Zlib::BufError) in Zlib::Deflate#deflate when using MJIT added

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned

[k0kubun (Takashi Kokubun)]

• Status changed from Assigned to Closed

I replaced the entire implementation in [Feature #19420], and I haven't seen this issue with today's implementation.