Bug #20796: Segmentation fault in rubyzip tests with ruby 3.4.0-preview2~659 e7cb70be4e on x86_64-darwin24

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.4.0dev (2024-09-03T05:25:25Z v3_4_0_preview2~659 e7cb70be4e) [x86_64-darwin24]
• Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONTNEED

[tikkss (Tsutomu Katsube)]

Steps to reproduce

$ git clone [e-mail]:rubyzip/rubyzip
$ cd rubyzip
$ bundle
$ bundle exec rake

Expected behavior

The tests should complete successfully.

Actual behavior

A segmentation fault occurs during the test run.
This issue does not occur with ruby 3.4.0-preview1 or other Ruby versions.

Console dump

See attached crash.log and ruby-2024-10-13-071029.ips for full details

$ bundle exec rake TESTOPTS="-v --seed=1" 2>crash.log
Run options: -v --seed=1

# Running:

ZipFileTest#test_get_output_stream = 0.02 s = .
ZipFileTest#test_add_directory = 0.01 s = .
ZipFileTest#test_streaming = 0.01 s = .
ZipFileTest#test_nonexistant_zip = 0.00 s = .
ZipFileTest#test_open_buffer_without_block = 0.00 s = .
ZipFileTest#test_rename_to_existing_entry = 0.00 s = .
ZipFileTest#test_open_buffer_with_io_and_block = 0.00 s = .
ZipFileTest#test_write_buffer = 0.01 s = .
ZipFileTest#test_open_buffer_with_stringio = 0.00 s = .
ZipFileTest#test_odd_extra_field = 0.00 s = .
ZipFileTest#test_add_existing_entry_name = 0.00 s = .
ZipFileTest#test_open_buffer_no_op_does_not_change_file = 0.00 s = .
ZipFileTest#test_replace = 0.01 s = .
ZipFileTest#test_rename = 0.01 s = .
ZipFileTest#test_recover_permissions_after_add_files_to_archive = %

My environment

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.6.0.dev
  - RUBY VERSION: 3.4.0 (2024-10-07 patchlevel -1) [x86_64-darwin24]
  - INSTALLATION DIRECTORY: /Users/zzz/.rbenv/versions/3.4.0-preview2/lib/ruby/gems/3.4.0+0
  - USER INSTALLATION DIRECTORY: /Users/zzz/.gem/ruby/3.4.0+0
  - RUBY EXECUTABLE: /Users/zzz/.rbenv/versions/3.4.0-preview2/bin/ruby
  - GIT EXECUTABLE: /usr/local/bin/git
  - EXECUTABLE DIRECTORY: /Users/zzz/.rbenv/versions/3.4.0-preview2/bin
  - SPEC CACHE DIRECTORY: /Users/zzz/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /Users/zzz/.rbenv/versions/3.4.0-preview2/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-darwin-24
  - GEM PATHS:
     - /Users/zzz/.rbenv/versions/3.4.0-preview2/lib/ruby/gems/3.4.0+0
     - /Users/zzz/.gem/ruby/3.4.0+0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
     - "gem" => "--no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /Users/zzz/.rbenv/versions/3.4.0-preview2/bin
     - /usr/local/Cellar/rbenv/1.3.0/libexec
     - /Users/zzz/.local/bin
     - /Users/zzz/.rbenv/shims
     - /Users/zzz/.rbenv/bin
     - /Users/zzz/.nodenv/bin
     - /Users/zzz/bin
     - /Users/zzz/.cargo/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /System/Cryptexes/App/usr/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
     - /var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
     - /Library/Apple/usr/bin

$ cat Gemfile.lock
PATH
  remote: .
  specs:
    rubyzip (3.0.0.alpha)

GEM
  remote: https://rubygems.org/
  specs:
    ast (2.4.2)
    docile (1.4.1)
    json (2.7.2)
    language_server-protocol (3.17.0.3)
    minitest (5.22.3)
    parallel (1.26.3)
    parser (3.3.5.0)
      ast (~> 2.4.1)
      racc
    psych (5.1.2)
      stringio
    racc (1.8.1)
    rainbow (3.1.1)
    rake (13.1.0)
    rdoc (6.6.3.1)
      psych (>= 4.0.0)
    regexp_parser (2.9.2)
    rexml (3.3.8)
    rubocop (1.61.0)
      json (~> 2.3)
      language_server-protocol (>= 3.17.0)
      parallel (~> 1.10)
      parser (>= 3.3.0.2)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8, < 3.0)
      rexml (>= 3.2.5, < 4.0)
      rubocop-ast (>= 1.30.0, < 2.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 2.4.0, < 3.0)
    rubocop-ast (1.32.3)
      parser (>= 3.3.1.0)
    rubocop-performance (1.20.2)
      rubocop (>= 1.48.1, < 2.0)
      rubocop-ast (>= 1.30.0, < 2.0)
    rubocop-rake (0.6.0)
      rubocop (~> 1.0)
    ruby-progressbar (1.13.0)
    simplecov (0.22.0)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-html (0.13.1)
    simplecov-lcov (0.8.0)
    simplecov_json_formatter (0.1.4)
    stringio (3.1.1)
    unicode-display_width (2.6.0)

PLATFORMS
  ruby
  x86_64-darwin-24

DEPENDENCIES
  minitest (~> 5.22.0)
  rake (~> 13.1.0)
  rdoc (~> 6.6.2)
  rubocop (~> 1.61.0)
  rubocop-performance (~> 1.20.0)
  rubocop-rake (~> 0.6.0)
  rubyzip!
  simplecov (~> 0.22.0)
  simplecov-lcov (~> 0.8)

BUNDLED WITH
   2.6.0.dev

[tenderlovemaking (Aaron Patterson)]

I'm not able to reproduce this, but I'm on an ARM mac. It looks like this is an Intel machine running macos? Is that correct? Can you give us more information about your environment?

Thanks!

[tikkss (Tsutomu Katsube)]

Yes, that's correct. I'm using an Intel-based Mac running macOS. Here are the details of my environment:

• Architecture: Intel
• System Version: macOS Sequoia 15.0.1
• Build Version: 24A348
• Kernel Version: Darwin 24.0.0
• Model: MacBook Pro 13-inch, 2018, Four Thunderbolt 3 Ports
• Processor: 2.3 GHz Quad core Intel Core i5
• Graphics: Intel Iris Plus Graphics 655 1536MB
• Memory: 16GB 2133 MHz LPDDR3

Please let me know if you need any additional information.

Thanks!

[tikkss (Tsutomu Katsube)]

• File crash.file_test.log crash.file_test.log added

The segmentation fault does not reproduce when running the test individually:

$ bundle exec rake TESTOPTS="-v --seed=1 --name=ZipFileTest#test_recover_permissions_after_add_files_to_archive"
ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
/Users/zzz/.rbenv/versions/3.4.0-preview2/bin/ruby -w -I"lib:lib:test" /Users/zzz/.rbenv/versions/3.4.0-preview2/lib/ruby/gems/3.4.0+0/gems/rake-13.1.0/lib/rake/rake_test_loader.rb "test/basic_zip_file_test.rb" "test/bzip2_support_test.rb" "test/case_sensitivity_test.rb" "test/central_directory_entry_test.rb" "test/central_directory_test.rb" "test/constants_test.rb" "test/crypto/null_encryption_test.rb" "test/crypto/traditional_encryption_test.rb" "test/decompressor_test.rb" "test/deflater_test.rb" "test/dos_time_test.rb" "test/encryption_test.rb" "test/entry_set_test.rb" "test/entry_test.rb" "test/extra_field_test.rb" "test/extra_field_unknown_test.rb" "test/extra_field_ut_test.rb" "test/file_extract_directory_test.rb" "test/file_extract_test.rb" "test/file_options_test.rb" "test/file_permissions_test.rb" "test/file_split_test.rb" "test/file_test.rb" "test/filesystem/dir_test.rb" "test/filesystem/directory_iterator_test.rb" "test/filesystem/file_mutating_test.rb" "test/filesystem/file_nonmutating_test.rb" "test/filesystem/file_stat_test.rb" "test/inflater_test.rb" "test/input_stream_test.rb" "test/ioextras/abstract_input_stream_test.rb" "test/ioextras/abstract_output_stream_test.rb" "test/ioextras/fake_io_test.rb" "test/local_entry_test.rb" "test/output_stream_test.rb" "test/pass_thru_compressor_test.rb" "test/pass_thru_decompressor_test.rb" "test/path_traversal_test.rb" "test/samples/example_recursive_test.rb" "test/settings_test.rb" "test/stored_support_test.rb" "test/unicode_file_names_and_comments_test.rb" "test/zip64_full_test.rb" "test/zip64_support_test.rb" -v --seed=1 --name=ZipFileTest#test_recover_permissions_after_add_files_to_archive
Run options: -v --seed=1 --name=ZipFileTest#test_recover_permissions_after_add_files_to_archive

# Running:

ZipFileTest#test_recover_permissions_after_add_files_to_archive = 0.01 s = .

Finished in 0.011834s, 84.5023 runs/s, 253.5068 assertions/s.

1 runs, 3 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Unit Tests to /private/tmp/rubyzip/coverage.
Line Coverage: 59.45% (1268 / 2133)
Branch Coverage: 22.82% (149 / 653)
Lcov style coverage report generated for Unit Tests to coverage/lcov.info

However, it reproduces when expanding the scope to a test file:

$ bundle exec rake TEST="test/file_test.rb" TESTOPTS="-v --seed=1" 2>crash.file_test.log
Run options: -v --seed=1

# Running:

ZipFileTest#test_get_output_stream = 0.02 s = .
ZipFileTest#test_add_directory = 0.00 s = .
ZipFileTest#test_streaming = 0.00 s = .
ZipFileTest#test_nonexistant_zip = 0.00 s = .
ZipFileTest#test_open_buffer_without_block = 0.00 s = .
ZipFileTest#test_rename_to_existing_entry = 0.00 s = .
ZipFileTest#test_open_buffer_with_io_and_block = 0.00 s = .
ZipFileTest#test_write_buffer = 0.01 s = .
ZipFileTest#test_open_buffer_with_stringio = 0.00 s = .
ZipFileTest#test_odd_extra_field = 0.00 s = .
ZipFileTest#test_add_existing_entry_name = 0.00 s = .
ZipFileTest#test_open_buffer_no_op_does_not_change_file = 0.00 s = .
ZipFileTest#test_replace = 0.01 s = .
ZipFileTest#test_rename = 0.01 s = .
ZipFileTest#test_recover_permissions_after_add_files_to_archive = 0.01 s = .
ZipFileTest#test_get_input_stream_stored_with_gpflag_bit3 = 0.00 s = .
ZipFileTest#test_remove = 0.01 s = .
ZipFileTest#test_preserve_file_order = %

See attached crash.file_test.log for full details.
By the way, crash log does not exists in ~/Library/Logs/DiagnosticReports/.

Thanks!

[Earlopain (Earlopain _)]

This PR seems relevant, and fits in the (rather large) preview1/preview2 window: https://github.com/ruby/ruby/pull/11519

[tikkss (Tsutomu Katsube)]

• Subject changed from Segmentation fault in rubyzip tests with ruby 3.4.0-preview2 to Segmentation fault in rubyzip tests with ruby 3.4.0-preview2~659 e7cb70be4e on x86_64-darwin24
• ruby -v changed from ruby 3.4.0preview2 (2024-10-07 master 32c733f57b) +PRISM [x86_64-darwin24] to ruby 3.4.0dev (2024-09-03T05:25:25Z v3_4_0_preview2~659 e7cb70be4e) [x86_64-darwin24]

This PR seems relevant, and fits in the (rather large) preview1/preview2 window: https://github.com/ruby/ruby/pull/11519

Thanks for your sharing.

According to git bisect between preview1 and preview2, it is triggered since ruby 3.4.0dev (2024-09-03T05:25:25Z v3_4_0_preview2~659 e7cb70be4e) [x86_64-darwin24].
The commit (e7cb70be4e) was merged as part of https://github.com/ruby/ruby/pull/11519.

If you need more information, please tell me what information do you need.

[mame (Yusuke Endoh)]

I don't have an access to Intel mac, but I think https://github.com/ruby/ruby/pull/11519 has indeed an off-by-one bug. I could reproduce the issue on Linux with valgrind by forcing to use the wrong memrchr function.

$ valgrind ./miniruby -e 'p ("Barbaz" + "qux"*1000).rindex("barbaz")'
...
==4779== Invalid read of size 1
==4779==    at 0x3B26DE: my_memrchr (string.c:4529)
==4779==    by 0x3B26DE: str_rindex (string.c:4558)
==4779==    by 0x3CB46B: rb_str_rindex_m (string.c:4699)
==4779==    by 0x431A9C: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3794)
==4779==    by 0x452D9D: vm_sendish (vm_insnhelper.c:5961)
==4779==    by 0x452D9D: vm_exec_core (insns.def:898)
==4779==    by 0x4424A4: rb_vm_exec (vm.c:2564)
==4779==    by 0x23A004: rb_ec_exec_node (eval.c:281)
==4779==    by 0x23C0A8: ruby_run_node (eval.c:319)
==4779==    by 0x15FFC5: rb_main (main.c:43)
==4779==    by 0x15FFC5: main (main.c:62)
==4779==  Address 0x20682b6f is 1 bytes before a block of size 3,007 alloc'd
==4779==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==4779==    by 0x2614EA: rb_gc_impl_malloc (default.c:8186)
==4779==    by 0x3B3FB3: str_new0 (string.c:983)
==4779==    by 0x3B8686: rb_str_plus (string.c:2375)
==4779==    by 0x4593F3: vm_opt_plus (vm_insnhelper.c:6491)
==4779==    by 0x4593F3: vm_exec_core (insns.def:1244)
==4779==    by 0x4424A4: rb_vm_exec (vm.c:2564)
==4779==    by 0x23A004: rb_ec_exec_node (eval.c:281)
==4779==    by 0x23C0A8: ruby_run_node (eval.c:319)
==4779==    by 0x15FFC5: rb_main (main.c:43)
==4779==    by 0x15FFC5: main (main.c:62)
==4779==

https://github.com/ruby/ruby/pull/11923 fixes the off-by-one issue. I am not sure if this change fixes the issue that the OP reports. @tikkss Can you try this patch?

[tikkss (Tsutomu Katsube)]

Thank you very much!

I have confirmed a83c91dd7a addresses the issue on Intel mac:

$ ruby -v
ruby 3.4.0dev (2024-10-21T11:40:42Z master a83c91dd7a) +PRISM [x86_64-darwin24]

$ bundle exec rake
ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
/Users/zzz/.rbenv/versions/a83c91dd7a/bin/ruby -w -I"lib:lib:test" /Users/zzz/.rbenv/versions/a83c91dd7a/lib/ruby/gems/3.4.0+0/gems/rake-13.1.0/lib/rake/rake_test_loader.rb "test/basic_zip_file_test.rb" "test/bzip2_support_test.rb" "test/case_sensitivity_test.rb" "test/central_directory_entry_test.rb" "test/central_directory_test.rb" "test/constants_test.rb" "test/crypto/null_encryption_test.rb" "test/crypto/traditional_encryption_test.rb" "test/decompressor_test.rb" "test/deflater_test.rb" "test/dos_time_test.rb" "test/encryption_test.rb" "test/entry_set_test.rb" "test/entry_test.rb" "test/extra_field_test.rb" "test/extra_field_unknown_test.rb" "test/extra_field_ut_test.rb" "test/file_extract_directory_test.rb" "test/file_extract_test.rb" "test/file_options_test.rb" "test/file_permissions_test.rb" "test/file_split_test.rb" "test/file_test.rb" "test/filesystem/dir_test.rb" "test/filesystem/directory_iterator_test.rb" "test/filesystem/file_mutating_test.rb" "test/filesystem/file_nonmutating_test.rb" "test/filesystem/file_stat_test.rb" "test/inflater_test.rb" "test/input_stream_test.rb" "test/ioextras/abstract_input_stream_test.rb" "test/ioextras/abstract_output_stream_test.rb" "test/ioextras/fake_io_test.rb" "test/local_entry_test.rb" "test/output_stream_test.rb" "test/pass_thru_compressor_test.rb" "test/pass_thru_decompressor_test.rb" "test/path_traversal_test.rb" "test/samples/example_recursive_test.rb" "test/settings_test.rb" "test/stored_support_test.rb" "test/unicode_file_names_and_comments_test.rb" "test/zip64_full_test.rb" "test/zip64_support_test.rb"
/Users/zzz/.rbenv/versions/a83c91dd7a/lib/ruby/gems/3.4.0+0/gems/minitest-5.22.3/lib/minitest/mock.rb:33: warning: redefining 'object_id' may cause serious problems
Run options: --seed 39370

# Running:

............................................................................................................................................................................/Users/zzz/src/github.com/rubyzip/rubyzip/test/ioextras/abstract_output_stream_test.rb:55: warning: '$OUTPUT_RECORD_SEPARATOR' is deprecated
.............................................................................................................S...S........................................................................

Finished in 1.143906s, 312.9628 runs/s, 2212.5944 assertions/s.

358 runs, 2531 assertions, 0 failures, 0 errors, 2 skips

You have skipped tests. Run with --verbose for details.
Coverage report generated for Unit Tests to /Users/zzz/src/github.com/rubyzip/rubyzip/coverage.
Line Coverage: 96.44% (2057 / 2133)
Branch Coverage: 78.25% (511 / 653)
Lcov style coverage report generated for Unit Tests to coverage/lcov.info

[mame (Yusuke Endoh)]

• Status changed from Open to Closed
• Backport changed from 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN to 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONTNEED

Great to hear, thank you very much.

[tikkss (Tsutomu Katsube)]

Though this is just a bug report, it's my first contribution to Ruby Core.
I'm excited to know some new words such as off-one-by bug, valgrind, miniruby, OP reports that were unfamiliar to me.
Thanks!