Project

General

Profile

Actions

Bug #17864

closed

[BUG] try to mark T_NONE object (in Ractors)

Added by NuriYuri (Youri Nouri) almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]
[ruby-core:103842]

Description

Hello,
I tried to make an experiment with Ractors, computing SHA1 using Ractors (I was hopping to divide the computation time because SHA1 hashing does not release GVL and I'm running a "GUI" that has time constraint).
When I test with StringIO everything works fine, I can compute 400 StringIO that are like 100MB without issues. But when I use files (what I need to do in real environment) I face a internal bug (that doesn't not spawn in all attempt, it's very random).

And here's all the information about the bug:
ruby -v = ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]
Error log:

<OBJ_INFO:gc_mark_ptr@../ruby-3.0.0/gc.c:6106> 0x053fbf68 [0 M   ] T_NONE 
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105: [BUG] try to mark T_NONE object
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]

-- Control frame information -----------------------------------------------
c:0003 p:---- s:0017 e:000016 CFUNC  :read
c:0002 p:0056 s:0012 e:000011 BLOCK  D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `block in make_ractor'
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `read'

-- C level backtrace information -------------------------------------------
C:\WINDOWS\SYSTEM32\ntdll.dll(ZwWaitForSingleObject+0xc) [0x775729ac]
C:\WINDOWS\System32\KERNELBASE.dll(WaitForSingleObject+0x12) [0x767d0452]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_bugreport+0x669) [0x68134f29]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_source_location+0x53) [0x68118d83]
C:\WINDOWS\System32\KERNELBASE.dll(CreateEventA+0x24) [0x767e1914]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_proc_local_ep+0xd81) [0x680d8741]
C:\WINDOWS\SYSTEM32\ntdll.dll(RtlGetAppContainerNamedObjectPath+0x11e) [0x77567a4e]

-- Other runtime information -----------------------------------------------

* Loaded script: src/IODigester.ytest copy.rb

* Loaded features:

    0 enumerator.so
    1 thread.rb
    2 rational.so
    3 complex.so
    4 ruby2_keywords.rb
    5 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/encdb.so
    6 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/transdb.so
    7 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/rbconfig.rb
    8 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/compatibility.rb
    9 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults.rb
   10 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/deprecate.rb
   11 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/errors.rb
   12 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/exceptions.rb
   13 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/singleton.rb
   14 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime.rb
   15 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/msys2_installation.rb
   16 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/fiddle.so
   17 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/closure.rb
   18 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/function.rb
   19 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/version.rb
   20 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle.rb
   21 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/dll_directory.rb
   22 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/value.rb
   23 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/pack.rb
   24 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/struct.rb
   25 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/cparser.rb
   26 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/import.rb
   27 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/utf_16le.so
   28 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/utf_16_32.so
   29 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/win32/registry.rb
   30 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/single_byte.so
   31 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults/operating_system.rb
   32 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/basic_specification.rb
   33 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/stub_specification.rb
   34 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/text.rb
   35 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/user_interaction.rb
   36 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification_policy.rb
   37 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util/list.rb
   38 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/platform.rb
   39 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/version.rb
   40 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/requirement.rb
   41 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification.rb
   42 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util.rb
   43 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/dependency.rb
   44 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_gem.rb
   45 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/monitor.so
   46 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/monitor.rb
   47 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb
   48 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_warn.rb
   49 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems.rb
   50 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/path_support.rb
   51 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/version.rb
   52 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/core_ext/name_error.rb
   53 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/levenshtein.rb
   54 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/jaro_winkler.rb
   55 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checker.rb
   56 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
   57 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
   58 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers.rb
   59 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/method_name_checker.rb
   60 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/key_error_checker.rb
   61 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/null_checker.rb
   62 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/tree_spell_checker.rb
   63 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/require_path_checker.rb
   64 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/formatters/plain_formatter.rb
   65 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean.rb
   66 D:/nuriy/persoWork/game_launcher/src/IODigester.rb
   67 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest.so
   68 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/digest.rb
   69 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest/sha1.so

Note: I noticed while writing the thread that my ruby version in this project was not updated, I tried with ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [i386-mingw32] and I have nothing but Idk if that's luck or not. If someone can confirm that this bug is solved that would be nice because I really hope to use that in production even if Ractors are still experimental.

Additional info

Here's the class I wrote to handle the SHA1 computation over ractors:

# Class responsive of digesting data (hashing) over several CPU core using Ractors
#
# How to use ?
#   1. First of all encapsulate all your data into IO object (File or StringIO)
#   2. Then create a new IODigester object by passing the Digest class you need and all the IO to digest in a Hash
#   3. Call the start function to let the system instanciate everything
#     * You can check the overall progress using #progess, it's a number between 0 & 1
#   4. If failed? respond true, you can grab the failure info using #failures (key = IO key, value = StandardError object)
#
# @example Demo
#   digester = IODigester.new(Digest::SHA1, { 'test' => io1, 'test2' => io2 })
#   digester.start { |d| print format("\r%<progress>.2f%%", progress: d.progress * 100) }
#   digester.hashes # => { 'test' => '<SHA1 of test>', 'test2' => '<SHA1 of test2>' }
#
# @note IODigester requires the IO to respond to size and give accurate value for size (in bytes)!
class IODigester
  # Get the hashes
  # @return [Hash{ String => String }]
  attr_reader :hashes
  # Get the failures
  # @return [Hash{ String => StandardError }]
  attr_reader :failures

  # Create a new IODigester
  # @param klass [Class<Digest::Base>] digest class to use to digest data
  # @param ios [Hash{ String => IO }] io to digest (key can be symbol if you want)
  def initialize(klass, ios)
    @klass = klass
    @ios = ios.clone
    @progresses = @ios.keys.map { |k| [k, 0] }.to_h
    @hashes = {}
    @failures = {}
  end

  # Start the process
  # @return [self]
  # @yieldparam d [self] current digester
  def start(&block)
    raise 'Already started' if @ractors

    @ractors = @ios.map { |key, value| make_ractor(key, value) }
    process_ractors(block)
    return self
  end

  # Test if the process has failures
  # @return [Boolean]
  def failed?
    return !@failures.empty?
  end

  # Get the progress
  # @return [Float]
  def progress
    @progresses.reduce(0.0) { |prev, curr| prev + curr.last } / @progresses.size
  end

  private

  # Function that processes all the ractors (in a Thread in order to not lock the GVL)
  # @param block [Proc]
  def process_ractors(block)
    until @ractors.empty?
      begin
        ractor, object = Ractor.select(*@ractors)
        parse_ractor_response(ractor, object)
        block&.call(self)
      rescue Ractor::RemoteError => e
        @failures[e.ractor.name] = e.cause
        @ractors.delete(e.ractor)
      end
    end
  end

  # Function that analyzes the payload of the received object from Ractor.select
  # @param ractor [Ractor] ractor object that we're talking about
  # @param object [Array] response from the ractor
  def parse_ractor_response(ractor, object)
    message, body = object
    case message
    when :progress
      @progresses[ractor.name] = body
    when :result
      @progresses[ractor.name] = 1
      @hashes[ractor.name] = body
      @ractors.delete(ractor)
    end
  end

  # Function that creates the ractor
  # @param name [String] name of the io (hash key)
  # @param io [IO] io to process
  # @return [Ractor]
  def make_ractor(name, io)
    ractor = Ractor.new(@klass, io, name: name) do |digest_klass, io_to_process|
      close_incoming
      # @type [Integer]
      io_size = io_to_process.size
      io_read = 0
      # @type [Digest::Base]
      digest = digest_klass.new
      while io_size > io_read
        Ractor.yield [:progress, io_read.to_f / io_size]
        # @type [String]
        data = io_to_process.read(20_971_520)
        io_read += data.bytesize
        digest << data
      end
      Ractor.yield [:result, digest.hexdigest]
    end
    return ractor
  end
end

Here's the script I use to try to reproduce the bug:

require_relative 'IODigester'
require 'digest'

ios = (
  Dir['D:/nuriy/Work/pokemonsdk/Release/Data/*'] +
  Dir['D:/nuriy/Work/pokemonsdk/Release/pokemonsdk/**/*']
).reject { |f| File.directory?(f) }.map { |f| [f, File.new(f, 'rb')] }.to_h
digester = IODigester.new(Digest::SHA1, ios)
digester.start { |d| print format("\r%<progress>.2f%%", progress: d.progress * 100) }
p digester.hashes

(I'm not closing the files because I don't actually care in that test since the bug appears in "start")

Here's the files in case you need it: https://www.mediafire.com/file/wt0eqzthxpb1nc8/Release.7z/file (it's over 2MB)

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0