Bug #11277: "code converter not found" error with multi-thread (high occurrence rate since r50887)

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.3.0dev (2015-06-16) [x86_64-linux]
• Backport: 2.0.0: WONTFIX, 2.1: DONE, 2.2: DONE

[ngoto (Naohisa Goto)]

sparc Solaris 10 にて、r50887 以降、make test-all にて以下の3つのFailureが出るようになりました。

  1) Failure:
TestDir_M17N#test_filename_extutf8_inteucjp_unrepresentable [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:152]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

  2) Failure:
TestDir_M17N#test_filename_extutf8_inteucjp_representable [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:125]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

  3) Failure:
TestDir_M17N#test_filename_ext_euc_jp_and_int_utf_8 [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:246]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (EUC-JP to UTF-8) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

[ngoto (Naohisa Goto)]

• Related to Bug #11060: load(fifo) blocks whole process added

[ngoto (Naohisa Goto)]

• Subject changed from "code converter not found" Failure since r50887 on Solaris 10 to "code converter not found" Failure since r50887
• ruby -v changed from ruby 2.3.0dev (2015-06-10) [sparc64-solaris2.10] to ruby 2.3.0dev (2015-06-16) [sparc64-solaris2.10]

x86_64 Linux でも50%程度の発生率で再現できました。
r50920 にて確認しました。

$ echo a0 > /tmp/a0
$ echo a1 > /tmp/a1
$ ruby --disable=gems -EEUC-JP:UTF-8 -e 'Thread.new { File.read("/tmp/a0") }; Thread.new { File.read("/tmp/a1") }.join'
-e:1:in `read': code converter not found (EUC-JP to UTF-8) (Encoding::ConverterNotFoundError)
	from -e:1:in `block in <main>'

2スレッドが同時に同一のenc/transを要求するとダメなようです。

[ngoto (Naohisa Goto)]

• ruby -v changed from ruby 2.3.0dev (2015-06-16) [sparc64-solaris2.10] to ruby 2.3.0dev (2015-06-16) [x86_64-linux]

[ngoto (Naohisa Goto)]

Ruby 1.9.3 でも発生率は低くなりますが再現できました。
r50887 以降発生しやすくなっただけで、それ以前から問題があったようです。

[ngoto (Naohisa Goto)]

• Subject changed from "code converter not found" Failure since r50887 to "code converter not found" Failure (high occurrence rate since r50887)

[ngoto (Naohisa Goto)]

• Related to Feature #5654: Introduce global lock to avoid concurrent require added

[ngoto (Naohisa Goto)]

• Subject changed from "code converter not found" Failure (high occurrence rate since r50887) to "code converter not found" error with multi-thread (high occurrence rate since r50887)

たとえば #5654 のような何らかのロック機構が必要な気はしますが、
Encoding 関係は、事前にどのファイルにどのクラス・モジュールが格納されているかは完全に把握済、かつ、circular requireは絶対無い、という前提があるため、グローバルなロックまでは必要無く、クラス・モジュールオブジェクト単位の簡易なロックで十分かもしれません。

[ngoto (Naohisa Goto)]

• Related to Feature #5653: "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead) added

[ngoto (Naohisa Goto)]

エンコーディング関係は autoload は使わず自前でロード処理を実装しているようですが、関連として autoload の将来廃止の方向性を示している #5653 を挙げておきます。

[ngoto (Naohisa Goto)]

r50887 以降では、IOを使わなくても以下のようにすれば高確率で再現できました。

Converter が見つからない Error だけでなく、 Encoding のロードに失敗した Warning も出ているのがわかります。

% ruby --disable=gems -e '(0..2).collect { |_| Thread.new { p "\u3042".encode("EUC-JP") }}.each { |t| t.join }' 
-e:1: warning: failed to load encoding (EUC-JP); use ASCII-8BIT instead
-e:1:in `encode': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)
	from -e:1:in `block (2 levels) in <main>'

Linux と Solaris 両方で確認しています。

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset r51037.

---

transcode.c: fix race condition

• transcode.c (load_transcoder_entry): fix transcoder loading race
  condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN to 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: REQUIRED

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_2 at r51474.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: DONE to 2.0.0: WONTFIX, 2.1: DONE, 2.2: DONE

ruby_2_1 r51598 merged revision(s) 51037.