Project

General

Profile

Bug #11277

"code converter not found" error with multi-thread (high occurrence rate since r50887)

Added by ngoto (Naohisa Goto) almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.3.0dev (2015-06-16) [x86_64-linux]
[ruby-dev:49106]

Description

sparc Solaris 10 にて、r50887 以降、make test-all にて以下の3つのFailureが出るようになりました。

  1) Failure:
TestDir_M17N#test_filename_extutf8_inteucjp_unrepresentable [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:152]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

  2) Failure:
TestDir_M17N#test_filename_extutf8_inteucjp_representable [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:125]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

  3) Failure:
TestDir_M17N#test_filename_ext_euc_jp_and_int_utf_8 [/XXXXX-trunk-50887/test/ruby/test_dir_m17n.rb:246]:
assert_separately failed with error message.
<""> expected but was
<"/XXXXX-trunk-50887/test/lib/envutil.rb:72:in `read': code converter not found (EUC-JP to UTF-8) (Encoding::ConverterNotFoundError)\n\tfrom /XXXXX-trunk-50887/test/lib/envutil.rb:72:in `block in invoke_ruby'\n">.

Related issues

Related to Ruby trunk - Bug #11060: load(fifo) blocks whole processClosedActions
Related to Ruby trunk - Feature #5654: Introduce global lock to avoid concurrent requireAssignedActions
Related to Ruby trunk - Feature #5653: "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead)ClosedActions

Associated revisions

Revision 9a7e4bf9
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51037 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 51037
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Revision 51037
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Revision 51037
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Revision 51037
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Revision 51037
Added by nobu (Nobuyoshi Nakada) almost 4 years ago

transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Revision 43850e26
Added by nagachika (Tomoyuki Chikanaga) almost 4 years ago

merge revision(s) 51037: [Backport #11277]

    * transcode.c (load_transcoder_entry): fix transcoder loading race
      condition, by waiting in require.  [ruby-dev:49106] [Bug #11277]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_2@51474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 51474
Added by nagachika (Tomoyuki Chikanaga) almost 4 years ago

merge revision(s) 51037: [Backport #11277]

* transcode.c (load_transcoder_entry): fix transcoder loading race
  condition, by waiting in require.  [ruby-dev:49106] [Bug #11277]

Revision 7766666a
Added by usa (Usaku NAKAMURA) almost 4 years ago

merge revision(s) 51037: [Backport #11277]

    * transcode.c (load_transcoder_entry): fix transcoder loading race
      condition, by waiting in require.  [ruby-dev:49106] [Bug #11277]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_1@51598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 51598
Added by usa (Usaku NAKAMURA) almost 4 years ago

merge revision(s) 51037: [Backport #11277]

* transcode.c (load_transcoder_entry): fix transcoder loading race
  condition, by waiting in require.  [ruby-dev:49106] [Bug #11277]

History

#1

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Related to Bug #11060: load(fifo) blocks whole process added

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Subject changed from "code converter not found" Failure since r50887 on Solaris 10 to "code converter not found" Failure since r50887
  • ruby -v changed from ruby 2.3.0dev (2015-06-10) [sparc64-solaris2.10] to ruby 2.3.0dev (2015-06-16) [sparc64-solaris2.10]

x86_64 Linux でも50%程度の発生率で再現できました。
r50920 にて確認しました。

$ echo a0 > /tmp/a0
$ echo a1 > /tmp/a1
$ ruby --disable=gems -EEUC-JP:UTF-8 -e 'Thread.new { File.read("/tmp/a0") }; Thread.new { File.read("/tmp/a1") }.join'
-e:1:in `read': code converter not found (EUC-JP to UTF-8) (Encoding::ConverterNotFoundError)
    from -e:1:in `block in <main>'

2スレッドが同時に同一のenc/transを要求するとダメなようです。

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • ruby -v changed from ruby 2.3.0dev (2015-06-16) [sparc64-solaris2.10] to ruby 2.3.0dev (2015-06-16) [x86_64-linux]

Updated by ngoto (Naohisa Goto) almost 4 years ago

Ruby 1.9.3 でも発生率は低くなりますが再現できました。
r50887 以降発生しやすくなっただけで、それ以前から問題があったようです。

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Subject changed from "code converter not found" Failure since r50887 to "code converter not found" Failure (high occurrence rate since r50887)
#6

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Related to Feature #5654: Introduce global lock to avoid concurrent require added

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Subject changed from "code converter not found" Failure (high occurrence rate since r50887) to "code converter not found" error with multi-thread (high occurrence rate since r50887)

たとえば #5654 のような何らかのロック機構が必要な気はしますが、
Encoding 関係は、事前にどのファイルにどのクラス・モジュールが格納されているかは完全に把握済、かつ、circular requireは絶対無い、という前提があるため、グローバルなロックまでは必要無く、クラス・モジュールオブジェクト単位の簡易なロックで十分かもしれません。

#8

Updated by ngoto (Naohisa Goto) almost 4 years ago

  • Related to Feature #5653: "I strongly discourage the use of autoload in any standard libraries" (Re: autoload will be dead) added

Updated by ngoto (Naohisa Goto) almost 4 years ago

エンコーディング関係は autoload は使わず自前でロード処理を実装しているようですが、関連として autoload の将来廃止の方向性を示している #5653 を挙げておきます。

Updated by ngoto (Naohisa Goto) almost 4 years ago

r50887 以降では、IOを使わなくても以下のようにすれば高確率で再現できました。

Converter が見つからない Error だけでなく、 Encoding のロードに失敗した Warning も出ているのがわかります。

% ruby --disable=gems -e '(0..2).collect { |_| Thread.new { p "\u3042".encode("EUC-JP") }}.each { |t| t.join }' 
-e:1: warning: failed to load encoding (EUC-JP); use ASCII-8BIT instead
-e:1:in `encode': code converter not found (UTF-8 to EUC-JP) (Encoding::ConverterNotFoundError)
    from -e:1:in `block (2 levels) in <main>'

Linux と Solaris 両方で確認しています。

#11

Updated by nobu (Nobuyoshi Nakada) almost 4 years ago

  • Status changed from Open to Closed

Applied in changeset r51037.


transcode.c: fix race condition

  • transcode.c (load_transcoder_entry): fix transcoder loading race condition, by waiting in require. [ruby-dev:49106] [Bug #11277]

Updated by usa (Usaku NAKAMURA) almost 4 years ago

  • Backport changed from 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN to 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: REQUIRED

Updated by nagachika (Tomoyuki Chikanaga) almost 4 years ago

  • Backport changed from 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: REQUIRED to 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: DONE

Backported into ruby_2_2 at r51474.

#14

Updated by usa (Usaku NAKAMURA) almost 4 years ago

  • Backport changed from 2.0.0: WONTFIX, 2.1: REQUIRED, 2.2: DONE to 2.0.0: WONTFIX, 2.1: DONE, 2.2: DONE

ruby_2_1 r51598 merged revision(s) 51037.

Also available in: Atom PDF