https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112015-07-31T04:59:06ZRuby Issue Tracking SystemRuby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=536212015-07-31T04:59:06Zthedarkone (Vit Z)thedarkone2@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/5428">0001-load.c-unlock-the-new-shield.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/5428/0001-load.c-unlock-the-new-shield.patch">0001-load.c-unlock-the-new-shield.patch</a> added</li></ul><p>That broken rubyspec was written by me. The problem lies with repeatedly <code>autoload</code>ing the same <code>.rb</code> file, since this should be impossible, the spec manually deletes the loaded path from <code>$LOADED_FEATURES</code> and then re-declares the <code>autoload</code>, this is currently broken on MRI.</p>
<p>Here's a much smaller repro script:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">with_autoload_file</span><span class="p">(</span><span class="n">const_name</span><span class="p">,</span> <span class="n">file_name</span> <span class="o">=</span> <span class="s1">'foo.rb'</span><span class="p">)</span>
<span class="n">mangled_file_name</span> <span class="o">=</span> <span class="n">file_name</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sr">/\.rb\Z/</span><span class="p">,</span> <span class="s1">'____temp____autoload.rb'</span><span class="p">)</span> <span class="c1"># avoid accidentally overwriting any files</span>
<span class="no">File</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">mangled_file_name</span><span class="p">,</span> <span class="s2">"sleep 1; module </span><span class="si">#{</span><span class="n">const_name</span><span class="si">}</span><span class="s2">; end"</span><span class="p">)</span>
<span class="nb">autoload</span> <span class="n">const_name</span><span class="p">,</span> <span class="no">File</span><span class="p">.</span><span class="nf">expand_path</span><span class="p">(</span><span class="n">mangled_file_name</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sr">/\.rb\Z/</span><span class="p">,</span> <span class="s1">''</span><span class="p">))</span>
<span class="vg">$LOADED_FEATURES</span><span class="p">.</span><span class="nf">delete</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">expand_path</span><span class="p">(</span><span class="n">mangled_file_name</span><span class="p">))</span> <span class="k">if</span> <span class="vg">$LOADED_FEATURES</span><span class="p">.</span><span class="nf">include?</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">expand_path</span><span class="p">(</span><span class="n">mangled_file_name</span><span class="p">))</span>
<span class="k">yield</span>
<span class="k">ensure</span>
<span class="no">File</span><span class="p">.</span><span class="nf">delete</span><span class="p">(</span><span class="n">mangled_file_name</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">foo_ready</span> <span class="o">=</span> <span class="n">bar_waiting</span> <span class="o">=</span> <span class="n">bar_ready</span> <span class="o">=</span> <span class="kp">false</span>
<span class="n">t</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">pass</span> <span class="k">until</span> <span class="n">foo_ready</span>
<span class="no">Foo</span>
<span class="n">bar_waiting</span> <span class="o">=</span> <span class="kp">true</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">pass</span> <span class="k">until</span> <span class="n">bar_ready</span>
<span class="no">Bar</span>
<span class="k">end</span>
<span class="n">with_autoload_file</span><span class="p">(</span><span class="s1">'Foo'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">foo_ready</span> <span class="o">=</span> <span class="kp">true</span>
<span class="no">Foo</span>
<span class="k">end</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">pass</span> <span class="k">until</span> <span class="n">bar_waiting</span>
<span class="n">with_autoload_file</span><span class="p">(</span><span class="s1">'Bar'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">bar_ready</span> <span class="o">=</span> <span class="kp">true</span>
<span class="no">Bar</span>
<span class="k">end</span>
<span class="n">t</span><span class="p">.</span><span class="nf">join</span>
</code></pre>
<p>Running this results in an "<code>uninitialized constant Bar</code>" exception from the non-main thread.</p>
<p>If the last block is rearranged like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">with_autoload_file</span><span class="p">(</span><span class="s1">'Bar'</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Bar</span>
<span class="n">bar_ready</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
</code></pre>
<p>the script deadlocks (main thread deadlocks, while secondary thread <code>t</code> busy spins in <code>Thread.pass until bar_ready</code>).</p>
<p>If the last <code>autoload</code> block uses a different <code>.rb</code> file, everything works fine:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">with_autoload_file</span><span class="p">(</span><span class="s1">'Bar'</span><span class="p">,</span> <span class="s1">'bar.rb'</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Bar</span>
<span class="n">bar_ready</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
</code></pre>
<p>I think I've tracked the issue to an incorrectly locked <code>load_lock</code>'s <code>thread_shield</code>: when <code>rb_thread_shield_wait()</code> returns <code>Qfalse</code> the failed thread creates a new <code>thread_shield</code> via <code>rb_thread_shield_new()</code>, however because <code>rb_thread_shield_new()</code> automatically locks the newly created shield and the branch does not return a successful <code>ftptr</code>, the newly installed shield is then never unlocked.</p>
<p>The attached patch seems to fix the issue for me.</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=536222015-07-31T05:59:50Znobu (Nobuyoshi Nakada)nobu@ruby-lang.org
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/7530">Bug #7530</a>: Concurrent loads fail with mutex errors</i> added</li></ul> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=544302015-10-12T13:22:45ZEregon (Benoit Daloze)
<ul></ul><p>Could someone review the patch and apply it or find an alternative fix?</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=544562015-10-14T19:58:48Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p><a href="mailto:eregontp@gmail.com" class="email">eregontp@gmail.com</a> wrote:</p>
<blockquote>
<p>Could someone review the patch and apply it or find an alternative fix?</p>
</blockquote>
<p>Fwiw, I mentioned in <a href="https://blade.ruby-lang.org/ruby-core/70359">[ruby-core:70359]</a> that I tried it for [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a>]<br>
without success, but Redmine + list integration was broken at the<br>
time.</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=544582015-10-15T11:48:45ZEregon (Benoit Daloze)
<ul></ul><p>On Wed, Oct 14, 2015 at 9:56 PM, Eric Wong <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>Fwiw, I mentioned in <a href="https://blade.ruby-lang.org/ruby-core/70359">[ruby-core:70359]</a> that I tried it for [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a>]<br>
without success, but Redmine + list integration was broken at the<br>
time.</p>
</blockquote>
<p>Ah indeed I missed that, thanks.<br>
Did you try for this issue in particular?</p>
<p>About <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a>, I guess we need another fix then :/</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=544602015-10-16T01:58:29Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>Benoit Daloze <a href="mailto:eregontp@gmail.com" class="email">eregontp@gmail.com</a> wrote:</p>
<blockquote>
<p>On Wed, Oct 14, 2015 at 9:56 PM, Eric Wong <a href="mailto:normalperson@yhbt.net" class="email">normalperson@yhbt.net</a> wrote:</p>
<blockquote>
<p>Fwiw, I mentioned in <a href="https://blade.ruby-lang.org/ruby-core/70359">[ruby-core:70359]</a> that I tried it for [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a>]<br>
without success, but Redmine + list integration was broken at the<br>
time.</p>
</blockquote>
<p>Ah indeed I missed that, thanks.<br>
Did you try for this issue in particular?</p>
</blockquote>
<p>Ah, yes, the repro script in <a href="/issues/10892">[ruby-core:70197]</a> does get fixed on my<br>
machine with the patch. I don't understand this code enough to<br>
know if it breaks anything else, or if <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a> is a different bug<br>
or a different manifestation of the same bug.</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=718982018-05-08T01:27:18Zeugeneius (Eugene Kenny)
<ul></ul><p>The simpler repro script runs successfully from 2.3.0 onwards, and <code>git bisect</code> between 2.2.0 and 2.3.0 shows that r59221 (from <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: multi-threaded autoload sometimes fails (Closed)" href="https://bugs.ruby-lang.org/issues/11384">#11384</a>) fixed it.</p> Ruby master - Bug #10892: Deadlock in autoloadhttps://bugs.ruby-lang.org/issues/10892?journal_id=791922019-07-08T01:03:47Zjeremyevans0 (Jeremy Evans)merch-redmine@jeremyevans.net
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul>