https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112009-05-29T13:46:48ZRuby Issue Tracking SystemRuby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=41272009-05-29T13:46:48Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>=begin<br>
(1) Hash data structure changed between 1.8 and 1.9. It is difficult (and too drastic I think) to backport that.<br>
(2) I believe that calling Hash#merge inside a Hash#each block is already prohibited in 1.8. A detailed info on your "very specific conditions" can help us a lot.<br>
=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=41312009-05-30T02:05:50Zrunpaint (Run Paint Run Run)runrun@runpaint.org
<ul></ul><p>=begin</p>
<blockquote>
<p>I believe that calling Hash#merge inside a Hash#each block is already prohibited in 1.8.</p>
</blockquote>
<p>I'm afraid I do not understand. It is not prohibited in the sense it is made impossible, because the following code works as one would expect on 1.8.7:</p>
<pre><code> >> x={:glark => :quark}
=> {:glark=>:quark}
>> h = {:foo=>:bar}
=> {:foo=>:bar}
>> h.each { h = h.merge(x) }
=> {:foo=>:bar}
>> p h
{:foo=>:bar, :glark=>:quark}
=> nil
</code></pre>
<p>Or:</p>
<pre><code> >> h = {:foo=>:bar}
=> {:foo=>:bar}
>> x={:glark => :quark}
=> {:glark=>:quark}
>> h.each { h.merge!(x) }
=> {:foo=>:bar, :glark=>:quark}
</code></pre>
<p>It is not prohibited in the sense that it is described as inadvisable in the documentation, either.</p>
<blockquote>
<p>A detailed info on your "very specific conditions" can help us a lot.</p>
</blockquote>
<p>The information given in the original bug report was really all I could ascertain. I tried running under strace and a debugger, but couldn't crystallize the problem further. Maybe some worked examples under 1.8.7 IRB will illuminate the problem.</p>
<p>Using the example given in my original report:</p>
<blockquote>
<blockquote>
<p>hash = {1 => 2, 3 => 4, 5 => 6}<br>
=> {5=>6, 1=>2, 3=>4}<br>
big_hash = {}<br>
=> {}<br>
64.times { |k| big_hash[k.to_s] = k }<br>
=> 64<br>
hash.each { hash.merge!(big_hash) }<br>
RuntimeError: hash modified during iteration<br>
from (irb):4:in `each'<br>
from (irb):4</p>
</blockquote>
</blockquote>
<p>So, with those specific parameters, calling #merge! from with #each raises a RuntimeError. If this usage really was prohibited, that's what you would expect. However, if we modify the example so <em>hash</em> only has 4 elements the code runs as follows:</p>
<blockquote>
<blockquote>
<p>hash = {1 => 2, 3 => 4}<br>
=> {1=>2, 3=>4}<br>
big_hash = {}<br>
=> {}<br>
64.times { |k| big_hash[k.to_s] = k }<br>
=> 64<br>
hash.each { hash.merge!(big_hash) }<br>
=> {"6"=>6, "11"=>11, "22"=>22, "33"=>33, "44"=>44, "55"=>55, "7"=>7, "12"=>12, "23"=>23, "34"=>34, "45"=>45, "56"=>56, "8"=>8, "13"=>13, "24"=>24, "35"=>35, "46"=>46, "57"=>57, "9"=>9, "14"=>14, "25"=>25, "36"=>36, "47"=>47, "58"=>58, 1=>2, "15"=>15, "26"=>26, "37"=>37, "48"=>48, "59"=>59, "60"=>60, "0"=>0, "16"=>16, "27"=>27, "38"=>38, "49"=>49, "50"=>50, "61"=>61, "1"=>1, "17"=>17, "28"=>28, "39"=>39, "40"=>40, "51"=>51, "62"=>62, "2"=>2, "18"=>18, "29"=>29, "30"=>30, "41"=>41, "52"=>52, "63"=>63, 3=>4, "3"=>3, "19"=>19, "20"=>20, "31"=>31, "42"=>42, "53"=>53, "4"=>4, "10"=>10, "21"=>21, "32"=>32, "43"=>43, "54"=>54, "5"=>5}</p>
</blockquote>
</blockquote>
<p>Here, no RuntimeError was raised despite the only difference between the two examples being the size of <em>hash</em>.</p>
<p>Similarly, observe the result if we modify the original example by populating <em>big_hash</em> with only 63 entries, keeping everything else constant:</p>
<blockquote>
<blockquote>
<p>hash = {1 => 2, 3 => 4, 5 => 6}<br>
=> {5=>6, 1=>2, 3=>4}<br>
big_hash = {}<br>
=> {}<br>
63.times { |k| big_hash[k.to_s] = k }<br>
=> 63<br>
hash.each { hash.merge!(big_hash) }<br>
=> {"6"=>6, "11"=>11, "22"=>22, "33"=>33, "44"=>44, "55"=>55, 5=>6, "7"=>7, "12"=>12, "23"=>23, "34"=>34, "45"=>45, "56"=>56, "8"=>8, "13"=>13, "24"=>24, "35"=>35, "46"=>46, "57"=>57, "9"=>9, "14"=>14, "25"=>25, "36"=>36, "47"=>47, "58"=>58, 1=>2, "15"=>15, "26"=>26, "37"=>37, "48"=>48, "59"=>59, "60"=>60, "0"=>0, "16"=>16, "27"=>27, "38"=>38, "49"=>49, "50"=>50, "61"=>61, "1"=>1, "17"=>17, "28"=>28, "39"=>39, "40"=>40, "51"=>51, "62"=>62, "2"=>2, "18"=>18, "29"=>29, "30"=>30, "41"=>41, "52"=>52, 3=>4, "3"=>3, "19"=>19, "20"=>20, "31"=>31, "42"=>42, "53"=>53, "4"=>4, "10"=>10, "21"=>21, "32"=>32, "43"=>43, "54"=>54, "5"=>5}</p>
</blockquote>
</blockquote>
<p>Again, the code behaves as we expect. However, if we adjust the above example so that big_hash has 65 entries, the RuntimError is raised again:</p>
<blockquote>
<blockquote>
<p>hash = {1 => 2, 3 => 4, 5=>6}<br>
=> {5=>6, 1=>2, 3=>4}<br>
big_hash = {}<br>
=> {}<br>
65.times { |k| big_hash[k.to_s] = k }<br>
=> 65<br>
hash.each { hash.merge!(big_hash) }<br>
RuntimeError: hash modified during iteration<br>
from (irb):38:in `each'<br>
from (irb):38<br>
from :0</p>
</blockquote>
</blockquote>
<p>This is what I meant by "very specific conditions". I have no further insight or explanation into their cause. :-(</p>
<p>The bug, then, is that the RuntimeError is unexpected. Code that worked correctly one day, would start crashing another just because the dimensions of the data set changed. If Hash#merge! is generally dangerous inside iterator blocks, its usage should be prohibited consistently such that all of the above examples raise a RuntimeError, and the documentation must make this limitation clear. This is what happens when Hash#rehash is called inside of an iterator block: any and all usages of this combination raise a RuntimeError.</p>
<p>Thank you for looking into this, Shyouhei, I'm sorry I can't be of any more assistance.<br>
=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=41322009-05-30T10:20:28Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Assignee</strong> set to <i>shyouhei (Shyouhei Urabe)</i></li></ul><p>=begin<br>
Thank you. I'll take a closer look at the source code.<br>
=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=60042009-09-25T03:27:49Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>3</i></li><li><strong>Target version</strong> changed from <i>Ruby 1.8.7</i> to <i>Ruby 1.8.8</i></li></ul><p>=begin</p>
<p>=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=83202010-02-16T22:13:50Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>=begin<br>
Hi,</p>
<blockquote>
<p>hash = {1 => 2, 3 => 4, 5 => 6}<br>
big_hash = {}<br>
64.times { |k| big_hash[k.to_s] = k }<br>
hash.each { hash.merge!(big_hash) }</p>
<p>This raises a RuntimeError: "hash modified during iteration" on 1.8.6.368 and 1.8.7.72. It runs correctly on 1.9.1.129.</p>
</blockquote>
<p>It raises a RuntimeError on trunk. I guess it is by accident<br>
for the exception not to occur on 1.9.1.</p>
<p>By hashtable's nature, adding new keys to hash may cause rehash<br>
automatically, and the automatic rehash may cause the exception<br>
during iteration.</p>
<p>For compatibility reason, we cannot prohibit hash modification<br>
during iteration because there are many programs that do so,<br>
(e.g., rbconfig.rb), like this:</p>
<p>hash.each {|k, v| hash[k] = func(v) }</p>
<p>But I agree with Run Paint Run Run's opinion. It may lead to<br>
difficult bug to indeterminately fail to add a new key.</p>
<p>So, I propose to permit only updating value of existing key,<br>
and to always prohibit adding a new key:</p>
<p>hash = { 1=>2, 3=>4, 5=>6 }<br>
hash.each {|k, v| hash[k] = func(v) } #=> OK<br>
hash.each {|k, v| hash[k.to_s] = v } #=> always exception</p>
<p>This does not cause compatibility problem because this just<br>
raises exception that has already been occurred indeterminately.<br>
I'll commit the following patch to trunk unless anyone says an<br>
objection.</p>
<p>diff --git a/hash.c b/hash.c<br>
index d49d0ea..51537e9 100644<br>
--- a/hash.c<br>
+++ b/hash.c<br>
@@ -270,6 +270,14 @@ rb_hash_modify(VALUE hash)<br>
}</p>
<p>static void<br>
+hash_update(VALUE hash, VALUE key)<br>
+{</p>
<ul>
<li>if (RHASH(hash)->iter_lev > 0 && !st_lookup(RHASH(hash)->ntbl, key, 0)) {</li>
<li>rb_raise(rb_eRuntimeError, "can't add a new key into hash during iteration");</li>
<li>}<br>
+}</li>
<li>
</ul>
<p>+static void<br>
default_proc_arity_check(VALUE proc)<br>
{<br>
int n = rb_proc_arity(proc);<br>
@@ -1036,6 +1044,7 @@ VALUE<br>
rb_hash_aset(VALUE hash, VALUE key, VALUE val)<br>
{<br>
rb_hash_modify(hash);</p>
<ul>
<li>hash_update(hash, key);<br>
if (hash == key) {<br>
rb_raise(rb_eArgError, "recursive key for hash");<br>
}<br>
@@ -1630,6 +1639,7 @@ static int<br>
rb_hash_update_i(VALUE key, VALUE value, VALUE hash)<br>
{<br>
if (key == Qundef) return ST_CONTINUE;</li>
<li>hash_update(hash, key);<br>
st_insert(RHASH(hash)->ntbl, key, value);<br>
return ST_CONTINUE;<br>
}<br>
@@ -1641,6 +1651,7 @@ rb_hash_update_block_i(VALUE key, VALUE value, VALUE hash)<br>
if (rb_hash_has_key(hash, key)) {<br>
value = rb_yield_values(3, key, rb_hash_aref(hash, key), value);<br>
}</li>
<li>hash_update(hash, key);<br>
st_insert(RHASH(hash)->ntbl, key, value);<br>
return ST_CONTINUE;<br>
}</li>
</ul>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a><br>
=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=83212010-02-16T23:34:05Zmatz (Yukihiro Matsumoto)matz@ruby.or.jp
<ul></ul><p>=begin<br>
Hi,</p>
<p>In message "Re: <a href="https://blade.ruby-lang.org/ruby-core/28189">[ruby-core:28189]</a> [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Hash#merge! Inside Iterator Can Cause RuntimeError (Closed)" href="https://bugs.ruby-lang.org/issues/1535">#1535</a>] Hash#merge! Inside Iterator Can Cause RuntimeError"<br>
on Tue, 16 Feb 2010 22:13:51 +0900, Yusuke Endoh <a href="mailto:redmine@ruby-lang.org" class="email">redmine@ruby-lang.org</a> writes:</p>
<p>|So, I propose to permit only updating value of existing key,<br>
|and to always prohibit adding a new key:<br>
|<br>
| hash = { 1=>2, 3=>4, 5=>6 }<br>
| hash.each {|k, v| hash[k] = func(v) } #=> OK<br>
| hash.each {|k, v| hash[k.to_s] = v } #=> always exception<br>
|<br>
|This does not cause compatibility problem because this just<br>
|raises exception that has already been occurred indeterminately.<br>
|I'll commit the following patch to trunk unless anyone says an<br>
|objection.</p>
<p>I think it's a good idea. Go ahead.</p>
<pre><code> matz.
</code></pre>
<p>=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=83232010-02-17T02:43:58Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>=begin<br>
Hi,</p>
<p>2010/2/16 Yukihiro Matsumoto <a href="mailto:matz@ruby-lang.org" class="email">matz@ruby-lang.org</a>:</p>
<blockquote>
<p>|This does not cause compatibility problem because this just<br>
|raises exception that has already been occurred indeterminately.<br>
|I'll commit the following patch to trunk unless anyone says an<br>
|objection.</p>
<p>I think it's a good idea. Go ahead.</p>
</blockquote>
<p>Done.</p>
<p>Finally, rubyspec on trunk reports just one error now.</p>
<ol>
<li>
</ol>
<p>IO#reopen reassociates self with a new stream after some reads FAILED<br>
Expected "Line 3: Three\n"<br>
to equal "Line 1: One\n"</p>
<p>/home/mame/work/ruby/spec/rubyspec/core/io/reopen_spec.rb:125:in<br>
<code>block (2 levels) in <top (required)>' /home/mame/work/ruby/spec/rubyspec/core/io/reopen_spec.rb:4:in </code><top<br>
(required)>'</p>
<p>Finished in 115.282141 seconds</p>
<p>2884 files, 13880 examples, 170803 expectations, 1 failure, 0 errors</p>
<p>--<br>
Yusuke ENDOH <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p>
<p>=end</p> Ruby 1.8 - Bug #1535: Hash#merge! Inside Iterator Can Cause RuntimeErrorhttps://bugs.ruby-lang.org/issues/1535?journal_id=133652010-09-14T16:32:56Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>=begin</p>
<p>=end</p>