https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112019-09-04T00:50:34ZRuby Issue Tracking SystemRuby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=813792019-09-04T00:50:34Zvcsjones (Kevin Jones)
<ul></ul><p>Hello,</p>
<p>We started running in to this issue as well with Ruby 2.6.4. We are seeing consistent but non-deterministic string corruption in one of our applications. As an example, we see something like:</p>
<pre><code>undefined method `"defense_profile_extension_id\x00eq"'
</code></pre>
<p>Note the <code>\x00</code> in the string does not belong there, it should be a <code>_</code>. However these results are not consistent. Another run of the application with zero code changes produced this error:</p>
<pre><code>undefined method `"\x00\x00\x00\x00\x00\x00\x00\x00f_profile_extension_id_eq"
</code></pre>
<p>This string is being passed in to <code>public_send</code> elsewhere in another library, hence the "undefined method" error.</p>
<p>Unfortunately, we have yet to be able to create a small reproduction of this issue. We can make it happen consistently in a very large code base though. I did manage to run <code>git bisect</code> on a locally compiled ruby and determined that the change that introduced is <a href="https://github.com/ruby/ruby/commit/da36d5700d9e0e66411d93595b6f654c85853fa1" class="external">https://github.com/ruby/ruby/commit/da36d5700d9e0e66411d93595b6f654c85853fa1</a>.</p>
<p>When we revert that change, the issue can no longer be reproduced.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814102019-09-06T00:16:51Zalanwu (Alan Wu)
<ul></ul><p>I wonder if these bugs exist on master or are specific to 2.6.4.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814152019-09-06T07:26:09Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/15916">Bug #15916</a>: Memory leak in Regexp literal interpolation</i> added</li></ul> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814162019-09-06T07:26:45Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Backport</strong> changed from <i>2.5: UNKNOWN, 2.6: UNKNOWN</i> to <i>2.5: DONTNEED, 2.6: REQUIRED</i></li></ul> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814182019-09-06T10:40:23Zdeivid (David Rodríguez)
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/27421">@vcsjones (Kevin Jones)</a> You're right, my raw guess at the culprit was wrong since I just got the issue again with the ruby I built excluding the commits I mentioned. I'm going to build the same ruby you built now to confirm that the culprit is <a href="https://github.com/ruby/ruby/commit/da36d5700d9e0e66411d93595b6f654c85853fa1" class="external">https://github.com/ruby/ruby/commit/da36d5700d9e0e66411d93595b6f654c85853fa1</a> from my side too.</p>
<p>If this is the culprit, I'm guessing this is happening on master too, but I'll run my tests against master as well to confirm.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814192019-09-06T12:58:34Zhsbt (Hiroshi SHIBATA)hsbt@ruby-lang.org
<ul></ul><p>I faced the same error maybe.</p>
<p>Our company blog generated by middleman was failed with Ruby 2.6.4 and 2.7.0 built by master branch. This issue was completely reproduced with them. We could build with Ruby 2.5.6.</p>
<p>I will investigate the root causes of this issue.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814222019-09-06T14:35:41Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="Fix a use-after-free bug by avoiding rb_str_new_frozen `str2 = rb_str_new_frozen(str1)` seems to..." href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/ade1283ca276f7d589ffd3539fbc7b9817f682d5">git|ade1283ca276f7d589ffd3539fbc7b9817f682d5</a>.</p>
<hr>
<p>Fix a use-after-free bug by avoiding rb_str_new_frozen</p>
<p><code>str2 = rb_str_new_frozen(str1)</code> seems to make str1 a shared string that<br>
refers to str2, but str2 is not marked as STR_IS_SHARED_M nor<br>
STR_NOFREE.<br>
<code>rb_fstring(str2)</code> frees str2's ptr because it is not marked, and the<br>
free'ed pointer is the same as str1's ptr.<br>
After that, accessing str1 may cause use-after-free memory corruption.</p>
<p>I guess this is a bug of rb_str_new_frozen, but I'm completely unsure<br>
what it should be; the string states and flags are not documented.<br>
So, this is a workaround for [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: String corruption in 2.6.4 (Closed)" href="https://bugs.ruby-lang.org/issues/16136">#16136</a>]. I confirmed that rspec of<br>
activeadmin runs gracefully.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814232019-09-06T14:42:48Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>I've committed ade1283ca276f7d589ffd3539fbc7b9817f682d5, which is a workaround for the issue. I confirm that activeadmin test runs successfully with this patch.</p>
<p>I think that this issue is related to the fstring mechanism, but I have no idea what function is wrong. The internal state of a String object is too complicated and undocumented, so I cannot see the design, representation, and invariants. The fstring mechanism is too difficult for human (except <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/724">@normalperson (Eric Wong)</a>).</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814242019-09-06T15:13:11Zvcsjones (Kevin Jones)
<ul></ul><p>I applied the patch to ruby 2.6.4 and I cannot reproduce the issue any more. This patch appears to resolve the issue for 2.6.4. Thank you very much for taking the time to investigate the issue.</p>
<pre><code class="diff syntaxhl" data-language="diff"><span class="gh">diff --git a/symbol.c b/symbol.c
index f3506a02dd..a408ee052e 100644
</span><span class="gd">--- a/symbol.c
</span><span class="gi">+++ b/symbol.c
</span><span class="p">@@ -743,7 +743,8 @@</span> rb_str_intern(VALUE str)
enc = ascii;
}
else {
<span class="gd">- str = rb_str_new_frozen(str);
</span><span class="gi">+ str = rb_str_dup(str);
+ OBJ_FREEZE(str);
</span> }
str = rb_fstring(str);
type = rb_str_symname_type(str, IDSET_ATTRSET_FOR_INTERN);
</code></pre> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814252019-09-06T15:50:20Zdeivid (David Rodríguez)
<ul></ul><p>Wow! Thanks so much for providing this solution in such a timely manner! <3</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814352019-09-07T04:48:36Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/27421">@vcsjones (Kevin Jones)</a> Good to hear! Thank you for confirming. And your investigation by bisect was very helpful.<br>
<a class="user active user-mention" href="https://bugs.ruby-lang.org/users/7174">@deivid (David Rodríguez)</a> The most difficult part for me was to read .circleci/config.yml to reproduce the issue on my machine :-)</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814372019-09-07T06:53:28Zdeivid (David Rodríguez)
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a> I'm sorry you had to do extra work to figure that out! I'll post proper reproduction steps next time :)</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814462019-09-07T11:25:50Zgraywolf (Gray Wolf)
<ul></ul><p>Out of curiosity, why is this issue closed if it was not yet backported to 2.6 branch?</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814512019-09-07T13:43:32Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>graywolf (Gray Wolf) wrote:</p>
<blockquote>
<p>Out of curiosity, why is this issue closed if it was not yet backported to 2.6 branch?</p>
</blockquote>
<p>Don't worry. The branch maintainers, <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/404">@nagachika (Tomoyuki Chikanaga)</a> and <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/9">@usa (Usaku NAKAMURA)</a>, are going to check all tickets whose "Backport" field is "REQUIRED", regardless of their status.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=814732019-09-09T01:12:02Zhsbt (Hiroshi SHIBATA)hsbt@ruby-lang.org
<ul></ul><p>Our blog system was fixed at <a class="changeset" title="Fix a use-after-free bug by avoiding rb_str_new_frozen `str2 = rb_str_new_frozen(str1)` seems to..." href="https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/ade1283ca276f7d589ffd3539fbc7b9817f682d5">ade1283ca276f7d589ffd3539fbc7b9817f682d5</a></p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=815492019-09-14T02:25:04Znagachika (Tomoyuki Chikanaga)nagachika00@gmail.com
<ul><li><strong>Backport</strong> changed from <i>2.5: DONTNEED, 2.6: REQUIRED</i> to <i>2.5: DONTNEED, 2.6: DONE</i></li></ul><p>ruby_2_6 r67803 merged revision(s) ade1283ca276f7d589ffd3539fbc7b9817f682d5.</p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=818082019-10-01T13:50:45Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>FYI: Ruby 2.6.5 has been released, which includes the fix for this issue.</p>
<p><a href="https://www.ruby-lang.org/en/news/2019/10/01/ruby-2-6-5-released/" class="external">https://www.ruby-lang.org/en/news/2019/10/01/ruby-2-6-5-released/</a></p> Ruby master - Bug #16136: String corruption in 2.6.4https://bugs.ruby-lang.org/issues/16136?journal_id=818112019-10-01T19:31:17Zdeivid (David Rodríguez)
<ul></ul><p>Thanks so much! <3</p>