https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112020-09-03T18:42:51ZRuby Issue Tracking SystemRuby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874152020-09-03T18:42:51Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-6 priority-4 priority-default closed" href="/issues/2509">Feature #2509</a>: Recursive freezing?</i> added</li></ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874192020-09-03T19:00:25ZEregon (Benoit Daloze)
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/17100">Feature #17100</a>: Ractor: a proposal for a new concurrent abstraction without thread-safety issues</i> added</li></ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874202020-09-03T19:06:20ZEregon (Benoit Daloze)
<ul></ul><p>A dynamic call to <code>freeze</code> causes extra calls, and needs checks that it was indeed frozen.<br>
So for efficiency I think it would be better to mark as frozen internally without a call to <code>freeze</code> on every value.</p>
<p>Also a leaf <code>freeze</code> call could technically mutate the object referencing it and e.g., add a new @ivar, which would make it complicated to ensure everything is frozen<br>
(would need to mark as shallow-frozen first to prevent that, and only as deep-frozen once all contained values are deeply-frozen).</p>
<p>Is there a compelling reason to call a user-defined <code>freeze</code> for every value?</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874212020-09-03T19:20:55Zko1 (Koichi Sasada)
<ul></ul><p>One concern about the name "freeze" is, what happens on shareable objects on Ractors.<br>
For example, Ractor objects are shareable and they don't need to freeze to send beyond Ractor boundary.</p>
<p>I also want to introduce Mutable but shareable objects using STM (or something similar) writing protocol (shareable Hash). What happens on <code>deep_freeze</code>?</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874222020-09-03T19:27:04ZEregon (Benoit Daloze)
<ul></ul><p>Maybe we should have a method that ensure an object graph is shareable?<br>
Not sure what a good name for that would be (shareable is hard to spell).<br>
So that would noop for special shareable objects like you say, and freeze the rest.<br>
Note that a shareable Hash or so would need to only allow shareable key/value/elements whenever writing to it.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874242020-09-03T19:34:33Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Eregon (Benoit Daloze) wrote in <a href="#note-3">#note-3</a>:</p>
<blockquote>
<p>A dynamic call to <code>freeze</code> causes extra calls</p>
</blockquote>
<p>Yes</p>
<blockquote>
<p>and needs checks that it was indeed frozen.</p>
</blockquote>
<p>That won't add any noticeable overhead</p>
<blockquote>
<p>Is there a compelling reason to call a user-defined <code>freeze</code> for every value?</p>
</blockquote>
<p>Yes. Some objects may need to calculate lazy operations (<code>@cache ||= potentially_long_calculation</code>)</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874252020-09-03T19:39:19Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-4">#note-4</a>:</p>
<blockquote>
<p>One concern about the name "freeze" is, what happens on shareable objects on Ractors.<br>
For example, Ractor objects are shareable and they don't need to freeze to send beyond Ractor boundary.</p>
<p>I also want to introduce Mutable but shareable objects using STM (or something similar) writing protocol (shareable Hash). What happens on <code>deep_freeze</code>?</p>
</blockquote>
<p>I think these objects should stop the propagation. The name <code>make_shareable_via_ractor_by_deep_freezing_what_is_necessary</code> would be more accurate but too long π Maybe <code>ractorable</code>?</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874322020-09-03T20:44:02Zko1 (Koichi Sasada)
<ul></ul><blockquote>
<p>I think these objects should stop the propagation. The name make_shareable_via_ractor_by_deep_freezing_what_is_necessary would be more accurate but too long π<br>
Maybe ractorable?</p>
</blockquote>
<p>For Ractor, it is very clear. Another candidate is <code>to_shareable</code>?</p>
<p>However, leave from Ractor's perspective, it is strange name, like:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">CONST</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">].</span><span class="nf">to_shareable</span>
</code></pre>
<p>Maybe the author don't want to care about Ractor.<br>
The author want to declare "I don't touch it". So "deep_freeze" is better.</p>
<p>hmmm.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=874432020-09-04T02:55:49Zduerst (Martin DΓΌrst)duerst@it.aoyama.ac.jp
<ul></ul><p>marcandre (Marc-Andre Lafortune) wrote in <a href="#note-7">#note-7</a>:</p>
<blockquote>
<p>I think these objects should stop the propagation. The name <code>make_shareable_via_ractor_by_deep_freezing_what_is_necessary</code> would be more accurate but too long π Maybe <code>ractorable</code>?</p>
</blockquote>
<p>What about <code>ractor_freeze</code>?</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=876062020-09-20T17:57:19ZEregon (Benoit Daloze)
<ul></ul><p>Maybe <code>obj.shareable</code>? And that would then return as-is if already shareable, and make it shareable if not (deep freezing until it reaches already-shareable objects)</p>
<p>I don't like anything with "ractor" in the name, that becomes not descriptive of what it does and IMHO looks weird for e.g. gems not specifically caring about Ractor.</p>
<p>How about first having <code>deep_freeze</code> that just freezes everything (except an object's class)?</p>
<p>And then maybe an extra method to deal with already-shareable (if needed), which is probably much less frequently needed.<br>
(e.g. it seems rather uncommon that a module constant contains a Ractor reference)</p>
<p>The intention of <code>deep_freeze</code> is I think clear: make this object and whatever it refers to immutable, so it can be shared freely via reference without any copying.<br>
And it applies to many other things besides just Ractor, so it seems a good functionality to have in general.<br>
For instance, <a href="https://github.com/dkubb/ice_nine" class="external">https://github.com/dkubb/ice_nine</a> already exists, can define <code>#deep_freeze</code> and is <a href="https://github.com/dkubb/ice_nine/network/dependents" class="external">widely used</a>.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=876112020-09-21T13:50:07Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-8">#note-8</a>:</p>
<blockquote>
<p>For Ractor, it is very clear. Another candidate is <code>to_shareable</code></p>
</blockquote>
<p>Methods <code>to_...</code> should be reserved for conversion methods, i.e. methods that may return a different object than the receiver. What I am proposing would always return the receiver. <code>to_shareable</code> would only be an acceptable name if it returned a deeply frozen copy, but I don't think that's what we need most.</p>
<p>Eregon (Benoit Daloze) wrote in <a href="#note-10">#note-10</a>:</p>
<blockquote>
<p>Maybe <code>obj.shareable</code>?</p>
</blockquote>
<p><code>shareable</code> is good. It's more accurate than <code>deep_freeze</code> if we think of ractor sharable structures.</p>
<p>Another alternative would be <code>freeze(shareable: true)</code>. It has the advantage of no possible conflict. It could also raise if it fails to produce a shareable object, which should be rare enough to warrant an exception. Something like <code>shareable: :try</code> could provide a non-raising alternative.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=878642020-10-02T08:44:02Zkirs (Kir Shatrov)shatrov@me.com
<ul></ul><p>I'd really like to support this change for reasons I've describe in <a href="https://bugs.ruby-lang.org/issues/17180" class="external">https://bugs.ruby-lang.org/issues/17180</a></p>
<p>We can take something as simple as <code>URI::parse("http://example.com")</code> as an example. Right now that can't be used from a Ractor because <code>URI::parse</code> refers to a few internal constants like <code>URI::RFC3986_PARSER</code> that are not frozen (and thus not sharable).</p>
<p>If we went with an explicit <code>deep_freeze</code> we might need to update all those constants like <code>URI::RFC3986_PARSER</code> in the standard library to be also call <code>deep_freeze</code>. That doesn't have to be a bad thing but something to be aware of.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=878682020-10-02T16:10:24Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Maybe we can start with <code>Ractor.shareable(obj)</code>?</p>
<p>The major reason why we need it is that there is no way to create a Ractor sharable recursive structure currently.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">child</span> <span class="o">=</span> <span class="p">{</span><span class="ss">type: :foo</span><span class="p">}</span>
<span class="n">node</span> <span class="o">=</span> <span class="p">{</span><span class="ss">type: </span><span class="n">bar</span><span class="p">,</span> <span class="ss">children: </span><span class="p">[</span><span class="n">child</span><span class="p">].</span><span class="nf">freeze</span><span class="p">}.</span><span class="nf">freeze</span>
<span class="n">child</span><span class="p">[</span><span class="ss">:parent</span><span class="p">]</span> <span class="o">=</span> <span class="n">node</span>
<span class="n">child</span><span class="p">.</span><span class="nf">freeze</span>
</code></pre>
<p>At this point, everything is frozen, but neither child nor node are Ractor sharable.</p>
<p><code>Ractor.shareable(node)</code> would check for this, and set both <code>node</code> and <code>child</code>'s "shareable" flag; after that <code>Ractor.shareable?(node)</code> would return true.</p>
<p>API-wise, <code>Ractor.shareable(node)</code> might be a good start, or <code>node.freeze(shareable: true)</code>, or <code>node.deep_freeze</code></p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880472020-10-19T20:48:02Zko1 (Koichi Sasada)
<ul></ul><p>I implemented <code>Object#deep_freeze(skip_shareable: false)</code> for trial.<br>
<a href="https://github.com/ko1/ruby/pull/new/deep_freeze" class="external">https://github.com/ko1/ruby/pull/new/deep_freeze</a></p>
<ul>
<li>It doesn't call <code>#freeze</code>, but only set frozen flag.</li>
<li>It set frozen bit for any objects such as Binding, Thread, Queue and so on, but no effect except adding ivars, etc.</li>
<li>Two pass freezing method
<ul>
<li>(1) collect reachable objects</li>
<li>(2) freeze collected objects</li>
</ul>
</li>
</ul>
<p>This two pass strategy allows errors during collection phase (1). For example, if we introduce unable to "freeze" objects, we can stop collection and raise some error without any side-effect. However, if we call <code>#freeze</code>, we can't cancel intermediate state (some objects are frozen, some are not). I have no idea how to solve it, if we need to call <code>freeze</code> for each instance.</p>
<p><code>Object#deep_freeze(skip_shareable: true)</code> skips shareable objects to freeze. Maybe it is <code>ractor_freeze</code>, <code>shareable</code> and so on. I don't have strong opinion about this naming, but I try to unify with <code>Object#deep_freeze</code>.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880482020-10-19T20:53:12Zko1 (Koichi Sasada)
<ul></ul><p>Maybe we need to introduce new protocol for <code>T_DATA</code>.</p>
<p>For example...:</p>
<ul>
<li>
<code>Time</code> should be frozen.</li>
<li>
<code>Thread</code>, <code>Fiber</code> should not be frozen. At least they can't become shareable objects.</li>
<li>
<code>Proc</code> and <code>Binding</code> objects are also not frozen (error on lvar assignment for frozen?)</li>
<li>... maybe many others ...</li>
</ul>
<p>To make conservative, all of them are unable to freeze, or allow freezing but not shareable objects.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880492020-10-19T21:24:48Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>I implemented <code>Object#deep_freeze(skip_shareable: false)</code> for trial.<br>
<a href="https://github.com/ko1/ruby/pull/new/deep_freeze" class="external">https://github.com/ko1/ruby/pull/new/deep_freeze</a></p>
<ul>
<li>It doesn't call <code>#freeze</code>, but only set frozen flag.</li>
<li>It set frozen bit for any objects such as Binding, Thread, Queue and so on, but no effect except adding ivars, etc.</li>
<li>Two pass freezing method
<ul>
<li>(1) collect reachable objects</li>
<li>(2) freeze collected objects</li>
</ul>
</li>
</ul>
<p>This two pass strategy allows errors during collection phase (1). For example, if we introduce unable to "freeze" objects, we can stop collection and raise some error without any side-effect. However, if we call <code>#freeze</code>, we can't cancel intermediate state (some objects are frozen, some are not). I have no idea how to solve it, if we need to call <code>freeze</code> for each instance.</p>
<p><code>Object#deep_freeze(skip_shareable: true)</code> skips shareable objects to freeze. Maybe it is <code>ractor_freeze</code>, <code>shareable</code> and so on. I don't have strong opinion about this naming, but I try to unify with <code>Object#deep_freeze</code>.</p>
</blockquote>
<p>This is great! πͺ</p>
<p>Two major issues remain:</p>
<ol>
<li>
<p>This does not work for recursive structures, in the sense that they will not be marked as Ractor shareable. This is capital, particularly since it is the (only) part of this method that can not be done in pure Ruby.</p>
</li>
<li>
<p>There is no callback mechanism for user classes (since this does not call <code>#freeze</code>). Typical use case would be for expensive calculations that are done lazily. A class may want to pre-build them before caching. There should be some callback mechanism. My opinion is that we should worry about things working well for well-programmed cases <em>before</em> we worry about exceptions (e.g. cases that fail do deeply freeze).<br>
I think that freezing a class "behind its back" by not calling <code>#freeze</code> breaks the contract and could be a source of incompatibility.<br>
I see no incompatibility with the two pass system and calling <code>#freeze</code> on the list of reachable objects. I have no issue with a structure being half frozen if things fail; the developper was ready to deep freeze all of it, so having only half frozen , even though it clearly was not the intention, does not feel like a big issue.</p>
</li>
</ol> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880562020-10-20T00:43:16Zko1 (Koichi Sasada)
<ul></ul><p>marcandre (Marc-Andre Lafortune) wrote in <a href="#note-16">#note-16</a>:</p>
<blockquote>
<ol>
<li>This does not work for recursive structures, in the sense that they will not be marked as Ractor shareable. This is capital, particularly since it is the (only) part of this method that can not be done in pure Ruby.</li>
</ol>
</blockquote>
<p>Current implementation doesn't mark as shareable, but we can make it. The problem is what happens on half-frozen case... Half-shareable is not acceptable. If half-frozen is acceptable, freezing with traversing, and mark as shareable at second phase is possible.</p>
<blockquote>
<ol start="2">
<li>There is no callback mechanism for user classes (since this does not call <code>#freeze</code>). Typical use case would be for expensive calculations that are done lazily. A class may want to pre-build them before caching. There should be some callback mechanism. My opinion is that we should worry about things working well for well-programmed cases <em>before</em> we worry about exceptions (e.g. cases that fail do deeply freeze).<br>
I think that freezing a class "behind its back" by not calling <code>#freeze</code> breaks the contract and could be a source of incompatibility.<br>
I see no incompatibility with the two pass system and calling <code>#freeze</code> on the list of reachable objects. I have no issue with a structure being half frozen if things fail; the developper was ready to deep freeze all of it, so having only half frozen , even though it clearly was not the intention, does not feel like a big issue.</li>
</ol>
</blockquote>
<p><code>$ gem-codesearch 'def freeze' | gist -p</code> => <a href="https://gist.github.com/ko1/63b8d43218884249e782d63c2f27b927" class="external">https://gist.github.com/ko1/63b8d43218884249e782d63c2f27b927</a></p>
<p>I never realized that so many <code>freeze</code> redefinition are used. Checking the code, some of them freeze attribute objects, which they are frozen with <code>deep_freeze</code>. I can find some cases to calculate lazy (?).</p>
<p>mmm. Maybe this method is used with literals, so there is a chance to optimize for such cases.</p>
<p>I still not sure we can remain the half-frozen state, so I want to ask other comments.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880632020-10-20T10:00:24Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-17">#note-17</a>:</p>
<blockquote>
<p>I never realized that so many <code>freeze</code> redefinition are used. Checking the code, some of them freeze attribute objects, which they are frozen with <code>deep_freeze</code>. I can find some cases to calculate lazy (?).</p>
</blockquote>
<p>Indeed, I imagine that the majority are used to do a deep-freeze, so calling <code>#freeze</code> or not will not make a difference.</p>
<p>An example of lazy computation I can remember writing is a <a href="https://github.com/deep-cover/deep-cover/blob/master/core_gem/lib/deep_cover/memoize.rb" class="external">generic memoization module in DeepCover</a> which is then used <a href="https://github.com/deep-cover/deep-cover/blob/master/core_gem/lib/deep_cover/coverage/analysis.rb#L6-L7" class="external">in different places</a>. Yes, I'm lazy enough that I factorized the <code>@cache ||= </code> pattern π</p>
<blockquote>
<p>I still not sure we can remain the half-frozen state, so I want to ask other comments.</p>
</blockquote>
<p>How about 3 pass?</p>
<ol>
<li>Collect reachable objects; abort on non-freezable.</li>
<li>Call <code>#freeze</code> on reachable objects. Exception due to custom method failing => half frozen state (just write better code π )</li>
<li>On success, mark all objects as Ractor shareable.</li>
</ol> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880712020-10-20T20:23:27ZEregon (Benoit Daloze)
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>I implemented <code>Object#deep_freeze(skip_shareable: false)</code> for trial.</p>
</blockquote>
<p>This sounds good to me.</p>
<p>IMHO if something fails in the middle, it's OK for part of it to be frozen.<br>
The intention was to freeze the whole object graph anyway.<br>
The code using the object graph after <code>deep_freeze</code> assumes it was frozen, there shouldn't be any issue if some of it is frozen, the code expected that.<br>
(also such code would only run if the exception is caught)</p>
<p>In some languages with unification (I know of Oz), it's also accepted that if it fails in the middle, then the partial state is still there and not magically undone.</p>
<p>I don't think we should call user methods, that will require a lot more scanning (before & after calling user <code>freeze</code>), and if some user<code>freeze</code> would create new objects and attach them to the object graph, <code>deep_freeze</code> could never end.<br>
Is there a concrete case where it is problematic to not call user's <code>freeze</code>?</p>
<p>I think the design should allow to do everything in a single pass for efficiency, and notably use the "deep frozen/shareable" bit as a "already visited object" marker.<br>
This is also how I implemented "Deep Sharing" during my PhD in <a href="https://eregon.me/blog/assets/research/thread-safe-objects.pdf" class="external">https://eregon.me/blog/assets/research/thread-safe-objects.pdf</a> section 5.3, and I think we should design <code>deep_freeze</code> to allow for the same optimization to be used because it's a large performance gain compared to a generic routine.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880732020-10-20T20:42:47ZEregon (Benoit Daloze)
<ul></ul><p>A draft to make this in a single pass (but it's late, I might have missed something):</p>
<ul>
<li>
<p>if the object is already deeply-frozen/shareable, return, otherwise:</p>
</li>
<li>
<p>mark an object as frozen if not already (so the reachable objects from it don't change)</p>
</li>
<li>
<p>iterate children</p>
</li>
<li>
<p>mark the object as deeply frozen</p>
</li>
</ul>
<p>but this doesn't handle recursion.</p>
<p>So we could mark as deeply frozen first, and remember to undo that if we cannot freeze some object.<br>
However, is there any object that cannot be frozen? I would think not.</p>
<p>So:</p>
<ul>
<li>
<p>if the object is already deeply-frozen/shareable, return, otherwise:</p>
</li>
<li>
<p>mark an object as deeply frozen</p>
</li>
<li>
<p>iterate children and recurse</p>
</li>
</ul>
<p>Since that wouldn't call user methods, there is no need to worry about something observing an object marked as deeply frozen but not actually deeply frozen yet.<br>
However, a different Thread could see that, so it can be a problem without a GIL.</p>
<p>So, I think we could use an extra bit for "visited by deep_freeze":</p>
<ul>
<li>
<p>if the object is already deeply-frozen/shareable or deep_freeze-visited, return, otherwise:</p>
</li>
<li>
<p>mark an object as deep_freeze-visited and as frozen if not already (so the reachable objects from it don't change)</p>
</li>
<li>
<p>iterate children and recurse</p>
</li>
<li>
<p>mark an object as deeply frozen</p>
</li>
</ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880792020-10-20T21:42:19Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Eregon (Benoit Daloze) wrote in <a href="#note-19">#note-19</a>:</p>
<blockquote>
<p>Is there a concrete case where it is problematic to not call user's <code>freeze</code>?</p>
</blockquote>
<p>I thought I <a href="https://bugs.ruby-lang.org/issues/17145#note-18" class="external">already answered that</a>, but here's a short example:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">Stats</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="nf">freeze</span>
<span class="n">extended_results</span> <span class="c1"># build cache if need be</span>
<span class="k">super</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">extended_results</span>
<span class="vi">@extended_results_cache</span> <span class="o">||=</span> <span class="n">calc_extended_results</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">calc_extended_results</span>
<span class="c1"># lots of calculations</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>There is currently no way for <code>#frozen?</code> to be <code>true</code> (and <code>@instance ||= </code> to fail) without going through the <code>#freeze</code> method. I think we should not break that, especially if there is no callback mechanism to circumvent this.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880832020-10-21T05:23:45Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Looking at <code>def freeze</code> in the top ~400 gems, I found 64 in <code>sequel</code> gem alone, and 28 definitions in the rest π .</p>
<p>Excluding <code>sequel</code>, half do deep freeze. The other use cases:</p>
<p>Cache prebuilding:<br>
<a href="https://github.com/rails/rails/blob/master/activesupport/lib/active_support/time_with_zone.rb#L503-L507" class="external">https://github.com/rails/rails/blob/master/activesupport/lib/active_support/time_with_zone.rb#L503-L507</a><br>
<a href="https://github.com/jeremyevans/sequel/blob/master/lib/sequel/adapters/mysql.rb#L150" class="external">https://github.com/jeremyevans/sequel/blob/master/lib/sequel/adapters/mysql.rb#L150</a><br>
<a href="https://github.com/mongodb/mongoid/blob/master/lib/mongoid/criteria.rb#L239" class="external">https://github.com/mongodb/mongoid/blob/master/lib/mongoid/criteria.rb#L239</a><br>
<a href="https://github.com/sporkmonger/addressable/blob/master/lib/addressable/uri.rb#L846-L858" class="external">https://github.com/sporkmonger/addressable/blob/master/lib/addressable/uri.rb#L846-L858</a><br>
<a href="https://github.com/sporkmonger/addressable/blob/master/lib/addressable/template.rb#247-L250" class="external">https://github.com/sporkmonger/addressable/blob/master/lib/addressable/template.rb#247-L250</a></p>
<p>Instance variable dupping + freeze:<br>
<a href="https://github.com/rails/rails/blob/master/activemodel/lib/active_model/attributes.rb#L118" class="external">https://github.com/rails/rails/blob/master/activemodel/lib/active_model/attributes.rb#L118</a><br>
<a href="https://github.com/rails/rails/blob/master/activerecord/lib/active_record/core.rb#L494" class="external">https://github.com/rails/rails/blob/master/activerecord/lib/active_record/core.rb#L494</a></p>
<p>Another usecase is lazy defined methods (I presume like <code>OpenStruct</code> before the recent changes):<br>
<a href="https://github.com/jeremyevans/sequel/blob/master/lib/sequel/plugins/finder.rb#L167" class="external">https://github.com/jeremyevans/sequel/blob/master/lib/sequel/plugins/finder.rb#L167</a><br>
<a href="https://github.com/solnic/virtus/blob/master/liblib/virtus/instance_methods.rb#L148" class="external">https://github.com/solnic/virtus/blob/master/liblib/virtus/instance_methods.rb#L148</a></p>
<p>Cases not actually freezing the receiver (?):<br>
<a href="https://github.com/mongodb/mongoid/blob/master/lib/mongoid/document.rb#L55-L69" class="external">https://github.com/mongodb/mongoid/blob/master/lib/mongoid/document.rb#L55-L69</a><br>
<a href="https://github.com/datamapper/dm_core/blob/master/lib/dm-core/support/lazy_array.rb#L300-L313" class="external">https://github.com/datamapper/dm_core/blob/master/lib/dm-core/support/lazy_array.rb#L300-L313</a><br>
<a href="https://github.com/dtao/safe_yaml/blob/master/lib/safe_yaml/transform/transformation_map.rb#L24" class="external">https://github.com/dtao/safe_yaml/blob/master/lib/safe_yaml/transform/transformation_map.rb#L24</a><br>
<a href="https://github.com/paulelliott/fabrication/blob/master/lib/fabrication/schematic/manager.rb#L23" class="external">https://github.com/paulelliott/fabrication/blob/master/lib/fabrication/schematic/manager.rb#L23</a></p>
<p>I've been wanting for a long while to propose an API for caching methods, and that could be made Ractor compatible and would resolve most of these cases, but maybe it's too early to consider it?</p>
<p>I was thinking <code>cache_method :foo</code> could "magically" cache the result of <code>foo</code> on first call (per Ractor), store it and return that in the future, with no possibility to invalidate that cache or guarantee that the method <code>foo</code> couldn't be called a few times in case of race conditions.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880932020-10-21T13:28:50ZDan0042 (Daniel DeLorme)
<ul></ul><p>marcandre (Marc-Andre Lafortune) wrote in <a href="#note-22">#note-22</a>:</p>
<blockquote>
<p>I've been wanting for a long while to propose an API for caching methods, and that could be made Ractor compatible and would resolve most of these cases</p>
</blockquote>
<p>+1<br>
The <code>@var ||= expr</code> idiom works well for most cases of memoization, but breaks down for nil values and frozen objects. It's easy enough to handle these cases with a small memoization library (and in fact I do so in my code). But since frozen objects are now going to be everywhere because of ractor, I think it's time to introduce this in core.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880952020-10-21T18:46:10ZEregon (Benoit Daloze)
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/17274">Feature #17274</a>: Ractor.make_shareable(obj)</i> added</li></ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=880992020-10-21T19:03:23ZEregon (Benoit Daloze)
<ul></ul><p><a class="user active user-mention" href="https://bugs.ruby-lang.org/users/182">@marcandre (Marc-Andre Lafortune)</a> Thanks, I didn't get the use-case from the DeepCover above (I just missed the <code>freeze</code> definition).</p>
<p>I think the last variant of #20 works too if we call user <code>freeze</code> instead of internal freeze.<br>
As long we call it before iterating ivars/reachable_objects of that object, we should be fine.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881372020-10-23T20:00:06Zko1 (Koichi Sasada)
<ul></ul><p>Sorry I misunderstood this proposal.<br>
<code>Ractor.make_shareable(obj)</code> proposal (<a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Ractor.make_shareable(obj) (Closed)" href="https://bugs.ruby-lang.org/issues/17274">#17274</a>) satisfies the functionality proposed here.<br>
(I thought <code>deep_freeze(skip_shareable: false)</code> was the proposal. sorry I skipped to read).</p>
<p>We discussed about the name "deep_freeze", and Matz said <code>deep_freeze</code> should be only for freezing, not related to Ractor. So classes/module should be frozen if <code>[C].deep_freeze</code>. This is why I proposed a <code>Object#deep_freeze(skip_shareable: true)</code> and <code>Ractor.make_shareable(obj)</code>.</p>
<p>Anyway maybe the functionality should be okay (calling <code>freeze</code> by <code>make_shareable</code> proposal).</p>
<p>So naming issue is reamained?</p>
<ul>
<li>
<code>Object#deep_freeze</code> (matz doesn't like it)</li>
<li>
<code>Object#deep_freeze(skip_sharable: true)</code> (I don't know how Matz feel. And it is difficult to define Class/Module/... on <code>skip_sharable: false</code>)</li>
<li>
<code>Ractor.make_shareable(obj)</code> (clear for me, but it is a bit long)</li>
<li>
<code>Ractor.shareable!(obj)</code> (shorter. is it clear?)</li>
<li>
<code>Object#shareable!</code> (is it acceptable?)</li>
<li>... other ideas?</li>
</ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881592020-10-25T13:39:06ZEregon (Benoit Daloze)
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-4 priority-default closed" href="/issues/17273">Feature #17273</a>: shareable_constant_value pragma</i> added</li></ul> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881612020-10-25T13:45:19ZEregon (Benoit Daloze)
<ul></ul><p>ko1 (Koichi Sasada) wrote in <a href="#note-26">#note-26</a>:</p>
<blockquote>
<p>So classes/module should be frozen if <code>[C].deep_freeze</code>.</p>
</blockquote>
<p>I think it's very rare somebody would want that.<br>
Would that freeze the <code>Array</code> class too because the class is reachable from the object?</p>
<p>And since the superclass and the constants are reachable from a module, every single named module and constant would be frozen?<br>
It could be a useful thing, but IMHO that is never the intention of <code>obj.deep_freeze</code>.<br>
So stopping at the class seems natural (and also never freezing modules).<br>
(we could also have <code>skip_modules:</code> defaults to <code>true</code>)</p>
<p>Also shareable-but-not-immutable objects seem to have little point to freeze them (Ractor/Thread::TVar).<br>
So I think <code>skip_shareable</code> could default to <code>true</code> for <code>deep_freeze</code>:<br>
<a href="https://bugs.ruby-lang.org/issues/17273#note-9" class="external">https://bugs.ruby-lang.org/issues/17273#note-9</a></p>
<p>I don't think any name besides <code>deep_freeze</code> is going to be as intuitive.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881632020-10-25T19:33:33Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Not only should the default be <code>skip_shareable: true</code>, I don't really see the use-case for <code>skip_shareable: false</code>. Unless someone comes up with one, the option should probably be removed altogether, or maybe we could start without the option and see.</p>
<p>If <code>Object#make_shareable</code> is preferred, that's ok too, but I still think <code>deep_freeze</code> is clearer.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881672020-10-25T21:30:34ZDan0042 (Daniel DeLorme)
<ul></ul><p>Let's say that in the future Array becomes Ractor-safe (i.e. shareable). Would that mean then that <code>[].deep_freeze</code> would no longer freeze the array?</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881822020-10-26T05:28:47Zmarcandre (Marc-Andre Lafortune)marcandre-ruby-core@marc-andre.ca
<ul></ul><p>Dan0042 (Daniel DeLorme) wrote in <a href="#note-30">#note-30</a>:</p>
<blockquote>
<p>Let's say that in the future Array becomes Ractor-safe (i.e. shareable). Would that mean then that <code>[].deep_freeze</code> would no longer freeze the array?</p>
</blockquote>
<p>My understanding is that will never be the case. You can't have 1) fast and 2) concurrent access to mutable data. There might very well be a <code>SharedArray</code> class, but I don't see how <code>Array</code> could ever become shareable.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881882020-10-26T07:46:01ZEregon (Benoit Daloze)
<ul></ul><p>marcandre (Marc-Andre Lafortune) wrote in <a href="#note-31">#note-31</a>:</p>
<blockquote>
<p>My understanding is that will never be the case. You can't have 1) fast and 2) concurrent access to mutable data.</p>
</blockquote>
<p>Digressing a bit, I would say it's possible (<a href="https://eregon.me/blog/assets/research/thread-safe-collections.pdf" class="external">thread-safe Array</a>), but not with Ractor's model which requires copying or moving objects.<br>
So yes, I think a mutable Array will never become Ractor-shareable because it would be too incompatible (due to copying/moveing the elements or only accepting shareable values).</p>
<p>Also if we have a type for which it becomes problematic, we could always add some keyword argument to handle it specially.</p>
<p>I think freezing all named modules and their constants could possibly be something useful for Ractor, but that IMHO should be its own method, like <code>Ractor.freeze_all_modules</code> or so. I guess in practice it doesn't work well at all, as it becomes very likely with more code, that some code needs mutable modules (e.g., @ivars on modules, <code>autoload</code> & lazy method definitions, ...).<br>
Maybe <code>Ractor.freeze_all_constants</code> is more useful. But again with more code there is a high chance something expects a mutable constant, and that would work fine if non-main Ractors don't use it.</p>
<p>Anyway, they are special cases, and I think for #deep_freeze, it's just not interesting to freeze modules.<br>
BTW, <a href="https://github.com/dkubb/ice_nine" class="external">https://github.com/dkubb/ice_nine</a> does not freeze modules either.<br>
(<code>ruby -rice_nine -e 'IceNine.deep_freeze(Enumerable); p Enumerable.frozen?'</code> => <code>false</code>)</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881932020-10-26T07:58:08Zmatz (Yukihiro Matsumoto)matz@ruby.or.jp
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Rejected</i></li></ul><p>This looks very interesting, but it would introduce quite big incompatibility. I don't want to break existing code.</p>
<p>Matz.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=881962020-10-26T09:02:59Zmatz (Yukihiro Matsumoto)matz@ruby.or.jp
<ul></ul><p>I made a mistake in the previous message (the reply was for another issue). Sorry for confusion.</p>
<p>As far as I undestand, the behavior of <code>freeze</code> and <code>make_shareable</code> are different. So making <code>deep_freeze</code> Ractor-aware does not make sense.</p>
<p>Matz.</p> Ruby master - Feature #17145: Ractor-aware `Object#deep_freeze`https://bugs.ruby-lang.org/issues/17145?journal_id=882022020-10-26T13:36:13ZDan0042 (Daniel DeLorme)
<ul></ul><p>marcandre (Marc-Andre Lafortune) wrote in <a href="#note-31">#note-31</a>:</p>
<blockquote>
<p>There might very well be a <code>SharedArray</code> class, but I don't see how <code>Array</code> could ever become shareable.</p>
</blockquote>
<p>Sorry, I think my point was not very clear. Replace "Array" with any class name. If <code>deep_freeze</code> does not freeze the object (because it's shareable), that makes its semantics kinda confusing. But it seems Matz already thinks so too.</p>