Ruby Issue Tracking System: Issueshttps://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112024-03-26T05:41:23ZRuby Issue Tracking System
Redmine Ruby master - Bug #20393 (Closed): `after_fork_ruby` clears all pending interrupts for both paren...https://bugs.ruby-lang.org/issues/203932024-03-26T05:41:23Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In the following program, the behaviour of the parent process is affected by whether <code>Process.fork</code> is invoked or not.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Thread</span><span class="p">.</span><span class="nf">handle_interrupt</span><span class="p">(</span><span class="no">RuntimeError</span> <span class="o">=></span> <span class="ss">:never</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">raise</span><span class="p">(</span><span class="no">RuntimeError</span><span class="p">,</span> <span class="s2">"Queued error"</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"Pending interrupt: </span><span class="si">#{</span><span class="no">Thread</span><span class="p">.</span><span class="nf">pending_interrupt?</span><span class="si">}</span><span class="s2">"</span> <span class="c1"># true</span>
<span class="n">pid</span> <span class="o">=</span> <span class="no">Process</span><span class="p">.</span><span class="nf">fork</span> <span class="k">do</span>
<span class="nb">puts</span> <span class="s2">"Pending interrupt (child process): </span><span class="si">#{</span><span class="no">Thread</span><span class="p">.</span><span class="nf">pending_interrupt?</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">handle_interrupt</span><span class="p">(</span><span class="no">RuntimeError</span> <span class="o">=></span> <span class="ss">:immediate</span><span class="p">){}</span>
<span class="k">end</span>
<span class="n">_</span><span class="p">,</span> <span class="n">status</span> <span class="o">=</span> <span class="no">Process</span><span class="p">.</span><span class="nf">waitpid2</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"Child process status: </span><span class="si">#{</span><span class="n">status</span><span class="p">.</span><span class="nf">inspect</span><span class="si">}</span><span class="s2">"</span>
<span class="nb">puts</span> <span class="s2">"Pending interrupt: </span><span class="si">#{</span><span class="no">Thread</span><span class="p">.</span><span class="nf">pending_interrupt?</span><span class="si">}</span><span class="s2">"</span> <span class="c1"># false</span>
<span class="k">end</span>
<span class="nb">puts</span> <span class="s2">"Exiting..."</span>
</code></pre>
<p>I don't think the parent process pending interrupts should be cleared by <code>after_fork_ruby</code>:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="k">static</span> <span class="kt">void</span>
<span class="nf">after_fork_ruby</span><span class="p">(</span><span class="n">rb_pid_t</span> <span class="n">pid</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">rb_threadptr_pending_interrupt_clear</span><span class="p">(</span><span class="n">GET_THREAD</span><span class="p">());</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// child</span>
<span class="n">clear_pid_cache</span><span class="p">();</span>
<span class="n">rb_thread_atfork</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="c1">// parent</span>
<span class="n">after_exec</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>How about this implementation:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="k">static</span> <span class="kt">void</span>
<span class="nf">after_fork_ruby</span><span class="p">(</span><span class="n">rb_pid_t</span> <span class="n">pid</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// child</span>
<span class="n">rb_threadptr_pending_interrupt_clear</span><span class="p">(</span><span class="n">GET_THREAD</span><span class="p">());</span>
<span class="n">clear_pid_cache</span><span class="p">();</span>
<span class="n">rb_thread_atfork</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="c1">// parent</span>
<span class="n">after_exec</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
<p>cc <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a></p> Ruby master - Feature #20298 (Open): Introduce `Time()` type-cast / constructor.https://bugs.ruby-lang.org/issues/202982024-02-26T00:34:27Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Many Ruby primitive types have constructors, e.g. <code>Integer(...)</code>, <code>String(...)</code>, <code>Float(...)</code>, etc. These usually convert from some subset of types, e.g. <code>Float(1) -> 1.0</code> will convert an Integer to a Float, and <code>Float("1") -> 1.0</code> will parse a String to a Float, and similar for other type casts/constructors.</p>
<p>I'd like to propose we introduce something similar for <code>Time</code> (and possibly this extends to <code>Date</code>/<code>DateTime</code> in a follow up proposal).</p>
<p>Suggested implementation could look something like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">Time</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">case</span> <span class="n">value</span>
<span class="k">when</span> <span class="no">Time</span>
<span class="n">value</span>
<span class="k">when</span> <span class="no">Integer</span>
<span class="no">Time</span><span class="p">.</span><span class="nf">at</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">when</span> <span class="no">String</span> <span class="c1"># The format is assumed to be the result of `Time#to_s`.</span>
<span class="no">Time</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">value</span><span class="p">.</span><span class="nf">to_time</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>Alternatively, we might like to be a little more specific with the <code>else</code> clause/error handling.</p>
<a name="Background"></a>
<h2 >Background<a href="#Background" class="wiki-anchor">¶</a></h2>
<p>In a project, I need to support multiple serialization formats/coders, including MessagePack, Marshal and JSON.</p>
<p>While Marshal and MessagePack are capable of serializing <code>Time</code> instances, <code>JSON</code> is not.</p>
<p>The code looks a bit like this:</p>
<pre><code>data = fetch_job_data
job = @coder.load(data)
scheduled_at = Time(job[:scheduled_at]) # Hypothetical type-cast as outlined above
</code></pre>
<a name="Additional-Observations"></a>
<h2 >Additional Observations<a href="#Additional-Observations" class="wiki-anchor">¶</a></h2>
<p>While some primitive data types accept themselves as arguments and construct by copy, e.g. <code>Array.new(Array.new)</code>, others do not, e.g. <code>Hash</code> and <code>Time</code>. Perhaps <code>Time.new(Time.new)</code> should behave similarly to <code>Array.new(Array.new)</code> - the outer instance is a copy of the inner.</p> Ruby master - Bug #20288 (Closed): `rb_fiber_scheduler_close` exceptions are not handled in `rb_f...https://bugs.ruby-lang.org/issues/202882024-02-21T08:45:33Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>The code responsible for setting a new scheduler on a thread does not properly handle exceptions raised by <code>rb_fiber_scheduler_close</code>. If <code>rb_fiber_scheduler_close</code> raised an exception, the assignment <code>thread->scheduler = scheduler</code> would not be executed. This leaves the thread in an undefined state because it might still hold a reference to the old scheduler, which was supposed to be closed and replaced.</p>
<a name="Steps-to-Reproduce"></a>
<h2 >Steps to Reproduce:<a href="#Steps-to-Reproduce" class="wiki-anchor">¶</a></h2>
<ol>
<li>Define a custom fiber scheduler that raises an exception in its close method.</li>
<li>Set the custom scheduler on a thread.</li>
<li>Attempt to replace the custom scheduler with a different scheduler.</li>
<li>Observe that if the close method of the original scheduler raises an exception, the thread's scheduler reference is not updated.</li>
</ol>
<a name="Proposed-Fix"></a>
<h2 >Proposed Fix:<a href="#Proposed-Fix" class="wiki-anchor">¶</a></h2>
<p>The use of <code>rb_ensure</code> can be introduced to wrap the call to <code>rb_fiber_scheduler_close</code> to ensure that, regardless of whether an exception is raised, <code>thread->scheduler</code> is set to <code>Qnil</code>, and then <code>thread->scheduler = scheduler</code> is safely executed to update the thread's scheduler reference.</p> Ruby master - Bug #20286 (Closed): TracePoint does not emit `thread_end` event when thread exits ...https://bugs.ruby-lang.org/issues/202862024-02-21T05:40:34Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Using TracePoint to trace <code>thread_begin</code> and <code>thread_end</code> events fails to emit the <code>thread_end</code> event when an exception (e.g., Interrupt) is raised within a thread. This behavior occurs because the exception handling bypasses the internal thread finishing logic, including trace point and fiber scheduler cleanup code. This issue affects the ability to accurately monitor thread lifecycle events in scenarios involving exception handling or abrupt thread terminations.</p>
<a name="Steps-to-Reproduce"></a>
<h2 >Steps to Reproduce:<a href="#Steps-to-Reproduce" class="wiki-anchor">¶</a></h2>
<ol>
<li>Set up <code>TracePoint</code> to trace <code>thread_begin</code> and <code>thread_end</code> events.</li>
<li>Create a new thread that raises an exception.</li>
<li>Join the thread and observe that only the <code>thread_begin</code> event is emitted without a corresponding <code>thread_end</code> event.</li>
</ol>
<a name="Example-Code"></a>
<h2 >Example Code<a href="#Example-Code" class="wiki-anchor">¶</a></h2>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">TracePoint</span><span class="p">.</span><span class="nf">trace</span><span class="p">(</span><span class="ss">:thread_begin</span><span class="p">,</span> <span class="ss">:thread_end</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">tp</span><span class="o">|</span>
<span class="nb">p</span> <span class="p">[</span><span class="n">tp</span><span class="p">.</span><span class="nf">event</span><span class="p">,</span> <span class="n">tp</span><span class="p">.</span><span class="nf">lineno</span><span class="p">,</span> <span class="n">tp</span><span class="p">.</span><span class="nf">path</span><span class="p">,</span> <span class="n">tp</span><span class="p">.</span><span class="nf">defined_class</span><span class="p">,</span> <span class="n">tp</span><span class="p">.</span><span class="nf">method_id</span><span class="p">]</span>
<span class="k">end</span>
<span class="n">thread</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="k">raise</span> <span class="no">Interrupt</span>
<span class="k">end</span>
<span class="n">thread</span><span class="p">.</span><span class="nf">join</span>
</code></pre>
<a name="Current-Behavior"></a>
<h3 >Current Behavior:<a href="#Current-Behavior" class="wiki-anchor">¶</a></h3>
<p>The <code>TracePoint</code> emits the <code>thread_begin</code> event but fails to emit the <code>thread_end</code> event when an exception is raised within the thread, indicating an incomplete tracing of thread lifecycle events.</p>
<p>I've confirmed this as far back as Ruby 2.6.</p>
<pre><code>> ruby ./test.rb
[:thread_begin, 0, nil, nil, nil]
#<Thread:0x000000010384b5a8 ./test.rb:5 run> terminated with exception (report_on_exception is true):
./test.rb:6:in `block in <main>': Interrupt (Interrupt)
./test.rb:6:in `block in <main>': Interrupt (Interrupt)
</code></pre>
<a name="Expected-Behavior"></a>
<h3 >Expected Behavior:<a href="#Expected-Behavior" class="wiki-anchor">¶</a></h3>
<p>The <code>TracePoint</code> should emit both <code>thread_begin</code> and <code>thread_end</code> events, accurately reflecting the lifecycle of the thread, even when an exception is raised within the thread.</p>
<pre><code>> ruby ./test.rb
[:thread_begin, 0, nil, nil, nil]
[:thread_end, 0, nil, nil, nil]
#<Thread:0x0000000105435db8 ./test.rb:5 run> terminated with exception (report_on_exception is true):
./test.rb:6:in 'block in <main>': Interrupt (Interrupt)
./test.rb:6:in 'block in <main>': Interrupt (Interrupt)
</code></pre>
<a name="Possible-Fix"></a>
<h2 >Possible Fix<a href="#Possible-Fix" class="wiki-anchor">¶</a></h2>
<p>Changing the implementation of <code>thread_do_start</code> to have what amounts to an "ensure" block.</p>
<pre><code class="c syntaxhl" data-language="c"><span class="k">static</span> <span class="kt">void</span>
<span class="nf">thread_do_start</span><span class="p">(</span><span class="n">rb_thread_t</span> <span class="o">*</span><span class="n">th</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">native_set_thread_name</span><span class="p">(</span><span class="n">th</span><span class="p">);</span>
<span class="n">VALUE</span> <span class="n">result</span> <span class="o">=</span> <span class="n">Qundef</span><span class="p">;</span>
<span class="n">rb_execution_context_t</span> <span class="o">*</span><span class="n">ec</span> <span class="o">=</span> <span class="n">th</span><span class="o">-></span><span class="n">ec</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">state</span><span class="p">;</span>
<span class="n">EXEC_EVENT_HOOK</span><span class="p">(</span><span class="n">ec</span><span class="p">,</span> <span class="n">RUBY_EVENT_THREAD_BEGIN</span><span class="p">,</span> <span class="n">th</span><span class="o">-></span><span class="n">self</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">Qundef</span><span class="p">);</span>
<span class="n">EC_PUSH_TAG</span><span class="p">(</span><span class="n">ec</span><span class="p">);</span>
<span class="k">if</span> <span class="p">((</span><span class="n">state</span> <span class="o">=</span> <span class="n">EC_EXEC_TAG</span><span class="p">())</span> <span class="o">==</span> <span class="n">TAG_NONE</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">th</span><span class="o">-></span><span class="n">invoke_type</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="n">thread_invoke_type_proc</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">thread_do_start_proc</span><span class="p">(</span><span class="n">th</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">thread_invoke_type_ractor_proc</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">thread_do_start_proc</span><span class="p">(</span><span class="n">th</span><span class="p">);</span>
<span class="n">rb_ractor_atexit</span><span class="p">(</span><span class="n">th</span><span class="o">-></span><span class="n">ec</span><span class="p">,</span> <span class="n">result</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">thread_invoke_type_func</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">th</span><span class="o">-></span><span class="n">invoke_arg</span><span class="p">.</span><span class="n">func</span><span class="p">.</span><span class="n">func</span><span class="p">)(</span><span class="n">th</span><span class="o">-></span><span class="n">invoke_arg</span><span class="p">.</span><span class="n">func</span><span class="p">.</span><span class="n">arg</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="n">thread_invoke_type_none</span><span class="p">:</span>
<span class="n">rb_bug</span><span class="p">(</span><span class="s">"unreachable"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">EC_POP_TAG</span><span class="p">();</span>
<span class="n">VALUE</span> <span class="n">errinfo</span> <span class="o">=</span> <span class="n">ec</span><span class="o">-></span><span class="n">errinfo</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">NIL_P</span><span class="p">(</span><span class="n">errinfo</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="n">RB_TYPE_P</span><span class="p">(</span><span class="n">errinfo</span><span class="p">,</span> <span class="n">T_OBJECT</span><span class="p">))</span> <span class="p">{</span>
<span class="n">ec</span><span class="o">-></span><span class="n">errinfo</span> <span class="o">=</span> <span class="n">Qnil</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">rb_fiber_scheduler_set</span><span class="p">(</span><span class="n">Qnil</span><span class="p">);</span>
<span class="n">EXEC_EVENT_HOOK</span><span class="p">(</span><span class="n">th</span><span class="o">-></span><span class="n">ec</span><span class="p">,</span> <span class="n">RUBY_EVENT_THREAD_END</span><span class="p">,</span> <span class="n">th</span><span class="o">-></span><span class="n">self</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">Qundef</span><span class="p">);</span>
<span class="n">ec</span><span class="o">-></span><span class="n">errinfo</span> <span class="o">=</span> <span class="n">errinfo</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">EC_JUMP_TAG</span><span class="p">(</span><span class="n">ec</span><span class="p">,</span> <span class="n">state</span><span class="p">);</span>
<span class="n">th</span><span class="o">-></span><span class="n">value</span> <span class="o">=</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
<p>It's possible <code>rb_fiber_scheduler_set(Qnil);</code> can emit an exception itself. How do we write the code to handle that case?</p> Ruby master - Feature #20282 (Open): Enhancing Ruby's Coverage with Per-Test Coverage Reportshttps://bugs.ruby-lang.org/issues/202822024-02-19T22:51:10Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.</p>
<a name="Objectives"></a>
<h2 >Objectives<a href="#Objectives" class="wiki-anchor">¶</a></h2>
<p>The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas:</p>
<ol>
<li>
<p>Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process.</p>
</li>
<li>
<p>Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable.</p>
</li>
<li>
<p>Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience.</p>
</li>
</ol>
<a name="Proposed-Solutions"></a>
<h2 >Proposed Solutions<a href="#Proposed-Solutions" class="wiki-anchor">¶</a></h2>
<p>The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named <code>Coverage::Capture</code>. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Coverage::Capture</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">start</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">tap</span><span class="p">(</span><span class="o">&</span><span class="ss">:start</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Start receiving coverage callbacks.</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="k">end</span>
<span class="c1"># Stop receiving coverage callbacks.</span>
<span class="k">def</span> <span class="nf">stop</span>
<span class="k">end</span>
<span class="c1"># User-overridable statement coverage callback.</span>
<span class="k">def</span> <span class="nf">statement</span><span class="p">(</span><span class="n">iseq</span><span class="p">,</span> <span class="n">location</span><span class="p">)</span>
<span class="n">fetch</span><span class="p">(</span><span class="n">iseq</span><span class="p">)</span><span class="o">&</span><span class="p">.</span><span class="nf">statement_coverage</span><span class="p">.</span><span class="nf">increment</span><span class="p">(</span><span class="n">location</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Additional methods for branch/declaration coverage would follow a similar pattern.</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">MyCoverageCapture</span> <span class="o"><</span> <span class="no">Coverage</span><span class="o">::</span><span class="no">Capture</span>
<span class="c1"># Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc.</span>
<span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">iseq</span><span class="p">)</span>
<span class="vi">@coverage</span><span class="p">[</span><span class="n">iseq</span><span class="p">]</span> <span class="o">||=</span> <span class="no">Coverage</span><span class="p">.</span><span class="nf">default_coverage</span><span class="p">(</span><span class="n">iseq</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Usage example:</span>
<span class="n">my_coverage_capture</span> <span class="o">=</span> <span class="no">MyCoverageCapture</span><span class="p">.</span><span class="nf">start</span>
<span class="c1"># Execute test suite or specific tests</span>
<span class="n">my_coverage_capture</span><span class="p">.</span><span class="nf">stop</span>
<span class="c1"># Access detailed coverage data</span>
<span class="nb">puts</span> <span class="n">my_coverage_capture</span><span class="p">.</span><span class="nf">coverage</span><span class="p">.</span><span class="nf">statement_coverage</span>
</code></pre>
<p>In addition, we'd need a well defined interface for <code>Coverage.default_coverage</code>, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: <a href="https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc7010216eef4/src/vscode-dts/vscode.proposed.testCoverage.d.ts" class="external">https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc7010216eef4/src/vscode-dts/vscode.proposed.testCoverage.d.ts</a> - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1"># Hypothetical interface (mostly copied from vscode's proposed interface):</span>
<span class="k">module</span> <span class="nn">Coverage</span>
<span class="c1"># Contains coverage metadata for a file</span>
<span class="k">class</span> <span class="nc">Target</span>
<span class="nb">attr_reader</span> <span class="ss">:instruction_sequence</span>
<span class="nb">attr_accessor</span> <span class="ss">:statement_coverage</span><span class="p">,</span> <span class="ss">:branch_coverage</span><span class="p">,</span> <span class="ss">:declaration_coverage</span><span class="p">,</span> <span class="ss">:detailed_coverage</span>
<span class="c1"># @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location.</span>
<span class="c1"># Similar structures for other coverage data.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">instruction_sequence</span><span class="p">,</span> <span class="n">statement_coverage</span><span class="p">,</span> <span class="n">branch_coverage</span><span class="o">=</span><span class="kp">nil</span><span class="p">,</span> <span class="n">declaration_coverage</span><span class="o">=</span><span class="kp">nil</span><span class="p">)</span>
<span class="vi">@instruction_sequence</span> <span class="o">=</span> <span class="n">instruction_sequence</span>
<span class="vi">@statement_coverage</span> <span class="o">=</span> <span class="n">statement_coverage</span>
<span class="vi">@branch_coverage</span> <span class="o">=</span> <span class="n">branch_coverage</span>
<span class="vi">@declaration_coverage</span> <span class="o">=</span> <span class="n">declaration_coverage</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Coverage information for a single statement or line.</span>
<span class="k">class</span> <span class="nc">StatementCoverage</span>
<span class="c1"># The number of times this statement was executed, or a boolean indicating</span>
<span class="c1"># whether it was executed if the exact count is unknown. If zero or false,</span>
<span class="c1"># the statement will be marked as un-covered.</span>
<span class="nb">attr_accessor</span> <span class="ss">:executed</span>
<span class="c1"># Statement location (line number? or range? or position? AST?)</span>
<span class="nb">attr_accessor</span> <span class="ss">:location</span>
<span class="c1"># Coverage from branches of this line or statement. If it's not a</span>
<span class="c1"># conditional, this will be empty.</span>
<span class="nb">attr_accessor</span> <span class="ss">:branches</span>
<span class="c1"># Initializes a new instance of the StatementCoverage class.</span>
<span class="c1">#</span>
<span class="c1"># @parameter executed [Number, Boolean] The number of times this statement was executed, or a</span>
<span class="c1"># boolean indicating whether it was executed if the exact count is unknown. If zero or false,</span>
<span class="c1"># the statement will be marked as un-covered.</span>
<span class="c1">#</span>
<span class="c1"># @parameter location [Position, Range] The statement position.</span>
<span class="c1">#</span>
<span class="c1"># @parameter branches [Array(BranchCoverage)] Coverage from branches of this line.</span>
<span class="c1"># If it's not a conditional, this should be omitted.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">executed</span><span class="p">,</span> <span class="n">location</span><span class="p">,</span> <span class="n">branches</span><span class="o">=</span><span class="p">[])</span>
<span class="vi">@executed</span> <span class="o">=</span> <span class="n">executed</span>
<span class="vi">@location</span> <span class="o">=</span> <span class="n">location</span>
<span class="vi">@branches</span> <span class="o">=</span> <span class="n">branches</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Coverage information for a branch</span>
<span class="k">class</span> <span class="nc">BranchCoverage</span>
<span class="c1"># The number of times this branch was executed, or a boolean indicating</span>
<span class="c1"># whether it was executed if the exact count is unknown. If zero or false,</span>
<span class="c1"># the branch will be marked as un-covered.</span>
<span class="nb">attr_accessor</span> <span class="ss">:executed</span>
<span class="c1"># Branch location.</span>
<span class="nb">attr_accessor</span> <span class="ss">:location</span>
<span class="c1"># Label for the branch, used in the context of "the ${label} branch was</span>
<span class="c1"># not taken," for example.</span>
<span class="nb">attr_accessor</span> <span class="ss">:label</span>
<span class="c1"># Initializes a new instance of the BranchCoverage class.</span>
<span class="c1">#</span>
<span class="c1"># @param executed [Number, Boolean] The number of times this branch was executed, or a</span>
<span class="c1"># boolean indicating whether it was executed if the exact count is unknown. If zero or false,</span>
<span class="c1"># the branch will be marked as un-covered.</span>
<span class="c1">#</span>
<span class="c1"># @param location [Position, Range] (optional) The branch position.</span>
<span class="c1">#</span>
<span class="c1"># @param label [String] (optional) Label for the branch, used in the context of</span>
<span class="c1"># "the ${label} branch was not taken," for example.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">executed</span><span class="p">,</span> <span class="n">location</span><span class="o">=</span><span class="kp">nil</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="kp">nil</span><span class="p">)</span>
<span class="vi">@executed</span> <span class="o">=</span> <span class="n">executed</span>
<span class="vi">@location</span> <span class="o">=</span> <span class="n">location</span>
<span class="vi">@label</span> <span class="o">=</span> <span class="n">label</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Coverage information for a declaration</span>
<span class="k">class</span> <span class="nc">DeclarationCoverage</span>
<span class="c1"># Name of the declaration. Depending on the reporter and language, this</span>
<span class="c1"># may be types such as functions, methods, or namespaces.</span>
<span class="nb">attr_accessor</span> <span class="ss">:name</span>
<span class="c1"># The number of times this declaration was executed, or a boolean</span>
<span class="c1"># indicating whether it was executed if the exact count is unknown. If</span>
<span class="c1"># zero or false, the declaration will be marked as un-covered.</span>
<span class="nb">attr_accessor</span> <span class="ss">:executed</span>
<span class="c1"># Declaration location.</span>
<span class="nb">attr_accessor</span> <span class="ss">:location</span>
<span class="c1"># Initializes a new instance of the DeclarationCoverage class.</span>
<span class="c1">#</span>
<span class="c1"># @param name [String] Name of the declaration.</span>
<span class="c1">#</span>
<span class="c1"># @param executed [Number, Boolean] The number of times this declaration was executed, or a</span>
<span class="c1"># boolean indicating whether it was executed if the exact count is unknown. If zero or false,</span>
<span class="c1"># the declaration will be marked as un-covered.</span>
<span class="c1">#</span>
<span class="c1"># @param location [Position, Range] The declaration position.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">executed</span><span class="p">,</span> <span class="n">location</span><span class="p">)</span>
<span class="vi">@name</span> <span class="o">=</span> <span class="nb">name</span>
<span class="vi">@executed</span> <span class="o">=</span> <span class="n">executed</span>
<span class="vi">@location</span> <span class="o">=</span> <span class="n">location</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>By following this format, we will be compatible with a wide range of external tools.</p> Ruby master - Misc #20279 (Closed): Is the implementation of `respond_to_missing?` in BasicObject...https://bugs.ruby-lang.org/issues/202792024-02-19T05:18:57Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Considering the documentation here: <a href="https://ruby-doc.org/3.2.2/BasicObject.html" class="external">https://ruby-doc.org/3.2.2/BasicObject.html</a></p>
<p>Introduced in: <a href="https://github.com/ruby/ruby/commit/3eb7d2b33e3f8555d81db5369eb6fb7100a91e63" class="external">https://github.com/ruby/ruby/commit/3eb7d2b33e3f8555d81db5369eb6fb7100a91e63</a></p>
<p>I wondered if <code>or super</code> is correct in <code>respond_to_missing?</code>.</p>
<p>For example:</p>
<pre><code>irb(main):001* class MyObjectSystem < BasicObject
irb(main):002* DELEGATE = [:puts, :p]
irb(main):003*
irb(main):004* def method_missing(name, *args, &block)
irb(main):005* return super unless DELEGATE.include? name
irb(main):006* ::Kernel.send(name, *args, &block)
irb(main):007* end
irb(main):008*
irb(main):009* public def respond_to_missing?(name, include_private = false)
irb(main):010* DELEGATE.include?(name) or super
irb(main):011* end
irb(main):012> end
=> :respond_to_missing?
irb(main):013> MyObjectSystem.new.respond_to_missing?(:foo)
(irb):5:in `method_missing': super: no superclass method `respond_to_missing?' for an instance of MyObjectSystem (NoMethodError)
from (irb):10:in `respond_to_missing?'
from (irb):13:in `<main>'
from <internal:kernel>:187:in `loop'
from /home/samuel/.gem/ruby/3.3.0/gems/irb-1.11.2/exe/irb:9:in `<top (required)>'
from /home/samuel/.gem/ruby/3.3.0/bin/irb:25:in `load'
from /home/samuel/.gem/ruby/3.3.0/bin/irb:25:in `<main>'
</code></pre>
<p>It looks wrong to me.</p>
<p>In addition, I'd like to know in what situations <code>BasicObject</code> should define <code>respond_to_missing?</code> - because I was under the impression it was called by <code>method_missing</code>. Does <code>BasicObject#method_missing</code> have this behaviour? Maybe we can improve the documentation cc <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/52355">@burdettelamar (Burdette Lamar)</a></p> Ruby master - Bug #20231 (Closed): Don't wait in io_binwrite_string if not necessary.https://bugs.ruby-lang.org/issues/202312024-02-01T01:52:02Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>PR: <a href="https://github.com/ruby/ruby/pull/9792" class="external">https://github.com/ruby/ruby/pull/9792</a></p>
<p>Writing to a buffered IO can result in the entire internal buffer being flushed, which causes <code>io_binwrite_string_internal</code> to return 0. In that case, we were setting <code>errno = EAGAIN</code>. This causes <code>rb_io_maybe_wait_writable</code> to be invoked, however we should immediately retry <code>io_binwrite_string_internal</code> instead.</p>
<p>The reason why calling <code>rb_io_maybe_wait_writable</code> is a bad idea in general, is that not all IO can go via this mechanism in every situation - in other words, <code>kqueue</code> does not support <code>kevent("/dev/null", writable)</code> and returns errno=22 <code>EINVAL</code>. The same applies to some kinds of pipes, TTYs, etc.</p>
<p>Indirectly responsible for <a href="https://github.com/socketry/async/issues/301" class="external">https://github.com/socketry/async/issues/301</a>.</p> Ruby master - Feature #20215 (Open): Introduce `IO#readable?`https://bugs.ruby-lang.org/issues/202152024-01-26T05:19:16Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There are some cases where, as an optimisation, it's useful to know whether more data is potentially available.</p>
<p>We already have <code>IO#eof?</code> but the problem with using <code>IO#eof?</code> is that it can block indefinitely for sockets.</p>
<p>Therefore, code which uses <code>IO#eof?</code> to determine if there is potentially more data, may hang.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">make_request</span><span class="p">(</span><span class="n">path</span> <span class="o">=</span> <span class="s2">"/"</span><span class="p">)</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">connect_remote_host</span>
<span class="c1"># HTTP/1.0 request:</span>
<span class="n">client</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="s2">"GET </span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="s2"> HTTP/1.0</span><span class="se">\r\n\r\n</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># Read response</span>
<span class="n">client</span><span class="p">.</span><span class="nf">gets</span><span class="p">(</span><span class="s2">"</span><span class="se">\r\n</span><span class="s2">"</span><span class="p">)</span> <span class="c1"># => "HTTP/1.0 200 OK\r\n"</span>
<span class="c1"># Assuming connection close, there are two things the server can do:</span>
<span class="c1"># 1. peer.close</span>
<span class="c1"># 2. peer.write(...); peer.close</span>
<span class="k">if</span> <span class="n">client</span><span class="p">.</span><span class="nf">eof?</span> <span class="c1"># <--- Can hang here!</span>
<span class="nb">puts</span> <span class="s2">"Connection closed"</span>
<span class="c1"># Avoid yielding as we know there definitely won't be any data.</span>
<span class="k">else</span>
<span class="nb">puts</span> <span class="s2">"Connection open, data may be available..."</span>
<span class="c1"># There might be data available, so yield.</span>
<span class="k">yield</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">ensure</span>
<span class="n">client</span><span class="o">&</span><span class="p">.</span><span class="nf">close</span>
<span class="k">end</span>
<span class="n">make_request</span> <span class="k">do</span> <span class="o">|</span><span class="n">client</span><span class="o">|</span>
<span class="nb">puts</span> <span class="n">client</span><span class="p">.</span><span class="nf">read</span> <span class="c1"># <--- Prefer to wait here.</span>
<span class="k">end</span>
</code></pre>
<p>The proposed <code>IO#readable?</code> is similar to <code>IO#eof?</code> but rather than blocking, would simply return false. The expectation is the user will subsequently call <code>read</code> which may then wait.</p>
<p>The proposed implementation would look something like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">IO</span>
<span class="k">def</span> <span class="nf">readable?</span>
<span class="o">!</span><span class="nb">self</span><span class="p">.</span><span class="nf">closed?</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">BasicSocket</span>
<span class="c1"># Is it likely that the socket is still connected?</span>
<span class="c1"># May return false positive, but won't return false negative.</span>
<span class="k">def</span> <span class="nf">readable?</span>
<span class="k">return</span> <span class="kp">false</span> <span class="k">unless</span> <span class="k">super</span>
<span class="c1"># If we can wait for the socket to become readable, we know that the socket may still be open.</span>
<span class="n">result</span> <span class="o">=</span> <span class="nb">self</span><span class="p">.</span><span class="nf">recv_nonblock</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="no">MSG_PEEK</span><span class="p">,</span> <span class="ss">exception: </span><span class="kp">false</span><span class="p">)</span>
<span class="c1"># No data was available - newer Ruby can return nil instead of empty string:</span>
<span class="k">return</span> <span class="kp">false</span> <span class="k">if</span> <span class="n">result</span><span class="p">.</span><span class="nf">nil?</span>
<span class="c1"># Either there was some data available, or we can wait to see if there is data avaialble.</span>
<span class="k">return</span> <span class="o">!</span><span class="n">result</span><span class="p">.</span><span class="nf">empty?</span> <span class="o">||</span> <span class="n">result</span> <span class="o">==</span> <span class="ss">:wait_readable</span>
<span class="k">rescue</span> <span class="no">Errno</span><span class="o">::</span><span class="no">ECONNRESET</span>
<span class="c1"># This might be thrown by recv_nonblock.</span>
<span class="k">return</span> <span class="kp">false</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>For <code>IO</code> itself, when there is buffered data, <code>readable?</code> would also return true immediately, similar to <code>eof?</code>. This is not shown in the above implementation as I'm not sure if there is any Ruby method which exposes "there is buffered data".</p> Ruby master - Feature #20105 (Open): Introduce `IO::Stream` or something similar.https://bugs.ruby-lang.org/issues/201052024-01-01T07:43:50Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Ruby's IO class has a general model for streaming IO, including some hidden classes like <code>IO::generic_readable</code> and <code>IO::generic_writable</code>.</p>
<p>As Ruby's core IO classes evolve, gems like <code>openssl</code> (see <code>OpenSSL::Buffering</code>) need to be updated to support changes to the interface.</p>
<p>As it stands, there are changes in <code>IO</code> which are not copied to <code>OpenSSL::Buffering</code>. I'd like to propose we introduce some shared interface for streams that is used by <code>IO</code>, <code>Socket</code>, and <code>OpenSSL</code> to start with. The general interface would be similar to <code>IO</code> and allow code like <code>OpenSSL</code> to avoid re-implementing the <code>IO</code> interface.</p>
<p>I don't have a strong idea for the interface yet, but it would probably look something like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">IO::Stream</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="ss">buffered: </span><span class="kp">true</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">buffer</span><span class="o">=</span><span class="kp">nil</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">write</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">buffer</span><span class="o">=</span><span class="kp">nil</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Include general operations from IO, like gets, puts, etc</span>
<span class="k">end</span>
</code></pre>
<p>I think ideally we'd try implement with pure Ruby and a first goal would be to replace <code>OpenSSL::Buffering</code>.</p> Ruby master - Feature #20102 (Open): Introduce `Fiber#resuming?`https://bugs.ruby-lang.org/issues/201022023-12-28T07:25:59Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There are some tricky edge cases when using <code>Fibre#raise</code> and <code>Fiber#kill</code>, e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">fiber</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="n">killer</span> <span class="o">=</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">fiber</span><span class="p">.</span><span class="nf">raise</span><span class="p">(</span><span class="s2">"Stop"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">fiber</span> <span class="o">=</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">killer</span><span class="p">.</span><span class="nf">resume</span>
<span class="k">end</span>
<span class="n">fiber</span><span class="p">.</span><span class="nf">resume</span>
<span class="c1"># 4:in `raise': attempt to raise a resuming fiber (FiberError)</span>
<span class="c1"># 4:in `block in <main>'</span>
</code></pre>
<p>Async has to deal with this edge case explicitly by rescuing the exception:</p>
<p><a href="https://github.com/socketry/async/blob/ffd019d9c1d547926a28fe8f36bf7bfe91d8a168/lib/async/task.rb#L226-L233" class="external">https://github.com/socketry/async/blob/ffd019d9c1d547926a28fe8f36bf7bfe91d8a168/lib/async/task.rb#L226-L233</a></p>
<p>I'd like to avoid doing that and instead just ask "Can I kill/raise on this fiber right now?" which is determined by whether the fiber itself can be resumed or transferred to.</p>
<p>To address this, I'd like to introduce <code>Fiber#resuming?</code>:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="cm">/*
* call-seq: fiber.resumed? -> true or false
*
* Whether the fiber is currently resumed.
*/</span>
<span class="n">VALUE</span>
<span class="nf">rb_fiber_resuming_p</span><span class="p">(</span><span class="n">VALUE</span> <span class="n">fiber_value</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="n">rb_fiber_struct</span> <span class="o">*</span><span class="n">fiber</span> <span class="o">=</span> <span class="n">fiber_ptr</span><span class="p">(</span><span class="n">fiber_value</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">FIBER_TERMINATED_P</span><span class="p">(</span><span class="n">fiber</span><span class="p">))</span> <span class="k">return</span> <span class="n">RUBY_Qfalse</span><span class="p">;</span>
<span class="k">return</span> <span class="n">RBOOL</span><span class="p">(</span><span class="n">fiber</span><span class="o">-></span><span class="n">resuming_fiber</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<p>See the PR: <a href="https://github.com/ruby/ruby/pull/9382" class="external">https://github.com/ruby/ruby/pull/9382</a></p> Ruby master - Bug #20086 (Closed): Windows memory mapped file `IO::Buffer` is buggy.https://bugs.ruby-lang.org/issues/200862023-12-26T11:38:46Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>We confirmed a bug in memory mapped files on Windows.</p>
<p>See <a href="https://github.com/ruby/ruby/pull/9358" class="external">https://github.com/ruby/ruby/pull/9358</a> for the fix.</p>
<p>We don't need to backport the logging.</p> Ruby master - Bug #20042 (Closed): ObjectSpace finalizer can cause segfaulthttps://bugs.ruby-lang.org/issues/200422023-12-05T20:47:47Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Finalizer</span>
<span class="k">def</span> <span class="nf">call</span> <span class="c1"># <- missing (id) argument</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">object</span> <span class="o">=</span> <span class="no">Object</span><span class="p">.</span><span class="nf">new</span>
<span class="no">ObjectSpace</span><span class="p">.</span><span class="nf">define_finalizer</span><span class="p">(</span><span class="n">object</span><span class="p">,</span> <span class="no">Finalizer</span><span class="p">.</span><span class="nf">new</span><span class="p">)</span>
<span class="n">object</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="no">GC</span><span class="p">.</span><span class="nf">start</span> <span class="c1"># segfaults here</span>
</code></pre> Ruby master - Bug #19857 (Rejected): Eval coverage is reset after each `eval`.https://bugs.ruby-lang.org/issues/198572023-08-30T07:39:36Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It seems like <code>eval</code> based coverage is reset every time eval is invoked.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'coverage'</span>
<span class="k">def</span> <span class="nf">measure</span><span class="p">(</span><span class="n">flag</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span>
<span class="n">c</span><span class="p">.</span><span class="nf">class_eval</span><span class="p">(</span><span class="o"><<~</span><span class="no">RUBY</span><span class="p">,</span> <span class="s2">"foo.rb"</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span><span class="sh">
def foo(flag)
if flag
puts "foo"
else
puts "bar"
end
end
</span><span class="no"> RUBY</span>
<span class="k">return</span> <span class="n">c</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">foo</span><span class="p">(</span><span class="n">flag</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">Coverage</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="ss">lines: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">eval: </span><span class="kp">true</span><span class="p">)</span>
<span class="c1"># Depending on the order of these two operations, different coverage is calculated, because the evaluation of the code is considered different, even if the content/path is the same.</span>
<span class="n">measure</span><span class="p">(</span><span class="kp">false</span><span class="p">)</span>
<span class="n">measure</span><span class="p">(</span><span class="kp">true</span><span class="p">)</span>
<span class="nb">p</span> <span class="no">Coverage</span><span class="p">.</span><span class="nf">result</span>
</code></pre>
<p>Further investigation is required.</p> Ruby master - Bug #19845 (Closed): `Fiber[key] = value` fails if key is not interned.https://bugs.ruby-lang.org/issues/198452023-08-24T01:33:03Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>See <a href="https://github.com/ruby/ruby/pull/8273" class="external">https://github.com/ruby/ruby/pull/8273</a> for the fix.</p> Ruby master - Feature #19742 (Open): Introduce `Module#anonymous?`https://bugs.ruby-lang.org/issues/197422023-06-21T10:47:33Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As a follow-on <from <a href="https://bugs.ruby-lang.org/issues/19521%3E" class="external">https://bugs.ruby-lang.org/issues/19521></a>, I'd like propose we introduce <code>Module#anonymous?</code>.</p>
<p>In some situations, like logging/formatting, serialisation/deserialization, debugging or meta-programming, we might like to know if a class is a proper constant or not.</p>
<p>However, this brings about some other issues which might need to be discussed.</p>
<p>After assigning a constant, then removing it, the internal state of Ruby still believes that the class name is permanent, even thought it's no longer true.</p>
<p>e.g.</p>
<pre><code>m = Module.new
m.anonymous? # true
M = m
m.anonyomous # false
Object.send(:remove_const, :M)
M # uninitialized constant M (NameError)
m.anonymous? # false
</code></pre>
<p>Because RCLASS data structure is not updated after the constant is removed, internally the state still has a "permanent class name".</p>
<p>I want to use this proposal to discuss this issue and whether there is anything we should do about such behaviour (or even if it's desirable).</p>
<p>Proposed PR: <a href="https://github.com/ruby/ruby/pull/7966" class="external">https://github.com/ruby/ruby/pull/7966</a></p>
<p>cc <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/1241">@fxn (Xavier Noria)</a></p> Ruby master - Bug #19738 (Closed): `ObjectSpace.each_object.to_a` crashes in `make runirb`.https://bugs.ruby-lang.org/issues/197382023-06-20T02:22:45Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code>> make runirb
RUBY_ON_BUG='gdb -x ./.gdbinit -p' ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems -r irb -e 'IRB.start("make runirb")'
irb(main):001:0> ObjectSpace.each_object.to_a
Assertion Failed: ./vm_method.c:1366:callable_method_entry_or_negative:RB_TYPE_P(klass, T_CLASS) || RB_TYPE_P(klass, T_ICLASS)
ruby 3.3.0dev (2023-06-19T19:19:45Z master 9ff4399dec) [arm64-darwin22]
</code></pre>
<p>I have no idea why.</p> Ruby master - Feature #19717 (Open): `ConditionVariable#signal` is not fair when the wakeup is co...https://bugs.ruby-lang.org/issues/197172023-06-07T10:04:38Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>For background, see this issue <a href="https://github.com/socketry/async/issues/99" class="external">https://github.com/socketry/async/issues/99</a>.</p>
<p>It looks like <code>ConditionVariable#signal</code> is not fair, if the calling thread immediately reacquires the resource.</p>
<p>I've given a detailed reproduction here as it's non-trivial: <a href="https://github.com/ioquatix/ruby-condition-variable-timeout" class="external">https://github.com/ioquatix/ruby-condition-variable-timeout</a>.</p>
<p>Because the spurious wakeup occurs, the thread is pushed to the back of the waitq, which means any other waiting thread will acquire the resource, and that thread will perpetually be at the back of the queue.</p>
<p>I believe the solution is to change <code>ConditionVarialbe#signal</code> should only remove the thread from the waitq if it's possible to acquire the lock. Otherwise it should be left in place, so that the order is retained, this should result in fair scheduling.</p> Ruby master - Bug #19709 (Closed): `Thread.join(timeout)` hangs in fiber scheduler.https://bugs.ruby-lang.org/issues/197092023-06-03T01:43:14Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Unfortunately the following script can hang:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require_relative</span> <span class="s1">'test/fiber/scheduler'</span>
<span class="n">scheduler</span> <span class="o">=</span> <span class="no">Scheduler</span><span class="p">.</span><span class="nf">new</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">set_scheduler</span> <span class="n">scheduler</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">schedule</span> <span class="k">do</span>
<span class="n">thread</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span><span class="p">{</span><span class="nb">sleep</span><span class="p">}</span>
<span class="n">thread</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
<p>The termination condition in the <code>thread_join</code> is not expressed correctly.</p> Ruby master - Feature #19642 (Open): Remove vectored read/write from `io.c`.https://bugs.ruby-lang.org/issues/196422023-05-15T06:33:11Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><a href="https://github.com/socketry/async/issues/228#issuecomment-1546789910" class="external">https://github.com/socketry/async/issues/228#issuecomment-1546789910</a> is a comment from the kernel developer who tells us that <code>writev</code> is always worse than <code>write</code> system call.</p>
<p>A large amount of complexity in <code>io.c</code> comes from optional support from <code>writev</code>.</p>
<p>So, I'd like to remove support for <code>writev</code>.</p>
<p>I may try to measure the performance before/after. However it may not show much difference, except that the implementation in <code>io.c</code> can be much simpler.</p> Ruby master - Bug #19640 (Closed): `IO#puts` can generate zero length iov which can cause rb_bug ...https://bugs.ruby-lang.org/issues/196402023-05-14T11:47:05Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In the fiber scheduler, <code>IO#puts ""</code> or <code>IO#puts nil</code> can generate a zero length <code>iov</code> which causes <code>io_binwritev_internal</code> to fail since the result is zero.</p>
<p>We need to fix <code>IO#puts</code> so that it does not generate zero length writes, but also we fix <code>io_binwritev_internal</code> to handle this case more robustly.</p>
<p>Fix: <a href="https://github.com/ruby/ruby/pull/7806/files" class="external">https://github.com/ruby/ruby/pull/7806/files</a></p> Ruby master - Feature #19607 (Open): Introduce `Hash#symbolize_keys`.https://bugs.ruby-lang.org/issues/196072023-04-18T11:02:29Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>This is a very common operation.</p>
<p>It can currently be implemented using <code>Hash#transform_keys(&:to_sym)</code>.</p>
<p>It's currently provided by Rails as <code>Hash#symbolize_keys</code> and <code>Hash#symbolize_keys!</code>.</p>
<p>Proposed implementation is identical to Rails implementation: <a href="https://github.com/rails/rails/blob/539144d2d61770dab66c8643e744441e52538e09/activesupport/lib/active_support/core_ext/hash/keys.rb#L20-L37" class="external">https://github.com/rails/rails/blob/539144d2d61770dab66c8643e744441e52538e09/activesupport/lib/active_support/core_ext/hash/keys.rb#L20-L37</a></p>
<p>For completeness we could also consider adding <code>stringify_keys</code> but I think that's less frequently used.</p> Ruby master - Bug #19546 (Closed): IO::Buffer is incorrectly invoking fiber scheduler interface.https://bugs.ruby-lang.org/issues/195462023-03-25T01:48:17Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>(Original bug report: <a href="https://github.com/socketry/async/issues/224" class="external">https://github.com/socketry/async/issues/224</a>)</p>
<p>The <code>IO::Buffer</code> implementation incorrectly calls <code>rb_fiber_scheduler_io_write</code> with the <code>size_t</code>s converted to <code>VALUE</code>s: <a href="https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/io_buffer.c#L2607" class="external">https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/io_buffer.c#L2607</a></p>
<p>But that function expects <code>size_t</code>s: <a href="https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/scheduler.c#L542" class="external">https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/scheduler.c#L542</a></p>
<p>(And probably the same for the other scheduler methods + data types.)</p>
<p>Fixed by <a href="https://github.com/ruby/ruby/pull/7593" class="external">https://github.com/ruby/ruby/pull/7593</a>.</p> Ruby master - Feature #19521 (Closed): Support for `Module#name=` and `Class#name=`.https://bugs.ruby-lang.org/issues/195212023-03-09T08:59:15Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>See <a href="https://bugs.ruby-lang.org/issues/19450" class="external">https://bugs.ruby-lang.org/issues/19450</a> for previous discussion and motivation.</p>
<p><a href="https://github.com/ruby/ruby/pull/7483" class="external">This proposal</a> introduces <code>Module#name=</code> (and thus also <code>Class#name=</code>) to set the temporary class name. The name assignment has no effect if the module/class already has a permanent name.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">c</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"fake name"</span>
<span class="k">end</span>
<span class="n">c</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span>
<span class="n">c</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"fake name"</span>
</code></pre>
<p>Alternatively, we could use <code>set_name</code>:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Class</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">set_name</span> <span class="s2">"fake_name"</span>
<span class="k">end</span>
</code></pre>
<p>Setting the name of a class changes its current name, irrespective of whether it's been assigned a permanent name, or has nested modules/classes which have cached a previous name. We might like to limit the cases where a name is set, e.g. only once, only if the name is nil, or only if it's not already permanent. There is no real harm in any of those options, just inconsistency.</p>
<a name="Example-usage"></a>
<h2 >Example usage<a href="#Example-usage" class="wiki-anchor">¶</a></h2>
<p>The current Ruby test suite has code which shows the usefulness of this new method:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"> <span class="k">def</span> <span class="nf">labeled_module</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">singleton_class</span><span class="p">.</span><span class="nf">class_eval</span> <span class="p">{</span>
<span class="n">define_method</span><span class="p">(</span><span class="ss">:to_s</span><span class="p">)</span> <span class="p">{</span><span class="nb">name</span><span class="p">}</span>
<span class="k">alias</span> <span class="nb">inspect</span> <span class="nb">to_s</span>
<span class="k">alias</span> <span class="nb">name</span> <span class="nb">to_s</span>
<span class="p">}</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_module</span>
<span class="k">def</span> <span class="nf">labeled_class</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">superclass</span> <span class="o">=</span> <span class="no">Object</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">superclass</span><span class="p">)</span> <span class="k">do</span>
<span class="n">singleton_class</span><span class="p">.</span><span class="nf">class_eval</span> <span class="p">{</span>
<span class="n">define_method</span><span class="p">(</span><span class="ss">:to_s</span><span class="p">)</span> <span class="p">{</span><span class="nb">name</span><span class="p">}</span>
<span class="k">alias</span> <span class="nb">inspect</span> <span class="nb">to_s</span>
<span class="k">alias</span> <span class="nb">name</span> <span class="nb">to_s</span>
<span class="p">}</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_class</span>
</code></pre>
<p>The updated code would look like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"> <span class="k">def</span> <span class="nf">labeled_module</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="nb">name</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">labeled_class</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">superclass</span> <span class="o">=</span> <span class="no">Object</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">superclass</span><span class="p">)</span> <span class="k">do</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="nb">name</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_class</span>
</code></pre>
<p>Because the name cannot be set as part of <code>.new</code>, we have to have a separate block to set the name, before calling <code>class_eval</code>. I think the ergonomics and performance of this are slightly worse than the <a href="https://bugs.ruby-lang.org/issues/19520" class="external">counter proposal</a>.</p> Ruby master - Feature #19520 (Rejected): Support for `Module.new(name)` and `Class.new(superclass...https://bugs.ruby-lang.org/issues/195202023-03-09T08:50:18Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>See <a href="https://bugs.ruby-lang.org/issues/19450" class="external">https://bugs.ruby-lang.org/issues/19450</a> for previous discussion and motivation.</p>
<p><a href="https://github.com/ruby/ruby/pull/7376" class="external">This proposal</a> introduces the <code>name</code> parameter to <code>Class.new</code> and <code>Module.new</code>:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">superclass</span><span class="p">,</span> <span class="nb">name</span><span class="p">)</span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
</code></pre>
<p>As a slight change, we could use keyword arguments instead.</p>
<a name="Example-usage"></a>
<h2 >Example usage<a href="#Example-usage" class="wiki-anchor">¶</a></h2>
<p>The current Ruby test suite has code which shows the usefulness of this new method:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"> <span class="k">def</span> <span class="nf">labeled_module</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">singleton_class</span><span class="p">.</span><span class="nf">class_eval</span> <span class="p">{</span>
<span class="n">define_method</span><span class="p">(</span><span class="ss">:to_s</span><span class="p">)</span> <span class="p">{</span><span class="nb">name</span><span class="p">}</span>
<span class="k">alias</span> <span class="nb">inspect</span> <span class="nb">to_s</span>
<span class="k">alias</span> <span class="nb">name</span> <span class="nb">to_s</span>
<span class="p">}</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_module</span>
<span class="k">def</span> <span class="nf">labeled_class</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">superclass</span> <span class="o">=</span> <span class="no">Object</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">superclass</span><span class="p">)</span> <span class="k">do</span>
<span class="n">singleton_class</span><span class="p">.</span><span class="nf">class_eval</span> <span class="p">{</span>
<span class="n">define_method</span><span class="p">(</span><span class="ss">:to_s</span><span class="p">)</span> <span class="p">{</span><span class="nb">name</span><span class="p">}</span>
<span class="k">alias</span> <span class="nb">inspect</span> <span class="nb">to_s</span>
<span class="k">alias</span> <span class="nb">name</span> <span class="nb">to_s</span>
<span class="p">}</span>
<span class="nb">class_eval</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">block</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_class</span>
</code></pre>
<p>The updated code would look like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"> <span class="k">def</span> <span class="nf">labeled_module</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Module</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">labeled_class</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">superclass</span> <span class="o">=</span> <span class="no">Object</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">superclass</span><span class="p">,</span> <span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="k">end</span>
<span class="kp">module_function</span> <span class="ss">:labeled_class</span>
</code></pre> Ruby master - Bug #19480 (Closed): invalid keeping_mutexes: Attempt to unlock a mutex which is no...https://bugs.ruby-lang.org/issues/194802023-03-07T06:08:42Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>The following program has a race condition due to IO write locks:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require_relative</span> <span class="s1">'lib/async'</span>
<span class="k">def</span> <span class="nf">wait_for_interrupt</span><span class="p">(</span><span class="n">thread_index</span><span class="p">,</span> <span class="n">repeat</span><span class="p">)</span>
<span class="n">sequence</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">events</span> <span class="o">=</span> <span class="no">Thread</span><span class="o">::</span><span class="no">Queue</span><span class="p">.</span><span class="nf">new</span>
<span class="n">reactor</span> <span class="o">=</span> <span class="no">Async</span><span class="o">::</span><span class="no">Reactor</span><span class="p">.</span><span class="nf">new</span>
<span class="n">thread</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">events</span><span class="p">.</span><span class="nf">pop</span>
<span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">thread_index</span><span class="si">}</span><span class="s2">+</span><span class="si">#{</span><span class="n">repeat</span><span class="si">}</span><span class="s2"> Sending Interrupt!"</span>
<span class="n">reactor</span><span class="p">.</span><span class="nf">interrupt</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">reactor</span><span class="p">.</span><span class="nf">async</span> <span class="k">do</span>
<span class="n">events</span> <span class="o"><<</span> <span class="kp">true</span>
<span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">thread_index</span><span class="si">}</span><span class="s2">+</span><span class="si">#{</span><span class="n">repeat</span><span class="si">}</span><span class="s2"> Reactor ready!"</span>
<span class="c1"># Wait to be interrupted:</span>
<span class="nb">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"</span><span class="si">#{</span><span class="n">thread_index</span><span class="si">}</span><span class="s2">+</span><span class="si">#{</span><span class="n">repeat</span><span class="si">}</span><span class="s2"> Missing interrupt!"</span>
<span class="k">end</span>
<span class="n">reactor</span><span class="p">.</span><span class="nf">run</span>
<span class="n">thread</span><span class="p">.</span><span class="nf">join</span>
<span class="k">end</span>
<span class="mi">64</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">thread_index</span><span class="o">|</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="mi">1000</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">repeat</span><span class="o">|</span>
<span class="n">wait_for_interrupt</span><span class="p">(</span><span class="n">thread_index</span><span class="p">,</span> <span class="n">repeat</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">.</span><span class="nf">each</span><span class="p">(</span><span class="o">&</span><span class="ss">:join</span><span class="p">)</span>
</code></pre>
<p>The error manifests:</p>
<pre><code>58+407 Reactor ready!
58+407 Sending Interrupt!
46+396 Sending Interrupt!
58+408 Sending Interrupt!
46+397 Reactor ready!
#<Thread:0x00007f9f681ca8e0 ./test-segfault.rb:11 run> terminated with exception (report_on_exception is true):
52+425 Sending Interrupt!
/home/samuel/Projects/socketry/async/lib/async/scheduler.rb:131:in `unblock': undefined method `wakeup' for nil:NilClass (NoMethodError)
@selector.wakeup
^^^^^^^
from ./test-segfault.rb:13:in `write'
from ./test-segfault.rb:13:in `puts'
from ./test-segfault.rb:13:in `puts'
from ./test-segfault.rb:13:in `block in wait_for_interrupt'
[BUG] invalid keeping_mutexes: Attempt to unlock a mutex which is not locked
ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0001 p:---- s:0003 e:000002 DUMMY [FINISH]
-- C level backtrace information -------------------------------------------
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(rb_print_backtrace+0xd) [0x560fabaeff41] /home/samuel/src/ruby-3.2.1/vm_dump.c:785
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(rb_vm_bugreport) /home/samuel/src/ruby-3.2.1/vm_dump.c:1080
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(bug_report_end+0x0) [0x560fabca13ab] /home/samuel/src/ruby-3.2.1/error.c:790
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(rb_bug_without_die) /home/samuel/src/ruby-3.2.1/error.c:790
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(die+0x0) [0x560fab8d73b8] /home/samuel/src/ruby-3.2.1/error.c:798
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(rb_bug) /home/samuel/src/ruby-3.2.1/error.c:800
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(thread_start_func_2+0x630) [0x560faba84fa0] /home/samuel/src/ruby-3.2.1/thread.c:434
/home/samuel/.rubies/ruby-3.2.1/bin/ruby(thread_start_func_1+0xdb) [0x560faba857ab] /home/samuel/src/ruby-3.2.1/thread_pthread.c:1170
/usr/lib/libc.so.6(0x7f9f78321bb5) [0x7f9f78321bb5]
/usr/lib/libc.so.6(0x7f9f783a3d90) [0x7f9f783a3d90]
</code></pre> Ruby master - Feature #19466 (Closed): Class.new takes a block, why doesn't Module.new take a block?https://bugs.ruby-lang.org/issues/194662023-02-25T02:37:51Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Class</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="c1">#... equivalent to class_eval</span>
<span class="k">end</span>
</code></pre>
<p>So, why don't we introduce:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Module</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="c1">#... equivalent to class_eval</span>
<span class="k">end</span>
</code></pre> Ruby master - Bug #19461 (Closed): Time.local performance tanks in forked process (on macOS only?)https://bugs.ruby-lang.org/issues/194612023-02-23T11:23:10Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>The following program demonstrates a performance regression in forked child processes when invoking <code>Time.local</code>:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'benchmark'</span>
<span class="nb">require</span> <span class="s1">'time'</span>
<span class="k">def</span> <span class="nf">sir_local_alot</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">Benchmark</span><span class="p">.</span><span class="nf">measure</span> <span class="k">do</span>
<span class="mi">10_000</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span>
<span class="n">tm</span> <span class="o">=</span> <span class="o">::</span><span class="no">Time</span><span class="p">.</span><span class="nf">local</span><span class="p">(</span><span class="mi">2023</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="vg">$stderr</span><span class="p">.</span><span class="nf">puts</span> <span class="n">result</span>
<span class="k">end</span>
<span class="n">sir_local_alot</span>
<span class="n">pid</span> <span class="o">=</span> <span class="nb">fork</span> <span class="k">do</span>
<span class="n">sir_local_alot</span>
<span class="k">end</span>
<span class="no">Process</span><span class="p">.</span><span class="nf">wait</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
</code></pre>
<p>On Linux the performance is similar, but on macOS, the performance is over 100x worse on my M1 laptop.</p> Ruby master - Feature #19453 (Closed): Move `Fiber.current` into core.https://bugs.ruby-lang.org/issues/194532023-02-20T23:48:10Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>i.e. don't <code>require 'fiber'</code> to use <code>Fiber.current</code>. Are there other methods we should consider too?</p> Ruby master - Feature #19452 (Open): `Thread::Backtrace::Location` should have column information...https://bugs.ruby-lang.org/issues/194522023-02-20T07:06:22Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I discussed this with <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/18">@mame (Yusuke Endoh)</a> and it would be pretty useful if we could also get the column information from exception backtrace location, even if it was slow.</p>
<p>A POC:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Thread::Backtrace::Location</span>
<span class="k">if</span> <span class="k">defined?</span><span class="p">(</span><span class="no">RubyVM</span><span class="o">::</span><span class="no">AbstractSyntaxTree</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">first_column</span>
<span class="no">RubyVM</span><span class="o">::</span><span class="no">AbstractSyntaxTree</span><span class="p">.</span><span class="nf">of</span><span class="p">(</span><span class="nb">self</span><span class="p">,</span> <span class="ss">keep_script_lines: </span><span class="kp">true</span><span class="p">).</span><span class="nf">first_column</span>
<span class="k">end</span>
<span class="k">else</span>
<span class="k">def</span> <span class="nf">first_column</span>
<span class="k">raise</span> <span class="no">NotImplementedError</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>It would be good to have a standard interface, so we follow the same interface as <a href="https://bugs.ruby-lang.org/issues/19451" class="external">https://bugs.ruby-lang.org/issues/19451</a> and vice versa where it makes sense. I'll investigate it.</p> Ruby master - Feature #19451 (Open): Extract path and line number from SyntaxError?https://bugs.ruby-lang.org/issues/194512023-02-20T06:58:04Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There doesn't seem to be any official way to extract the path and line number from a syntax error.</p>
<p>There are two ways I can see us dong this:</p>
<ul>
<li>Provide explicit <code>path</code> and <code>line_number</code> attributes.</li>
<li>Prepend them to <code>backtrace_locations</code>.</li>
</ul>
<p>The nice thing about the latter approach is that it will just work with existing tools which understand how to highlight code based on backtrace locations.</p>
<p>Maybe we should do both?</p> Ruby master - Feature #19450 (Closed): Is there an official way to set a class name without setti...https://bugs.ruby-lang.org/issues/194502023-02-20T01:04:30Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>This is the best I could come up with:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">klass</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span>
<span class="no">Object</span><span class="p">.</span><span class="nf">const_set</span><span class="p">(</span><span class="s2">"Klass"</span><span class="p">,</span> <span class="n">klass</span><span class="p">)</span>
<span class="no">Object</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="ss">:remove_const</span><span class="p">,</span> <span class="s2">"Klass"</span><span class="p">)</span>
<span class="nb">puts</span> <span class="n">klass</span><span class="p">.</span><span class="nf">new</span>
<span class="c1"># => #<Klass:0x0000000100a9d688></span>
</code></pre>
<p>Can we do better?</p>
<p>What about something like:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">name: </span><span class="s2">"Klass"</span><span class="p">)</span>
</code></pre>
<p>or</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Class</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">name</span>
<span class="s2">"Klass"</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>etc</p> Ruby master - Misc #19421 (Open): Distribution documentationhttps://bugs.ruby-lang.org/issues/194212023-02-07T05:03:49Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I use Ruby a lot, on a lot of different systems, and help people and companies use it, including developers who install it on their systems.</p>
<p>Over time, I found that installing Ruby isn't always easy. Part of this is due to package management. There are many systems, and Ruby has had some tricky migrations (e.g. OpenSSL is probably one of the most painful ones that lots of developers have trouble with).</p>
<p>Arch Linux has been stuck on Ruby 3.0 for a long time, which could be considered surprising given that Arch Linux is often on the bleeding edge of releases. I personally use Arch too. So I decided to ask, what is holding them up from making a release?</p>
<p>I found out they had many questions about how to distribute Ruby correctly. When I listened to those questions I felt that there are many ambiguities in how we build and package Ruby for operating system packages. This isn't to say that there isn't a good way to do it, just that we as a core team might be able to improve our communication about how Ruby is evolving and the implications for package managers (if any).</p>
<p>I've introduced <code>doc/distribution.md</code> as an effort to start having better documentation for people distributing Ruby. There are many ways which people distribute Ruby, and many "partial" documentation or assumptions being made about how to distribute Ruby and I'd like to provide a convenient standard location that can help people build package for Ruby distribution. Ultimately this makes my job easier because the latest versions of Ruby will be easier to install, so that's what I care about, but I don't care about what specifically is in the document, except that I think we should listen to the kinds of questions being asked and, in the best interest of Ruby, provide guidance.</p>
<p>There was a lot of good discussion on the PR, but my goal is not to make a finished document, but instead plant the seed so it can grow.</p>
<p><a href="https://github.com/ruby/ruby/pull/6856" class="external">https://github.com/ruby/ruby/pull/6856</a></p>
<p>Some follow up discussion is required:</p>
<ul>
<li>
<p>What is the best practice for building source packages. The documentation I wrote from this was removed as "out of scope" but I disagree with that (<a href="https://github.com/ruby/ruby/commit/c35ebed895e1a3f7bced3db50ea0db8f284744e8" class="external">https://github.com/ruby/ruby/commit/c35ebed895e1a3f7bced3db50ea0db8f284744e8</a>). I don't have a strong opinion about what it should look like, but I think we should give a clear example of how to build source packages like what I wrote.</p>
</li>
<li>
<p>Related to the above, what is the official location for source tarballs?</p>
</li>
<li>
<p>What optional dependencies are required for building vs distributing Ruby. Arch Linux currently lists: <code>doxygen gdbm graphviz libffi libyaml openssl ttf-dejavu tk</code> as dependencies but it's not clear if this list is up to date, or what the expectations are. When someone installs Ruby (e.g. <code>apt-get install ruby</code>) what dependencies should be installed? I think it would be helpful to list expected dependencies (from a system package POV), etc.</p>
</li>
<li>
<p>Is Ruby needed for building Ruby? Should source packages install Ruby before building from source? If so, what versions are supported?</p>
</li>
<li>
<p>Clear guidance on gems that are distributed an alongside Ruby, and how security changes to gems are managed.</p>
</li>
</ul>
<p>Even if we have more detailed documentation elsewhere, let's summarise it and then cross-reference it. People who build packages to distribute and install Ruby should feel supported, they are very important to our community. To this end, I established <code>#distribution</code> channel on Slack for discussion. We should listen to the questions being asked and use those questions to drive improvements to documentation.</p> Ruby master - Bug #19389 (Closed): StringIO gets(..., chomp: true) behaves differently to File/IO.https://bugs.ruby-lang.org/issues/193892023-01-28T10:05:28Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It seems like <code>StringIO</code> and <code>File</code> <code>#gets(..., chomp: true)</code> have different behaviour.</p>
<pre><code>irb(main):003:0> io = StringIO.new("foo\r\nbar\r\n\r\n")
=> #<StringIO:0x00000001077e2018>
irb(main):004:0> io.gets("\r\n", chomp: true)
=> "foo"
irb(main):005:0> io.gets("\r\n", chomp: true)
=> "bar"
irb(main):006:0> io.gets("\r\n", chomp: true)
=> "\r\n"
</code></pre>
<p>compared with File:</p>
<pre><code>irb(main):008:0> io = Tempfile.new
=> #<File:/var/folders/cm/p8lkj40s0vz_vgx_b5df9dkm0000gn/T/20230128-37714-107ims>
irb(main):009:0> io.write("foo\r\nbar\r\n\r\n")
=> 12
irb(main):012:0> io.rewind
=> 0
irb(main):013:0> io.gets("\r\n", chomp: true)
=> "foo"
irb(main):014:0> io.gets("\r\n", chomp: true)
=> "bar"
irb(main):015:0> io.gets("\r\n", chomp: true)
=> ""
</code></pre> Ruby master - Bug #19360 (Closed): Enabling coverage with `-r` option isn't sufficient to interce...https://bugs.ruby-lang.org/issues/193602023-01-21T11:10:48Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>For some reason, Ruby's coverage library doesn't work when the file is loaded from the command line. In the below example, test2.rb loads test.rb. If you run test2.rb with coverage enabled, it will report coverage for test.rb but not test2.rb. If you run test.rb directly, no coverage is reported.</p>
<pre><code>samuel@aiko ~/P/i/autocoverage> ruby -r "./autocoverage.rb" test.rb
Hello World
{}
samuel@aiko ~/P/i/autocoverage> ruby -r "./autocoverage.rb" test2.rb
Hello World
{"/home/samuel/Projects/ioquatix/autocoverage/test.rb"=>{:lines=>[1, 1, nil, nil, 1], :branches=>{}, :methods=>{[Object, :main, 1, 0, 3, 3]=>1}}}
</code></pre>
<p>The same problem affects simplecov.</p>
<pre><code>ruby -r "./simplecov.rb" test.rb
... similar results in coverage directory ...
</code></pre>
<p>See <a href="https://github.com/ioquatix/autocoverage" class="external">https://github.com/ioquatix/autocoverage</a> for a complete reproduction.</p> Ruby master - Feature #19333 (Open): Setting (Fiber Local|Thread Local|Fiber Storage) to nil shou...https://bugs.ruby-lang.org/issues/193332023-01-11T22:21:06Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As it stands, Fiber Locals, Thread Locals and Fiber Storage have no way of deleting key-value associations.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="mi">100</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="nb">name</span> <span class="o">=</span> <span class="ss">:"variable-</span><span class="si">#{</span><span class="n">i</span><span class="si">}</span><span class="ss">"</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">[</span><span class="nb">name</span><span class="p">]</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">end</span>
</code></pre>
<p>Because of this, dynamically generated associations can leak over time. This is worse for things like Threads that might be pooled (or maybe an argument against user-space pooling).</p>
<p>In any case, having a way to delete those associations would allow application code to at least delete the associations when they no longer make sense.</p>
<p>I propose that assigning <code>nil</code> to "locals" or "storage" should effectively delete them.</p>
<p>e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="mi">100</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="nb">name</span> <span class="o">=</span> <span class="ss">:"variable-</span><span class="si">#{</span><span class="n">i</span><span class="si">}</span><span class="ss">"</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">[</span><span class="nb">name</span><span class="p">]</span> <span class="o">=</span> <span class="mi">10</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">[</span><span class="nb">name</span><span class="p">]</span> <span class="o">=</span> <span class="kp">nil</span> <span class="c1"># delete association</span>
<span class="k">end</span>
</code></pre>
<p>A more invasive alternative would be to define new interfaces like <code>Thread::Local</code>, <code>Fiber::Local</code> and <code>Fiber::Storage::Local</code> (or something) which correctly clean up on GC.</p> Ruby master - Feature #19306 (Open): Expand zlib interfacehttps://bugs.ruby-lang.org/issues/193062023-01-04T07:02:07Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Introduce some extra methods like <code>window_size</code>, <code>level</code> (compression/decompression), enhance <code>inspect</code> and various other quality of life improvements.</p> Ruby master - Feature #19135 (Closed): Support `UNIXSocket` on Windowshttps://bugs.ruby-lang.org/issues/191352022-11-17T20:09:27Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In recent versions of Windows, the required parts are now supported: <a href="https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/" class="external">https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/</a></p>
<p>This enables UNIXSocket, UNIXServer and UNIXSocket.pair on Windows. The anonymous socket is emulated on Windows using a temporary file, but it's good enough and the best we can do. The semantics is not completely identical.</p>
<p>Because these constants are now available, more tests are run on Windows.</p> Ruby master - Bug #19101 (Closed): madvise(free) was broken in 3.1?https://bugs.ruby-lang.org/issues/191012022-11-03T19:35:45Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><a href="https://github.com/ruby/ruby/commit/77f3319071e600a2aafaa9863b892dfd3c1da343#r88774579" class="external">https://github.com/ruby/ruby/commit/77f3319071e600a2aafaa9863b892dfd3c1da343#r88774579</a></p>
<p>We need to investigate why this was done, how to fix it, and how to prevent regressions in the future.</p> Ruby master - Feature #19094 (Closed): `sleep(nil)` vs `sleep()` and replicating the default impl...https://bugs.ruby-lang.org/issues/190942022-10-31T05:01:04Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code>> sleep(nil)
(irb):1:in `sleep': can't convert NilClass into time interval (TypeError)
</code></pre>
<p>However, I feel that this makes implementing a compatible sleep method a little difficult.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">sleep</span><span class="p">(</span><span class="n">time</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
<span class="k">if</span> <span class="n">time</span>
<span class="n">sleep_some</span><span class="p">(</span><span class="n">time</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">sleep_forever</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>Can we consider allowing <code>sleep(nil)</code> and <code>sleep(false)</code> to be the same as <code>sleep()</code> to simplify this behaviour?</p>
<p>Otherwise, it's hard to proxy (e.g. fiber scheduler, or even just a plain old variable argument).</p>
<p>e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Sleeper</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">time</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
<span class="vi">@time</span> <span class="o">=</span> <span class="n">time</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">sleep</span>
<span class="no">Kernel</span><span class="o">::</span><span class="nb">sleep</span><span class="p">(</span><span class="vi">@time</span><span class="p">)</span> <span class="c1"># Hard to get the behaviour of `sleep()` here without explicitly handling/checking.</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre> Ruby master - Feature #19078 (Closed): Introduce `Fiber#storage` for inheritable fiber-scoped var...https://bugs.ruby-lang.org/issues/190782022-10-22T21:56:41Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Pull Request: <a href="https://github.com/ruby/ruby/pull/6612" class="external">https://github.com/ruby/ruby/pull/6612</a></p>
<p>This is an evolution of the previous ideas:</p>
<ul>
<li><a href="https://bugs.ruby-lang.org/issues/19058" class="external">https://bugs.ruby-lang.org/issues/19058</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/19062" class="external">https://bugs.ruby-lang.org/issues/19062</a></li>
</ul>
<p>This PR introduces fiber scoped variables, and is a solution for problems like <a href="https://github.com/ioquatix/ioquatix/discussions/17" class="external">https://github.com/ioquatix/ioquatix/discussions/17</a>.</p>
<p>The main interface is:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="no">Fiber</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="c1"># => value</span>
</code></pre>
<p>The variables are scoped (local to) a fiber and inherited into child fibers and threads.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">[</span><span class="ss">:request_id</span><span class="p">]</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">hex</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="nb">p</span> <span class="no">Fiber</span><span class="p">[</span><span class="ss">:request_id</span><span class="p">]</span> <span class="c1"># prints the above request id</span>
<span class="k">end</span>
</code></pre>
<p>The fiber scoped variables are stored and can be accessed:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">storage</span> <span class="c1"># => returns a Hash (copy) of the internal storage.</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">storage</span><span class="o">=</span> <span class="c1"># => assigns a Hash (copy) to the internal storage.</span>
</code></pre>
<p>Fiber itself has one new keyword argument:</p>
<pre><code>Fiber.new(..., storage: hash, false, undef, nil)
</code></pre>
<p>This can control how the fiber variables are setup in a child context.</p>
<p>To minimise the performance overhead of some of the implementation choices, we are also simultaneously implementing <a href="https://bugs.ruby-lang.org/issues/19077" class="external">https://bugs.ruby-lang.org/issues/19077</a>.</p>
<a name="Examples"></a>
<h2 >Examples<a href="#Examples" class="wiki-anchor">¶</a></h2>
<a name="Request-loop"></a>
<h3 >Request loop<a href="#Request-loop" class="wiki-anchor">¶</a></h3>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="k">while</span> <span class="n">request</span> <span class="o">=</span> <span class="n">queue</span><span class="p">.</span><span class="nf">pop</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">storage: </span><span class="p">{</span><span class="ss">id: </span><span class="no">SecureRandom</span><span class="p">.</span><span class="nf">hex</span><span class="p">(</span><span class="mi">16</span><span class="p">)})</span> <span class="k">do</span>
<span class="n">handle_request</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>OR</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="k">while</span> <span class="n">request</span> <span class="o">=</span> <span class="n">queue</span><span class="p">.</span><span class="nf">pop</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">storage</span> <span class="o">=</span> <span class="p">{</span><span class="ss">id: </span><span class="no">SecureRandom</span><span class="p">.</span><span class="nf">hex</span><span class="p">(</span><span class="mi">16</span><span class="p">)}</span>
<span class="n">handle_request</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre> Ruby master - Feature #19077 (Open): Introduce `Hash#dup` copy on write.https://bugs.ruby-lang.org/issues/190772022-10-22T21:55:25Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Pull Request: <a href="https://github.com/ruby/ruby/pull/6613" class="external">https://github.com/ruby/ruby/pull/6613</a>.</p> Ruby master - Feature #19062 (Closed): Introduce `Fiber#locals` for shared inheritable state.https://bugs.ruby-lang.org/issues/190622022-10-16T10:31:04Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>After exploring <a href="https://bugs.ruby-lang.org/issues/19058" class="external">https://bugs.ruby-lang.org/issues/19058</a>, I felt uncomfortable about the performance of copying lots of inheritable attributes. Please review that issue for the background and summary of the problem.</p>
<a name="Proposal"></a>
<h2 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h2>
<p>Introduce <code>Fiber#locals</code> which is a hash table of local attributes which are inherited by child fibers.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">locals</span><span class="p">[</span><span class="ss">:x</span><span class="p">]</span> <span class="o">=</span> <span class="mi">10</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">pp</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">locals</span><span class="p">[</span><span class="ss">:x</span><span class="p">]</span> <span class="c1"># => 10</span>
<span class="k">end</span>
</code></pre>
<p>It's possible to reset <code>Fiber.current.locals</code>, e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">accept_connection</span><span class="p">(</span><span class="n">peer</span><span class="p">)</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">locals: </span><span class="kp">nil</span><span class="p">)</span> <span class="k">do</span> <span class="c1"># This causes a new hash table to be allocated.</span>
<span class="c1"># Generate a new request id for all fibers nested in this one:</span>
<span class="no">Fiber</span><span class="p">[</span><span class="ss">:request_id</span><span class="p">]</span> <span class="o">=</span> <span class="no">SecureRandom</span><span class="p">.</span><span class="nf">hex</span><span class="p">(</span><span class="mi">32</span><span class="p">)</span>
<span class="vi">@app</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="k">end</span><span class="p">.</span><span class="nf">resume</span>
<span class="k">end</span>
</code></pre>
<p>A high level overview of the proposed changes:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Fiber</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="o">...</span><span class="p">,</span> <span class="ss">locals: </span><span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">locals</span><span class="p">)</span>
<span class="vi">@locals</span> <span class="o">=</span> <span class="n">locals</span> <span class="o">||</span> <span class="no">Hash</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="nb">attr_accessor</span> <span class="ss">:locals</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">[]</span> <span class="n">key</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">locals</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">[]=</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">locals</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>See the pull request <a href="https://github.com/ruby/ruby/pull/6566" class="external">https://github.com/ruby/ruby/pull/6566</a> for the full proposed implementation.</p>
<a name="Expected-Usage"></a>
<h2 >Expected Usage<a href="#Expected-Usage" class="wiki-anchor">¶</a></h2>
<p>Currently, a lot of libraries use <code>Thread.current[:x]</code> which is unexpectedly "fiber local". A common bug shows up when lazy enumerators are used, because it may create an internal fiber. Because <code>locals</code> are inherited, code which uses <code>Fiber[:x]</code> will not suffer from this problem.</p>
<p>Any program that uses true thread locals for per-request state, can adopt the proposed <code>Fiber#locals</code> and get similar behaviour, without breaking on per-fiber servers like Falcon, because Falcon can "reset" <code>Fiber.current.locals</code> for each request fiber, while servers like Puma won't have to do that and will retain thread-local behaviour.</p>
<p>Libraries like ActiveRecord can adopt <code>Fiber#locals</code> to avoid the need for users to opt into different "IsolatedExecutionState" models, since it can be transparently handled by the web server (see <a href="https://github.com/rails/rails/pull/43596" class="external">https://github.com/rails/rails/pull/43596</a> for more details).</p>
<p>We hope by introducing <code>Fiber#locals</code>, we can avoid all the confusion and bugs of the past designs.</p> Ruby master - Feature #19060 (Closed): Introduce `Fiber::Scheduler#io_select` for non-blocking `I...https://bugs.ruby-lang.org/issues/190602022-10-15T05:08:02Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It's relatively straight forward to implement the interface for the hook, but the implementation is tricky. However, I would prefer if we at least had some basic support for this, as it's still fairly common in existing code.</p>
<p>Pull Request: <a href="https://github.com/ruby/ruby/pull/6559" class="external">https://github.com/ruby/ruby/pull/6559</a></p> Ruby master - Feature #19059 (Open): Introduce top level `module TimeoutError` for aggregating va...https://bugs.ruby-lang.org/issues/190592022-10-15T03:04:07Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>This proposal was originally part of <a href="https://bugs.ruby-lang.org/issues/18630" class="external">https://bugs.ruby-lang.org/issues/18630</a> but was removed because we could not decide on the name.</p>
<p>Introduce the following:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">module</span> <span class="nn">TimeoutError</span>
<span class="k">end</span>
<span class="no">IO</span><span class="o">::</span><span class="no">TimeoutError</span><span class="p">.</span><span class="nf">include</span><span class="p">(</span><span class="no">TimeoutError</span><span class="p">)</span>
<span class="no">Regexp</span><span class="o">::</span><span class="no">TimeoutError</span><span class="p">.</span><span class="nf">include</span><span class="p">(</span><span class="no">TimeoutError</span><span class="p">)</span>
<span class="c1"># Maybe?</span>
<span class="no">Timeout</span><span class="o">::</span><span class="no">Error</span><span class="p">.</span><span class="nf">include</span><span class="p">(</span><span class="no">TimeoutError</span><span class="p">)</span>
</code></pre>
<p>It may be easier for users.</p>
<p>This was discussed before with the following conclusion:</p>
<ul>
<li>Top level <code>TimeoutError</code> is available.</li>
<li>Using a module for a <code>TimeoutError</code> may not be consistent with other top level <code>class #{thing}Error</code>.</li>
</ul> Ruby master - Feature #19058 (Closed): Introduce `Fiber.inheritable` attributes/variables for dea...https://bugs.ruby-lang.org/issues/190582022-10-15T02:52:09Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There are many instances of programs using globals, thread locals and fiber locals for shared state. Unfortunately these programs often have bugs or unusual behaviour when objects are used on different threads or fibers.</p>
<p>Here is a simple example of the kind of problem that can occur:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Users</span>
<span class="k">def</span> <span class="nf">each</span>
<span class="k">return</span> <span class="n">to_enum</span> <span class="k">unless</span> <span class="nb">block_given?</span>
<span class="nb">p</span> <span class="ss">each: </span><span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span>
<span class="k">yield</span> <span class="s2">"A"</span>
<span class="k">yield</span> <span class="s2">"B"</span>
<span class="k">yield</span> <span class="s2">"C"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="nb">p</span> <span class="no">Users</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">each</span><span class="p">.</span><span class="nf">zip</span><span class="p">(</span><span class="no">Users</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">each</span><span class="p">)</span>
</code></pre>
<p>When the enumeration depends on a fiber-local connection instance (e.g. <code>Thread.current[:connection]</code>) it could unexpectedly break the operation because within the enumerate, the fiber is different leading to missing connection.</p>
<p>In all these cases, the problem can be solved by not relying on implicit/invisible state. Unfortunately, many programs take advantage of process (global), thread or fiber local variables to create more ergonomic interfaces, e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">DB</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">,</span> <span class="n">host</span><span class="p">)</span>
<span class="no">DB</span><span class="p">.</span><span class="nf">query</span><span class="p">(</span><span class="s2">"SELECT * FROM BUGS"</span><span class="p">);</span> <span class="c1"># implicit dependency on connection (ideally from shared pool).</span>
</code></pre>
<p>Ruby provides several, somewhat confusing options.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">[</span><span class="ss">:x</span><span class="p">]</span> <span class="c1"># Fiber local.</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">thread_variable_get</span><span class="p">(</span><span class="ss">:x</span><span class="p">)</span> <span class="c1"># Thread local.</span>
<span class="k">class</span> <span class="nc">Thread</span>
<span class="kp">attr</span> <span class="ss">:x</span>
<span class="k">end</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">x</span> <span class="c1"># thread local</span>
<span class="k">class</span> <span class="nc">Fiber</span>
<span class="kp">attr</span> <span class="ss">:x</span>
<span class="k">end</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">x</span> <span class="c1"># fiber local</span>
</code></pre>
<p>Over the years there have been multiple issues about the above behaviour:</p>
<ul>
<li><a href="https://bugs.ruby-lang.org/issues/1717" class="external">https://bugs.ruby-lang.org/issues/1717</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/7097" class="external">https://bugs.ruby-lang.org/issues/7097</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/8215" class="external">https://bugs.ruby-lang.org/issues/8215</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/13893" class="external">https://bugs.ruby-lang.org/issues/13893</a></li>
<li>Rails is also working around this issue: <a href="https://github.com/rails/rails/pull/43596" class="external">https://github.com/rails/rails/pull/43596</a>
</li>
</ul>
<p>The heart of the issue is: there should be a consistent and convenient way to define state attached to the current execution context, which gets inherited by child execution contexts. Essentially:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">let</span><span class="p">(</span><span class="ss">x: </span><span class="mi">10</span><span class="p">)</span> <span class="k">do</span>
<span class="c1"># In every execution context created within this block, unless otherwise changed, `x` is bound to the value 10. That means, threads, fibers, etc.</span>
<span class="k">end</span>
</code></pre>
<p>This is sometimes referred to as dynamic scope. This proposal is to introduce a similar dynamic scope for fiber inheritable attributes.</p>
<a name="Proposal"></a>
<h2 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h2>
<p>We have several units of execution in Ruby, implicitly a process, which has threads which has fibers. Internally, Ruby has an "execution context". Each fiber has an execution context. Each thread has a main fiber and execution context. I propose to introduce an interface for defining a per-fiber context which is semantically very similar to dynamic variables. True dynamic variables are stack frame scoped, but my proposal doesn't go that far. This proposal is similar to Kotlin's Coroutine Contexts <a href="https://kotlinlang.org/docs/coroutine-context-and-dispatchers.html" class="external">https://kotlinlang.org/docs/coroutine-context-and-dispatchers.html</a> and JEP 429: Extent-Local Variables <a href="https://openjdk.org/jeps/429" class="external">https://openjdk.org/jeps/429</a>.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'fiber'</span>
<span class="c1"># Compatible shim to introduce proposed behaviour:</span>
<span class="k">class</span> <span class="nc">Fiber</span>
<span class="k">module</span> <span class="nn">Inheritable</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="k">super</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">inherit_attributes_from</span><span class="p">(</span><span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">prepended</span><span class="p">(</span><span class="n">klass</span><span class="p">)</span>
<span class="n">klass</span><span class="p">.</span><span class="nf">extend</span><span class="p">(</span><span class="no">Singleton</span><span class="p">)</span>
<span class="n">klass</span><span class="p">.</span><span class="nf">instance_variable_set</span><span class="p">(</span><span class="ss">:@inheritable_attributes</span><span class="p">,</span> <span class="no">Hash</span><span class="p">.</span><span class="nf">new</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">module</span> <span class="nn">Singleton</span>
<span class="k">def</span> <span class="nf">inheritable</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="ss">default: </span><span class="kp">nil</span><span class="p">)</span>
<span class="vi">@inheritable_attributes</span><span class="p">[</span><span class="ss">:"@</span><span class="si">#{</span><span class="n">key</span><span class="si">}</span><span class="ss">"</span><span class="p">]</span> <span class="o">=</span> <span class="n">default</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">inheritable_attributes</span>
<span class="vi">@inheritable_attributes</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">inherit_attributes_from</span><span class="p">(</span><span class="n">fiber</span><span class="p">)</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">inheritable_attributes</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="nb">name</span><span class="p">,</span> <span class="n">default</span><span class="o">|</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">fiber</span><span class="p">.</span><span class="nf">instance_variable_get</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span> <span class="o">||</span> <span class="n">default</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">instance_variable_set</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">unless</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">respond_to?</span><span class="p">(</span><span class="ss">:inheritable</span><span class="p">)</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">prepend</span><span class="p">(</span><span class="no">Fiber</span><span class="o">::</span><span class="no">Inheritable</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
<p>This allows you to write the following code:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Fiber</span>
<span class="n">inheritable</span> <span class="nb">attr_accessor</span> <span class="ss">:connection</span>
<span class="k">end</span>
<span class="c1"># When lazy enumerator creates internal fiber, the connection and related state will be inherited correctly:</span>
<span class="nb">p</span> <span class="no">User</span><span class="p">.</span><span class="nf">first</span><span class="p">(</span><span class="mi">100</span><span class="p">).</span><span class="nf">find_each</span><span class="p">.</span><span class="nf">zip</span><span class="p">(</span><span class="no">Post</span><span class="p">.</span><span class="nf">first</span><span class="p">(</span><span class="mi">100</span><span class="p">).</span><span class="nf">find_each</span><span class="p">)</span>
</code></pre>
<p>This proposed implementation was discussed here too: <a href="https://github.com/socketry/fiber-local/pull/1" class="external">https://github.com/socketry/fiber-local/pull/1</a>.</p>
<p>Some open questions:</p>
<ul>
<li>Should we introduce the same interface for Thread?</li>
<li>(or) Should Thread.new's main fiber inherit from the current Fiber?</li>
<li>Kotlin's coroutine context can be shared but is also immutable. This makes it a little bit harder to use. Should we consider their design more closely?</li>
<li>Should we have options to bypass inheriting attributes? e.g. <code>Fiber.new(inherit_attributes_from: ...)</code>.</li>
</ul> Ruby master - Feature #19057 (Assigned): Hide implementation of `rb_io_t`.https://bugs.ruby-lang.org/issues/190572022-10-15T01:54:57Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In order to make improvements to the IO implementation like <a href="https://bugs.ruby-lang.org/issues/18455" class="external">https://bugs.ruby-lang.org/issues/18455</a>, we need to add new fields to <code>struct rb_io_t</code>.</p>
<p>By the way, ending types in <code>_t</code> is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop <code>_t</code> where possible during this conversion.</p>
<p>Anyway, we should try to hide the implementation of <code>struct rb_io</code>. Ideally, we don't expose any of it, but the problem is backwards compatibility.</p>
<p>So, in order to remain backwards compatibility, we should expose some fields of <code>struct rb_io</code>, the most commonly used one is <code>fd</code> and <code>mode</code>, but several others are commonly used.</p>
<p>There are many fields which should not be exposed because they are implementation details.</p>
<a name="Current-proposal"></a>
<h2 >Current proposal<a href="#Current-proposal" class="wiki-anchor">¶</a></h2>
<p>The current proposed change <a href="https://github.com/ruby/ruby/pull/6511" class="external">https://github.com/ruby/ruby/pull/6511</a> creates two structs:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="c1">// include/ruby/io.h</span>
<span class="cp">#ifndef RB_IO_T
</span><span class="k">struct</span> <span class="n">rb_io</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">fd</span><span class="p">;</span>
<span class="c1">// ... public fields ...</span>
<span class="p">};</span>
<span class="cp">#else
</span><span class="k">struct</span> <span class="n">rb_io</span><span class="p">;</span>
<span class="cp">#endif
</span>
<span class="c1">// internal/io.h</span>
<span class="cp">#define RB_IO_T
</span><span class="k">struct</span> <span class="n">rb_io</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">fd</span><span class="p">;</span>
<span class="c1">// ... public fields ...</span>
<span class="c1">// ... private fields ...</span>
<span class="p">};</span>
</code></pre>
<p>However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/772">@Eregon (Benoit Daloze)</a> + <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/73">@tenderlovemaking (Aaron Patterson)</a> have both given some kind of approval.</p>
<p>That being said, maybe it's not safe.</p>
<p>There are two alternatives:</p>
<a name="Hide-all-details"></a>
<h2 >Hide all details<a href="#Hide-all-details" class="wiki-anchor">¶</a></h2>
<p>We can make public <code>struct rb_io</code> completely invisible.</p>
<pre><code class="c syntaxhl" data-language="c"><span class="c1">// include/ruby/io.h</span>
<span class="cp">#define RB_IO_HIDDEN
</span><span class="k">struct</span> <span class="n">rb_io</span><span class="p">;</span>
<span class="kt">int</span> <span class="nf">rb_ioptr_descriptor</span><span class="p">(</span><span class="k">struct</span> <span class="n">rb_io</span> <span class="o">*</span><span class="n">ioptr</span><span class="p">);</span> <span class="c1">// accessor for previously visible state.</span>
<span class="c1">// internal/io.h</span>
<span class="k">struct</span> <span class="n">rb_io</span> <span class="p">{</span>
<span class="c1">// ... all fields ...</span>
<span class="p">};</span>
</code></pre>
<p>This would only be forwards compatible, and code would need to feature detect like this:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="cp">#ifdef RB_IO_HIDDEN
#define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor
#else
#define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr)
#endif
</span></code></pre>
<a name="Nested-public-interface"></a>
<h2 >Nested public interface<a href="#Nested-public-interface" class="wiki-anchor">¶</a></h2>
<p>Alternatively, we can nest the public fields into the private struct:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="c1">// include/ruby/io.h</span>
<span class="k">struct</span> <span class="n">rb_io_public</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">fd</span><span class="p">;</span>
<span class="c1">// ... public fields ...</span>
<span class="p">};</span>
<span class="c1">// internal/io.h</span>
<span class="cp">#define RB_IO_T
</span><span class="k">struct</span> <span class="n">rb_io</span> <span class="p">{</span>
<span class="k">struct</span> <span class="n">rb_io_public</span> <span class="n">public</span><span class="p">;</span>
<span class="c1">// ... private fields ...</span>
<span class="p">};</span>
</code></pre>
<a name="Considerations"></a>
<h2 >Considerations<a href="#Considerations" class="wiki-anchor">¶</a></h2>
<p>I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us.</p>
<p>I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing.</p> Ruby master - Feature #19056 (Open): Introduce `Fiber.annotation` for attaching messages to fibers.https://bugs.ruby-lang.org/issues/190562022-10-14T23:09:31Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It's useful to know what a fiber is doing especially when they have a temporal execution (i.e. sockets connecting vs connected, binding vs accepting, queue popping, etc)</p>
<p>Let's introduce <code>Fiber.annotate</code> and <code>Fiber#annotation</code> for logging a short message attached to Fibers.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">.</span><span class="nf">annotate</span> <span class="s2">"Counting to 10"</span>
<span class="mi">10</span><span class="p">.</span><span class="nf">times</span><span class="p">{</span><span class="o">|</span><span class="no">I</span><span class="o">|</span> <span class="nb">puts</span> <span class="no">I</span><span class="p">}</span>
<span class="c1"># Fiber.current.annotation => "Counting to 10"</span>
</code></pre>
<p>Pull Request: <a href="https://github.com/ruby/ruby/pull/6554" class="external">https://github.com/ruby/ruby/pull/6554</a></p> Ruby master - Bug #19026 (Closed): Add `Coverage.supported?(x)` to detect support for `eval` cove...https://bugs.ruby-lang.org/issues/190262022-09-28T10:52:40Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Introduce the following interface:</p>
<pre><code>Coverage.supported?(eval) -> true/false
</code></pre>
<p>We can also check <code>lines</code> <code>branches</code> and <code>methods</code>?</p> Ruby master - Feature #19019 (Open): Nicely formatted exception messages in HTMLhttps://bugs.ruby-lang.org/issues/190192022-09-23T07:52:32Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>We have made a lot of improvements to exception formatting. See <a href="https://bugs.ruby-lang.org/issues/18296" class="external">https://bugs.ruby-lang.org/issues/18296</a> for details.</p>
<p>I'd like us to consider adding support for HTML formatting of messages, e.g.</p>
<pre><code>exception.full_message(highlight: :html)
</code></pre>
<p>or something to that effect.</p>
<p>Another option is to convert terminal style errors to html by converting control characters. However, I'm less optimistic about parsing sequences like <code>^^^^^</code> to apply a style to the above line.</p>
<p>Maybe it's sufficient when <code>highlight: true</code> is specified to avoid <code>^^^^^</code> characters.</p>
<p>When I've personally implemented this in the past, I've used a formatting object, e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">exception</span><span class="p">.</span><span class="nf">full_message</span><span class="p">(</span><span class="ss">highlight: </span><span class="no">HTMLFormatter</span><span class="p">.</span><span class="nf">new</span><span class="p">)</span>
</code></pre>
<p>The formatter has a rich interface for printing, and uses a set of abstractions to map to the underlying output.</p>
<p>e.g.</p>
<pre><code>formatter.puts(:exception, "NoMethodError", :reset, ": ", :message, "undefined method 'bar' for ", :object, "#<Foo...>")
formatter.puts(:code, "x = foo", :error, ".bar, :reset)
</code></pre>
<p>This can be easily mapped to HTML, XTerm, plain text, json, etc.</p> Ruby master - Feature #19008 (Closed): Introduce coverage support for `eval`.https://bugs.ruby-lang.org/issues/190082022-09-17T09:56:04Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I'd like to introduce coverage support for <code>eval</code>. I mostly only care about the case where an explicit path is given, and I'd even be okay to only handle the case where the line number is the default (0).</p>
<p><a href="https://github.com/ruby/ruby/pull/6396" class="external">https://github.com/ruby/ruby/pull/6396</a></p>
<p>This is an incredibly useful feature for computing coverage of ERB templates and other similar things.</p> Ruby master - Bug #18886 (Closed): Struct aref and aset don't trigger any tracepoints.https://bugs.ruby-lang.org/issues/188862022-06-29T02:19:28Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Given the following program, <code>thing.name</code> and <code>thing.shape</code> don't trigger <code>c_call</code> trace points (or any trace points actually).</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">pp</span> <span class="no">RUBY_VERSION</span>
<span class="n">trace_point</span> <span class="o">=</span> <span class="no">TracePoint</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:line</span><span class="p">,</span> <span class="ss">:call</span><span class="p">,</span> <span class="ss">:c_call</span><span class="p">,</span> <span class="ss">:a_call</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">trace</span><span class="o">|</span>
<span class="nb">puts</span> <span class="n">trace</span><span class="p">.</span><span class="nf">event</span>
<span class="k">if</span> <span class="n">trace</span><span class="p">.</span><span class="nf">event</span> <span class="o">==</span> <span class="ss">:call</span>
<span class="c1"># Ruby doesn't always mark call-sites in sub-expressions, so we use this approach to compute a call site and mark it:</span>
<span class="k">if</span> <span class="n">location</span> <span class="o">=</span> <span class="n">caller_locations</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">).</span><span class="nf">first</span> <span class="ow">and</span> <span class="n">path</span> <span class="o">=</span> <span class="n">location</span><span class="p">.</span><span class="nf">path</span>
<span class="nb">puts</span> <span class="s2">"> </span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="n">location</span><span class="p">.</span><span class="nf">lineno</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="n">trace</span><span class="p">.</span><span class="nf">event</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">if</span> <span class="n">path</span> <span class="o">=</span> <span class="n">trace</span><span class="p">.</span><span class="nf">path</span>
<span class="nb">puts</span> <span class="s2">"= </span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="n">trace</span><span class="p">.</span><span class="nf">lineno</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="n">trace</span><span class="p">.</span><span class="nf">event</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">trace_point</span><span class="p">.</span><span class="nf">enable</span>
<span class="c1"># This will trigger call trace points</span>
<span class="k">class</span> <span class="nc">Thing</span>
<span class="k">def</span> <span class="nf">name</span>
<span class="ss">:cat</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">shape</span>
<span class="ss">:square</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">thing</span> <span class="o">=</span> <span class="no">Thing</span><span class="p">.</span><span class="nf">new</span>
<span class="c1"># Thing = Struct.new(:name, :shape)</span>
<span class="c1"># thing = Thing.new(:cat, :rectangle)</span>
<span class="p">[</span>
<span class="ss">name: </span><span class="n">thing</span><span class="p">.</span><span class="nf">name</span><span class="p">,</span>
<span class="ss">shape: </span><span class="n">thing</span><span class="p">.</span><span class="nf">shape</span><span class="p">,</span>
<span class="p">]</span>
</code></pre>
<a name="Current-HEAD"></a>
<h2 >Current HEAD<a href="#Current-HEAD" class="wiki-anchor">¶</a></h2>
<pre><code>= ../test.rb:30:line:
= ../test.rb:30:c_call:new
= ../test.rb:30:c_call:inherited
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:const_added
= ../test.rb:31:line:
= ../test.rb:31:c_call:new
= ../test.rb:31:c_call:initialize
= ../test.rb:34:line:
</code></pre>
<a name="Proposed-PR"></a>
<h2 >Proposed PR<a href="#Proposed-PR" class="wiki-anchor">¶</a></h2>
<pre><code>= ../test.rb:30:line:
= ../test.rb:30:c_call:new
= ../test.rb:30:c_call:inherited
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:singleton_method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:method_added
= ../test.rb:30:c_call:const_added
= ../test.rb:31:line:
= ../test.rb:31:c_call:new
= ../test.rb:31:c_call:initialize
= ../test.rb:34:line:
= ../test.rb:34:c_call:name
= ../test.rb:35:c_call:shape
</code></pre>
<p>The reason is the internal implementation of struct doesn't have trace point instrumentation in <code>vm_call_opt_struct_aset</code> or <code>vm_call_opt_struct_aref</code>.</p>
<p>Proposed fix: <a href="https://github.com/ruby/ruby/pull/6071" class="external">https://github.com/ruby/ruby/pull/6071</a> but this would need a review, maybe <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/1604">@jeremyevans0 (Jeremy Evans)</a> and <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a> can help.</p> Ruby master - Bug #18810 (Closed): Make `Kernel#p` interruptable.https://bugs.ruby-lang.org/issues/188102022-05-28T03:04:18Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>While figuring out <a href="https://bugs.ruby-lang.org/issues/18465" class="external">https://bugs.ruby-lang.org/issues/18465</a> I found a test which fails when <code>rb_io_flush</code> becomes blocking.: <a href="https://github.com/ruby/ruby/commit/fe6b2e20e9f17ed2c2900aa72994e075ffdc7124" class="external">https://github.com/ruby/ruby/commit/fe6b2e20e9f17ed2c2900aa72994e075ffdc7124</a></p>
<p>It seems unusual to me that <code>Kernel#p</code> is uninterruptible (unique among all Ruby methods). I'd like to make <code>Kernel#p</code> interruptible.</p> Ruby master - Bug #18782 (Closed): Race conditions in autoload when loading the same feature with...https://bugs.ruby-lang.org/issues/187822022-05-14T23:01:50Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I have identified several race conditions in the autoload code.</p>
<ol>
<li>It's possible to race on adding and then deleting items in <code>autoload_featuremap</code>. When this happens, two threads will try to load the same file with different autoload data and deadlock.</li>
<li>When finishing autoload, it's necessary to clear <code>ele->state</code> before setting constants. If this is not synchronised, a thread can see the cleared <code>ele->state</code> before seeing the constants and assume the constant is not being autoloaded and then fail with <code>NameError</code>.</li>
</ol>
<p>This test case can reproduce both cases:</p>
<pre><code># test.rb
autoload_path = File.join(__dir__, "foobar.rb")
File.write(autoload_path, 'module Foo; end; module Bar; end')
100_000.times do
$stderr.puts "--------------------"
autoload :Foo, autoload_path
autoload :Bar, autoload_path
t1 = Thread.new {Foo}
t2 = Thread.new {Bar}
t1.join
t2.join
Object.send(:remove_const, :Foo)
Object.send(:remove_const, :Bar)
$LOADED_FEATURES.delete(autoload_path)
end
</code></pre>
<p>Example failure of case (1):</p>
<pre><code>-------------------- (success)
autoload_by_someone_else ele=0x55f33b806a30 ele->state=(nil)
autoload_by_someone_else ele=0x55f33b806a30 ele->state=(nil)
check_autoload_required 2
autoload_by_someone_else ele=0x55f33b806a30 ele->state=0x7fdd678be780
check_autoload_required 4
autoload_by_someone_else ele=0x55f33b806a30 ele->state=0x7fdd678be780
check_autoload_required 4
ele=0x55f33b806a30 ele->state=0x7fdd678be780 = NULL
check_autoload_required 4
-------------------- (failure)
autoload_by_someone_else ele=0x55f33b806a30 ele->state=(nil)
autoload_by_someone_else ele=0x55f33b6e8f40 ele->state=(nil)
check_autoload_required 2
check_autoload_required 3
autoload_by_someone_else ele=0x55f33b806a30 ele->state=0x7fdd6779d780
check_autoload_required 1
autoload_by_someone_else ele=0x55f33b806a30 ele->state=0x7fdd6779d780
check_autoload_required 1
ele=0x55f33b806a30 ele->state=0x7fdd6779d780 = NULL
ele=0x55f33b6e8f40 ele->state=0x7fdd678be780 = NULL
../test.rb:12:in `join': No live threads left. Deadlock? (fatal)
3 threads, 3 sleeps current:0x000055f33b771250 main thread:0x000055f33b66e090
* #<Thread:0x00007fdd6a2cb0b0 sleep_forever>
rb_thread_t:0x000055f33b66e090 native:0x00007fdd6a71c3c0 int:0
* #<Thread:0x00007fdd676e0090 ../test.rb:9 sleep_forever>
rb_thread_t:0x000055f33b770ff0 native:0x00007fdd6789e640 int:1 mutex:0x000055f33b7c5100 cond:1
depended by: tb_thread_id:0x000055f33b66e090
* #<Thread:0x00007fdd676e1238 ../test.rb:10 sleep_forever>
rb_thread_t:0x000055f33b771250 native:0x00007fdd679bf640 int:0
from ../test.rb:12:in `block in <main>'
from ../test.rb:4:in `times'
from ../test.rb:4:in `<main>'
make: *** [uncommon.mk:1250: runruby] Error 1
</code></pre>
<p>Example failure of case (2):</p>
<pre><code>[0x7f175fe5b0c8] rb_autoload_str mod=Object id=Foo file="/home/samuel/Projects/ioquatix/ruby/foobar.rb"
[0x7f175fe5b0c8] rb_autoload_str const_set mod=Object id=Foo file="/home/samuel/Projects/ioquatix/ruby/foobar.rb"
[0x7f175fe5b0c8] rb_autoload_str mod=Object id=Bar file="/home/samuel/Projects/ioquatix/ruby/foobar.rb"
[0x7f175fe5b0c8] rb_autoload_str const_set mod=Object id=Bar file="/home/samuel/Projects/ioquatix/ruby/foobar.rb"
[0x7f175fe61d88] rb_const_search_from value == Qundef -> autoloading
[0x7f175fe61e78] rb_const_search_from value == Qundef -> autoloading
[0x7f175fe61e78] Assigning constants...
[0x7f175fe61d88] rb_const_search_from value == Qundef -> autoloading
[0x7f175fe61e78] autoload_const_set name=:Foo value=Foo
[0x7f175fe61e78] autoload_const_set name=:Bar value=Bar
#<Thread:0x00007f175fe61d88 ../test.rb:11 run> terminated with exception (report_on_exception is true):
../test.rb:11:in `block (2 levels) in <main>': uninitialized constant Bar (NameError)
../test.rb:11:in `block (2 levels) in <main>': uninitialized constant Bar (NameError)
make: *** [uncommon.mk:1250: runruby] Error 1
</code></pre>
<p>These failures are very uncommon but it does impact Ruby as far back as 2.7, and probably earlier.</p> Ruby master - Bug #18663 (Closed): Autoload doesn't work with fiber context switch.https://bugs.ruby-lang.org/issues/186632022-03-25T20:12:27Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As discussed most recently here: <a href="https://github.com/ruby/debug/issues/580" class="external">https://github.com/ruby/debug/issues/580</a></p>
<p>The following program appears to work:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'tempfile'</span>
<span class="no">Tempfile</span><span class="p">.</span><span class="nf">create</span><span class="p">([</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'.rb'</span><span class="p">])</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">file</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="o"><<~</span><span class="no">RUBY</span><span class="p">)</span><span class="sh">
#
$stderr.puts 1; q = Queue.new
$stderr.puts 2; t = Thread.new{q.pop}
$stderr.puts 3; q << :sig
$stderr.puts 4; t.join
sleep 1
class C
end
</span><span class="no"> RUBY</span>
<span class="n">file</span><span class="p">.</span><span class="nf">close</span>
<span class="nb">autoload</span> <span class="ss">:C</span><span class="p">,</span> <span class="n">file</span><span class="p">.</span><span class="nf">path</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">threads</span> <span class="o">=</span> <span class="mi">3</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="vg">$stderr</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"LOADING C"</span>
<span class="vg">$stderr</span><span class="p">.</span><span class="nf">puts</span> <span class="no">C</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">threads</span><span class="p">.</span><span class="nf">each</span><span class="p">(</span><span class="o">&</span><span class="ss">:join</span><span class="p">)</span>
<span class="k">end</span><span class="p">.</span><span class="nf">join</span>
<span class="k">end</span>
</code></pre>
<p>This one doesn't:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'tempfile'</span>
<span class="nb">require_relative</span> <span class="s1">'lib/async'</span>
<span class="no">Tempfile</span><span class="p">.</span><span class="nf">create</span><span class="p">([</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'.rb'</span><span class="p">])</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">file</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="o"><<~</span><span class="no">RUBY</span><span class="p">)</span><span class="sh">
#
$stderr.puts 1; q = Queue.new
$stderr.puts 2; t = Thread.new{q.pop}
$stderr.puts 3; q << :sig
$stderr.puts 4; t.join
class C
end
</span><span class="no"> RUBY</span>
<span class="n">file</span><span class="p">.</span><span class="nf">close</span>
<span class="nb">autoload</span> <span class="ss">:C</span><span class="p">,</span> <span class="n">file</span><span class="p">.</span><span class="nf">path</span>
<span class="no">Async</span> <span class="k">do</span> <span class="o">|</span><span class="n">task</span><span class="o">|</span>
<span class="mi">3</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">task</span><span class="p">.</span><span class="nf">async</span> <span class="k">do</span>
<span class="vg">$stderr</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"LOADING C"</span>
<span class="vg">$stderr</span><span class="p">.</span><span class="nf">puts</span> <span class="no">C</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">.</span><span class="nf">wait</span>
<span class="k">end</span>
</code></pre>
<p>Semantically, they should be very similar. It feels like someone is checking the current thread rather than the current fiber or there is a poor implementation of locking somewhere, however I don't actually know for sure yet, investigation is required.</p> Ruby master - Feature #18630 (Closed): Introduce general `IO#timeout` and `IO#timeout=` for block...https://bugs.ruby-lang.org/issues/186302022-03-14T02:43:43Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I would like us to consider introducing a general IO timeout for all (non-)blocking operations, specified per-IO instance. It's useful for ensuring programs don't stop responding or spend an unreasonable amount of time waiting for IO operations.</p>
<p>There are effectively two kinds of interfaces that we need to address:</p>
<ul>
<li>Those that already have a timeout argument (e.g. <code>wait_readable</code>) and we follow the existing semantics.</li>
<li>Those that don't have a timeout argument or timeout semantics (e.g. <code>puts</code>, <code>gets</code>), and thus probably need to raise an exception on timeout.</li>
</ul>
<p>We have three possible kinds of exceptions we could raise:</p>
<ul>
<li><code>Errno::ETIMEDOUT</code></li>
<li>
<code>Timeout::Error</code> (from <code>timeout.rb</code>)</li>
<li>Introduce <code>IO::Timeout</code> or something similar.</li>
</ul>
<p>Timeout isn't necessarily an error condition. There are different arguments for whether we should define:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">IO::Timeout</span> <span class="o"><</span> <span class="no">Exception</span>
<span class="k">end</span>
<span class="c1"># or</span>
<span class="k">class</span> <span class="nc">IO::Timeout</span> <span class="o"><</span> <span class="no">StandardError</span>
<span class="k">end</span>
</code></pre>
<p>I believe the latter (<code>StandardError</code>) is more practical but I'm open to either option. I might have more specific arguments later why one is better than the other after testing in a practical system.</p>
<p>There is already a PR to try it out: <a href="https://github.com/ruby/ruby/pull/5653" class="external">https://github.com/ruby/ruby/pull/5653</a></p> Ruby master - Bug #18465 (Closed): Make `IO#write` atomic.https://bugs.ruby-lang.org/issues/184652022-01-09T06:33:20Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Right now, <code>IO#write</code> including everything that calls it including <code>IO#puts</code>, has a poorly specified behaviour w.r.t. other fibers/threads that call <code>IO#write</code> at the same time.</p>
<p>Internally, we have a write lock, however it's only used to lock against individual writes rather than the whole operation. From a user point of view, there is some kind of atomicity, but it's not clearly defined and depends on many factors, e.g. whether <code>write</code> or <code>writev</code> is used internally.</p>
<p>We propose to make <code>IO#write</code> an atomic operation, that is, <code>IO#write</code> on a synchronous/buffered IO will always perform the write operation using a lock around the entire operation.</p>
<p>In theory, this should actually be more efficient than the current approach which may acquire and release the lock several times per operation, however in practice I'm sure it's almost unnoticeable.</p>
<p>Where it does matter, is when interleaved operations invoke the fiber scheduler. By using a single lock around the entire operation, rather than one or more locks around the system calls, the entire operation is more predictable and behaves more robustly.</p> Ruby master - Bug #18455 (Open): `IO#close` has poor performance and difficult to understand sema...https://bugs.ruby-lang.org/issues/184552022-01-01T07:13:08Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><code>IO#close</code> should be responsible for closing the file descriptor referred to by the IO instance. When dealing with buffered IO, one can also expect this to flush the internal buffers if possible.</p>
<p>Currently, all blocking IO operations release the GVL and perform the blocking system call using <code>rb_thread_io_blocking_region</code>. The current implementation takes a file descriptor and adds an entry to the VM global <code>waiting_fds</code> list. When the operation is completed, the entry is removed from <code>waiting_fds</code>.</p>
<p>When calling <code>IO#close</code>, this list is traversed and any threads performing blocking operations with a matching file descriptor are interrupted. The performance of this is O(number of blocking IO operations) which in practice the performance of <code>IO#close</code> can take milliseconds with 10,000 threads performing blocking IO. This performance is unacceptable.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'benchmark'</span>
<span class="k">class</span> <span class="nc">Reading</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@r</span><span class="p">,</span> <span class="vi">@w</span> <span class="o">=</span> <span class="no">IO</span><span class="p">.</span><span class="nf">pipe</span>
<span class="vi">@thread</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="vi">@r</span><span class="p">.</span><span class="nf">read</span>
<span class="k">rescue</span> <span class="no">IOError</span>
<span class="c1"># Ignore.</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">attr</span> <span class="ss">:r</span>
<span class="kp">attr</span> <span class="ss">:w</span>
<span class="kp">attr</span> <span class="ss">:thread</span>
<span class="k">def</span> <span class="nf">join</span>
<span class="vi">@thread</span><span class="p">.</span><span class="nf">join</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">measure</span><span class="p">(</span><span class="n">count</span> <span class="o">=</span> <span class="mi">10</span><span class="p">)</span>
<span class="n">readings</span> <span class="o">=</span> <span class="n">count</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span>
<span class="no">Reading</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="nb">sleep</span> <span class="mi">10</span>
<span class="n">duration</span> <span class="o">=</span> <span class="no">Benchmark</span><span class="p">.</span><span class="nf">measure</span> <span class="k">do</span>
<span class="n">readings</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">reading</span><span class="o">|</span>
<span class="n">reading</span><span class="p">.</span><span class="nf">r</span><span class="p">.</span><span class="nf">close</span>
<span class="n">reading</span><span class="p">.</span><span class="nf">w</span><span class="p">.</span><span class="nf">close</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">average</span> <span class="o">=</span> <span class="p">(</span><span class="n">duration</span><span class="p">.</span><span class="nf">total</span> <span class="o">/</span> <span class="n">count</span><span class="p">)</span> <span class="o">*</span> <span class="mf">1000.0</span>
<span class="n">pp</span> <span class="ss">count: </span><span class="n">count</span><span class="p">,</span> <span class="ss">average: </span><span class="nb">sprintf</span><span class="p">(</span><span class="s2">"%0.2fms"</span><span class="p">,</span> <span class="n">average</span><span class="p">)</span>
<span class="n">readings</span><span class="p">.</span><span class="nf">each</span><span class="p">(</span><span class="o">&</span><span class="ss">:join</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">measure</span><span class="p">(</span> <span class="mi">10</span><span class="p">)</span>
<span class="n">measure</span><span class="p">(</span> <span class="mi">100</span><span class="p">)</span>
<span class="n">measure</span><span class="p">(</span> <span class="mi">1000</span><span class="p">)</span>
<span class="n">measure</span><span class="p">(</span><span class="mi">10000</span><span class="p">)</span>
</code></pre>
<p>In addition, the semantics of this operation are confusing at best. While Ruby programs are dealing with IO instances, the VM is dealing with file descriptors, in effect performing some internal de-duplication of IO state. In practice, this leads to strange behaviour:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="n">r</span><span class="p">,</span> <span class="n">w</span> <span class="o">=</span> <span class="no">IO</span><span class="p">.</span><span class="nf">pipe</span>
<span class="n">r2</span> <span class="o">=</span> <span class="no">IO</span><span class="p">.</span><span class="nf">for_fd</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="nf">to_i</span><span class="p">)</span>
<span class="n">pp</span> <span class="ss">r: </span><span class="n">r</span><span class="p">,</span> <span class="ss">r2: </span><span class="n">r2</span>
<span class="n">t</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">r2</span><span class="p">.</span><span class="nf">read</span> <span class="k">rescue</span> <span class="kp">nil</span>
<span class="n">r2</span><span class="p">.</span><span class="nf">read</span> <span class="c1"># EBADF</span>
<span class="k">end</span>
<span class="nb">sleep</span> <span class="mf">0.5</span>
<span class="n">r</span><span class="p">.</span><span class="nf">close</span>
<span class="n">t</span><span class="p">.</span><span class="nf">join</span> <span class="k">rescue</span> <span class="kp">nil</span>
<span class="n">pp</span> <span class="ss">r: </span><span class="n">r</span><span class="p">,</span> <span class="ss">r2: </span><span class="n">r2</span>
<span class="c1"># r is closed, r2 is valid but will raise EBADF on any operation.</span>
</code></pre>
<p>In addition, this confusing behaviour extends to Ractor and state is leaked between the two:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">r</span><span class="p">,</span> <span class="n">w</span> <span class="o">=</span> <span class="no">IO</span><span class="p">.</span><span class="nf">pipe</span>
<span class="n">ractor</span> <span class="o">=</span> <span class="no">Ractor</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="nf">to_i</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">fd</span><span class="o">|</span>
<span class="n">r2</span> <span class="o">=</span> <span class="no">IO</span><span class="p">.</span><span class="nf">for_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
<span class="n">r2</span><span class="p">.</span><span class="nf">read</span>
<span class="c1"># r2.read # EBADF</span>
<span class="k">end</span>
<span class="nb">sleep</span> <span class="mf">0.5</span>
<span class="n">r</span><span class="p">.</span><span class="nf">close</span>
<span class="n">pp</span> <span class="ss">take: </span><span class="n">ractor</span><span class="p">.</span><span class="nf">take</span>
</code></pre>
<p>I propose the following changes to simplify the semantics and improve performance:</p>
<ul>
<li>Move the semantics of <code>waiting_fds</code> from per-fd to per-IO. This means that <code>IO#close</code> only interrupts blocking operations performed on the same IO instance rather than ANY IO which refers to the same file descriptor. I think this behaviour is easier to understand and still protects against the vast majority of incorrect usage.</li>
<li>Move the details of <code>struct rb_io_t</code> to <code>internal/io.h</code> so that the implementation details are not part of the public interface.</li>
</ul>
<a name="Benchmarks"></a>
<h2 >Benchmarks<a href="#Benchmarks" class="wiki-anchor">¶</a></h2>
<p>Before:</p>
<pre><code>{:count=>10, :average=>"0.19ms"}
{:count=>100, :average=>"0.11ms"}
{:count=>1000, :average=>"0.18ms"}
{:count=>10000, :average=>"1.16ms"}
</code></pre>
<p>After:</p>
<pre><code>{:count=>10, :average=>"0.20ms"}
{:count=>100, :average=>"0.11ms"}
{:count=>1000, :average=>"0.15ms"}
{:count=>10000, :average=>"0.68ms"}
</code></pre>
<p>After investigating this further I found that the <code>rb_thread_io_blocking_region</code> using <code>ubf_select</code> can be incredibly slow, proportional to the number of threads. I don't know whether it's advisable but:</p>
<pre><code class="c syntaxhl" data-language="c"> <span class="n">BLOCKING_REGION</span><span class="p">(</span><span class="n">blocking_node</span><span class="p">.</span><span class="kr">thread</span><span class="p">,</span> <span class="p">{</span>
<span class="n">val</span> <span class="o">=</span> <span class="n">func</span><span class="p">(</span><span class="n">data1</span><span class="p">);</span>
<span class="n">saved_errno</span> <span class="o">=</span> <span class="n">errno</span><span class="p">;</span>
<span class="p">},</span> <span class="nb">NULL</span> <span class="cm">/* ubf_select */</span><span class="p">,</span> <span class="n">blocking_node</span><span class="p">.</span><span class="kr">thread</span><span class="p">,</span> <span class="n">FALSE</span><span class="p">);</span>
</code></pre>
<p>Disabling the UBF function and relying on <code>read(fd, ...)</code>/<code>write(fd, ...)</code> blocking operations to fail when <code>close(fd)</code> is invoked might be sufficient? This needs more investigation but after making this change, we have constant-time IO#close.</p>
<pre><code>{:count=>10, :average=>"0.13ms"}
{:count=>100, :average=>"0.06ms"}
{:count=>1000, :average=>"0.04ms"}
{:count=>10000, :average=>"0.09ms"}
</code></pre>
<p>Which is ideally what we want.</p> Ruby master - Bug #18443 (Closed): IO read/write/wait hook bug fixes.https://bugs.ruby-lang.org/issues/184432021-12-27T22:29:07Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>After testing Ruby 3.1.0 extensively, I found some obscure bugs which should be fixed. This should be back ported to 3.1.1 ASAP.</p>
<ul>
<li>
<code>console.c</code> incorrect <code>rb_io_wait</code> argument.</li>
<li>
<code>rb_read_internal</code> incorrect logic when handling partial non-blocking reads in <code>io.c</code>.</li>
<li>Prefer <code>wait_readable</code> rather than <code>IO.select</code> in irb implementation.</li>
</ul>
<p>Please see <a href="https://github.com/ruby/ruby/pull/5353" class="external">https://github.com/ruby/ruby/pull/5353</a> for details.</p> Ruby master - Feature #18411 (Closed): Introduce `Fiber.blocking` for disabling scheduler.https://bugs.ruby-lang.org/issues/184112021-12-16T07:58:58Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>When implementing pure-ruby IO scheduler, we may need to invoke some Ruby IO operations without entering the scheduler.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">def</span> <span class="nf">io_write</span><span class="p">(</span><span class="n">fiber</span><span class="p">,</span> <span class="n">io</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="n">offset</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="n">length</span> <span class="o">></span> <span class="mi">0</span>
<span class="c1"># From offset until the end:</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="n">buffer</span><span class="p">.</span><span class="nf">to_str</span><span class="p">(</span><span class="n">offset</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="k">case</span> <span class="n">result</span> <span class="o">=</span> <span class="n">io</span><span class="p">.</span><span class="nf">write_nonblock</span><span class="p">(</span><span class="n">chunk</span><span class="p">,</span> <span class="ss">exception: </span><span class="kp">false</span><span class="p">)</span>
<span class="k">when</span> <span class="ss">:wait_readable</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">io_wait</span><span class="p">(</span><span class="n">fiber</span><span class="p">,</span> <span class="n">io</span><span class="p">,</span> <span class="no">IO</span><span class="o">::</span><span class="no">READABLE</span><span class="p">)</span>
<span class="k">when</span> <span class="ss">:wait_writable</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">io_wait</span><span class="p">(</span><span class="n">fiber</span><span class="p">,</span> <span class="n">io</span><span class="p">,</span> <span class="no">IO</span><span class="o">::</span><span class="no">WRITABLE</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">offset</span> <span class="o">+=</span> <span class="n">result</span>
<span class="n">length</span> <span class="o">-=</span> <span class="n">result</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">return</span> <span class="n">offset</span>
<span class="k">end</span>
</code></pre>
<p>There are some cases where even in this code <code>read_nonblock</code> can invoke fiber scheduler creating infinite recursion.</p>
<p>Therefore, I propose to introduce <code>Fiber.blocking{...}</code> which has almost identical implementation to <code>Fiber.new(blocking: true) {}.resume</code>.</p>
<p>In the above code, we change the line:</p>
<pre><code> case result = io.write_nonblock(chunk, exception: false)
</code></pre>
<p>to</p>
<pre><code> case result = Fiber.blocking{io.write_nonblock(chunk, exception: false)}
</code></pre>
<p>This ensures that <code>write_nonblock</code> can never enter the scheduler again.</p> Ruby master - Bug #18389 (Closed): `binding.irb` can fail in some classes that implement `context...https://bugs.ruby-lang.org/issues/183892021-12-05T22:15:48Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Foo</span>
<span class="k">def</span> <span class="nf">boop</span>
<span class="nb">binding</span><span class="p">.</span><span class="nf">irb</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">context</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">print</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">Foo</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">boop</span>
</code></pre>
<p>It fails with:</p>
<pre><code>> ruby ./test.rb
From: ./test.rb @ line 3 :
1: class Foo
2: def boop
=> 3: binding.irb
4: end
5:
6: def context
7: end
8:
./test.rb:9:in `print': wrong number of arguments (given 1, expected 0) (ArgumentError)
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb/extend-command.rb:238:in `install_alias_method'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb/extend-command.rb:252:in `block in extend_object'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb/extend-command.rb:251:in `each'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb/extend-command.rb:251:in `extend_object'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb.rb:466:in `extend'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb.rb:466:in `initialize'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb.rb:959:in `new'
from /Users/samuel/.gem/ruby/3.0.3/gems/irb-1.3.7/lib/irb.rb:959:in `irb'
from <internal:prelude>:5:in `irb'
from ./test.rb:3:in `boop'
from ./test.rb:13:in `<main>'
</code></pre>
<p>I suggest that <code>binding.irb</code> should be a little bit less invasive to avoid these kinds of issues.</p> Ruby master - Bug #18363 (Closed): `make clean` should consistently handle generated files otherw...https://bugs.ruby-lang.org/issues/183632021-11-25T06:33:28Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Running <code>make clean</code> will cause <code>ext/ripper/ripper.c</code> to be regenerated but not <code>parse.c</code> and this causes problems if bison is incompatible with previous version, because <code>ripper.c</code> will be regenerated with a different version of bison.</p>
<p>nobu: Ah, yes, concatenating <code>ripper.y</code> and <code>ripper.c</code> to <code>$cleanfiles</code>, in <code>ext/ripper/extconf.rb</code>.</p>
<p>samuel: Can this solve the problem so make clean does not remove it?</p>
<p>nobu: I think so. <code>eventids*.c</code> also should be removed by distclean instead.</p> Ruby master - Feature #18296 (Closed): Custom exception formatting should override `Exception#ful...https://bugs.ruby-lang.org/issues/182962021-11-10T11:49:32Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>After discussing with <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/772">@Eregon (Benoit Daloze)</a>, we came to the conclusion that the current implementation of <code>did_you_mean</code> and <code>error_highlight</code> could avoid many issues by using <code>Exception#full_message</code>.</p>
<p>We propose to introduce a more nuanced interface:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Exception</span>
<span class="k">def</span> <span class="nf">full_message</span><span class="p">(</span><span class="ss">highlight: </span><span class="n">bool</span><span class="p">,</span> <span class="ss">order: </span><span class="p">[</span><span class="ss">:top</span> <span class="ow">or</span> <span class="ss">:bottom</span><span class="p">],</span> <span class="o">**</span><span class="n">options</span><span class="p">)</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">module</span> <span class="nn">DidYouMean</span>
<span class="k">module</span> <span class="nn">Formatter</span>
<span class="k">def</span> <span class="nf">full_message</span><span class="p">(</span><span class="n">highlight</span><span class="p">:,</span> <span class="ss">did_you_mean: </span><span class="kp">true</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">)</span>
<span class="n">buffer</span> <span class="o">=</span> <span class="k">super</span><span class="p">(</span><span class="ss">highlight: </span><span class="n">highlight</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">).</span><span class="nf">dup</span>
<span class="n">buffer</span> <span class="o"><<</span> <span class="s2">"extra stuff"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">Exception</span><span class="p">.</span><span class="nf">prepend</span> <span class="no">DidYouMean</span><span class="o">::</span><span class="no">Formatter</span>
<span class="k">module</span> <span class="nn">ErrorHighlight</span>
<span class="k">module</span> <span class="nn">Formatter</span>
<span class="k">def</span> <span class="nf">full_message</span><span class="p">(</span><span class="n">highlight</span><span class="p">:,</span> <span class="ss">error_highlight: </span><span class="kp">true</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">)</span>
<span class="c1"># same as above</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">Exception</span><span class="p">.</span><span class="nf">prepend</span> <span class="no">ErrorHighlight</span><span class="o">::</span><span class="no">Formatter</span>
</code></pre> Ruby master - Feature #18258 (Closed): Ractor.shareable? can be slow and mutates internal object ...https://bugs.ruby-lang.org/issues/182582021-10-21T03:19:30Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>On my computer, even with a relatively small object graph,<code>Ractor.shareable?</code> can be quite slow (around 1-2ms). The following example creates an object graph with ~40k objects as an example, and on my computer takes around 20ms to execute <code>Ractor.shareable?</code>. Because the object cannot be marked as <code>RB_FL_SHAREABLE</code> because it contains mutable state, every time we check <code>Ractor.shareable?</code> it will perform the same object traversal which is the slow path.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'benchmark'</span>
<span class="k">class</span> <span class="nc">Borked</span>
<span class="k">def</span> <span class="nf">freeze</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Nested</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">count</span><span class="p">,</span> <span class="n">top</span> <span class="o">=</span> <span class="kp">true</span><span class="p">)</span>
<span class="k">if</span> <span class="n">count</span> <span class="o">></span> <span class="mi">0</span>
<span class="vi">@nested</span> <span class="o">=</span> <span class="n">count</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">map</span><span class="p">{</span><span class="no">Nested</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">count</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="kp">false</span><span class="p">).</span><span class="nf">freeze</span><span class="p">}.</span><span class="nf">freeze</span>
<span class="k">end</span>
<span class="k">if</span> <span class="n">top</span>
<span class="vi">@borked</span> <span class="o">=</span> <span class="no">Borked</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">attr</span> <span class="ss">:nested</span>
<span class="kp">attr</span> <span class="ss">:borked</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"Creating nested object of size N=</span><span class="si">#{</span><span class="n">n</span><span class="si">}</span><span class="s2">"</span>
<span class="n">nested</span> <span class="o">=</span> <span class="no">Nested</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">n</span><span class="p">).</span><span class="nf">freeze</span>
<span class="n">shareable</span> <span class="o">=</span> <span class="kp">false</span>
<span class="n">result</span> <span class="o">=</span> <span class="no">Benchmark</span><span class="p">.</span><span class="nf">measure</span> <span class="k">do</span>
<span class="n">shareable</span> <span class="o">=</span> <span class="no">Ractor</span><span class="p">.</span><span class="nf">shareable?</span><span class="p">(</span><span class="n">nested</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">pp</span> <span class="ss">result: </span><span class="n">result</span><span class="p">,</span> <span class="ss">shareable: </span><span class="n">shareable</span>
<span class="k">end</span>
<span class="nb">test</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
</code></pre>
<p>I propose we change <code>Ractor.shareable?</code> to only check <code>RB_FL_SHAREABLE</code> which gives (1) predictable and fast performance in every case and (2) avoids mutating internal object flags when performing what looks like a read-only operation.</p>
<p>I respect that one way of looking at <code>Ractor.shareable?</code> is as a cache for object state. But this kind of cache can lead to unpredictable performance.</p>
<p>As a result, something like <code>String#freeze</code> would not create objects that can be shared with Ractor. However, I believe we can mitigate this by tweaking <code>String#freeze</code> to also set <code>RB_FL_SHAREABLE</code> if possible. I believe we should apply this to more objects. It will lead to more predictable performance for Ruby.</p>
<p>Since there are few real-world examples of Ractor, it's hard to find real world example of the problem. However, I believe such an issue will prevent Ractor usage as even relatively small object graphs (~1000 objects) can cause 1-2ms of latency, and this particular operation does not release the GVL either which means it stalls the entire VM.</p>
<p>This issue came from discussion regarding <a href="https://bugs.ruby-lang.org/issues/18035" class="external">https://bugs.ruby-lang.org/issues/18035</a> where we are considering using <code>RB_FL_SHAREABLE</code> as a flag for immutability. By fixing this issue, we make it easier to implement model for immutability because we don't need to introduce new flags and can instead reuse existing flags.</p> Ruby master - Feature #18227 (Open): Static class initialization.https://bugs.ruby-lang.org/issues/182272021-09-27T03:49:35Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As a follow on from <a href="https://bugs.ruby-lang.org/issues/18189" class="external">https://bugs.ruby-lang.org/issues/18189</a> I would like to propose some kind of static class initialization. I'll investigate whether it's possible and create a PR.</p> Ruby master - Feature #18194 (Closed): No easy way to format exception messages per thread/fiber ...https://bugs.ruby-lang.org/issues/181942021-09-26T20:44:51Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In the new error highlighting gem, formatting exception messages appears to be per-process which is insufficiently nuanced for existing use cases.</p>
<p>As in:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">TerminalColorFormatter</span>
<span class="k">def</span> <span class="nf">message_for</span><span class="p">(</span><span class="n">spot</span><span class="p">)</span>
<span class="c1"># How do we know the output format here? Maybe it's being written to a log file?</span>
<span class="s2">"..."</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">ErrorHighlight</span><span class="p">.</span><span class="nf">formatter</span> <span class="o">=</span> <span class="no">TerminalColorFormatter</span><span class="p">.</span><span class="nf">new</span>
</code></pre>
<p>But we won't know until the time we actually write the error message whether terminal codes are suitable or available. Or an error message might be formatted for both the terminal and a log file, which have different formatting requirements. There are many consumers of error messages an some of them produce text, or HTML, or JSON, etc.</p>
<p>Because of this design we are effectively forcing everyone to parse the default text output if they want to do any kind of formatting, which will ossify the format and make it impossible in practice for anyone to use anything but the default <code>ErrorHighlight.format</code>. For what is otherwise a really fantastic idea, this implementation concerns me greatly.</p>
<p>I would like us to consider introducing sufficient metadata on the exception object so that complete formatting can be implemented by an output layer (e.g. logger, terminal wrapper, etc). This allows the output layer to intelligently format the output in a suitable way, or capture the metadata to allow for processing elsewhere.</p>
<p>In addition, to simplify this general usage, we might like to introduce <code>Exception#formatted_message</code>.</p>
<p>In order to handle default formatting requirements, we need to provide a hook for formatting uncaught exceptions. This would be excellent for many different use cases (e.g. HoneyBadger type systems), and I suggest we think about the best interface. Probably a thread-local with some default global implementation makes sense... maybe even something similar to <code>at_exit { ... $! ... }</code>.</p> Ruby master - Bug #18189 (Closed): `rb_cString` can be NULL during `Init_Object`https://bugs.ruby-lang.org/issues/181892021-09-23T06:45:10Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It's possible for <code>rb_cString</code> to be NULL during <code>Init_Object</code> and thus <code>Init_class_hierarchy</code> which means that <code>rb_fstring_lit</code>, which invokes <code>setup_fake_str</code>, invokes <code>RBASIC_SET_CLASS_RAW(..., NULL)</code> (or possibly just something totally random if it's not zero initialized!).</p>
<p>Later on in <code>register_fstring</code> we have an assertion which also fails to detect the abnormality:</p>
<pre><code>assert(RBASIC_CLASS(args.fstr) == rb_cString);
</code></pre>
<p>Because both are NULL. Oops.</p>
<p>It seems that later on, <code>rb_cString</code> is set on that specific fstring. But in my own usage of <code>rb_define_module_under</code> during <code>InitVM_Object</code>, this creates invalid class names which fail when passed into Ruby land.</p> Ruby master - Bug #18084 (Closed): `JSON.dump` can crash VM.https://bugs.ruby-lang.org/issues/180842021-08-18T04:17:20Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="nb">require</span> <span class="s1">'json'</span>
<span class="n">x</span> <span class="o">=</span> <span class="p">{};</span> <span class="n">x</span><span class="p">[</span><span class="ss">:x</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
<span class="no">JSON</span><span class="p">.</span><span class="nf">dump</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="c1"># => nil</span>
<span class="no">JSON</span><span class="p">.</span><span class="nf">dump</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="c1"># => SIGILL</span>
</code></pre>
<p>Results from our friendly all ruby bot:</p>
<pre><code>:x: 1.8
-:1:in `require': no such file to load -- json (LoadError)
from -:1
exit: 1
:x: 1.9
/build-all-ruby/1.9.3-p551/lib/ruby/1.9.1/json/common.rb:216: stack level too deep (SystemStackError)
exit: 1
:x: 2.0
/build-all-ruby/2.0.0-p648/lib/ruby/2.0.0/json/common.rb:224: stack level too deep (SystemStackError)
exit: 1
:x: 2.1
/build-all-ruby/2.1.10/lib/ruby/2.1.0/json/common.rb:223: stack level too deep (SystemStackError)
exit: 1
:x: 2.2
/build-all-ruby/2.2.10/lib/ruby/2.2.0/json/common.rb:223:in `encode': stack level too deep (SystemStackError)
from /build-all-ruby/2.2.10/lib/ruby/2.2.0/json/common.rb:223:in `generate'
from /build-all-ruby/2.2.10/lib/ruby/2.2.0/json/common.rb:223:in `generate'
from /build-all-ruby/2.2.10/lib/ruby/2.2.0/json/common.rb:394:in `dump'
from -:5:in `<main>'
Show more
exit: 1
:x: 2.3
/build-all-ruby/2.3.8/lib/ruby/2.3.0/json/common.rb:224:in `encode': stack level too deep (SystemStackError)
from /build-all-ruby/2.3.8/lib/ruby/2.3.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.3.8/lib/ruby/2.3.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.3.8/lib/ruby/2.3.0/json/common.rb:395:in `dump'
from -:5:in `<main>'
Show more
exit: 1
:x: 2.4
/build-all-ruby/2.4.10/lib/ruby/2.4.0/json/common.rb:224:in `encode': stack level too deep (SystemStackError)
from /build-all-ruby/2.4.10/lib/ruby/2.4.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.4.10/lib/ruby/2.4.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.4.10/lib/ruby/2.4.0/json/common.rb:394:in `dump'
from -:5:in `<main>'
Show more
exit: 1
:x: 2.5
/build-all-ruby/2.5.8/lib/ruby/2.5.0/json/common.rb:224:in `generate': stack level too deep (SystemStackError)
from /build-all-ruby/2.5.8/lib/ruby/2.5.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.5.8/lib/ruby/2.5.0/json/common.rb:394:in `dump'
from -:5:in `<main>'
exit: 1
:x: 2.6
/build-all-ruby/2.6.6/lib/ruby/2.6.0/json/common.rb:224:in `generate': stack level too deep (SystemStackError)
from /build-all-ruby/2.6.6/lib/ruby/2.6.0/json/common.rb:224:in `generate'
from /build-all-ruby/2.6.6/lib/ruby/2.6.0/json/common.rb:394:in `dump'
from -:5:in `<main>'
exit: 1
:x: 2.7 -- 58bd943436 (2021-08-17T17:25:19Z)
-: machine stack overflow in critical region (fatal)
exit: 1
</code></pre>
<p>So in the best case, we got <code>SystemStackError</code>, and from 2.7+ we get a fatal error.</p>
<p>Do we want to improve this behaviour or is this expected, and if so, should we add documentation to this effect?</p> Ruby master - Feature #18083 (Open): Capture error in ensure block.https://bugs.ruby-lang.org/issues/180832021-08-18T03:03:42Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>As discussed in <a href="https://bugs.ruby-lang.org/issues/15567" class="external">https://bugs.ruby-lang.org/issues/15567</a> there are some tricky edge cases.</p>
<p>As a general model, something like the following would be incredibly useful:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">begin</span>
<span class="o">...</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">error</span>
<span class="n">pp</span> <span class="s2">"error occurred"</span> <span class="k">if</span> <span class="n">error</span>
<span class="k">end</span>
</code></pre>
<p>Currently you can get similar behaviour like this:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">begin</span>
<span class="o">...</span>
<span class="k">rescue</span> <span class="no">Exception</span> <span class="o">=></span> <span class="n">error</span>
<span class="k">raise</span>
<span class="k">ensure</span>
<span class="n">pp</span> <span class="s2">"error occurred"</span> <span class="k">if</span> <span class="n">error</span>
<span class="k">end</span>
</code></pre>
<p>The limitation of this approach is it only works if you don't need any other <code>rescue</code> clause. Otherwise, it may not work as expected or require extra care. Also, Rubocop will complain about it.</p>
<p>Using <code>$!</code> can be buggy if you call some method from <code>rescue</code> or <code>ensure</code> clause, since it would be set already. It was discussed extensively in <a href="https://bugs.ruby-lang.org/issues/15567" class="external">https://bugs.ruby-lang.org/issues/15567</a> if you want more details.</p> Ruby master - Bug #18048 (Closed): Thread#join can break with fiber scheduler unblock fails or bl...https://bugs.ruby-lang.org/issues/180482021-07-27T06:57:50Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In addition to <a href="https://bugs.ruby-lang.org/issues/17666" class="external">https://bugs.ruby-lang.org/issues/17666</a> we found several more cases that need to be addressed.</p>
<p>Fix potential hang when joining threads.</p>
<p>If the thread termination invokes user code after <code>th->status</code> becomes<br>
<code>THREAD_KILLED</code>, and the user unblock function causes that <code>th->status</code> to<br>
become something else (e.g. <code>THREAD_RUNNING</code>), threads waiting in<br>
<code>thread_join_sleep</code> will hang forever. We move the unblock function call<br>
to before the thread status is updated, and allow threads to join as soon<br>
as <code>th->value</code> becomes defined.</p>
<p>Wake up join list within thread EC context. (<a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: sample list.rb generates warnings (Closed)" href="https://bugs.ruby-lang.org/issues/4471">#4471</a>)</p>
<p>If <code>rb_fiber_scheduler_unblock</code> raises an exception, it can result in a<br>
segfault if <code>rb_threadptr_join_list_wakeup</code> is not within a valid EC. This<br>
change moves <code>rb_threadptr_join_list_wakeup</code> into the thread's top level EC<br>
which initially caused an infinite loop because on exception will retry. We<br>
explicitly remove items from the thread's join list to avoid this situation.</p>
<p>These are already fixed on master branch. Here is a PR for backport: <a href="https://github.com/ruby/ruby/pull/4686" class="external">https://github.com/ruby/ruby/pull/4686</a></p> Ruby master - Bug #18036 (Closed): Pthread fibers become invalid on fork - different from normal ...https://bugs.ruby-lang.org/issues/180362021-07-09T09:45:30Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Fork is notoriously hard to use correctly and I most cases we should be encouraging <code>Process#spawn</code>. However, it does have use cases for example pre-fork model of server design. So there are some valid usage at least.</p>
<p>We recently introduced non-native fiber based on pthread which is generally more compatible than copy coroutine w.r.t. the overall burden on the implementation. However, it has one weak point, which is that pthreads become invalid on fork, and thus fibers become invalid on fork. That means that the following program can become invalid:</p>
<pre><code>Fiber.new do
fork
end.resume
</code></pre>
<p>It will create two threads, the main thread and the thread for the fiber. When the child begins execution, it will be within the child pthread, but the parent pthread is no longer valid, i.e. it's gone.</p>
<p>I see a couple of options here (not mutually exclusive):</p>
<ul>
<li>Combining Fibers and fork is invalid. Fork only works from main fiber.</li>
<li>Ignore the problem and expect users of fork to be aware that the program can potentially enter an invalid state - okay for <code>fork-exec</code> but not much else.</li>
<li>Terminate all non-current fibers as we do for threads, and possibly fail if the current fiber exits for some reason.</li>
</ul>
<p>Because pthread coroutine should be very uncommon, I don't think we should sacrifice the general good qualities of the fiber semantic model for some obscure case. Maybe it would be sufficient to have a warning (not printed by default unless running on pthread coroutines), that fork within a non-main fiber can have undefined results.</p> Ruby master - Feature #18035 (Open): Introduce general model/semantic for immutability.https://bugs.ruby-lang.org/issues/180352021-07-09T08:10:55Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It would be good to establish some rules around mutability, immutability, frozen, and deep frozen in Ruby.</p>
<p>I see time and time again, incorrect assumptions about how this works in production code. Constants that aren't really constant, people using <code>#freeze</code> incorrectly, etc.</p>
<p>I don't have any particular preference but:</p>
<ul>
<li>We should establish consistent patterns where possible, e.g.
<ul>
<li>Objects created by <code>new</code> are mutable.</li>
<li>Objects created by literal are immutable.</li>
</ul>
</li>
</ul>
<p>We have problems with how <code>freeze</code> works on composite data types, e.g. <code>Hash#freeze</code> does not impact children keys/values, same for Array. Do we need to introduce <code>freeze(true)</code> or <code>#deep_freeze</code> or some other method?</p>
<p>Because of this, frozen does not necessarily correspond to immutable. This is an issue which causes real world problems.</p>
<p>I also propose to codify this where possible, in terms of "this class of object is immutable" should be enforced by the language/runtime, e.g.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">module</span> <span class="nn">Immutable</span>
<span class="k">def</span> <span class="nf">new</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="k">super</span><span class="p">.</span><span class="nf">freeze</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">MyImmutableObject</span>
<span class="kp">extend</span> <span class="no">Immutable</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="vi">@x</span> <span class="o">=</span> <span class="n">x</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">freeze</span>
<span class="k">return</span> <span class="nb">self</span> <span class="k">if</span> <span class="nb">frozen?</span>
<span class="vi">@x</span><span class="p">.</span><span class="nf">freeze</span>
<span class="k">super</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">o</span> <span class="o">=</span> <span class="no">MyImmutableObject</span><span class="p">.</span><span class="nf">new</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span>
<span class="nb">puts</span> <span class="n">o</span><span class="p">.</span><span class="nf">frozen?</span>
</code></pre>
<p>Finally, this area has an impact to thread and fiber safe programming, so it is becoming more relevant and I believe that the current approach which is rather adhoc is insufficient.</p>
<p>I know that it's non-trivial to retrofit existing code, but maybe it can be done via magic comment, etc, which we already did for frozen string literals.</p>
<p>Proposed PR: <a href="https://github.com/ruby/ruby/pull/4879" class="external">https://github.com/ruby/ruby/pull/4879</a></p> Ruby master - Feature #18020 (Closed): Introduce `IO::Buffer` for fiber scheduler.https://bugs.ruby-lang.org/issues/180202021-07-03T07:24:10Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>After continuing to build out the fiber scheduler interface and the specific hooks required for <code>io_uring</code>, I found some trouble within the implementation of <code>IO</code>.</p>
<p>I found that in some cases, we need to read into the internal IO buffers directly. I tried creating a "fake string" in order to transit back into the Ruby fiber scheduler interface and this did work to a certain extent, but I was told we cannot expose fake string to Ruby scheduler interface.</p>
<p>So, after this, and many other frustrations with using <code>String</code> as a IO buffer, I decided to implement a low level <code>IO::Buffer</code> based on my needs for high performance IO, and as part of the fiber scheduler interface.</p>
<p>Here is roughly the interface implemented by the scheduler w.r.t. the buffer:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Scheduler</span>
<span class="c1"># @parameter buffer [IO::Buffer] Buffer for reading into.</span>
<span class="k">def</span> <span class="nf">io_read</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="c1"># implementation provided by `read` system call, IO_URING_READV, etc.</span>
<span class="k">end</span>
<span class="c1"># @parameter buffer [IO::Buffer] Buffer for writing from.</span>
<span class="k">def</span> <span class="nf">io_write</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="c1"># implementation provided by `write` system call, IO_URING_WRITEV, etc.</span>
<span class="k">end</span>
<span class="c1"># Potential new hooks (Socket#recvmsg, sendmsg, etc):</span>
<span class="k">def</span> <span class="nf">io_recvmsg</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>In reviewing other language designs, I found that this design is very similar to Crystal's IO buffering strategy.</p>
<p>The proposed implementation provides enough of an interface to implement both native schedulers as well as pure Ruby schedulers. It also provides some extra functionality for interpreting the data in the buffer. This is mostly for testing and experimentation, although it might make sense to expose this interface for binary protocols like HTTP/2, QUIC, WebSockets, etc.</p>
<a name="Proposed-Solution"></a>
<h2 >Proposed Solution<a href="#Proposed-Solution" class="wiki-anchor">¶</a></h2>
<p>We introduce new class <code>IO::Buffer</code>.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">IO::Buffer</span>
<span class="c1"># @returns [IO::Buffer] A buffer with the contents of the string data.</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">for</span><span class="p">(</span><span class="n">string</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">PAGE_SIZE</span> <span class="o">=</span> <span class="c1"># ... operating system page size</span>
<span class="c1"># @returns [IO::Buffer] A buffer with the contents of the file mapped to memory.</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">map</span><span class="p">(</span><span class="n">file</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Flags for buffer state.</span>
<span class="no">EXTERNAL</span> <span class="o">=</span> <span class="c1"># The buffer is from external memory.</span>
<span class="no">INTERNAL</span> <span class="o">=</span> <span class="c1"># The buffer is from internal memory (malloc).</span>
<span class="no">MAPPED</span> <span class="o">=</span> <span class="c1"># The buffer is from mapped memory (mmap, VirtualAlloc, etc)</span>
<span class="no">LOCKED</span> <span class="o">=</span> <span class="c1"># The buffer is locked for usage (cannot be resized)</span>
<span class="no">PRIVATE</span> <span class="o">=</span> <span class="c1"># The buffer is mapped as copy-on-write.</span>
<span class="no">IMMUTABLE</span> <span class="o">=</span> <span class="c1"># The buffer cannot be modified.</span>
<span class="c1"># @returns [IO::Buffer] A buffer with the specified size, allocated according to the given flags.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">flags</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># @returns [Integral] The size of the buffer</span>
<span class="kp">attr</span> <span class="ss">:size</span>
<span class="c1"># @returns [String] A brief summary and hex dump of the buffer.</span>
<span class="k">def</span> <span class="nf">inspect</span>
<span class="k">end</span>
<span class="c1"># @returns [String] A brief summary of the buffer.</span>
<span class="k">def</span> <span class="nf">to_s</span>
<span class="k">end</span>
<span class="c1"># Flag predicates:</span>
<span class="k">def</span> <span class="nf">external?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">internal?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">mapped?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">locked?</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">immutable?</span>
<span class="k">end</span>
<span class="c1"># Flags for endian/byte order:</span>
<span class="no">LITTLE_ENDIAN</span> <span class="o">=</span> <span class="c1"># ...</span>
<span class="no">BIG_ENDIAN</span> <span class="o">=</span> <span class="c1"># ...</span>
<span class="no">HOST_ENDIAN</span> <span class="o">=</span> <span class="c1"># ...</span>
<span class="no">NETWORK_ENDIAN</span><span class="o">=</span> <span class="c1"># ...</span>
<span class="c1"># Lock the buffer (prevent resize, unmap, changes to base and size).</span>
<span class="k">def</span> <span class="nf">lock</span>
<span class="k">raise</span> <span class="s2">"Already locked!"</span> <span class="k">if</span> <span class="n">flags</span> <span class="o">&</span> <span class="no">LOCKED</span>
<span class="n">flags</span> <span class="o">|=</span> <span class="no">LOCKED</span>
<span class="k">end</span>
<span class="c1"># Unlock the buffer.</span>
<span class="k">def</span> <span class="nf">unlock</span>
<span class="k">raise</span> <span class="s2">"Not locked!"</span> <span class="k">unless</span> <span class="n">flags</span> <span class="o">&</span> <span class="no">LOCKED</span>
<span class="n">flags</span> <span class="o">|=</span> <span class="o">~</span><span class="no">LOCKED</span>
<span class="k">end</span>
<span class="sr">//</span> <span class="no">Manipulation</span><span class="p">:</span>
<span class="c1"># @returns [IO::Buffer] A slice of the buffer's data. Does not copy.</span>
<span class="k">def</span> <span class="nf">slice</span><span class="p">(</span><span class="n">offset</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># @returns [String] A binary string starting at offset, length bytes.</span>
<span class="k">def</span> <span class="nf">to_str</span><span class="p">(</span><span class="n">offset</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Copy the specified string into the buffer at the given offset.</span>
<span class="k">def</span> <span class="nf">copy</span><span class="p">(</span><span class="n">string</span><span class="p">,</span> <span class="n">offset</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Compare two buffers.</span>
<span class="k">def</span> <span class="nf"><</span><span class="o">=></span><span class="p">(</span><span class="n">other</span><span class="p">)</span>
<span class="k">end</span>
<span class="kp">include</span> <span class="no">Comparable</span>
<span class="c1"># Resize the buffer, preserving the given length (if non-zero).</span>
<span class="k">def</span> <span class="nf">resize</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">preserve</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Clear the buffer to the specified value.</span>
<span class="k">def</span> <span class="nf">clear</span><span class="p">(</span><span class="n">value</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">offset</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="p">(</span><span class="vi">@size</span> <span class="o">-</span> <span class="n">offset</span><span class="p">))</span>
<span class="k">end</span>
<span class="c1"># Data Types:</span>
<span class="c1"># Lower case: little endian.</span>
<span class="c1"># Upper case: big endian (network endian).</span>
<span class="c1">#</span>
<span class="c1"># :U8 | unsigned 8-bit integer.</span>
<span class="c1"># :S8 | signed 8-bit integer.</span>
<span class="c1">#</span>
<span class="c1"># :u16, :U16 | unsigned 16-bit integer.</span>
<span class="c1"># :s16, :S16 | signed 16-bit integer.</span>
<span class="c1">#</span>
<span class="c1"># :u32, :U32 | unsigned 32-bit integer.</span>
<span class="c1"># :s32, :S32 | signed 32-bit integer.</span>
<span class="c1">#</span>
<span class="c1"># :u64, :U64 | unsigned 64-bit integer.</span>
<span class="c1"># :s64, :S64 | signed 64-bit integer.</span>
<span class="c1">#</span>
<span class="c1"># :f32, :F32 | 32-bit floating point number.</span>
<span class="c1"># :f64, :F64 | 64-bit floating point number.</span>
<span class="c1"># Get the given data type at the specified offset.</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">type</span><span class="p">,</span> <span class="n">offset</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Set the given value as the specified data type at the specified offset.</span>
<span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="n">type</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>The C interface provides a few convenient methods for accessing the underlying data buffer:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="kt">void</span> <span class="nf">rb_io_buffer_get_mutable</span><span class="p">(</span><span class="n">VALUE</span> <span class="n">self</span><span class="p">,</span> <span class="kt">void</span> <span class="o">**</span><span class="n">base</span><span class="p">,</span> <span class="kt">size_t</span> <span class="o">*</span><span class="n">size</span><span class="p">);</span>
<span class="kt">void</span> <span class="nf">rb_io_buffer_get_immutable</span><span class="p">(</span><span class="n">VALUE</span> <span class="n">self</span><span class="p">,</span> <span class="k">const</span> <span class="kt">void</span> <span class="o">**</span><span class="n">base</span><span class="p">,</span> <span class="kt">size_t</span> <span class="o">*</span><span class="n">size</span><span class="p">);</span>
</code></pre>
<p>In the fiber scheduler, it is used like this:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">VALUE</span>
<span class="nf">rb_fiber_scheduler_io_read_memory</span><span class="p">(</span><span class="n">VALUE</span> <span class="n">scheduler</span><span class="p">,</span> <span class="n">VALUE</span> <span class="n">io</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">base</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">size</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">length</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">VALUE</span> <span class="n">buffer</span> <span class="o">=</span> <span class="n">rb_io_buffer_new</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">RB_IO_BUFFER_LOCKED</span><span class="p">);</span>
<span class="n">VALUE</span> <span class="n">result</span> <span class="o">=</span> <span class="n">rb_fiber_scheduler_io_read</span><span class="p">(</span><span class="n">scheduler</span><span class="p">,</span> <span class="n">io</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">length</span><span class="p">);</span>
<span class="n">rb_io_buffer_free</span><span class="p">(</span><span class="n">buffer</span><span class="p">);</span>
<span class="k">return</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
<p>This function is invoked from <code>io.c</code> at various places to fill the buffer. We specifically the <code>(base, size)</code> tuple, along with <code>length</code> which is the <em>minimum</em> length required and assists with efficient non-blocking implementation.</p>
<p>The <code>uring.c</code> implementation in the event gem uses this interface like so:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">VALUE</span> <span class="nf">Event_Backend_URing_io_read</span><span class="p">(</span><span class="n">VALUE</span> <span class="n">self</span><span class="p">,</span> <span class="n">VALUE</span> <span class="n">fiber</span><span class="p">,</span> <span class="n">VALUE</span> <span class="n">io</span><span class="p">,</span> <span class="n">VALUE</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">VALUE</span> <span class="n">_length</span><span class="p">)</span> <span class="p">{</span>
<span class="k">struct</span> <span class="n">Event_Backend_URing</span> <span class="o">*</span><span class="n">data</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
<span class="n">TypedData_Get_Struct</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="k">struct</span> <span class="n">Event_Backend_URing</span><span class="p">,</span> <span class="o">&</span><span class="n">Event_Backend_URing_Type</span><span class="p">,</span> <span class="n">data</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">descriptor</span> <span class="o">=</span> <span class="n">RB_NUM2INT</span><span class="p">(</span><span class="n">rb_funcall</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">id_fileno</span><span class="p">,</span> <span class="mi">0</span><span class="p">));</span>
<span class="kt">void</span> <span class="o">*</span><span class="n">base</span><span class="p">;</span>
<span class="kt">size_t</span> <span class="n">size</span><span class="p">;</span>
<span class="n">rb_io_buffer_get_mutable</span><span class="p">(</span><span class="n">buffer</span><span class="p">,</span> <span class="o">&</span><span class="n">base</span><span class="p">,</span> <span class="o">&</span><span class="n">size</span><span class="p">);</span>
<span class="kt">size_t</span> <span class="n">offset</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">size_t</span> <span class="n">length</span> <span class="o">=</span> <span class="n">NUM2SIZET</span><span class="p">(</span><span class="n">_length</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="n">length</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">size_t</span> <span class="n">maximum_size</span> <span class="o">=</span> <span class="n">size</span> <span class="o">-</span> <span class="n">offset</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">result</span> <span class="o">=</span> <span class="n">io_read</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">fiber</span><span class="p">,</span> <span class="n">descriptor</span><span class="p">,</span> <span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="n">base</span><span class="o">+</span><span class="n">offset</span><span class="p">,</span> <span class="n">maximum_size</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">result</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">offset</span> <span class="o">+=</span> <span class="n">result</span><span class="p">;</span>
<span class="k">if</span> <span class="p">((</span><span class="kt">size_t</span><span class="p">)</span><span class="n">result</span> <span class="o">></span> <span class="n">length</span><span class="p">)</span> <span class="k">break</span><span class="p">;</span>
<span class="n">length</span> <span class="o">-=</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="o">-</span><span class="n">result</span> <span class="o">==</span> <span class="n">EAGAIN</span> <span class="o">||</span> <span class="o">-</span><span class="n">result</span> <span class="o">==</span> <span class="n">EWOULDBLOCK</span><span class="p">)</span> <span class="p">{</span>
<span class="n">Event_Backend_URing_io_wait</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">fiber</span><span class="p">,</span> <span class="n">io</span><span class="p">,</span> <span class="n">RB_INT2NUM</span><span class="p">(</span><span class="n">READABLE</span><span class="p">));</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">rb_syserr_fail</span><span class="p">(</span><span class="o">-</span><span class="n">result</span><span class="p">,</span> <span class="n">strerror</span><span class="p">(</span><span class="o">-</span><span class="n">result</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">SIZET2NUM</span><span class="p">(</span><span class="n">offset</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
<a name="Buffer-Allocation"></a>
<h2 >Buffer Allocation<a href="#Buffer-Allocation" class="wiki-anchor">¶</a></h2>
<p>The Linux kernel provides some advanced mechanisms for registering buffers for asynchronous I/O to reduce per-operation overhead.</p>
<blockquote>
<p>The io_uring_register() system call registers user buffers or files for use in an io_uring(7) instance referenced by fd. Registering files or user buffers allows the kernel to take long term references to internal data structures or create long term mappings of application memory, greatly reducing per-I/O overhead.</p>
</blockquote>
<p>With appropriate support, we can use <code>IORING_OP_PROVIDE_BUFFERS</code> to efficiently manage buffers in applications which are dealing with lots of sockets. See <a href="https://lore.kernel.org/io-uring/20200228203053.25023-1-axboe@kernel.dk/T/" class="external">https://lore.kernel.org/io-uring/20200228203053.25023-1-axboe@kernel.dk/T/</a> for more details about how it works. I'm still exploring the performance implications of this, but the proposed implementation provides sufficient meta-data for us to explore this in real world schedulers.</p>
<p>PR: <a href="https://github.com/ruby/ruby/pull/4621" class="external">https://github.com/ruby/ruby/pull/4621</a></p> Ruby master - Feature #18015 (Closed): Replace copy coroutine with pthread implementation.https://bugs.ruby-lang.org/issues/180152021-06-30T04:29:19Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>The copy coroutine is unique in that it uses a private stack area. It is also tricky to implement as it uses setjmp/longjmp to update the stack & instruction pointer. Because of this, pointers to stack allocated data are only valid while the fiber is running, and this makes implementation tricky for any code which uses stack allocated structs.</p>
<p>Previously we introduced some macros for dealing with this - copy coroutines fall back to malloc/free rather than using stack locals, but the chance if this causing problems is very high. In addition, rb_protect prevents us from switching fibers, which also prevents the fiber scheduler from operating within this context.</p>
<p>We propose to remove the copy coroutine implementation and replace it with a pthread-based cooperative scheduled implementation. This implementation uses one native thread per fiber, but only allows one of them to execute at a time using a mutex. This approach should be more consistent with the other native implementations and allow us to simplify a number of implementation details.</p> Ruby master - Feature #18005 (Closed): Enable non-blocking `binding.irb`.https://bugs.ruby-lang.org/issues/180052021-06-23T10:28:05Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>This is a multi-faceted issue.</p>
<p>Firstly, we need to make some library changes, remove <code>IO.select</code>, etc from <code>reline</code> and so on.</p>
<p>Then, we need to make <code>$stdin</code> non-blocking (maybe by default - was planned).</p>
<p>Finally, we need to figure out whether we can relax <code>rb_protect</code> to allow fiber transfer.</p> Ruby master - Bug #18003 (Closed): Rework internal IO to directly invoke scheduler with IO object.https://bugs.ruby-lang.org/issues/180032021-06-22T11:01:31Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Many functions are still going through C functions which take file descriptors fd argument, and internally we need to create an IO object to handle this correctly in the fiber scheduler. This is slightly inefficient, so we should rework these code paths to use the updated interfaces which handle IO instances directly.</p> Ruby master - Bug #17827 (Closed): Monitor is not fiber safehttps://bugs.ruby-lang.org/issues/178272021-04-25T21:30:36Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>According to our discussion here <a href="https://github.com/rspec/rspec-support/issues/501" class="external">https://github.com/rspec/rspec-support/issues/501</a> it seems like typical implementation of per-thread reentrant mutex is no longer valid and can lead to some deadlock situation.</p>
<pre><code>#!/usr/bin/env ruby
require 'monitor'
def monitor_failure
m = Monitor.new
f1 = Fiber.new do
m.synchronize do
puts "f1 A"
Fiber.yield
puts "f1 B"
end
end
f2 = Fiber.new do
m.synchronize do
puts "f2 A"
# Fiber.yield
f1.resume
puts "f2 B"
end
end
f1.resume
f2.resume
end
monitor_failure
</code></pre> Ruby master - Feature #17470 (Closed): Introduce non-blocking `Timeout.timeout`https://bugs.ruby-lang.org/issues/174702020-12-26T07:42:23Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In this bug report, user complained that <code>Timeout.timeout</code> does not work correctly with scheduler: <a href="https://github.com/socketry/async-io/issues/43" class="external">https://github.com/socketry/async-io/issues/43</a></p>
<p>We should introduce non-blocking timeout.</p>
<p>I propose the following:</p>
<pre><code>rb_fiber_scheduler_with_timeout(VALUE scheduler, VALUE timeout, VALUE block)
</code></pre>
<p>We can directly modify <code>Timeout.timeout</code> to invoke this hook.</p> Ruby master - Feature #17370 (Closed): Introduce non-blocking `Addrinfo.getaddrinfo` and related ...https://bugs.ruby-lang.org/issues/173702020-12-05T12:18:27Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>We would like to introduce a new scheduler hook for non-blocking <code>getaddrinfo</code>.</p>
<pre><code>class Scheduler
def address_resolve(...)
[array of addrinfo objects]
end
</code></pre>
<p><a href="https://github.com/bruno-/ruby/pull/1" class="external">https://github.com/bruno-/ruby/pull/1</a></p>
<p>This is a work in progress.</p> Ruby master - Feature #17369 (Closed): Introduce non-blocking `Process.wait`, `Kernel.system` and...https://bugs.ruby-lang.org/issues/173692020-12-05T12:10:34Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><a href="https://github.com/ruby/ruby/pull/3853" class="external">https://github.com/ruby/ruby/pull/3853</a></p>
<p>This PR introduces optional hooks to the scheduler interface for handling <code>Process.wait</code>, <code>Kernel.system</code> and other related methods (<code>waitpid</code>, <code>wait2</code>, etc).</p>
<p>It funnels all methods through a new interface <code>Process::Status.wait</code> which is almost identical to <code>Process.wait</code> except for several key differences:</p>
<ul>
<li>The return value is a single instance of <code>Process::Status</code>.</li>
<li>It does not set thread local <code>$?</code>.</li>
</ul>
<p>This is necessary for keeping the scheduler interface simple (and side effects are generally bad anyway).</p> Ruby master - Bug #17107 (Rejected): Backtick in backtrace is a little bit annoyinghttps://bugs.ruby-lang.org/issues/171072020-08-08T02:30:54Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>In Ruby exception backtrace (and other places), the method name uses a quoting style of a backtick followed by a text/name followed by an apostrophy.</p>
<pre><code>in `<main>'
</code></pre>
<p>The quoting style is not safe, as in, it cannot be relied upon for parsing, because it's easy to inject quotes into the output stream:</p>
<p>e.g.</p>
<pre><code>irb(main):006:0> self.send(:"I'm Great")
Traceback (most recent call last):
4: from /tmp/64556713-b3a1-40a1-b458-bfed3fec55b2:8:in `<main>'
3: from <internal:prelude>:20:in `irb'
2: from /tmp/64556713-b3a1-40a1-b458-bfed3fec55b2:5:in `<main>'
1: from /tmp/64556713-b3a1-40a1-b458-bfed3fec55b2:6:in `rescue in <main>'
NoMethodError (undefined method `I'm Great' for main:Object)
</code></pre>
<p>Note the last line now has ambiguous quoting.</p>
<ul>
<li>Should we fix this quoting style?</li>
<li>Should we remove this quoting style?</li>
<li>Should we choose some other quoting style?</li>
</ul>
<p>As a specific point:</p>
<ul>
<li>Can we use two apostrophes rather than a backtick/apostrophe? Backticks often have a special meaning (e.g. markdown) which makes it somewhat annoying to copy and paste.</li>
</ul> Ruby master - Misc #17050 (Closed): profiler gemhttps://bugs.ruby-lang.org/issues/170502020-07-25T00:03:02Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I would like to reuse profiler gem namespace:</p>
<p><a href="https://rubygems.org/gems/profiler/versions" class="external">https://rubygems.org/gems/profiler/versions</a></p>
<p>It currently conflicts with <code>profile/lib/profiler.rb</code> however this is no longer part of stdlib.</p>
<p>cc <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a></p> Ruby master - Feature #16962 (Feedback): Make IO.for_fd autoclose option default to falsehttps://bugs.ruby-lang.org/issues/169622020-06-16T11:54:36Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I discussed this with <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/772">@Eregon (Benoit Daloze)</a> and I think the goal here is to try and figure out a way these interfaces can be a bit less confusing.</p>
<a name="1-I-dont-understand-this-behaviour"></a>
<h2 >1. I don't understand this behaviour:<a href="#1-I-dont-understand-this-behaviour" class="wiki-anchor">¶</a></h2>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">STDOUT</span><span class="p">.</span><span class="nf">close</span>
<span class="no">STDOUT</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"Hello World"</span>
<span class="c1"># => closed stream</span>
</code></pre>
<p>vs</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">IO</span><span class="p">.</span><span class="nf">for_fd</span><span class="p">(</span><span class="no">STDOUT</span><span class="p">.</span><span class="nf">fileno</span><span class="p">,</span> <span class="ss">autoclose: </span><span class="kp">true</span><span class="p">).</span><span class="nf">close</span>
<span class="no">STDOUT</span><span class="p">.</span><span class="nf">puts</span> <span class="s2">"Hello World"</span>
<span class="c1"># => Hello World</span>
</code></pre>
<h2>2. <code>IO.for_fd(..., autoclose: true/false)</code>
</h2>
<p>The documentation for <code>autoclose</code> is:</p>
<blockquote>
<p>If the value is false, the fd will be kept open after this IO instance gets finalized.</p>
</blockquote>
<p>But it also seems to affect <code>#close</code> - i.e. calling close does not close underlying file descriptor.</p>
<p>Should we fix the documentation or is the implementation wrong? Maybe the name <code>autoclose:</code> is very confusing. My initial interpretation was it was just 'automatically close this I/O when it is garbage collected'.</p>
<a name="3-IOfor_fd-autoclose-false-default"></a>
<h2 >3. <code>IO.for_fd(..., autoclose: false)</code> default<a href="#3-IOfor_fd-autoclose-false-default" class="wiki-anchor">¶</a></h2>
<p>In most cases, it seems like <code>autoclose: false</code> would make more sense as the default, since the file descriptor must come from some other place.</p> Ruby master - Bug #16908 (Closed): Strange behaviour of Hash#shift when used with `default_proc`.https://bugs.ruby-lang.org/issues/169082020-05-23T02:15:53Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I don't have any strong opinion about this, but I observed the following behaviour which I thought was confusing. Maybe it's okay, or maybe we should change it to be more consistent.</p>
<pre><code>hash = Hash.new{|k,v| k[v] = 0}
hash.shift # => 0
hash.shift # => [nil, 0]
</code></pre>
<p>My feeling was, both cases should return <code>[nil, 0]</code>.</p> Ruby master - Feature #16815 (Closed): Implement Fiber#backtracehttps://bugs.ruby-lang.org/issues/168152020-04-24T08:53:14Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Previously discussed here: <a href="https://bugs.ruby-lang.org/issues/8215" class="external">https://bugs.ruby-lang.org/issues/8215</a></p>
<p>Add a new method <code>Fiber#backtrace</code>. It would produce a backtrace similar to how <code>Thread#backtrace</code> works.</p> Ruby master - Misc #16802 (Closed): Prefer use of RHS assigment in documentationhttps://bugs.ruby-lang.org/issues/168022020-04-21T05:13:44Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Many documentation uses some format like this:</p>
<pre><code> Hash.new -> hash
</code></pre>
<p>Now that RHS assignment is a thing, we can make the documentation valid Ruby code:</p>
<pre><code> Hash.new => hash
</code></pre>
<p>There is some discussion here: <a href="https://github.com/ruby/ruby/pull/3026" class="external">https://github.com/ruby/ruby/pull/3026</a></p>
<p>I would like to start changing the existing documentation to use this, and in addition, make an automated sweep of the entire code base to update to the new syntax.</p> Ruby master - Feature #16786 (Closed): Light-weight scheduler for improved concurrency.https://bugs.ruby-lang.org/issues/167862020-04-14T14:38:58Zioquatix (Samuel Williams)samuel@oriontransfer.net
<a name="Abstract"></a>
<h1 >Abstract<a href="#Abstract" class="wiki-anchor">¶</a></h1>
<p>We propose to introduce a light weight fiber scheduler, to improve the concurrency of Ruby code with minimal changes.</p>
<a name="Background"></a>
<h1 >Background<a href="#Background" class="wiki-anchor">¶</a></h1>
<p>We have been discussing and considering options to improve Ruby scalability for several years. More context can be provided by the following discussions:</p>
<ul>
<li><a href="https://bugs.ruby-lang.org/issues/14736" class="external">https://bugs.ruby-lang.org/issues/14736</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/13618" class="external">https://bugs.ruby-lang.org/issues/13618</a></li>
</ul>
<p>The final Ruby Concurrency report provides some background on the various issues considered in the latest iteration: <a href="https://www.codeotaku.com/journal/2020-04/ruby-concurrency-final-report/index" class="external">https://www.codeotaku.com/journal/2020-04/ruby-concurrency-final-report/index</a></p>
<a name="Proposal"></a>
<h1 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h1>
<p>We propose to introduce the following concepts:</p>
<ul>
<li>A <code>Scheduler</code> interface which provides hooks for user-supplied event loops.</li>
<li>Non-blocking <code>Fiber</code> which can invoke the scheduler when it would otherwise block.</li>
</ul>
<a name="Scheduler"></a>
<h2 >Scheduler<a href="#Scheduler" class="wiki-anchor">¶</a></h2>
<p>The per-thread fiber scheduler interface is used to intercept blocking operations. A typical implementation would be a wrapper for a gem like EventMachine or Async. This design provides separation of concerns between the event loop implementation and application code. It also allows for layered schedulers which can perform instrumentation, enforce constraints (e.g. during testing) and provide additional logging. You can see a <a href="https://github.com/socketry/async/pull/56" class="external">sample implementation here</a>.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Scheduler</span>
<span class="c1"># Wait for the given file descriptor to become readable.</span>
<span class="k">def</span> <span class="nf">wait_readable</span><span class="p">(</span><span class="n">io</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Wait for the given file descriptor to become writable.</span>
<span class="k">def</span> <span class="nf">wait_writable</span><span class="p">(</span><span class="n">io</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Wait for the given file descriptor to match the specified events within</span>
<span class="c1"># the specified timeout.</span>
<span class="c1"># @param event [Integer] a bit mask of +IO::WAIT_READABLE+,</span>
<span class="c1"># `IO::WAIT_WRITABLE` and `IO::WAIT_PRIORITY`.</span>
<span class="c1"># @param timeout [#to_f] the amount of time to wait for the event.</span>
<span class="k">def</span> <span class="nf">wait_any</span><span class="p">(</span><span class="n">io</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">timeout</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Sleep the current task for the specified duration, or forever if not</span>
<span class="c1"># specified.</span>
<span class="c1"># @param duration [#to_f] the amount of time to sleep.</span>
<span class="k">def</span> <span class="nf">wait_sleep</span><span class="p">(</span><span class="n">duration</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># The Ruby virtual machine is going to enter a system level blocking</span>
<span class="c1"># operation.</span>
<span class="k">def</span> <span class="nf">enter_blocking_region</span>
<span class="k">end</span>
<span class="c1"># The Ruby virtual machine has completed the system level blocking</span>
<span class="c1"># operation.</span>
<span class="k">def</span> <span class="nf">exit_blocking_region</span>
<span class="k">end</span>
<span class="c1"># Intercept the creation of a non-blocking fiber.</span>
<span class="k">def</span> <span class="nf">fiber</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">blocking: </span><span class="kp">false</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Invoked when the thread exits.</span>
<span class="k">def</span> <span class="nf">run</span>
<span class="c1"># Implement event loop here.</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>A thread has a non-blocking fiber scheduler. All blocking operations on non-blocking fibers are hooked by the scheduler and the scheduler can switch to another fiber. If any mutex is acquired by a fiber, then a scheduler is not called; the same behaviour as blocking Fiber.</p>
<p>Schedulers can be written in Ruby. This is a desirable property as it allows them to be used in different implementations of Ruby easily.</p>
<p>To enable non-blocking fiber switching on blocking operations:</p>
<ul>
<li>Specify a scheduler: <code>Thread.current.scheduler = Scheduler.new</code>.</li>
<li>Create several non-blocking fibers: <code>Fiber.new(blocking:false) {...}</code>.</li>
<li>As the main fiber exits, <code>Thread.current.scheduler.run</code> is invoked which<br>
begins executing the event loop until all fibers are finished.</li>
</ul>
<a name="TimeDuration-Arguments"></a>
<h3 >Time/Duration Arguments<a href="#TimeDuration-Arguments" class="wiki-anchor">¶</a></h3>
<p>Tony Arcieri suggested against using floating point values for time/durations, because they can accumulate rounding errors and other issues. He has a wealth of experience in this area so his advice should be considered carefully. However, I have yet to see these issues happen in an event loop. That being said, round tripping between <code>struct timeval</code> and <code>double</code>/<code>VALUE</code> seems a bit inefficient. One option is to have an opaque argument that responds to <code>to_f</code> as well as potentially <code>seconds</code> and <code>microseconds</code> or some other such interface (could be opaque argument supported by <code>IO.select</code> for example).</p>
<a name="File-Descriptor-Arguments"></a>
<h3 >File Descriptor Arguments<a href="#File-Descriptor-Arguments" class="wiki-anchor">¶</a></h3>
<p>Because of the public C interface we may need to support a specific set of wrappers for CRuby.</p>
<pre><code class="c syntaxhl" data-language="c"><span class="kt">int</span> <span class="nf">rb_io_wait_readable</span><span class="p">(</span><span class="kt">int</span><span class="p">);</span>
<span class="kt">int</span> <span class="nf">rb_io_wait_writable</span><span class="p">(</span><span class="kt">int</span><span class="p">);</span>
<span class="kt">int</span> <span class="nf">rb_wait_for_single_fd</span><span class="p">(</span><span class="kt">int</span> <span class="n">fd</span><span class="p">,</span> <span class="kt">int</span> <span class="n">events</span><span class="p">,</span> <span class="k">struct</span> <span class="n">timeval</span> <span class="o">*</span><span class="n">tv</span><span class="p">);</span>
</code></pre>
<p>One option is to introduce hooks specific to CRuby:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">class</span> <span class="nc">Scheduler</span>
<span class="c1"># Wrapper for rb_io_wait_readable(int) C function.</span>
<span class="k">def</span> <span class="nf">wait_readable_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
<span class="n">wait_readable</span><span class="p">(</span><span class="o">::</span><span class="no">IO</span><span class="p">.</span><span class="nf">from_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="ss">autoclose: </span><span class="kp">false</span><span class="p">))</span>
<span class="k">end</span>
<span class="c1"># Wrapper for rb_io_wait_readable(int) C function.</span>
<span class="k">def</span> <span class="nf">wait_writable_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
<span class="n">wait_writable</span><span class="p">(</span><span class="o">::</span><span class="no">IO</span><span class="p">.</span><span class="nf">from_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="ss">autoclose: </span><span class="kp">false</span><span class="p">))</span>
<span class="k">end</span>
<span class="c1"># Wrapper for rb_wait_for_single_fd(int) C function.</span>
<span class="k">def</span> <span class="nf">wait_for_single_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">duration</span><span class="p">)</span>
<span class="n">wait_any</span><span class="p">(</span><span class="o">::</span><span class="no">IO</span><span class="p">.</span><span class="nf">from_fd</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="ss">autoclose: </span><span class="kp">false</span><span class="p">),</span> <span class="n">events</span><span class="p">,</span> <span class="n">duration</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
<p>Alternatively, in CRuby, it may be possible to map from <code>fd</code> -> <code>IO</code> instance. Most C schedulers only care about file descriptor, so such a mapping will introduce a small performance penalty. In addition, most C level schedulers will not care about <code>IO</code> instance.</p>
<a name="Non-blocking-Fiber"></a>
<h2 >Non-blocking Fiber<a href="#Non-blocking-Fiber" class="wiki-anchor">¶</a></h2>
<p>We propose to introduce per-fiber flag <code>blocking: true/false</code>.</p>
<p>A fiber created by <code>Fiber.new(blocking: true)</code> (the default <code>Fiber.new</code>) becomes a "blocking Fiber" and has no changes from current Fiber implementation. This includes the root fiber.</p>
<p>A fiber created by <code>Fiber.new(blocking: false)</code> becomes a "non-blocking Fiber" and it will be scheduled by the per-thread scheduler when the blocking operations (blocking I/O, sleep, and so on) occurs.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">blocking: </span><span class="kp">false</span><span class="p">)</span> <span class="k">do</span>
<span class="nb">puts</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">blocking?</span> <span class="c1"># false</span>
<span class="c1"># May invoke `Thread.scheduler&.wait_readable`.</span>
<span class="n">io</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="c1"># May invoke `Thread.scheduler&.wait_writable`.</span>
<span class="n">io</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="c1"># Will invoke `Thread.scheduler&.wait_sleep`.</span>
<span class="nb">sleep</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="k">end</span><span class="p">.</span><span class="nf">resume</span>
</code></pre>
<p>Non-blocking fibers also supports <code>Fiber#resume</code>, <code>Fiber#transfer</code> and <code>Fiber.yield</code> which are necessary to create a scheduler.</p>
<a name="Fiber-Method"></a>
<h3 >Fiber Method<a href="#Fiber-Method" class="wiki-anchor">¶</a></h3>
<p>We also introduce a new method which simplifes the creation of these non-blocking fibers:</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="no">Fiber</span> <span class="k">do</span>
<span class="nb">puts</span> <span class="no">Fiber</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">blocking?</span> <span class="c1"># false</span>
<span class="k">end</span>
</code></pre>
<p>This method invokes <code>Scheduler#fiber(...)</code>. The purpose of this method is to allow the scheduler to internally decide the policy for when to start the fiber, and whether to use symmetric or asymmetric fibers.</p>
<p>If no scheduler is specified, it is a error: <code>RuntimeError.new("No scheduler is available")</code>.</p>
<p>In the future we may expand this to support some kind of default scheduler.</p>
<a name="Non-blocking-IO"></a>
<h2 >Non-blocking I/O<a href="#Non-blocking-IO" class="wiki-anchor">¶</a></h2>
<p><code>IO#nonblock</code> is an existing interface to control whether I/O uses blocking or non-blocking system calls. We can take advantage of this:</p>
<ul>
<li>
<code>IO#nonblock = false</code> prevents that particular IO from utilising the scheduler. This should be the default for <code>stderr</code>.</li>
<li>
<code>IO#nonblock = true</code> enables that particular IO to utilise the scheduler. We should enable this where possible.</li>
</ul>
<p>As proposed by Eric Wong, we believe that making I/O non-blocking by default is the right approach. We have expanded his work in the current implementation. By doing this, when the user writes <code>Fiber do ... end</code> they are guaranteed the best possible concurrency possible, without any further changes to code. As an example, one of the tests shows <code>Net::HTTP.get</code> being used in this way with no further modifications required.</p>
<p>To support this further, consider the counterpoint, that <code>Net::HTTP.get(..., blocking: false)</code> is required for concurrent requests. Library code may not expose the relevant options, sevearly limiting the user's ability to improve concurrency, even if that is what they desire.</p>
<a name="Implementation"></a>
<h1 >Implementation<a href="#Implementation" class="wiki-anchor">¶</a></h1>
<p>We have an evolving implementation here: <a href="https://github.com/ruby/ruby/pull/3032" class="external">https://github.com/ruby/ruby/pull/3032</a> which we will continue to update as the proposal changes.</p>
<a name="Evaluation"></a>
<h1 >Evaluation<a href="#Evaluation" class="wiki-anchor">¶</a></h1>
<p>This proposal provides the hooks for scheduling fibers. With regards to performance, there are several things to consider:</p>
<ul>
<li>The impact of the scheduler design on non-concurrent workloads. We believe it's acceptable.</li>
<li>The impact of the scheduler design on concurrent workloads. Our results are promising.</li>
<li>The impact of different event loops on throughput and latency. We have independent tests which confirm the scalability of the approach.</li>
</ul>
<p>We can control for the first two in this proposal, and depending on the design we may help or hinder the wrapper implementation.</p>
<p>In the tests, we provide a basic implementation using <code>IO.select</code>. As this proposal is finalised, we will introduce some basic benchmarks using this approach.</p>
<a name="Discussion"></a>
<h1 >Discussion<a href="#Discussion" class="wiki-anchor">¶</a></h1>
<p>The following points are good ones for discussion:</p>
<ul>
<li>Handling of file descriptors vs <code>IO</code> instances.</li>
<li>Handling of time/duration arguments.</li>
<li>General design and naming conventions.</li>
<li>Potential platform issues (e.g. CRuby vs JRuby vs TruffleRuby, etc).</li>
</ul>
<p>The following is planned to be described by <a class="user active user-mention" href="https://bugs.ruby-lang.org/users/772">@Eregon (Benoit Daloze)</a> in another design document:</p>
<ul>
<li>Semantics of non-blocking mutex (e.g. <code>Mutex.new(blocking: false)</code> or some other approach).</li>
</ul>
<p>In the future we hope to extend the scheduler to handle other blocking operations, including name resolution, file I/O (by <code>io_uring</code>) and others. We may need to introduce additional hooks. If these hooks are not defined on the scheduler implementation, we will revert back to the blocking implementation where possible.</p> Ruby master - Bug #16782 (Closed): `lock': deadlock; recursive locking (ThreadError) in 2.7.1https://bugs.ruby-lang.org/issues/167822020-04-13T03:42:18Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Using the latest <code>async-http</code> repo:</p>
<pre><code>% bundle exec rspec spec/async/http/performance_spec.rb
Traceback (most recent call last):
11: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/task.rb:258:in `block in make_fiber'
10: from /home/samuel/.gem/ruby/2.7.1/gems/async-io-1.28.0/lib/async/io/socket.rb:73:in `block in accept'
9: from /home/samuel/.gem/ruby/2.7.1/gems/async-io-1.28.0/lib/async/io/server.rb:32:in `block in accept_each'
8: from /home/samuel/Documents/socketry/async-http/lib/async/http/server.rb:67:in `accept'
7: from /home/samuel/Documents/socketry/async-http/lib/async/http/protocol/http1/connection.rb:80:in `close'
6: from /home/samuel/.gem/ruby/2.7.1/gems/protocol-http1-0.12.0/lib/protocol/http1/connection.rb:135:in `close'
5: from /home/samuel/.gem/ruby/2.7.1/gems/async-io-1.28.0/lib/async/io/stream.rb:224:in `close'
4: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/wrapper.rb:164:in `close'
3: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/wrapper.rb:215:in `cancel_monitor'
2: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/wrapper.rb:215:in `close'
1: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/wrapper.rb:215:in `deregister'
/home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/wrapper.rb:215:in `lock': SIGTERM (SignalException)
9: from /home/samuel/.gem/ruby/2.7.1/gems/async-container-0.14.1/lib/async/container/forked.rb:64:in `block in spawn'
8: from /home/samuel/.gem/ruby/2.7.1/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
7: from /home/samuel/.gem/ruby/2.7.1/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
6: from /home/samuel/.gem/ruby/2.7.1/gems/async-container-0.14.1/lib/async/container/forked.rb:67:in `block (2 levels) in spawn'
5: from /home/samuel/.gem/ruby/2.7.1/gems/async-container-0.14.1/lib/async/container/controller.rb:47:in `block in async'
4: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/reactor.rb:58:in `run'
3: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/reactor.rb:58:in `ensure in run'
2: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/reactor.rb:250:in `close'
1: from /home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/reactor.rb:250:in `close'
/home/samuel/.gem/ruby/2.7.1/gems/async-1.24.2/lib/async/reactor.rb:250:in `lock': deadlock; recursive locking (ThreadError)
runs benchmark
Finished in 4.5 seconds (files took 0.2483 seconds to load)
2 examples, 0 failures
% ruby --version
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
</code></pre>
<p>About 1 in 10-100 runs will trigger this result. I also tried current head and could reproduce:</p>
<pre><code>Running 2s test @ http://127.0.0.1:9294/
8 threads and 8 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.35ms 14.03ms 46.73ms 78.86%
Req/Sec 0.99k 454.60 2.30k 63.80%
16122 requests in 2.10s, 787.51KB read
Requests/sec: 7675.89
Transfer/sec: 374.94KB
4.66s error: Async::Task [oid=0x1360] [pid=69337] [2020-04-13 15:40:21 +1200]
| ThreadError: deadlock; recursive locking
| → /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:215 in `lock'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:215 in `deregister'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:215 in `close'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:215 in `cancel_monitor'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:164 in `close'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/shared_endpoint.rb:78 in `ensure in block (2 levels) in bind'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/shared_endpoint.rb:78 in `block (2 levels) in bind'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/task.rb:258 in `block in make_fiber'
| Caused by Async::Stop: Async::Stop
| → /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/task.rb:66 in `yield'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:233 in `wait_for'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/wrapper.rb:124 in `wait_readable'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/generic.rb:220 in `async_send'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/socket.rb:62 in `accept'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/server.rb:36 in `accept_each'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/shared_endpoint.rb:104 in `block in accept'
| /home/samuel/.gem/ruby/2.8.0/gems/async-io-1.28.0/lib/async/io/shared_endpoint.rb:76 in `block (2 levels) in bind'
| /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/task.rb:258 in `block in make_fiber'
Traceback (most recent call last):
9: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/forked.rb:64:in `block in spawn'
8: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
7: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
6: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/forked.rb:67:in `block (2 levels) in spawn'
5: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/controller.rb:47:in `block in async'
4: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:56:in `run'
3: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:234:in `run'
2: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:204:in `run_once'
1: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:204:in `select'
/home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:204:in `lock': SIGTERM (SignalException)
9: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/forked.rb:64:in `block in spawn'
8: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
7: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/group.rb:42:in `fork'
6: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/forked.rb:67:in `block (2 levels) in spawn'
5: from /home/samuel/.gem/ruby/2.8.0/gems/async-container-0.14.1/lib/async/container/controller.rb:47:in `block in async'
4: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:58:in `run'
3: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:58:in `ensure in run'
2: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:250:in `close'
1: from /home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:250:in `close'
/home/samuel/.gem/ruby/2.8.0/gems/async-1.24.2/lib/async/reactor.rb:250:in `lock': deadlock; recursive locking (ThreadError)
runs benchmark
Finished in 4.56 seconds (files took 0.31919 seconds to load)
2 examples, 0 failures
</code></pre>
<p>It's relatively straight forward to reproduce.</p> Ruby master - Feature #16602 (Rejected): Add support for `frozen_string_literals` to eval.https://bugs.ruby-lang.org/issues/166022020-02-01T07:29:40Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>Would it make sense for <code>eval(..., frozen_string_literal: true)</code> to exist?</p> Ruby master - Bug #16459 (Closed): <internal:trace_point>:346: [BUG] Segmentation fault at 0x0000...https://bugs.ruby-lang.org/issues/164592019-12-28T00:41:28Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>My code coverage gem which makes extensive use of <code>script_compiled</code> is causing segfault on 2.7.0 when accessing <code>event.instruction_sequence</code>. I didn't notice any issue in older rubies.</p>
<pre><code><internal:trace_point>:346: [BUG] Segmentation fault at 0x0000000000000008
ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0042 p:0003 s:0214 e:000213 METHOD <internal:trace_point>:346
c:0041 p:0050 s:0210 e:000207 BLOCK /home/samuel/.rvm/gems/ruby-2.7.0/gems/covered-0.13.1/lib/covered/source.rb:41 [FINISH]
c:0040 p:---- s:0202 e:000201 CFUNC :eval
c:0039 p:0026 s:0194 E:000578 BLOCK /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:5 [FINISH]
c:0038 p:---- s:0190 e:000189 CFUNC :catch
c:0037 p:0004 s:0186 E:0005a8 METHOD /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:4
c:0036 p:0105 s:0181 e:000180 METHOD /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:208
c:0035 p:0022 s:0169 e:000168 METHOD /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:184
c:0034 p:0018 s:0159 e:000158 BLOCK /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:154 [FINISH]
c:0033 p:---- s:0155 e:000154 CFUNC :each
c:0032 p:0005 s:0151 e:000150 METHOD /home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:152
c:0031 p:0029 s:0145 e:000144 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:82
c:0030 p:0125 s:0139 e:000138 CLASS /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:99
c:0029 p:0084 s:0136 e:000135 CLASS /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:46
c:0028 p:0007 s:0133 e:000132 CLASS /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:25
c:0027 p:0019 s:0130 e:000129 TOP /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:24 [FINISH]
c:0026 p:---- s:0127 e:000126 CFUNC :require_relative
c:0025 p:0017 s:0122 e:000121 TOP /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/socket.rb:24 [FINISH]
c:0024 p:---- s:0119 e:000118 CFUNC :require_relative
c:0023 p:0011 s:0114 e:000113 TOP /home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/endpoint.rb:22 [FINISH]
c:0022 p:---- s:0111 e:000110 CFUNC :require
c:0021 p:0005 s:0106 e:000105 TOP /home/samuel/Documents/socketry/async-http/lib/async/http/server.rb:23 [FINISH]
c:0020 p:---- s:0103 e:000102 CFUNC :require
c:0019 p:0005 s:0098 e:000097 TOP /home/samuel/Documents/socketry/async-http/spec/async/http/body/writable_examples.rb:21 [FINISH]
c:0018 p:---- s:0095 e:000094 CFUNC :require_relative
c:0017 p:0005 s:0090 e:000089 TOP /home/samuel/Documents/socketry/async-http/spec/async/http/body/slowloris_spec.rb:21 [FINISH]
c:0016 p:---- s:0087 e:000086 CFUNC :load
c:0015 p:0007 s:0082 e:000081 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:2076
c:0014 p:0023 s:0073 e:000072 BLOCK /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:1583 [FINISH]
c:0013 p:---- s:0068 e:000067 CFUNC :each
c:0012 p:0018 s:0064 e:000063 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:1581
c:0011 p:0009 s:0060 e:000059 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/covered-0.13.1/lib/covered/rspec.rb:45
c:0010 p:0036 s:0056 e:000055 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:102
c:0009 p:0007 s:0050 e:000049 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:86
c:0008 p:0065 s:0044 e:000043 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:71
c:0007 p:0020 s:0036 e:000035 METHOD /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:45
c:0006 p:0025 s:0031 e:000030 TOP /home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/exe/rspec:4 [FINISH]
c:0005 p:---- s:0028 e:000027 CFUNC :load
c:0004 p:0112 s:0023 E:001210 EVAL /home/samuel/.rvm/gems/ruby-2.7.0/bin/rspec:23 [FINISH]
c:0003 p:---- s:0018 e:000017 CFUNC :eval
c:0002 p:0189 s:0011 E:000768 EVAL /home/samuel/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24 [FINISH]
c:0001 p:0000 s:0003 E:000790 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
/home/samuel/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
/home/samuel/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
/home/samuel/.rvm/gems/ruby-2.7.0/bin/rspec:23:in `<main>'
/home/samuel/.rvm/gems/ruby-2.7.0/bin/rspec:23:in `load'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/exe/rspec:4:in `<top (required)>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:45:in `invoke'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:71:in `run'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:86:in `run'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/runner.rb:102:in `setup'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/covered-0.13.1/lib/covered/rspec.rb:45:in `load_spec_files'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:1581:in `load_spec_files'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:1581:in `each'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:1583:in `block in load_spec_files'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:2076:in `load_file_handling_errors'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/rspec-core-3.9.0/lib/rspec/core/configuration.rb:2076:in `load'
/home/samuel/Documents/socketry/async-http/spec/async/http/body/slowloris_spec.rb:21:in `<top (required)>'
/home/samuel/Documents/socketry/async-http/spec/async/http/body/slowloris_spec.rb:21:in `require_relative'
/home/samuel/Documents/socketry/async-http/spec/async/http/body/writable_examples.rb:21:in `<top (required)>'
/home/samuel/Documents/socketry/async-http/spec/async/http/body/writable_examples.rb:21:in `require'
/home/samuel/Documents/socketry/async-http/lib/async/http/server.rb:23:in `<top (required)>'
/home/samuel/Documents/socketry/async-http/lib/async/http/server.rb:23:in `require'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/endpoint.rb:22:in `<top (required)>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/endpoint.rb:22:in `require_relative'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/socket.rb:24:in `<top (required)>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/socket.rb:24:in `require_relative'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:24:in `<top (required)>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:25:in `<module:Async>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:46:in `<module:IO>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:99:in `<class:Generic>'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/async-io-1.27.1/lib/async/io/generic.rb:82:in `wraps'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:152:in `def_instance_delegators'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:152:in `each'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:154:in `block in def_instance_delegators'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:184:in `def_instance_delegator'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable.rb:208:in `_delegator_method'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:4:in `_valid_method?'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:4:in `catch'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:5:in `block in _valid_method?'
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/forwardable/impl.rb:5:in `eval'
/home/samuel/.rvm/gems/ruby-2.7.0/gems/covered-0.13.1/lib/covered/source.rb:41:in `block in initialize'
<internal:trace_point>:346:in `instruction_sequence'
-- Machine register context ------------------------------------------------
RIP: 0x00007f5ebba7c410 RBP: 0x00007f5ebb183718 RSP: 0x00007fff84e05468
RAX: 0x0000000000012007 RBX: 0x00007f5ebb1836e0 RCX: 0x00007f5ebbbfcee0
RDX: 0x0000000000002007 RDI: 0x0000000000000000 RSI: 0x0000556d3cb9c068
R8: 0x0000556d3cb9c068 R9: 0x00007f5ebb1836e0 R10: 0x0000000000000000
R11: 0x0000556d3cb9c068 R12: 0x0000556d3c5461e0 R13: 0x0000556d3b77a610
R14: 0x0000556d3b8ad490 R15: 0x00007f5ebb1836e0 EFL: 0x0000000000010202
-- C level backtrace information -------------------------------------------
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_vm_bugreport+0x573) [0x7f5ebbbfb4d3] vm_dump.c:755
[0x7f5ebba23557]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(sigsegv+0x49) [0x7f5ebbb5a279] signal.c:946
/usr/lib/libpthread.so.0(__restore_rt+0x0) [0x7f5ebb8bb930]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_iseqw_new+0x0) [0x7f5ebba7c410] iseq.c:1112
[0x7f5ebbbe30f4]
[0x7f5ebbbe727c]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_vm_invoke_proc+0x292) [0x7f5ebbbec5d2] vm.c:1044
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_proc_call_with_block+0x4d) [0x7f5ebbb04a5d] proc.c:1006
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(tp_call_trace+0x48) [0x7f5ebbbfbf58] vm_trace.c:1108
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(exec_hooks_body+0x86) [0x7f5ebbbfc0d6] vm_trace.c:295
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(exec_hooks_protected+0xb3) [0x7f5ebbbfc953] vm_trace.c:342
[0x7f5ebbbfe02a]
[0x7f5ebbbd95f2]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_f_eval+0x13a) [0x7f5ebbbea2aa] vm_eval.c:1633
[0x7f5ebbbd41ec]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(catch_i+0x2eb) [0x7f5ebbbeb4eb] vm.c:1044
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(vm_catch_protect+0xae) [0x7f5ebbbda35e] vm_eval.c:2308
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_catch_obj+0x2c) [0x7f5ebbbda45c] vm_eval.c:2334
[0x7f5ebbbd41ec]
[0x7f5ebbbe169d]
[0x7f5ebbbe727c]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_yield+0x293) [0x7f5ebbbeb853] vm.c:1044
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_ary_each+0x3c) [0x7f5ebb98df2c] array.c:2135
[0x7f5ebbbd41ec]
[0x7f5ebbbe169d]
[0x7f5ebbbe727c]
[0x7f5ebba83bb5]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_require_string+0x23) [0x7f5ebba842d3] load.c:1105
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
[0x7f5ebba83bb5]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_require_string+0x23) [0x7f5ebba842d3] load.c:1105
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
[0x7f5ebba83bb5]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_require_string+0x23) [0x7f5ebba842d3] load.c:1105
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
[0x7f5ebba83bb5]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_require_string+0x23) [0x7f5ebba842d3] load.c:1105
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
[0x7f5ebba83bb5]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_require_string+0x23) [0x7f5ebba842d3] load.c:1105
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
[0x7f5ebba827bb]
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(vm_call_opt_send+0x2ee) [0x7f5ebbbe99de] vm_insnhelper.c:2661
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_yield+0x293) [0x7f5ebbbeb853] vm.c:1044
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_ary_each+0x3c) [0x7f5ebb98df2c] array.c:2135
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe169d]
[0x7f5ebbbe727c]
[0x7f5ebba827bb]
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe727c]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_f_eval+0x175) [0x7f5ebbbea2e5] vm_eval.c:1646
[0x7f5ebbbd41ec]
[0x7f5ebbbe8eab]
[0x7f5ebbbe15e3]
[0x7f5ebbbe7a64]
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(rb_ec_exec_node+0xaa) [0x7f5ebba27f8a] eval.c:277
/home/samuel/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7(ruby_run_node+0x4a) [0x7f5ebba2d63a] eval.c:335
/home/samuel/.rvm/rubies/ruby-2.7.0/bin/ruby(main+0x5b) [0x556d3a14f0fb] ./main.c:50
</code></pre>
<p>I don't know why it's happening yet, but I'll try to work on short reproduction. As a side request, I didn't realise <code>TracePoint#instruction_sequence</code> was so expensive (constructing object). Maybe we can add <code>eval_path</code> along with <code>eval_source</code> to side step this issue entirely.</p> Ruby master - Bug #16026 (Rejected): `Set#count` performance issueshttps://bugs.ruby-lang.org/issues/160262019-07-27T06:00:19Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><code>Set#size</code> is O(1), but <code>Set#count</code> is O(N).</p>
<p>I would like to add <code>alias count size</code> to <code>class Set</code></p>
<p>Is it okay?</p> Ruby master - Bug #16009 (Closed): Performance regression in 2.7https://bugs.ruby-lang.org/issues/160092019-07-17T11:19:30Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code>Number is requests/s
dcf5c19c9f89d732da70a1a16a2fe60cd1999bcc 7561.82
c53f87943e53c96b86d50b496d2a410ff1245b4c 7534.09
d2003a6d392b3b0054d7528e2e731584196aefad 7467.44, 7458.18, 7111.09, 7293.74, 7493.13
f54aa6c5b286b2b44bcdb1958fc9b1ebfce3559e 7558.63
18e43e823106f15c8aaceb1f56874bdf67bc36a3 7439.23, 7356.25, 7437.61, 7489.07
7069f64c419ebb9a7fd3e48d81454148ed4b2fba 7580.31
a160b2f56716f70fa3e485ae89875da48baefc1d 7597.52
88449100bc6d23a00dbf3addb97665f4f606f2b8 7258.96, 7184.62, 7190.71 *** Performance regression
e0f0ab959e9a0fa3db8dfdb2a493b057d6e7541b 7145.69, 7061.17, 7217.51
4d9c3a8c2362b7d5973057435258e447ce733741 7276.46, 7203.8
498113d5d39a4227c2b9a9c11bea895fe316e6b9 7218.93
c55db6aa271df4a689dc8eb0039c929bf6ed43ff 7149.20
</code></pre> Ruby master - Feature #15997 (Closed): Improve performance of fiber creation by using pool alloca...https://bugs.ruby-lang.org/issues/159972019-07-12T01:21:24Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p><a href="https://github.com/ruby/ruby/pull/2224" class="external">https://github.com/ruby/ruby/pull/2224</a></p>
<p>This PR improves the performance of fiber allocation and reuse by implementing a better stack cache.</p>
<p>The fiber pool manages a singly linked list of fiber pool allocations. The fiber pool allocation contains 1 or more stack (typically more, e.g. 512). It uses N^2 allocation strategy, starting at 8 initial stacks, next is 8, 16, 32, etc.</p>
<pre><code>//
// base = +-------------------------------+-----------------------+ +
// |VM Stack |VM Stack | | |
// | | | | |
// | | | | |
// +-------------------------------+ | |
// |Machine Stack |Machine Stack | | |
// | | | | |
// | | | | |
// | | | . . . . | | size
// | | | | |
// | | | | |
// | | | | |
// | | | | |
// | | | | |
// +-------------------------------+ | |
// |Guard Page |Guard Page | | |
// +-------------------------------+-----------------------+ v
//
// +------------------------------------------------------->
//
// count
//
</code></pre>
<p>The performance improvement depends on usage:</p>
<pre><code>Calculating -------------------------------------
compare-ruby built-ruby
vm2_fiber_allocate 132.900k 180.852k i/s - 100.000k times in 0.752447s 0.552939s
vm2_fiber_count 5.317k 110.724k i/s - 100.000k times in 18.806479s 0.903145s
vm2_fiber_reuse 160.128 347.663 i/s - 200.000 times in 1.249003s 0.575269s
vm2_fiber_switch 13.429M 13.490M i/s - 20.000M times in 1.489303s 1.482549s
Comparison:
vm2_fiber_allocate
built-ruby: 180851.6 i/s
compare-ruby: 132899.7 i/s - 1.36x slower
vm2_fiber_count
built-ruby: 110724.3 i/s
compare-ruby: 5317.3 i/s - 20.82x slower
vm2_fiber_reuse
built-ruby: 347.7 i/s
compare-ruby: 160.1 i/s - 2.17x slower
vm2_fiber_switch
built-ruby: 13490282.4 i/s
compare-ruby: 13429100.0 i/s - 1.00x slower
</code></pre>
<p>This test is run on Linux server with 64GB memory and 4-core Xeon (Intel Xeon CPU E3-1240 v6 @ 3.70GHz). "compare-ruby" is <code>master</code>, and "built-ruby" is <code>master+fiber-pool</code>.</p>
<p>Additionally, we conservatively use <code>madvise(free)</code> to avoid swap space usage for unused fiber stacks. However, if you remove this requirement, we can get 6x - 10x performance improvement in <code>vm2_fiber_reuse</code> benchmark. There are some options to deal with this (e.g. moving it to <code>GC.compact</code>) but as this is still a net win, I'd like to merge this PR as is.</p> Ruby master - Feature #15894 (Closed): Remove support for IA64https://bugs.ruby-lang.org/issues/158942019-06-01T06:08:24Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There are many <code>#ifdef __ia64</code> in Ruby code base. At one point it must have been supported.</p>
<p>But as an architecture, it's EOL (Jan 30th 2019). Last CPU was released in 2017.</p>
<p>I can't even buy 2nd hand CPU for testing.</p>
<p>So, I propose we remove support for IA64.</p>
<p>The main motivation for me personally is I want to simplify and improve fiber implementation. But it's impossible for me to test IA64 architecture.</p> Ruby master - Bug #15634 (Closed): TracePoint seems to be skipping some methods.https://bugs.ruby-lang.org/issues/156342019-03-04T09:56:37Zioquatix (Samuel Williams)samuel@oriontransfer.net
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">trace_point</span> <span class="o">=</span> <span class="no">TracePoint</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:call</span><span class="p">,</span> <span class="ss">:return</span><span class="p">,</span> <span class="ss">:line</span><span class="p">,</span> <span class="ss">:c_call</span><span class="p">,</span> <span class="ss">:c_return</span><span class="p">,</span> <span class="ss">:b_call</span><span class="p">,</span> <span class="ss">:b_return</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">trace</span><span class="o">|</span>
<span class="nb">puts</span> <span class="p">[</span><span class="n">trace</span><span class="p">.</span><span class="nf">path</span><span class="p">,</span> <span class="n">trace</span><span class="p">.</span><span class="nf">lineno</span><span class="p">].</span><span class="nf">join</span><span class="p">(</span><span class="s2">":"</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">trace_point</span><span class="p">.</span><span class="nf">enable</span>
<span class="n">values</span> <span class="o">=</span> <span class="p">{</span><span class="ss">foo: </span><span class="mi">10</span><span class="p">}</span>
<span class="k">def</span> <span class="nf">shell_escape</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">x</span>
<span class="k">end</span>
<span class="n">values</span><span class="p">.</span><span class="nf">map</span><span class="p">{</span><span class="o">|</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="o">|</span> <span class="p">[</span>
<span class="n">key</span><span class="p">.</span><span class="nf">to_s</span><span class="p">.</span><span class="nf">upcase</span><span class="p">,</span>
<span class="n">shell_escape</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="c1"># TracePoint is never triggered for this line.</span>
<span class="p">]}</span>
</code></pre> Ruby master - Bug #15572 (Closed): `RubyVM::InstructionSequence` doesn't work with `extend`.https://bugs.ruby-lang.org/issues/155722019-01-29T21:25:05Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>For some reason,<code>RubyVM::InstructionSequence.extend</code> doesn't seem to work as expected.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="k">module</span> <span class="nn">Loader</span>
<span class="k">def</span> <span class="nf">load_iseq</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="nb">puts</span> <span class="n">path</span>
<span class="k">return</span> <span class="n">compile_file</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># This doesn't work?</span>
<span class="c1"># RubyVM::InstructionSequence.extend(Loader)</span>
<span class="c1"># This works:</span>
<span class="k">class</span> <span class="o"><<</span> <span class="no">RubyVM</span><span class="o">::</span><span class="no">InstructionSequence</span>
<span class="n">prepend</span> <span class="no">Loader</span>
<span class="k">end</span>
</code></pre> Ruby master - Feature #15567 (Rejected): Allow ensure to match specific situationshttps://bugs.ruby-lang.org/issues/155672019-01-26T08:31:43Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>There are some situations where <code>rescue Exception</code> or <code>ensure</code> are not sufficient to correctly, efficiently and easily handle abnormal flow control.</p>
<p>Take the following program for example:</p>
<pre><code>def doot
yield
ensure
# Did the function run to completion?
return "abnormal" if $!
end
puts doot{throw :foo}
puts doot{raise "Boom"}
puts doot{"Hello World"}
catch(:foo) do
puts doot{throw :foo}
end
</code></pre>
<p>Using <code>rescue Exception</code> is not sufficient as it is not invoked by <code>throw</code>.</p>
<p>Using <code>ensure</code> is inefficient because it's triggered every time, even though exceptional case might never happen or happen very infrequently.</p>
<p>I propose some way to limit the scope of the ensure block:</p>
<pre><code>def doot
yield
ensure when raise, throw
return "abnormal"
end
</code></pre>
<p>The scope should be one (or more) of <code>raise</code>, <code>throw</code>, <code>return</code>, <code>next</code>, <code>break</code>, <code>redo</code>, <code>retry</code> (everything in <code>enum ruby_tag_type</code> except all except for <code>RUBY_TAG_FATAL</code>).</p>
<p>Additionally, it might be nice to support the inverted pattern, i.e.</p>
<pre><code>def doot
yield
ensure when not return
return "abnormal"
end
</code></pre>
<p>Inverted patterns allow user to specify the behaviour without having problems if future scopes are introduced.</p>
<p><code>return</code> in this case matches both explicit and implicit.</p> Ruby master - Feature #15560 (Closed): Add support for read/write offsets.https://bugs.ruby-lang.org/issues/155602019-01-23T21:39:52Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>It would be nice if read/write/send/recv/etc methods could accept an offset argument.</p>
<p>e.g.</p>
<pre><code>socket = Socket.new(...)
buffer = String.b
socket.read(1024, buffer)
socket.read(1024, buffer, offset: buffer.bytesize)
</code></pre>
<p>The same for write, e.g.</p>
<pre><code>socket = Socket.new(...)
buffer = String.b
amount = socket.write(buffer)
socket.write(buffer, offset: amount)
</code></pre>
<p>Could also include "size:" so that we can selectively write parts of the buffer.</p> Ruby master - Feature #15456 (Open): Adopt some kind of consistent versioning mechanismhttps://bugs.ruby-lang.org/issues/154562018-12-23T21:29:22Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>After the discussion <a href="https://github.com/ruby/bigdecimal/issues/114" class="external">https://github.com/ruby/bigdecimal/issues/114</a> I feel like we would benefit from some consistent versioning mechanism across all of Ruby.</p>
<p>So far, I feel the majority of Ruby uses some form of semantic versioning.</p>
<p>For the sanity of all Ruby users, I think it would be a good policy to adopt this across core Ruby and standard gems.</p>
<p>There are some previous discussions around this:</p>
<ul>
<li><a href="https://bugs.ruby-lang.org/issues/9215" class="external">https://bugs.ruby-lang.org/issues/9215</a></li>
<li><a href="https://bugs.ruby-lang.org/projects/ruby/wiki/GeneralMaintenancePolicy" class="external">https://bugs.ruby-lang.org/projects/ruby/wiki/GeneralMaintenancePolicy</a></li>
<li><a href="https://bugs.ruby-lang.org/issues/8835" class="external">https://bugs.ruby-lang.org/issues/8835</a></li>
</ul>
<p>So, the questions are as follows:</p>
<ul>
<li>Can we adopt Semantic Versioning (or as much of it as possible) across Ruby?</li>
<li>Would such a change help users of Ruby?</li>
<li>Is there existing documentation about how version number works?</li>
<li>How does it deviate from Semantic Versioning?</li>
<li>Is this deviation important and worth the additional complexity for our users?</li>
</ul>
<p>As an aside:</p>
<ul>
<li>How do other implementations advertise compatibility with Ruby?</li>
<li>JRuby and RBX have totally different version numbers that are difficult to understand w.r.t. compatibility with mainline CRuby.</li>
</ul>
<p>My main concern is how difficult this is for everyone to keep track of and also the implied assumptions (e.g. breaking change if and only if major versions bump). If different parts of Ruby use different versioning scheme, it is hard for our users to define dependencies which don't cause broken software.</p> Ruby master - Bug #15417 (Closed): Pathname case insensitive comparisonhttps://bugs.ruby-lang.org/issues/154172018-12-15T04:35:06Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>While fixing some issues with Pathname, I noticed the following comparison:</p>
<pre><code> SAME_PATHS = if File::FNM_SYSCASE.nonzero?
# Avoid #zero? here because #casecmp can return nil.
proc {|a, b| a.casecmp(b) == 0}
else
proc {|a, b| a == b}
end
</code></pre>
<p>Firstly, this seems wrong to me because case sensitivity is per-mount not a global state for the entire system.</p>
<p>Secondly, it concerns me because sometimes this becomes security bug, e.g. path may or may not be the same, and could slip through some sanity check (e.g. git could checkout files to <code>.git</code> directory with case insensitive file system).</p>
<p>Unless string match exactly, we should leave it to file system to determine if the path is equivalent or not (e.g. in the case of <code>Pathname#relative_path_from</code>). Trying to be too clever might cause future pain.</p> Ruby master - Feature #15344 (Feedback): Being proactive about Ruby securityhttps://bugs.ruby-lang.org/issues/153442018-11-27T01:10:02Zioquatix (Samuel Williams)samuel@oriontransfer.net
<p>I would like to start a discussion relating to <a href="https://github.com/rubygems/rubygems/issues/2496" class="external">https://github.com/rubygems/rubygems/issues/2496</a></p>
<p>I don't know what has been done here already. I know from a computability POV, it's impossible to solve this problem.</p>
<p>That being said, it seems like we could do more.</p>
<p>Some parts of this problem relate to tooling (like RubyGems above).</p>
<p>Other parts relate to isolation within the interpreter.</p>
<p>As code bases get bigger, security is more of a concern, both in terms of how users interact with a system, and what they can do.</p>
<p>Dropping all but the necessary privileges is a great way to provide security at the interpreter level.</p>
<p>In BSD, they recently adopted this kind of model: <a href="https://www.openbsd.org/papers/BeckPledgeUnveilBSDCan2018.pdf" class="external">https://www.openbsd.org/papers/BeckPledgeUnveilBSDCan2018.pdf</a></p>
<p>It might be something which we can incorporate into Guilds.</p>
<p>User can create guild, pledge which APIs should be available, and then load 3rd party code.</p>
<p>Some code does not function well when it is isolated, Ruby often encourages monkey patching. That being said, in many large Ruby applications, business logic could be isolated. Some benefits might include reduced memory usage, improved security, etc. Apple adopted a similar policy with XPC (<a href="https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/DesigningDaemons.html" class="external">https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/DesigningDaemons.html</a>).</p>
<p>For example, it might look like:</p>
<pre><code>Guild.new do
Guild.drop(File)
Guild.drop(Socket)
require 'unstrusted/3rd/party/code`
Untrusted.all_your_base(are_belong_to_us) # Raise error or SIGABRT.
end
</code></pre>
<p>It would bring more purpose to Guild, not just for scalability, but also reliability, and security.</p>
<p>This is not the only way to solve this problem, I welcome any and all discussion.</p>