Project

General

Profile

Actions

Feature #10344

closed

[PATCH] Implement Fiber#raise

Added by nome (Knut Franke) over 9 years ago. Updated over 4 years ago.

Status:
Closed
Target version:
[ruby-core:65537]

Description

While it is possible to implement this in pure Ruby (by wrapping Fiber.yield and Fiber#resume), this feels like a low-level feature that ought to be provided out of the box. Also, the C implementation is more straight-forward, and more efficient. Unfortunately, it is not quite possible to implement this as a C extension module (without resorting to wrappers again); cf. the change to make_passing_arg().

Example usage:

fib = Fiber.new do
  counter = 0
  loop { counter += Fiber.yield }
  counter
end
fib.resume
fib.resume 10
fib.resume 100
fib.raise StopIteration # => 110

Files

0001-Implement-Fiber-raise.patch (4.12 KB) 0001-Implement-Fiber-raise.patch nome (Knut Franke), 10/09/2014 12:19 AM
0001-Implement-Fiber-raise.patch (3.51 KB) 0001-Implement-Fiber-raise.patch updated patch with some doc corrections nome (Knut Franke), 10/18/2014 01:16 PM
0001-Implement-Fiber-raise-in-ext-fiber.patch (3.6 KB) 0001-Implement-Fiber-raise-in-ext-fiber.patch variant adding Fiber#raise only in ext/fiber nome (Knut Franke), 10/18/2014 01:16 PM

Related issues 1 (0 open1 closed)

Related to Ruby master - Bug #595: Fiber ignores ensure clauseClosedioquatix (Samuel Williams)Actions

Updated by ko1 (Koichi Sasada) over 9 years ago

Do you have good example to use it?

Updated by funny_falcon (Yura Sokolov) over 9 years ago

Yes: it is missing piece to make "greenlet" port, in other words - green threads that mimics real threads.
To accomplish that there is a need to raise exception in a fiber waiting on event if event fails.
As Knut said it could be implemented in a Ruby, but it feels to be dirty way. If implemented in C it will lead to cleaner implementation.

Updated by nome (Knut Franke) over 9 years ago

For some more sophisticated examples, see https://github.com/nome/coroutines. The library does work with vanilla Ruby, but the patch improves performance.

Also, similar code can be simplified by using Fiber#raise. Compare e.g. the two implementation of Consumer::Yielder#await at
https://github.com/nome/coroutines/blob/master/lib/coroutines/base.rb

Updated by ko1 (Koichi Sasada) over 9 years ago

On 2014/10/12 1:28, wrote:

For some more sophisticated examples, see https://github.com/nome/coroutines. The library does work with vanilla Ruby, but the patch improves performance.

Also, similar code can be simplified by using Fiber#raise. Compare e.g. the two implementation of Consumer::Yielder#await at
https://github.com/nome/coroutines/blob/master/lib/coroutines/base.rb

I understand this feature helps some libraries. But I can't understand
why it is important.

I'm afraid that introducing such feature increases complexity of Fiber.
Basically, I want to recommend strongly that using Fiber as
semi-croutine, ristricted feature.

At least, such feature should be located at ext/fiber.

--
// SASADA Koichi at atdot dot net

Updated by funny_falcon (Yura Sokolov) over 9 years ago

At least, such feature should be located at ext/fiber.

even if ext/fiber it is still should be implemented in C .

I'm afraid that introducing such feature increases complexity of Fiber.

Not too much. Look at the patch: it uses same hook as rb_fiber_terminate.

By the way, why Thread has this feature?
http://www.ruby-doc.org/core-2.1.3/Thread.html#method-i-raise
And why Fiber hasn't? In fact, it is strange that this feature still not
presented, cause it is expected to exists.

Updated by nome (Knut Franke) over 9 years ago

I understand this feature helps some libraries. But I can't understand why it is important.

Without Fiber#raise, libraries need to use some rather inelegant and inefficient workarounds (wrapping or tagging exceptions, wrapping Fiber.yield etc). Also, having Thread#raise arguably leads one to expect the same functionality in Fiber. I'm not sure whether the feature qualifies as "important", but I still think it is worth including.

Basically, I want to recommend strongly that using Fiber as semi-croutine, ristricted feature.

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

At least, such feature should be located at ext/fiber.

I disagree, because the feature is applicable to the restricted (semi-coroutine) fibers available without requiring 'fiber', and indeed implementable on top of them. Nevertheless, I've attached a variant of the patch which adds the method only in ext/fiber (as a compromise solution).

Updated by ko1 (Koichi Sasada) over 9 years ago

Yura Sokolov wrote:

At least, such feature should be located at ext/fiber.

even if ext/fiber it is still should be implemented in C .

I agree.

I'm afraid that introducing such feature increases complexity of Fiber.

Not too much. Look at the patch: it uses same hook as rb_fiber_terminate.

I don't mention about implementation, but specification.

By the way, why Thread has this feature?
http://www.ruby-doc.org/core-2.1.3/Thread.html#method-i-raise
And why Fiber hasn't? In fact, it is strange that this feature still not
presented, cause it is expected to exists.

Thread#raise is one of inter communication method (but asynchronouse, difficult, unrecommended way).

Fibers can be under control without such communication.

Updated by ko1 (Koichi Sasada) over 9 years ago

Knut Franke wrote:

I understand this feature helps some libraries. But I can't understand why it is important.

Without Fiber#raise, libraries need to use some rather inelegant and inefficient workarounds (wrapping or tagging exceptions, wrapping Fiber.yield etc).

I understand.

Also, having Thread#raise arguably leads one to expect the same functionality in Fiber. I'm not sure whether the feature qualifies as "important", but I still think it is worth including.

I don't expect there is Fiber#raise. Fiber and Thread are different.

Basically, I want to recommend strongly that using Fiber as semi-croutine, ristricted feature.

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

Interesting. I don't know details of this library. Could you explain why it is important in this library?

Updated by nome (Knut Franke) over 9 years ago

Koichi Sasada wrote:

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

Interesting. I don't know details of this library. Could you explain why it is important in this library?

Consumer provides an abstraction for (semi-)coroutines that accept (consume) values; analogous to an Enumerator, which produces (enumerates) values. Since a consumer may need to do resource cleanup and/or produce a final result, we need some way to signal end of values. I think the most natural way to do this is to raise StopIteration in the consumer: The situation is analogous to calling Enumerator#next when no more values are available, and StopIteration terminates Kernel#loop, which often allows writing consumers very concisely.

In general, I think Fiber is a powerful primitive that can be used by libraries to build abstractions like Enumerator, Consumer and others. While Enumerator does not require the #raise feature, other abstractions can very well benefit from it.

Updated by yugui (Yuki Sonoda) over 6 years ago

  • Assignee set to ko1 (Koichi Sasada)
  • Target version set to 2.5

Let me add another use case which I discussed with naruse and akr in RubyKaigi.

Fiber#raise is helpful to improve compatibility of methods with blocks (internal iterators) and Enumerator (external iterators).
This happened for me when I tried to write an adapter between two independently-developed APIs.

Here's a simplified example below. You can also find the real example in https://github.com/supership-jp/activerecord-spanner-adapter/blob/master/lib/active_record/connection_adapters/spanner/client.rb.

Example

Suppose that we have a third-paty API, which we cannot control.

def with_transaction
  tx = Transaction.new
  begin
    yield tx
  rescue
    tx.rollback
  else
    tx.commit
  end
end

And now I need to begin a transaction in a method and then need to commit or rollback the transaction in other methods.

class TransactionManager
  def begin_transaction
     @iter = Transaction.enum_for(:begin_transaction)
     @tx = @iter.next # skip error handlings for simplicity
  end

  def commit_transaction
    loop { @iter.next }
  ensure
    @tx = nil
  end

  def abort_transaction(reason = RuntimeError.new)
    # ??? How can I let the iterator know the error raised
    @iter.raise(reason)
  end
  
  attr_reader :tx
end

In other words, internal iterators in Ruby can manage resources in it but external iterators can't even if the user tried to take responsibility of calling cleanup mechanism.
In the real usecase, with_transaction was an API given by google-cloud-spanner gem and the interface of the TransactionManager was the one I had to implement for ActiveRecord.

Proposal

Although I could certainly implement the adapter without the native Fiber#raise, it would be still helpful if internal iterators and external iterators in Ruby were more consistent.

Can we have Fiber#raise and Enumerator#raise on top of it to improve the consistency? ko1?

FYI: I've confirmed that the patch by nome still works fine for ruby-trunk. https://github.com/yugui/ruby/commit/5a04a8b49b5086686fd75c6635c95c12ccc6caa8

Actions #11

Updated by naruse (Yui NARUSE) over 6 years ago

  • Target version deleted (2.5)

Updated by ioquatix (Samuel Williams) over 5 years ago

I think this is a good idea. I already ended up implementing it by hand, but it's messy.

https://github.com/socketry/async/blob/e169aac7bc47f251ae7c6ebe94820b41bc2d063f/lib/async/task.rb#L43-L55

Updated by ioquatix (Samuel Williams) over 5 years ago

@ko1 (Koichi Sasada) do you think we can aim for 2.6? I don't mind reviewing this patch and merging it if you are happy with it.

Updated by ioquatix (Samuel Williams) about 5 years ago

  • Status changed from Open to Assigned
  • Assignee changed from ko1 (Koichi Sasada) to ioquatix (Samuel Williams)
  • Target version set to 2.7

We will experiment and aim to merge this into 2.7

Actions #15

Updated by ioquatix (Samuel Williams) about 5 years ago

  • Related to Bug #595: Fiber ignores ensure clause added
Actions #16

Updated by Anonymous about 5 years ago

  • Status changed from Assigned to Closed

Applied in changeset trunk|r66610.


Implement Fiber#raise. Fixes #10344.

This allows raising exceptions in another fiber, similarly to
Thread#raise.

Updated by ioquatix (Samuel Williams) about 5 years ago

@nome (Knut Franke) can you review the relevant commits and give me feedback?

Updated by naruse (Yui NARUSE) about 5 years ago

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

And also you need to add description to NEWS if you add new feature.

Updated by ko1 (Koichi Sasada) about 5 years ago

naruse (Yui NARUSE) wrote:

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

this is a support comment.

Matz agreed this feature several years ago (maybe I'm only person against, and I also agree to introduce this feature now, except naming issue. but nobody except me has no objection :p)
I'm not sure about naming issue, but he doesn't say anything last time.

Anyway, it is safe to discuss at next dev meeting (I will speak for Samuel).

Thanks,
Koichi

Updated by ioquatix (Samuel Williams) about 5 years ago

Thanks I didn’t know approval procedure.

I will update NEWS one I’m sure the feature has stabilised.

Updated by naruse (Yui NARUSE) about 5 years ago

ko1 (Koichi Sasada) wrote:

naruse (Yui NARUSE) wrote:

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

this is a support comment.

Matz agreed this feature several years ago (maybe I'm only person against, and I also agree to introduce this feature now, except naming issue. but nobody except me has no objection :p)
I'm not sure about naming issue, but he doesn't say anything last time.

Anyway, it is safe to discuss at next dev meeting (I will speak for Samuel).

Sure, thank you for comment.

ioquatix (Samuel Williams) wrote:

Thanks I didn’t know approval procedure.

I will update NEWS one I’m sure the feature has stabilised.

Anyway please update NEWS now to avoid forgetting adding the description.

Updated by decuplet (Nikita Shilnikov) almost 5 years ago

Shortly after I started to work on a library implementing algebraic effects (https://github.com/dry-rb/dry-effects) I stumbled upon lack of Fiber#raise. From the user POV it is important to signal improper use of effects with a meaningful stacktrace so they can quickly detect what's wrong. It's possible to emulate #raise with checks but it's quite inefficient and seems redundant. Thank you all folks for discussing and working on this, I hope it'll land in 2.7. If you're interested I can show a clear example but the task is almost identical to @ioquatix's one.

Updated by ioquatix (Samuel Williams) over 4 years ago

NEWS is updated.

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0