Feature #15567: Allow ensure to match specific situations

• Status: Rejected
• Assignee: ioquatix (Samuel Williams)
• Target version: -

[ioquatix (Samuel Williams)]

There are some situations where rescue Exception or ensure are not sufficient to correctly, efficiently and easily handle abnormal flow control.

Take the following program for example:

def doot
	yield
ensure
	# Did the function run to completion?
	return "abnormal" if $!
end

puts doot{throw :foo}
puts doot{raise "Boom"}
puts doot{"Hello World"}

catch(:foo) do
	puts doot{throw :foo}
end

Using rescue Exception is not sufficient as it is not invoked by throw.

Using ensure is inefficient because it's triggered every time, even though exceptional case might never happen or happen very infrequently.

I propose some way to limit the scope of the ensure block:

def doot
	yield
ensure when raise, throw
	return "abnormal"
end

The scope should be one (or more) of raise, throw, return, next, break, redo, retry (everything in enum ruby_tag_type except all except for RUBY_TAG_FATAL).

Additionally, it might be nice to support the inverted pattern, i.e.

def doot
	yield
ensure when not return
	return "abnormal"
end

Inverted patterns allow user to specify the behaviour without having problems if future scopes are introduced.

return in this case matches both explicit and implicit.

[Eregon (Benoit Daloze)]

I think the workaround using a variable set after yield (success = true) is not too bad, and clear enough. And the performance would be fine in this case (i.e. there would be no overhead if the JIT profiles the branch and only one branch is taken, like TruffleRuby).

Ensure should always be executed no matter the circumstances IMHO, so I don't like to make it conditional with extra syntax.
Not executing ensure when using break, etc sounds like a bug in the vast majority of cases.

[ioquatix (Samuel Williams)]

@Eregon (Benoit Daloze) - thanks for your response.

Ensure should always be executed no matter the circumstances IMHO, so I don't like to make it conditional with extra syntax.

This isn't about not executing ensure, but allowing user to handle situations like "returned normally" vs "non-standard flow control".

I think the workaround using a variable set after yield (success = true) is not too bad, and clear enough. And the performance would be fine in this case (i.e. there would be no overhead if the JIT profiles the branch and only one branch is taken, like TruffleRuby).

Personally, I think it's ugly, and I also think it is inefficient. I also don't think it's clearly conveying what the user is trying to do.

The problem is, people write code like this:

begin
  ..
ensure
  abnormal_path if $!
end

This actually wrong and fails in the case of throw wrapped in catch as given in the original examples.

It's actually not clear from the code if this is what the user wanted or not. Because you can't express clearly what you are actually interested in.

There should be no overhead when exiting normally in the following situation:

begin
  yield
ensure when not return
  return :abnormal
end

As soon as you write code like:

begin
  yield
  success = true
ensure
  return :abnormal unless success
end

you guarantee that the code must pass through the ensure block. But your intention is, only execute this code if the code didn't return normally (or some other flow control).

So, I don't think the argument about always executing ensure holds up - this isn't about changing the semantics of ensure but extending it to handle explicitly what people are already doing, albeit probably incorrectly and inefficiently.

[ioquatix (Samuel Williams)]

• Target version deleted (2.7)

[shevegen (Robert A. Heiler)]

I was about to write a long comment, but then I noticed that my reply would be too long, so I shortened
this. So just a few key statements:

I don't like the proposed syntax suggestion here in particular:

ensure when not return

The reason why I dislike it is partially due to the syntax verbosity; but also because, more significantly,
because it appears to do conditional checks in very different means than have existed before in ruby.
I would expect "when" usually in a case-structure.

There was this other recent suggestion to also add "when", to allow for chaining after the addition of
the alias "then" to "yield_self", leading to chains of .when .then .when .then; and I also am not a huge fan
of using "when" that way. Of course there is more than one way to do something, but my own personal
preference would be to keep "when" reserved for case-when layouts.

As for catch/throw: I think it is a lot more common to see begin/rescue than catch/throw. Even if this
may be slower (e. g. some people doing control flow logic with begin/rescue), I think the intent and
clarity of intent is much clearer with control flows based on begin/rescue (and ensure) style layout
of code. So to conclude and finish my comment, I think it would be better to not introduce a more
complicated ensure conditional clause as proposed in the suggestion here. IMO you can re-structure
your code to keep ensure simple, so it should not be necessary to add additional conditionals into
ensure, even more so when these are unusual in regards to syntax - it would be the first time that I
would see "ensure when" and I don't quite like that syntax suggestion in this context.

[ioquatix (Samuel Williams)]

@shevegen (Robert A. Heiler) - thanks for your feedback.

The reason why I dislike it is partially due to the syntax verbosity

With regards to syntax, it should use existing keywords, otherwise it can cause more problems. The reason for not is already explained, because otherwise user may have to enumerate all other tags which is inconvenient and may break if new tags are introduced in the future. If you can propose better syntax, please do.

but also because, more significantly, because it appears to do conditional checks in very different means than have existed before in ruby.

This isn't really a conditional check like if, and it's not even a big change internally, it simply exposes a bit more of the exception handling mechanisms. It should be more efficient, because ensure when not return can be handled as part of exceptional flow control.

It's more similar to rescue than ensure in it's overhead. In MRI, rescue has some overhead, but it's possible to implement zero-cost exception handling.

The reason why I think this is a good idea is because Ruby provides a lot of flow control statements, but handling these abnormal flow control is very tricky and easy to get wrong. Statements like ensure ... unless $! is actually incorrect as well as inefficient. rescue Exception doesn't handle all ways a function can exit.

I think the key things to think about would be:

• Is there a better syntax for this proposal?
• Does introducing such feature have any practical downside?
  • Implementation cost.
  • Maintenance cost.
  • Does it introduce technical debt?
• Does it solve a problem that can't be solved some other way?
  • ensure when not return can be partly solved some other way but with some overhead.
  • ensure when raise is similar to rescue Exception but doesn't require raise to resume default exception handling.
  • ensure when retry and other formulations (throw, return, next, break, redo, retry) have no equivalent right now.
  • ensure when ..., ..., ... is practically impossible, each case needs to be handled independently if possible.
• Even if there are existing solutions, does this proposed solution communicate more clearly the intention of the programmer?
• Do we want to expose this level of behaviour to user code?
• Could we limit what tags you can match? I suspect the most important one would be ensure when not return, but I can also imagine it could be pretty useful to catch things like retry and redo.

[ioquatix (Samuel Williams)]

One more thing to bear in mind is that you can avoid this feature if you don't like it. Because it's for some very specific situations. It doesn't affect existing code in any way. But for situations like mine where I have a handful of use cases, and efficiency is a concern (to an extent) as well as readability and correctness, I would welcome such a feature. If you don't need it you won't be affected by it, but if you need it, there is actually no other practical solution in the language right now.

[jeremyevans0 (Jeremy Evans)]

ioquatix (Samuel Williams) wrote:

One more thing to bear in mind is that you can avoid this feature if you don't like it. Because it's for some very specific situations. It doesn't affect existing code in any way. But for situations like mine where I have a handful of use cases, and efficiency is a concern (to an extent) as well as readability and correctness, I would welcome such a feature. If you don't need it you won't be affected by it, but if you need it, there is actually no other practical solution in the language right now.

In the very rare cases where you want to treat a non-local, non-exception exit differently from a normal exit, as Eregon mentioned, there is a simple existing practical solution that is unlikely to be significantly less efficient:

def doot
  ret = yield
  normal_exit = true
  ret
ensure                                                                                                                                                                                                              
  # Did the block return normally
  return "abnormal" if $! || !normal_exit
end

I think we should definitely not add syntax in an attempt to make it easier to treat a non-local, non-exception exit differently than a normal exit, as doing so is usually a mistake.

You may not consider the above approach a simple and practical solution. However, considering you have already conceded that your proposal is for "very specific situations" and you only have "handful of use cases", even if you consider it neither simple nor practical, it's still easy and possible. New syntax is not necessary to support what you want, and I do not think new syntax should be added for "very specific situations" with only a "handful of use cases" when it is already possible to get the behavior you want with existing syntax.

[ioquatix (Samuel Williams)]

So, @jeremyevans0 (Jeremy Evans), I agree with everything you say.

That being said, this proposal goes beyond just that single case. For example, there is no easy way to capture retry or redo exiting a block.

Looking at your example, e.g.

def doot
  ret = yield
  normal_exit = true
  ret
ensure                                                                                                                                                                                                              
  # Did the block return normally
  return "abnormal" if $! || !normal_exit
end

I certainly prefer

def doot
  yield
ensure when not return                                                                                                                                                                                                             
  return "abnormal"
end

It's clearer what's going on (especially if function is more complex than 1 line of actual code), it should be more efficiently implemented by VM since it only affects non-normal flow control and there is no conditional checks required.

[Eregon (Benoit Daloze)]

ioquatix (Samuel Williams) wrote:

For example, there is no easy way to capture retry or redo exiting a block.

Which is intended, it would break the semantics of those if they were caught, or it would be very surprising if some ensure blocks are not run in these cases even though they exit the scope.

I think there is < 1% case where it's sensible to not run ensure in this kind of conditions, and the workaround seems good enough for these rare cases.

it should be more efficiently implemented by VM since it only affects non-normal flow control and there is no conditional checks required.

Please don't assume that. A check will be needed too in the general case, and it can already optimize very well with the local variable (e.g., zero overhead on TruffleRuby if only using the normal path, or only the exception path).

[Eregon (Benoit Daloze)]

@ioquatix (Samuel Williams) Could you give one or two real-world examples where this would be useful?

[matz (Yukihiro Matsumoto)]

• Status changed from Open to Rejected

The language feature similar to ensure can be observed in many languages (unwind-protocol in Lisp, defer in Go, finally in Java and others), but neither of them considers the situation. That indicates there's no real-world use-case for it. So currently I believe there's no benefit enough the implementation cost and making the language more complex.

Matz.

[ioquatix (Samuel Williams)]

Okay, thanks for your time and input. That makes sense.

[ioquatix (Samuel Williams)]

I revisited this situation again.

This behaviour is very confusing:

def transaction
	begin
		puts "Begin Transaction"
		yield
	rescue
		puts "Abort Transaction"
	else
		puts "Commit Transaction"
	end
end

catch(:ball) do
	transaction do
		throw :ball
	end
end

In particular, trying to handle this by using ensure ... unless $! can fail in some situations due to scope of $!

def transaction
	begin
		puts "Begin Transaction"
		yield
	rescue
		puts "Abort Transaction"
	ensure
		puts "Commit Transaction" unless $!
	end
end

catch(:ball) do
	begin
		raise "Problem"
	rescue
		transaction do
			throw :ball
		end
	end
end

It seems like we must involve some elaborate state tracking in order to get the correct behaviour (calling either abort or commit predictably). So, I believe we should reconsider this feature, or some evolution of it.

[ioquatix (Samuel Williams)]

Here is working example:

def transaction
	failed = false
	
	begin
		puts "Begin Transaction"
		yield
	rescue Exception
		puts "Abort Transaction"
		failed = true
		raise
	ensure
		puts "Commit Transaction" unless failed
	end
end

catch(:ball) do
	begin
		raise "Problem"
	rescue
		transaction do
			throw :ball
		end
	end
end

This seems overly complex to me personally.

[jeremyevans0 (Jeremy Evans)]

ioquatix (Samuel Williams) wrote in #note-14:

Here is working example:

def transaction
	failed = false
	
	begin
		puts "Begin Transaction"
		yield
	rescue Exception
		puts "Abort Transaction"
		failed = true
		raise
	ensure
		puts "Commit Transaction" unless failed
	end
end

catch(:ball) do
	begin
		raise "Problem"
	rescue
		transaction do
			throw :ball
		end
	end
end

This seems overly complex to me personally.

You can simplify this slightly using a local variable:

def transaction
    failed = false

    begin
        puts "Begin Transaction"
        yield
    rescue Exception => exc
        puts "Abort Transaction"
        raise
    ensure
        puts "Commit Transaction" unless exc
    end
end

catch(:ball) do
    begin
        raise "Problem"
    rescue
        transaction do
            throw :ball
        end
    end
end

I think such code makes sense. Exceptions abort the transaction, but should be reraised and not swallowed, so the raise doesn't feel out of place.

Looking at my earlier example:

def doot
  ret = yield
  normal_exit = true
  ret
ensure                                                                                                                                                                                                              
  # Did the block return normally
  return "abnormal" if $! || !normal_exit
end

This is a case that wouldn't work correctly in the transaction scenario, where it is called inside an existing rescue block. So you would currently have to use:

def doot
  ret = yield
  normal_exit = true
  ret
rescue Exception => exc
  raise
ensure                                                                                                                                                                                                              
  # Did the block return normally
  return "abnormal" if exc || !normal_exit
end

About the only way I can think to make that easier would be for ensure to support =>:

def doot
  ret = yield
  normal_exit = true
  ret
ensure => exc                                                                                                                                                                                                        
  # Did the block return normally
  return "abnormal" if exc || !normal_exit
end

In this case exc would be nil if the code before ensure did not raise an exception, and the exception instance if it did.

However, I think the cases where this actually matters (ensure without rescue but that cares about whether an exception has been raised) are so few that we shouldn't consider language modifications to support them.

[Eregon (Benoit Daloze)]

What's wrong with this, which works fine?

def transaction
  begin
    puts "Begin Transaction"
    result = yield
    success = true
    result
  ensure
    if success
      puts "Commit Transaction"
    else
      puts "Abort Transaction"
    end
  end
end

catch(:ball) do
  transaction do
    throw :ball
  end
end

IMHO it's clear and explicit.
What we want to check here is ultimately if the block was successful (no error of any sort) and the local variable is pretty clear.
In fact, you might even want to allow the block to return :abort or so, and then it would simply be if success && result != :abort.

FWIW I think there are very few use cases where throw/catch are a good idea, but this applies to break and others.
Short answer: use ensure if you want to be sure something runs, rescue only captures Exception, as expected.

[ioquatix (Samuel Williams)]

I suspect that there is existing buggy code which has ensure blocks and checks $! which fails in the case I mentioned.

I think the ensure block should be able to capture the current exception without using a global.

I like the proposed syntax:

begin
  ...
ensure => exception
  if exception
    abort
  else
    commit
  end
end

It allows existing code that uses if $! to be retrofitted to do the correct thing.

[ioquatix (Samuel Williams)]

@Eregon (Benoit Daloze) throw should not abort the transaction, it should be committed.

[Eregon (Benoit Daloze)]

I suspect that there is existing buggy code which has ensure blocks and checks $! which fails in the case I mentioned.

Typically that's an anti-pattern.
In 99% cases, the ensure code should do its logic regardless whether there is an ongoing exception/unwind or not.

Seems Jeremy agrees in comment 7:

I think we should definitely not add syntax in an attempt to make it easier to treat a non-local, non-exception exit differently than a normal exit, as doing so is usually a mistake.

---

Eregon (Benoit Daloze) throw should not abort the transaction, it should be committed.

That is surprising to me.
If throw is performed in the middle of the transaction block then maybe only half the operations are done, committing in that case seems wrong.

transaction do
  update1
  some_call_that_ends_up_in_throw
  update2
end

[jeremyevans0 (Jeremy Evans)]

Eregon (Benoit Daloze) wrote in #note-19:

Eregon (Benoit Daloze) throw should not abort the transaction, it should be committed.

That is surprising to me.
If throw is performed in the middle of the transaction block then maybe only half the operations are done, committing in that case seems wrong.

transaction do
  update1
  some_call_that_ends_up_in_throw
  update2
end

Rails 7 is going to start treating non-local, non-exception exits as rolling back the transaction, the same as exception exits. I believe this is to work around the fact that Timeout uses throw (https://github.com/rails/rails/pull/29333).

Sequel won't make the same change. I think that non-local, non-exception exits should commit the transaction, and only exceptions should roll the transaction back. Changing the behavior just because of Timeout's design (even if Timeout is not used) seems to be a mistake to me. I certainly have a lot of code that uses throw to exit transactions and expects the transaction will be committed. Sinatra and Roda both use throw for returning responses to web requests, for example:

DB.transaction do
  DB[:table].insert

  unless update_related_table?
    redirect '/some-path' # throw
  end

  DB[:related_table].update
end

[Eregon (Benoit Daloze)]

Thanks for the link.

Is throw used for other things than redirect?

If redirect would use a regular exception, and for instance pass [] as the backtrace, it would probably be as efficient, but then gives the possibility to differentiate for example with rescue => e; e&.should_commit? or so.
throw/catch has AFAIK no way to communicate any information, so there is no way to know if the throw is meant as a convenience return like for redirect or as some kind of bailout which should not commit the transaction.
It can't even be differentiated from return/break so it seems a bad idea to use throw in the first place.

It feels weird that Timeout uses throw and not raise, and it seems counter-intuitive (everyone knows Timeout.timeout {} raises Timeout::Error, and exception, throw is unexpected)
It also adds quite some complexity: https://github.com/ruby/timeout/blob/4893cde0eda321448a1a86487ac9b571f6c35727/lib/timeout.rb#L29-L50
Does anyone know what's the point of that?

What I could find is https://github.com/ruby/timeout/commit/238c003c921e6e555760f8e96968562a622a99c4
and https://bugs.ruby-lang.org/issues/8730

IMHO rescue Exception without re-raise is always the bug.

[jeremyevans0 (Jeremy Evans)]

Eregon (Benoit Daloze) wrote in #note-21:

Thanks for the link.

Is throw used for other things than redirect?

Yes, it's used anytime you want to return a response immediately. I just used redirect as an example.

If redirect would use a regular exception, and for instance pass [] as the backtrace, it would probably be as efficient, but then gives the possibility to differentiate for example with rescue => e; e&.should_commit? or so.
throw/catch has AFAIK no way to communicate any information, so there is no way to know if the throw is meant as a convenience return like for redirect or as some kind of bailout which should not commit the transaction.
It can't even be differentiated from return/break so it seems a bad idea to use throw in the first place.

Changing a throw (which doesn't indicate error) to an exception (which indicates error) for a case that is not an error makes no sense to me. The problem is that Timeout uses throw instead of an exception for error cases (timeouts), not code that uses throw for non-error cases.

It feels weird that Timeout uses throw and not raise, and it seems counter-intuitive (everyone knows Timeout.timeout {} raises Timeout::Error, and exception, throw is unexpected)
It also adds quite some complexity: https://github.com/ruby/timeout/blob/4893cde0eda321448a1a86487ac9b571f6c35727/lib/timeout.rb#L29-L50
Does anyone know what's the point of that?

I believe this is because exception handling inside the Timeout.timeout block doesn't interfere with timeout handling.

What I could find is https://github.com/ruby/timeout/commit/238c003c921e6e555760f8e96968562a622a99c4
and https://bugs.ruby-lang.org/issues/8730

Yep, that pretty much confirms it. I don't agree with the tradeoff. Using throw for error cases is wrong, IMO. It would be better to have cases where Timeout didn't work due to improper rescuing, than not being able to use throw correctly because of Timeout's implementation. There are already places where Timeout doesn't work correctly, after all. A timeout error in my mind is more similar to a TERM signal, and Ruby handles that using an exception.

I should point out that this is not just a throw issue, it occurs for any non local exit. Here's an example using return. We have to use a transaction around the initial retrieval of the record due to the use of FOR UPDATE, because in certain conditions we want to update the record later, and don't want any modifications to the record in between. We need the transaction to commit and not rollback on exit, otherwise we lose the update to the associated object.

def foo
  record = nil

  transaction do
    return :a unless record  = dataset.for_update.first

    record.associated_object.update()

    return :b if record.some_condition?
 
    record.update() if record.some_other_condition?
  end
  
  record
end

While you can generally rewrite code to avoid non-local exits like this, often the non-local exit approach is simplest.

[Eregon (Benoit Daloze)]

A timeout error in my mind is more similar to a TERM signal, and Ruby handles that using an exception.

I think that's a good point.
I agree for similar things like Ctrl+C/SIGINT which is an Interrupt exception (< SignalException < Exception), and Kernel#exit which is a SystemExit exception (< Exception).

Nothing can guarantee to bubble through user code without stopping in between (it's always possible to hang/loop via ensure),
except a fatal error that would immediately end the process or skip even ensure blocks and always reach the C main().
So proper exception handling (e.g. no rescue Exception; # ignore, ensure code must be bounded in time/steps) is always needed for correctness, and if Ctrl+C/exit work with an exception, so should Timeout.timeout.
Maybe we should open a new issue so that Timeout.timeout uses an Exception and not throw?

---

I'm not sure if throw is always used as non-local control flow, and not as some sort of exception (e.g., to indicate some 403).

BTW I guess we could replace throw with a non-local return using handle_request(proc { return }) instead of catch, but it wouldn't change anything related to this issue, both are considered non-local exits and non-distinguishable.
That's an argument for treating throw as control flow much like non-local return and not as exceptional though.

[ioquatix (Samuel Williams)]

Thanks for the discussion. I'll try to summarise my position briefly.

Maybe we should open a new issue so that Timeout.timeout uses an Exception and not throw?

I agree Timeout should raise an exception internally and we should change this. Happy to work on a PR with you all. I am very strongly in favour of this. If we decide to do this, we should let Rails team know about our intended change so they don't end up working themselves into a corner.

Yes, it's used anytime you want to return a response immediately. I just used redirect as an example.

Yes, I also do this and agree with Jeremy's position here. While I do respect everyone's input, to me, exception = abort transaction, throw/break/next = commit.

Therefore, I do feel strongly that we need some precise way to capture the exception as we go through the ensure block. The previous suggestion makes sense to me, as in:

begin
  raise "Boom"
ensure => exception
end

This actually makes it very easy to do the correct thing. The alternatives are tricky to get right:

failed = false
begin
rescue Exception # "Exception" usually missing... so buggy.
  # error
  failed = true
  raise
ensure
  if failed
    abort
  else
    commit
  end
end

I'm pretty sure I've got bugs in my code due to this, due to mis-understanding the flow control. So, I think we should make this easier for developers to get right.

[ioquatix (Samuel Williams)]

• Status changed from Rejected to Open

[ioquatix (Samuel Williams)]

Here are some examples of situations where $! is used inside ensure block. In every case, there is a potential buggy behaviour.

https://github.com/drbrain/net-http-persistent/blob/75574f2546a08aa2663b06a2e005bcf2ee304f13/lib/net/http/persistent.rb#L230

https://github.com/copiousfreetime/launchy/blob/80e723ebf2d9e73d35ae5de31a73268eb6c46ecc/lib/launchy.rb#L36-L40

https://github.com/rails/spring/blob/577cf01f232bb6dbd0ade7df2df2ac209697e741/lib/spring/application.rb#L296-L301

https://github.com/ahoward/open4/blob/2bc378285dcec9c88613fd4def57a58606c0680b/lib/open4.rb#L189-L192

I can find more, but I think it's sufficient to show the problem.

[ioquatix (Samuel Williams)]

Also there is a problem with the proposed work around:

begin
rescue Exception => exception
  raise
ensure
  if exception
    abort
  else
    commit
  end
end

This only works provided you don't have any other rescue blocks, which you can see in some of the above examples may be an issue, or at least require careful implementation.

[ioquatix (Samuel Williams)]

• Status changed from Open to Rejected

I will make a more limited proposal.

[Eregon (Benoit Daloze)]

• Related to Feature #18083: Capture error in ensure block. added