Feature #22392: Allow refinements to define && and ||

• Status: Rejected
• Assignee: -
• Target version: 4.1

[shugo (Shugo Maeda)]

Abstract

I propose to allow refinements to define && and ||, so that DSLs can build expressions such as name == "x" && age > 20. The right-hand side is evaluated before the method is called, so short-circuiting is not customizable.

Motivation

Query builders and similar DSLs want to capture boolean expressions written in a block:

User.where { name == "x" && age > 20 }

Today such DSLs have to use & and | instead, which bind tighter than == and >, so every operand needs parentheses: (name == "x") & (age > 20).

Proposal

&& and || can be defined as methods in refinements:

module QueryDSL
  refine Condition do
    def &&(other) = And.new(self, other)
    def ||(other) = Or.new(self, other)
  end
end

def User.where(&block) = build(block.refined(QueryDSL).call)

User.where { name == "x" && age > 20 }  # => And(Eq(name, "x"), Gt(age, 20))

Semantics of a && b (a || b is symmetric):

• If the class of a has && defined by a refinement active in the current scope, b is evaluated and a.&&(b) is the result.
• Otherwise, the behavior is unchanged.

With a hook, the result is always a.&&(b), which never short-circuits.

Details:

• && and || can be defined only in refinements. Defining them elsewhere raises NameError. A hook changes when the right-hand side is evaluated, so its effect should be limited to code that opted in by using or Proc#refined, rather than reaching every method that receives such an object. There are no default methods such as Object#&&, so super is not available in a hook.
• The hooks also apply in conditions such as if a && b, so that x = a && b; if x and if a && b behave the same.
  Conditions such as if a && b, and statements that discard the value, do not use the hooks and compile as today; only the value of a && b can reach a hook. A hook cannot change how a condition is tested, so hooking conditions would only add cost.
• and and or behave the same as && and ||.
• &&= and ||= do not use the hooks, to keep the memoization idiom.
• def &&, def ||, :&& and :|| become valid syntax.

Prior art

Scala and Swift treat && and || as ordinary methods, so DSLs such as Slick define them on their own types, and the compiler resolves them statically. C# lets a type define &, |, true and false, and expands x && y to T.false(x) ? x : x & y. C++ allows overloading && globally, which is discouraged because short-circuiting is lost for every user of the type. Python, Kotlin, Rust and Julia don't allow it; their DSLs use & and | with the same precedence problem as in Ruby, and PEP 335, which proposed overloadable and/or, was rejected for the cost it added to every and/or and for its complexity.

This proposal follows Scala and Swift in dispatching on the receiver, at run time since Ruby is dynamically typed, and limits the definitions to refinements so that, unlike in C++, code outside the DSL keeps short-circuiting. Code that doesn't use the hooks pays nothing, unlike in PEP 335. The right-hand side is evaluated eagerly rather than passed lazily: a hook that builds an expression needs its value in any case, so laziness would add complexity without a use.

[alanwu (Alan Wu)]

Please no. Changing non-method call operations to be method calls has disastrous time and space cost for everyone involved, JIT or not. If accepted, this would be yet another case of refinement making people that do not use it pay in terms of compute.

[kddnewton (Kevin Newton)]

Please don't do this. This is going to start impacting &&= and ||=, which already have very complex semantics depending on what is on the left-hand side of the expression. This is going to be very difficult to reason about.

[shugo (Shugo Maeda)]

• Description updated (diff)

alanwu (Alan Wu) wrote in #note-1:

Please no. Changing non-method call operations to be method calls has disastrous time and space cost for everyone involved, JIT or not. If accepted, this would be yet another case of refinement making people that do not use it pay in terms of compute.

Fair point.

I've implemented a PoC: https://github.com/ruby/ruby/pull/19133
I tried to reduce time and space cost for code that does not use the feature, and there are some numbers in the PR description, but the cost for non-users is not zero.

To reduce the cost more, I have changed the proposal: conditions such as if a && b and statements that discard the value compile exactly as today and never reach a hook, because I could not find a use case for them.

kddnewton (Kevin Newton) wrote in #note-2:

Please don't do this. This is going to start impacting &&= and ||=, which already have very complex semantics depending on what is on the left-hand side of the expression. This is going to be very difficult to reason about.

I agree that supporting &&= and ||= is too complex, so this proposal does not change their behavior.

[matheusrich (Matheus Richard)]

IMHO this is already pretty readable with current & and | as we can see in practice in ROM. This seems to add much confusion for no additional clarity.

[mame (Yusuke Endoh)]

I do understand the wish. I have wanted to write name == "x" && age > 20 in a DSL myself, more than once.

If today were April 1st, I'd be all for it. (cf. #16746)

As an aside, mostly to make @alanwu (Alan Wu) nervous: I'd rather have user-defined falsy values.

[alanwu (Alan Wu)]

BTW, the cost analysis to non-users is incomplete in the PoC. Quantifying JIT time cost in terms of extra instructions doesn't talk about the optimizations otherwise possible that this inhibits.
The space analysis doesn't talk about all the extra call caches that non-users end up allocating, and there's nothing about the extra space it takes to store the JIT assumptions.
Claude chose a BOP speculation scheme, which is a performance cliff if you're unlucky enough to take a dependency that merely defines a refinement, not using it. There's also a brownout from all the recompilations that follow.
I'm not kidding when I say it's a disaster.

It's a fundamental language change to strip a common operator of keyword status, arguably a bigger change than adding a new keyword because it undermines existing knowledge.

Hiding behind refinements doesn't help because refinements has a design that doesn't isolate well. Because of the late binding on Module#using, any way you implement it you end up making non-users pay. Proc#refined builds on the leakage and completely shatters lexicality.

This is really a language extension that should be known at parse time, with say, a pragma, to isolate the costs. But then everyone chooses to speak a different Ruby dialect like they do it in the Haskell community.

[shugo (Shugo Maeda)]

alanwu (Alan Wu) wrote in #note-6:

BTW, the cost analysis to non-users is incomplete in the PoC.

You're right that the analysis is incomplete.
As you say, a parse-time switch would be ideal, but I cannot think of a way to do that in Ruby other than a magic comment, which I don't like.

To be honest, I think this feature is difficult to justify, but the motivation is real, as mame mentioned. I have put it on the agenda of the next developer meeting, but I'm leaning toward withdrawing it after the discussion there.

[matz (Yukihiro Matsumoto)]

I do sympathize with examples like User.where { name == "x" && age > 20 }, but I think it is difficult to accept this as things stand.

&& and || are control structures, and with this change the reader can no longer tell from the code whether they short-circuit. The cost for programs that do not use the feature, as @alanwu (Alan Wu) pointed out, is also a concern.

Matz.

[shugo (Shugo Maeda)]

• Status changed from Open to Rejected

Thank you for the discussion at the developer meeting.

At the meeting, mame suggested an alternative: lhs && rhs as lhs.&&() { rhs }, which I had rejected myself before proposing this.
Anyway, I understand that it is difficult to make this change to Ruby now.

[mame (Yusuke Endoh)]

class BasicObject
  def && = self ? yield : self
  def || = self ? self : yield
end

1 && 2      #=> 2
nil && 2    #=> nil
false && 2  #=> false

1 || 2      #=> 1
nil || 2    #=> 2
false || 2  #=> 2

Very cool! :-)