Feature #22132
Updated by shugo (Shugo Maeda) 3 months ago
## Abstract
How about adding an expression form of `for` that desugars into nested `flat_map`/`map` and `filter` calls.
Here's an example, which computes all pairs of numbers between `0` and `n-1` whose sum is equal to a given value `v`:
```ruby
def foo(n, v)
for i in 0...n,
j in 0...n when i + j == v then
[i, j]
end
end
p foo(10, 10) #=> [[1, 9], [2, 8], [3, 7]...]
```
The above code is desugared as follows:
```ruby
def foo(n, v)
(0...n).flat_map { |i|
(0...n).filter { |j|
i + j == v
}.map { |j|
[i, j]
}
}
end
p foo(10, 10)
```
## Background and Motivation
Some other languages have syntactic sugar that flattens nested code.
For example, Scala has for comprehensions:
```scala
def foo(n: Int, v: Int) =
for i <- 0 until n
j <- 0 until n if i + j == v
yield (i, j)
```
Haskell has do notation:
```haskell
foo :: Int -> Int -> [(Int, Int)]
foo n v = do
i <- [0 .. n-1]
j <- [0 .. n-1]
guard (i + j == v)
pure (i, j)
```
Blocks are often nested deeply in Ruby, so such syntactic sugar is useful.
## Use cases
For comprehensions can be used to flatten nested blocks.
For example,
```ruby
(1..).lazy.flat_map { |z|
(1..z).lazy.flat_map { |x|
(x..z).lazy.filter { |y|
x**2 + y**2 == z**2
}.map { |y|
[x, y, z]
}
}
}.take(3).force
```
can be flattened as follows:
```ruby
for z in (1..).lazy,
x in (1..z).lazy,
y in (x..z).lazy when x**2 + y**2 == z**2 then
[x, y, z]
end.take(3).force
```
For comprehensions can be used not only for Enumerable objects, but also for other objects with lawful `flat_map` and `map` (i.e., any [monad](https://wiki.haskell.org/index.php?title=Monad) whose `map` is the functor map derived from `flat_map`). For example, `flat_map` and `map` must cohere so that nested calls can be flattened:
```ruby
m.flat_map(&f).map(&g) == m.flat_map { |x| f.(x).map(&g) }
```
For example, for comprehensions can be used with [a parser-combinator library](https://github.com/shugo/packrat_parser):
```ruby
class SimpleCalcParser < PackratParser
def additive
for x in multitive << term("+"), y in additive then x + y end |
for x in multitive << term("-"), y in additive then x - y end |
multitive
end
```
Here `<<` is equivalent to Scala's `<~`: it runs both parsers but keeps only the left operand's value.
## Why `then` and `when`?
Scala's `yield` conflicts with the existing `yield` keyword in Ruby, so I chose `then`.
While a bare `for ... then` (no guard) reads a little unnaturally in English, Ruby already gives `then` a value-producing meaning (e.g., `Kernel#then`), so I consider it acceptable.
The guard keyword is not `if` but `when`, because `if` after the source would be ambiguous with the modifier `if`.
## Limitations
The right operand of `in` is `arg_value`, not `expr_value`, to avoid conflicts, so unparenthesized method calls (command calls) must be parenthesized.
## Backward compatibility
* `for x in xs do ... end` (and the newline form) is unchanged: it still iterates via `each` and returns the collection.
* All of the new forms (`for ... then`, `for ... ,`, `for ... when`) were `SyntaxError` before, so no existing program changes meaning.
## Implementation
Implemented in both parsers (`parse.y` and Prism) so behavior is identical either way: PoC: https://github.com/ruby/ruby/pull/17500
### Desugaring
A comprehension desugars to a chain of `filter` / `flat_map` / `map` sends, one stage per iterator:
```ruby
for x It's currently implemented only in xs when x.even?, y `parse.y`, not in ys when x < y then [x, y] end
# => xs.filter { |x| x.even? }
# .flat_map { |x| ys.filter { |y| x < y }.map { |y| [x, y] } }
```
- A single iterator uses `map`; with multiple iterators, all but the innermost use `flat_map`.
- A `when` guard applies `filter` to the preceding iterator's collection. Later iterators may reference variables bound
by earlier ones.
- Iterator variables are block parameters, Prism yet, so they do **not** leak into the enclosing scope (unlike legacy `for`). The g
uard's `filter` block and the `map`/`flat_map` block are siblings, so a temporary assigned in a guard is not visible in
the body.
### Parsers
- **`parse.y`**: a dedicated node `NODE_FOR_COMP` (one per iterator) is built at parse time, and `compile.c` emits the `
filter`/`flat_map`/`map` sends, mirroring how legacy `for` keeps `NODE_FOR` and defers the `each` send. This keeps `--du
mp=parsetree` honest and requires the scopes to be built at parse time (`compile.c` cannot allocate AST nodes).
- **Prism**: source-faithful `ForComprehensionNode` / `ForComprehensionIteratorNode`, with `prism_compile.c` folding the
same sends. Currently against the vendored `prism/`; upstreaming to `ruby/prism` is follow-up.
### Diagnostics (parse-time, both parsers)
`break`, a non-local loop variable, a self-referential (circular) iterator, implicit block params (`it`/`_1`), and a mis
sing `then`/`end` are all rejected. `--parser=parse.y`.
## Open questions
* Variable scope: A for comprehension currently scopes the loop variables unlike `for ... do`. Should it leak them?
* Guards desugar to `filter`, not a lazy `withFilter` as in Scala. So in the strict case, each guard builds one intermediate array; users who want fusion can add `.lazy`. Is `filter` acceptable, or should we add a `with_filter`-like method?