Feature #8840

Yielder#state

Added by Marc-Andre Lafortune 8 months ago. Updated 3 months ago.

[ruby-core:56894]
Status:Feedback
Priority:Normal
Assignee:Yukihiro Matsumoto
Category:core
Target version:current: 2.2.0

Description

Defining an Enumerator that require a state is currently troublesome. For example, it is not really possible to define an equivalent of Lazy#drop in Ruby without making an assumption on the implementation.

To address this, I propose that we

(a) guarantee that a new Yielder object will be given for each enumeration
(b) add a 'state' attribute to Yielder.

This way, one could implement Lazy#drop in a way similar to:

class Enumerator::Lazy < Enumerator
def drop(n)
n = n.to_i
Lazy.new(self) do |yielder, values|
yielder.state ||= n
if yielder.state > 0
yielder.state -= 1
else
yielder.yield(
values)
end
end
end
end

Note that (a) is currently true for Ruby MRI, JRuby and Rubinius, but it is not explicit in the documentation.

state.pdf (87.8 KB) Marc-Andre Lafortune, 08/31/2013 07:57 AM


Related issues

Related to ruby-trunk - Bug #7696: Lazy enumerators with state can't be rewound Closed 01/15/2013

History

#1 Updated by Marc-Andre Lafortune 8 months ago

#2 Updated by Yukihiro Matsumoto 8 months ago

  • Status changed from Open to Feedback

I understand the motivation, and how it works. It is very simple.
But I hesitate to introduce state easily in this age of functional programming.
Let me think for a while.

And tell me if anyone has better idea to address the issue.

Matz.

#3 Updated by Magnus Holm 8 months ago

On Sat, Aug 31, 2013 at 12:52 AM, marcandre (Marc-Andre Lafortune) <
ruby-core@marc-andre.ca> wrote:

Defining an Enumerator that require a state is currently troublesome. For
example, it is not really possible to define an equivalent of Lazy#drop in
Ruby without making an assumption on the implementation.

Can't you just use the closure?

class Enumerator::Lazy < Enumerator
def drop(n)
n = n.to_i
Lazy.new(self) do |yielder, values|
if n > 0
n -= 1
else
yielder.yield(
values)
end
end
end
end

#4 Updated by Marc-Andre Lafortune 8 months ago

judofyr (Magnus Holm) wrote:

Can't you just use the closure?

Your example will fail if iterated a second time.
It will also not work correctly when using rewind and next. Check #7696.

#5 Updated by Akira Tanaka 8 months ago

2013/9/1 marcandre (Marc-Andre Lafortune) ruby-core@marc-andre.ca:

Your example will fail if iterated a second time.
It will also not work correctly when using rewind and next. Check #7696.

How about adding a method to wrap an enumerator to add a state?

% ruby -e '
class Enumerator
def withstate(init)
Enumerator.new {|y|
state = init.dup
self.each {|v|
y.yield [state, v]
}
}
end
end
class Enumerator::Lazy
def drop2(n)
e = with
state([n])
Enumerator.new {|y|
e.each {|remain, v|
if remain[0] == 0
y.yield v
else
remain[0] -= 1
end
}
}
end
end
e = (1..42).lazy
p e.drop2(40).toa
p e.drop2(40).to
a
'
[41, 42]
[41, 42]

--
Tanaka Akira

#6 Updated by Yui NARUSE 7 months ago

Need marcandre's reply

#7 Updated by Marc-Andre Lafortune 7 months ago

I'm sorry for my late reply, I'm way back on many things I want to do.

The proposition of with_state is interesting, but I personally find it leads to complex/convoluted solutions and is cumbersome to use. Note that the given implementation of drop2 is slightly incomplete as it needs to return a lazy enumerator, so Enumerator.new needs to be followed by a call to lazy.

#8 Updated by Akira Tanaka 7 months ago

marcandre (Marc-Andre Lafortune) wrote:

The proposition of with_state is interesting, but I personally find it leads to complex/convoluted solutions and is cumbersome to use. Note that the given implementation of drop2 is slightly incomplete as it needs to return a lazy enumerator, so Enumerator.new needs to be followed by a call to lazy.

Would you explain the incompleteness concretely?
I couldn't understand.

#9 Updated by Marc-Andre Lafortune 7 months ago

akr (Akira Tanaka) wrote:

Would you explain the incompleteness concretely?

Sure. With your code above:

e.drop2(40).map(&:odd?) # => [true, false]
# expected lazy enumerator, as with original drop:
e.drop(40).map(&:odd?)  # => #<Enumerator::Lazy: #<Enumerator::Lazy: #<Enumerator::Lazy: 1..42>:drop(40)>:map> 

Here is another implementation using with_state that returns a lazy enumerator:

class Enumerator::Lazy
  def drop3(n)
    Lazy.new(with_state(remain: n)) do |y, (state, v)|
      if state[:remain] == 0
        y.yield v
      else
        state[:remain] -= 1
      end
    end
  end
end

This implementation doesn't look so bad. It's probably quite a bit slower than using a Yielder#state method though.

#10 Updated by Akira Tanaka 7 months ago

2013/10/3 marcandre (Marc-Andre Lafortune) ruby-core@marc-andre.ca:

Issue #8840 has been updated by marcandre (Marc-Andre Lafortune).

akr (Akira Tanaka) wrote:

Would you explain the incompleteness concretely?

Sure. With your code above:

e.drop2(40).map(&:odd?) # => [true, false]
# expected lazy enumerator, as with original drop:
e.drop(40).map(&:odd?)  # => #<Enumerator::Lazy: #<Enumerator::Lazy: #<Enumerator::Lazy: 1..42>:drop(40)>:map>

Here is another implementation using with_state that returns a lazy enumerator:

class Enumerator::Lazy
  def drop3(n)
    Lazy.new(with_state(remain: n)) do |y, (state, v)|
      if state[:remain] == 0
        y.yield v
      else
        state[:remain] -= 1
      end
    end
  end
end

This implementation doesn't look so bad. It's probably quite a bit slower than using a Yielder#state method though.

Thank you. I understand.

I still like with_state than Yielder#state
because it limits stateful behaviors into a method.
--
Tanaka Akira

#11 Updated by Hiroshi SHIBATA 3 months ago

  • Target version changed from 2.1.0 to current: 2.2.0

Also available in: Atom PDF