Project

General

Profile

Actions

Feature #18275

open

Add an option to define_method to not capture the surrounding environment

Added by vinistock (Vinicius Stock) about 1 month ago. Updated 5 days ago.

Status:
Open
Priority:
Normal
Target version:
-
[ruby-core:105833]

Description

Invoking define_method will capture the surrounding environment, making sure we have access to anything defined in that surrounding scope. However, that’s not always necessary. There are uses for define_method where the surrounding environment is not needed.

Always capturing the surrounding environment slows down even the methods that don’t need access to it. Additionally, it prevents methods created using define_method to exist in all Ractors in a program.

If we could add an option to disable capturing the surrounding environment for define_method, we could make it so that it creates the dynamic method in all Ractors.

There could also be some performance benefits for the usages that do not need the surrounding environment. By not having to keep references to the surrounding scope, the GC could let go of locals from that environment, which might benefit GC as well.

Another option could be to accept the list of locals that the define_method invocation will need, as a way of letting go of references that are no longer needed.

Examples:

# Current behavior
#
# All of the surrounding environment is captured and references are kept for the locals
# The method created only exists in the current Ractor, due to possible references to the captured variables

some_random_thing = "a" * 10000
some_captured_block = -> { ... }

define_method(:my_method, &some_captured_block)
# Enable/disable all option
#
# Add an option that allows disabling capturing the surrounding environment completely
# The method created exists in all Ractors and none of the references are kept

some_random_thing = "a" * 10000
some_captured_block = -> { ... }

define_method(:my_method, capture_environment: false, &some_captured_block)
# Choose variables option
#
# Add an option that allows indicating which locals are needed for a define_method invocation
# The method created exists in all Ractors if no locals are needed
# The method is created only in the current Ractor if at least one local is needed
# All “unneeded” locals are let go

some_random_thing = "a" * 10000 # kept because `my_method` needs it
another_random_thing = "b" * 10000 # not kept
some_captured_block = -> { ... }

define_method(:my_method, needs: [:some_random_thing], &some_captured_block)

Updated by shyouhei (Shyouhei Urabe) about 1 month ago

Can you tell us why you cannot define your methods using normal def statements? As far as I read your proposal there seems no need of define_method to creep in.

Updated by Eregon (Benoit Daloze) about 1 month ago

In general if define_method takes an existing Proc, it should IMHO behave exactly like calling that Proc.
If we do anything special, every call like that to define_method would become very expensive as we'd need to e.g. change bytecodes of that Proc, and suddenly we'd have different behavior (e.g., regarding assignment, eval inside that Proc, etc).

There is already a way to do this via define_method(:name, &Ractor.make_shareable(proc)), isn't it? (if captured variables are needed, otherwise def is better of course).

Updated by vinistock (Vinicius Stock) about 1 month ago

Regarding the examples, they are over simplified for the sake of understanding. Sure, you wouldn't need define_method in those cases. I'm talking more generally when someone needs to capture a block or surrounding locals to use in define_method.

Maybe here's a better example

def self.test(name, &block)
  define_method("test_#{name}", &block)
end

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

Could we make invocations to define_method with shareable procs create the method in all Ractors?

Additionally, is there a way to mark an inline block as shareable or would we always need to assign it to a variable? For example

define_method(:blah) do
  # How do I make this shareable without first assigning it to a local?
end

Updated by nobu (Nobuyoshi Nakada) about 1 month ago

  • Description updated (diff)

vinistock (Vinicius Stock) wrote in #note-3:

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

Could we make invocations to define_method with shareable procs create the method in all Ractors?

I can't get your point.
Methods are unrelated to Ractor.

Additionally, is there a way to mark an inline block as shareable or would we always need to assign it to a variable? For example

define_method(:blah) do
  # How do I make this shareable without first assigning it to a local?
end
define_method(:blah, &Ractor.make_shareable(proc do
end))

Updated by byroot (Jean Boussier) about 1 month ago

Maybe it's orthogonal, but what if define_method accepted any object responding to call?

e.g.

class SomeCallback
  def initialize(foo, bar)
     @foo = foo
     @bar = bar
  end

  def call
    @foo.call(@bar)
  end
end

define_method(:my_method, SomeCallback.new(some_random_thing, 42))

This would allow tighter control on what's captured, and also be helpful for many other use cases such as delegation etc.

Updated by vinistock (Vinicius Stock) about 1 month ago

I can't get your point.
Methods are unrelated to Ractor.

If we create a method using define_method, it only exists in the Ractor that made the invocation. Trying to invoke that method from a different Ractor throws an error (something like defined in another Ractor). This can be limiting for certain types of Ractor applications.

Let's continue with the previous example of a test framework.

class Test
  def self.test(name, &block)
    define_method("test_#{name}", &block)
  end
end

class MyTest < Test
  test "that it works" do
    # ...
  end
end

In a scenario like that, all methods defined by the test singleton method will be created when loading the test classes, such as MyTest. This typically means that the methods will end up being defined in the main Ractor while requiring files.

If we try to create a worker pool of Ractors to run the tests, none of the Ractors have access to the methods that were defined invoking test.

I imagined that the reason for that is because define_method captures the surrounding environment. That's why I thought, maybe if we don't capture it we could have the dynamic be accessible to all Ractors.

Is there a different reason behind why other Ractors can't invoke methods dynamically defined by another Ractor?

Updated by Eregon (Benoit Daloze) about 1 month ago

  • Status changed from Open to Rejected

vinistock (Vinicius Stock) wrote in #note-3:

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

That's not true, methods are the same for all Ractors, always.
They might throw an exception on non-main Ractor, and the way to fix that is to use Ractor.make_shareable.

Updated by ko1 (Koichi Sasada) 5 days ago

  • Assignee set to ko1 (Koichi Sasada)
  • Status changed from Rejected to Open

Eregon's comment #7 is true. This is why Ractor.make_shareable(proc_object) is supported.

So

class C
  a = 10
  define_method(:show_a, &Ractor.make_shareable(pr))
end

Ractor.new{ C.new.show_a }.take
#=> 10

works completely.

On the other hands, writing Ractor.make_shareable() is long, so define_method with keyword argument makes easier to write.

class C
  a = 10
  pr = Proc.new{p a}
  define_method(:show_a, shareable: true){p a}
end

for example.

Updated by Eregon (Benoit Daloze) 5 days ago

+1 for define_method(:show_a, shareable: true){p a}.
That nicely avoids the issue of whether Ractor.make_shareable(some_Proc) should make the receiver shareable, which is in the unique case of define_method does not matter.

Updated by Dan0042 (Daniel DeLorme) 5 days ago

Does Ractor.make_shareable(proc{}) actually do anything? It seem to only check if the proc is (can be?) shareable and raise an error otherwise, but it doesn't freeze any objects like Ractor.make_shareable(obj) usually does.

So in that case, why is there a need to explicitly specify shareable or not for define_method ? If the proc can be shareable, why not automatically define the method as shareable?

edit: sorry, Ractor.make_shareable(proc{}) of course makes a copy of the context. My brain must have been elsewhere.

Actions

Also available in: Atom PDF