Project

General

Profile

Actions

Feature #17795

open

Around `Process.fork` callbacks API

Added by byroot (Jean Boussier) 5 months ago. Updated 2 months ago.

Status:
Open
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:103400]

Description

Replaces: https://bugs.ruby-lang.org/issues/5446

Context

Ruby code in production is very often running in a forking setup (puma, unicorn, etc), and it is common some types of libraries to need to know when the Ruby process was forked. For instance:

  • Most database clients, ORMs or other libraries keeping a connection pool might need to close connections before the fork happens.
  • Libraries relying on some kind of dispatcher thread might need to restart the thread in the forked children, and clear any internal buffer (e.g. statsd clients, newrelic_rpm).

This need is only for forking the whole ruby process, extensions doing a fork(2) + exec(2) combo etc are not a concern, this aim at only catching kernel.fork, Process.fork and maybe Process.daemon..
The use case is for forks that end up executing Ruby code.

Current solutions

Right now this use case is handled in several ways.

Rely on the integrating code to call a before_fork or after_fork callback.

Some libraries simply rely on documentation and require the user to use the hooks provided by their forking server.

Examples:

Continuously check Process.pid

Some libraries chose to instead keep the process PID in a variable, and to regularly compare it to Process.pid to detect forked children.
Unfortunately Process.pid is relatively slow on Linux, and these checks tend to be in tight loops, so it's not uncommon when using these libraries
to spend 1 or 2% of runtime in Process.pid.

Examples:

Continuously check Thread#alive?

Similar to checking Process.pid, but for the background thread use case. Thread#alive? is regularly checked, and if the thread is dead, it is assumed that the process was forked.
It's much less costly than a Process.pid, but also a bit less reliable as the thread could have died for other reasons. It also delays re-creating the thread to the next check rather than immediately upon forking.

Examples:

Decorate Kernel.fork and Process.fork

Another solution is to prepend a module in Process and Kernel, to decorate the fork method and implement your own callback. It works well, but is made difficult by Kernel.fork.

Examples:

Proposals

I see two possible features to improve this situation:

Fork callbacks

One solution would be for Ruby to expose a callback API for these two events, similar to Kernel.at_exit.

Most implementations of this functionnality in other languages (C's pthread_atfork, Python's os.register_at_fork) expose 3 callbacks:

  • prepare or before executed in the parent process before the fork(2)
  • parent or after_in_parent executed in the parent process after the fork(2)
  • child or after_in_child executed in the child process after the fork(2)

A direct translation of such API in Ruby could look like Process.at_fork(prepare: Proc, parent: Proc, child: Proc) if inspired by pthread_atfork.

Or alternatively each callback could be exposed idependently: Process.before_fork {}, Process.after_fork_parent {}, Process.after_fork_child {}.

Also note that similar APIs don't expose any way to unregister callbacks, and expect users to use weak references or to not hold onto objects that should be garbage collected.

Pseudo code:

module Process
  @prepare = []
  @parent = []
  @child = []

  def self.at_fork(prepare: nil, parent: nil, child: nil)
    @prepare.unshift(prepare) if prepare # prepare callbacks are executed in reverse registration order
    @parent << parent if parent
    @child << child if child
  end

  def self.fork
    @prepare.each(&:call)
    if pid = Primitive.fork
      @parent.each(&:call) # We could consider passing the pid here.
    else
      @child.each(&:call)
    end
  end
end

Make Kernel.fork a delegator

A simpler change would be to just make Kernel.fork a delegator to Process.fork. This would make it much easier to prepend a module on Process for each library to implement its own callback.

Proposed patch: https://github.com/ruby/ruby/pull/4361


Related issues

Related to Ruby master - Feature #5446: at_fork callback APIClosedkosaki (Motohiro KOSAKI)Actions
Actions

Also available in: Atom PDF