Project

General

Profile

Bug #14471

DRb mixing up function return values between PIDs after fork()

Added by jrafanie (Joe Rafaniello) over 1 year ago. Updated about 13 hours ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin16]
[ruby-core:85524]

Description

This is likely a duplicate of https://bugs.ruby-lang.org/issues/2718, although the Backport191 project in redmine appears to be dead.

We have found the exact same problem with modern rubies such as 2.3, 2.4, 2.5.

The summary is that due to fork and sharing of established connections in the DRbConn pool, the DRb server sometimes sends DRb client 1's message to client 2 and client 2's message to client 1.

My colleague Adam Grare and I have narrowed it down to the "global" DRb::DRbConn class instance variable @pool being inherited in child forked processes.

Our workaround[1] was to close all existing connections in the inherited pool in each fork to ensure each child fork process would get brand new connections:

DRb::DRbConn.instance_variable_get(:@mutex).synchronize do
  DRb::DRbConn.instance_variable_get(:@pool).each(&:close)
end

The attached client and server and log demonstrates the problem and fails on all modern ruby versions. The source of these can be found in a gist here: [2]. Note, we have tested this on Mac OSX and debian.

It would be great if connection pools would not be shared in forked processes or give us a public interface to clear the connection pool.

[1] https://github.com/ManageIQ/manageiq/pull/16953
[2] https://gist.github.com/agrare/d9484884bd297b1615814128129cfc5c


Files

echo_server.rb (134 Bytes) echo_server.rb jrafanie (Joe Rafaniello), 02/13/2018 05:06 PM
echo_client.rb (2.22 KB) echo_client.rb jrafanie (Joe Rafaniello), 02/13/2018 05:06 PM
Example_Run.log (615 Bytes) Example_Run.log jrafanie (Joe Rafaniello), 02/13/2018 05:06 PM

Related issues

Is duplicate of Ruby master - Bug #2718: DRb mixing up function return values between PIDs after fork()ClosedActions

Associated revisions

Revision d0ed935d
Added by jeremyevans (Jeremy Evans) about 13 hours ago

Fix some DRb issues (#2552)

  • Handle BasicObject in drb

Also fix a bug in rescue clause of any_to_s because sprintf
does not handle the %l modifier.

Fixes [Bug #7833]

  • Do not send a reply to the client if there is a connection error

This allows for normal TCP shutdown (fin-ack-fin-ack instead of
fin-ack-push-rst).

Patch from pierre@mouraf.org (Pierre-Alexandre Meyer).

Fixes [Bug #2339]

  • Detect fork and do not reuse forked connections in drb

This associates each DRbConn with a pid, and if the pid changes,
it closes any DRbConns in the pool with a pid that no longer
matches. This fixes DRb servers from sending messages intended
for one client to another client after forking.

Fixes [Bug #2718]
Fixes [Bug #14471]

History

#1

Updated by jeremyevans0 (Jeremy Evans) about 2 months ago

  • Is duplicate of Bug #2718: DRb mixing up function return values between PIDs after fork() added
#2

Updated by jeremyevans (Jeremy Evans) about 13 hours ago

  • Status changed from Open to Closed

Applied in changeset git|d0ed935d5bf8c3fce9800742a36e44fb7f63dda4.


Fix some DRb issues (#2552)

  • Handle BasicObject in drb

Also fix a bug in rescue clause of any_to_s because sprintf
does not handle the %l modifier.

Fixes [Bug #7833]

  • Do not send a reply to the client if there is a connection error

This allows for normal TCP shutdown (fin-ack-fin-ack instead of
fin-ack-push-rst).

Patch from pierre@mouraf.org (Pierre-Alexandre Meyer).

Fixes [Bug #2339]

  • Detect fork and do not reuse forked connections in drb

This associates each DRbConn with a pid, and if the pid changes,
it closes any DRbConns in the pool with a pid that no longer
matches. This fixes DRb servers from sending messages intended
for one client to another client after forking.

Fixes [Bug #2718]
Fixes [Bug #14471]

Also available in: Atom PDF