Project

General

Profile

Actions

Bug #17669

closed

An exception still breaks monitor state and causes deadlock in 2.6.7

Added by yoheimuta (Yohei Yoshimuta) over 1 year ago. Updated 12 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.6.7p153 (2021-01-31 revision 67892) [x86_64-darwin19]
[ruby-core:102706]

Description

lib/monitor.rb provides Monitor.

However, its state handling is weak for interrupts caused by Thread.kill for example timeout libraries
even after introducing some uses of Thread.handle_interrupt at https://bugs.ruby-lang.org/issues/15992.

Actually, timeout exception may happen everywhere.
If it raised when the thread is executing right before the begin block,

  def mon_synchronize
    # Prevent interrupt on handling interrupts; for example timeout errors
    # it may break locking state.
->  Thread.handle_interrupt(Exception => :never){ mon_enter }
    begin
      yield
    ensure
      Thread.handle_interrupt(EXCEPTION_NEVER){ mon_exit }
    end
  end

it breaks the state of the monitor and it causes deadlock.

I can confirm that this happens either in 2.6.7 head and 2.6.6 release.

/bin/bash -c \
"date; ruby -v; ruby reproducible.rb; tail -n 10 /tmp/tmp.txt; date;" | tee ruby:2.6.7-macosx.log
docker run -it --rm -v `pwd`:`pwd` -w `pwd` ruby:2.6.6-alpine3.13 /bin/ash -c \
"date; ruby -v; ruby reproducible.rb; tail -n 10 /tmp/tmp.txt; date;" | tee ruby:2.6.6-alpine3.13.log

Technically, 2.5.8 is also reproducible because it shares the same releated code.
Incidentally, this doesn't happen in either 2.7.2 and 3.0.0 because the monitor was reimplemented in C.

Our production busy puma servers have suffered this weakness susceptible to timeouts, which frequently causes completely hung worker threads in a process.

The commit https://github.com/ruby/ruby/pull/4204/commits/e99c823f16918677b823255c44142910e02922c1 should fix this issue.


Files

reproducible.rb (1.71 KB) reproducible.rb yoheimuta (Yohei Yoshimuta), 03/02/2021 09:48 AM
ruby_2.6.6-alpine3.13.log (12.8 KB) ruby_2.6.6-alpine3.13.log yoheimuta (Yohei Yoshimuta), 03/02/2021 09:48 AM
ruby_2.6.7-macosx.log (3.73 KB) ruby_2.6.7-macosx.log yoheimuta (Yohei Yoshimuta), 03/02/2021 09:48 AM

Updated by Eregon (Benoit Daloze) over 1 year ago

This is the same bug that @headius (Charles Nutter) reported in https://github.com/ruby/monitor/issues/2.

I'd like to ask to make that repository public (currently it's private).
If the concern is that it might be confusing as the recent monitor stdlib does not use that source, how about renaming that repository, e.g. to monitor-rb?

Actions #2

Updated by jeremyevans0 (Jeremy Evans) over 1 year ago

  • Status changed from Open to Closed
  • Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.5: UNKNOWN, 2.6: REQUIRED, 2.7: DONTNEED, 3.0: DONTNEED

Updated by Eregon (Benoit Daloze) 12 months ago

It looks like this was missed to be backported (https://github.com/ruby/ruby/blob/ruby_2_6/lib/monitor.rb#L230-L239 does not have the fix).
The Backport field looks correct to me though.

Updated by Eregon (Benoit Daloze) 12 months ago

Ah, 2.6.8 is in security maintenance, so maybe this is not considered then?

Actions

Also available in: Atom PDF