Project

General

Profile

Bug #14432

Updated by gfx (Goro FUJI) over 1 year ago

We notice that sidekiq (sidekiq pro) workers crash by `rb_bug_errno()`.

The very line that the issue raised is here: https://github.com/ruby/ruby/blob/v2_4_2/thread_pthread.c#L260

```c
static void
native_mutex_destroy(pthread_mutex_t *lock)
{
int r = pthread_mutex_destroy(lock);
mutex_debug("destroy", lock);
if (r != 0) {
rb_bug_errno("pthread_mutex_destroy", r); // HERE!
}
}
```

log/sidekiq.log:

```
[BUG] pthread_mutex_destroy: Device or resource busy (EBUSY)
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
-- Control frame information -----------------------------------------------
-- C level backtrace information -------------------------------------------
/usr/local/rbenv/versions/2.4.2/bin/ruby(rb_vm_bugreport+0xcf4) [0x558eb7577584] vm_dump.c:684
/usr/local/rbenv/versions/2.4.2/bin/ruby(rb_bug+0xd0) [0x558eb756bcf0] error.c:491
/usr/local/rbenv/versions/2.4.2/bin/ruby(rb_bug_errno+0x3a) [0x558eb756be9a] error.c:520
/usr/local/rbenv/versions/2.4.2/bin/ruby(thread_start_func_2+0x98d) [0x558eb74a0e9d] thread_pthread.c:260
/usr/local/rbenv/versions/2.4.2/bin/ruby(thread_start_func_1+0xd0) [0x558eb74a0fd0] thread_pthread.c:887
/lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xca) [0x7f9c1bf536ba]
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x6d) [0x7f9c1b54441d]
-- Other runtime information -----------------------------------------------
* Loaded script: sidekiq 5.0.5 kibela [1 of 5 busy]
* Loaded features:
0 enumerator.so
1 thread.rb
2 rational.so
3 complex.so
4 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/x86_64-linux/enc/encdb.so
5 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/x86_64-linux/enc/trans/transdb.so
6 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/unicode_normalize.rb
7 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/x86_64-linux/rbconfig.rb
8 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/compatibility.rb
9 /usr/local/rbenv/versions/2.4.2/lib/ruby/2.4.0/rubygems/defaults.rb
... omit about 5000 lines
```

Unfortunately, it occurs only in the production environment, so the condition is not clear. We also issued it to the Sidekiq Pro support team, but it seems a bug in cruby.

versions:

* ruby 2.4.2
* rails 5.1.4
* sidekiq 5.0.5
* sidekiq-pro 3.7.0

sidekiq's concurrency: 1 (see https://github.com/mperham/sidekiq/wiki/Advanced-Options for details)

Back