Actions
Bug #21524
closed
SIGABRT of fluend while being terminated
Status:
Rejected
Assignee:
-
Target version:
-
ruby -v:
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]
Description
The trap / core dump was for fluentd process while being terminated as part of rollingupdate in kubernetes platform. fluentd operator pod dumped core.
Following is the backtrace
Core was generated by `/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd -c /fluentd/'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007e0f4bbd6fd1 in raise () from /lib/libc.so.6
[Current thread is 1 (LWP 4159)]
#0 0x00007e0f4bbd6fd1 in raise () from /lib/libc.so.6
#1 0x00007e0f4bbc0546 in abort () from /lib/libc.so.6
#2 0x00007e0f4bf7d269 in ruby_abort () at signal.c:979
#3 check_reserved_signal_ (name=name@entry=0x7e0f4c233526 "SEGV", name_len=name_len@entry=4) at signal.c:1012
#4 0x00007e0f4c15003d in sigsegv (sig=11, info=0x7e0f38c484b0, ctx=0x7e0f38c48380) at signal.c:944
#5 <signal handler called>
#6 0x00007e0f4c03904a in ruby_sized_xfree (size=<optimized out>, x=0x7e0f38c49060) at gc.c:10178
#7 ruby_sized_xfree (size=0, x=0x7e0f38c49060) at gc.c:10175
#8 ruby_xfree (x=0x7e0f38c49060) at gc.c:10185
#9 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#10 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#11 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Full BT
(gdb) info threads
Id Target Id Frame
* 1 LWP 4159 0x00007e0f4bbd6fd1 in raise () from /lib/libc.so.6
2 LWP 27 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
3 LWP 3475 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
4 LWP 3690 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
5 LWP 3709 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
6 LWP 3805 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
7 LWP 3951 0x00007e0f4c328d8a in je_arena_bin_index (bin=<optimized out>, arena=<optimized out>) at include/jemalloc/internal/arena.h:791
8 LWP 3732 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
9 LWP 3680 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
10 LWP 3899 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
11 LWP 3807 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
12 LWP 3961 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
13 LWP 4007 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
14 LWP 4027 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
15 LWP 4184 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
16 LWP 4095 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
17 LWP 4296 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
18 LWP 4137 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
19 LWP 4298 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
20 LWP 4278 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
21 LWP 4183 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
22 LWP 4195 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
23 LWP 3958 0x00007e0f4c37f902 in mmap64 () from /lib64/ld-linux-x86-64.so.2
24 LWP 4179 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
(gdb) thread apply all bt
Thread 24 (LWP 4179):
#0 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
#1 0x00007e0f4c32f79e in je_pages_purge (addr=addr@entry=0x7e0eafa1d000, length=<optimized out>) at src/chunk_mmap.c:134
#2 0x00007e0f4c328e31 in arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0eaf800000, arena=0x7e0f4b7a5dc0) at src/arena.c:905
#3 arena_chunk_purge (all=<optimized out>, chunk=0x7e0eaf800000, arena=<optimized out>) at src/arena.c:995
#4 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#5 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#6 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9c800000, run=run@entry=0x7e0e9c854000, bin=bin@entry=0x7e0f4b7a6010) at src/arena.c:1842
#7 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=0x7e0e9c800000, ptr=ptr@entry=0x7e0e9c8547a0, mapelm=<optimized out>) at src/arena.c:1896
#8 0x00007e0f4c3506af in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=2, tbin=<optimized out>) at src/tcache.c:127
#9 je_tcache_destroy (tcache=0x7e0f3483c000) at src/tcache.c:325
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 23 (LWP 3958):
#0 0x00007e0f4c37f902 in mmap64 () from /lib64/ld-linux-x86-64.so.2
#1 0x00007e0f4c373e4d in _dl_sysdep_read_whole_file () from /lib64/ld-linux-x86-64.so.2
#2 0x00007e0f4c37aa98 in _dl_load_cache_lookup () from /lib64/ld-linux-x86-64.so.2
#3 0x00007e0f4c36c9b7 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#4 0x00007e0f4c3770f8 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#5 0x00007e0f4bcd3d80 in _dl_catch_exception () from /lib/libc.so.6
#6 0x00007e0f4c376c6c in _dl_open () from /lib64/ld-linux-x86-64.so.2
#7 0x00007e0f4bcd321d in do_dlopen () from /lib/libc.so.6
#8 0x00007e0f4bcd3d80 in _dl_catch_exception () from /lib/libc.so.6
#9 0x00007e0f4bcd3e3f in _dl_catch_error () from /lib/libc.so.6
#10 0x00007e0f4bcd32f7 in dlerror_run () from /lib/libc.so.6
#11 0x00007e0f4bcd3386 in __libc_dlopen_mode () from /lib/libc.so.6
#12 0x00007e0f4bca6675 in init () from /lib/libc.so.6
#13 0x00007e0f4bf13a9f in __pthread_once_slow () from /lib/libpthread.so.0
#14 0x00007e0f4bca67b4 in backtrace () from /lib/libc.so.6
#15 0x00007e0f4c1efb47 in rb_print_backtrace () at vm_dump.c:755
#16 rb_vm_bugreport (ctx=0x7e0f0ac36380) at vm_dump.c:941
#17 0x00007e0f4c0175f7 in rb_bug_for_fatal_signal (default_sighandler=0x0, sig=sig@entry=11, ctx=ctx@entry=0x7e0f0ac36380, fmt=fmt@entry=0x7e0f4c23352b "Segmentation fault at %p") at error.c:660
#18 0x00007e0f4c150069 in sigsegv (sig=11, info=0x7e0f0ac364b0, ctx=0x7e0f0ac36380) at signal.c:946
#19 <signal handler called>
#20 0x00007e0f4c03904a in ruby_sized_xfree (size=<optimized out>, x=0x7e0f0ac3f060) at gc.c:10178
#21 ruby_sized_xfree (size=0, x=0x7e0f0ac3f060) at gc.c:10175
#22 ruby_xfree (x=0x7e0f0ac3f060) at gc.c:10185
#23 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#24 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#25 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 22 (LWP 4195):
#0 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
#1 0x00007e0f4c32f79e in je_pages_purge (addr=addr@entry=0x7e0f22c5c000, length=<optimized out>) at src/chunk_mmap.c:134
#2 0x00007e0f4c328e31 in arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0f22c00000, arena=0x7e0f4b67acc0) at src/arena.c:905
#3 arena_chunk_purge (all=<optimized out>, chunk=0x7e0f22c00000, arena=<optimized out>) at src/arena.c:995
#4 arena_purge (arena=arena@entry=0x7e0f4b67acc0, all=all@entry=false) at src/arena.c:1075
#5 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b67acc0) at src/arena.c:793
#6 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b67acc0, chunk=chunk@entry=0x7e0f22c00000, run=run@entry=0x7e0f22daf000, bin=bin@entry=0x7e0f4b67bb88) at src/arena.c:1842
#7 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b67acc0, chunk=0x7e0f22c00000, ptr=ptr@entry=0x7e0f22dafd00, mapelm=<optimized out>) at src/arena.c:1896
#8 0x00007e0f4c3506af in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=23, tbin=<optimized out>) at src/tcache.c:127
#9 je_tcache_destroy (tcache=0x7e0f30445000) at src/tcache.c:325
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 21 (LWP 4183):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e94000000, run=run@entry=0x7e0e942d8000, bin=bin@entry=0x7e0f4b7a6990) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e94000000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c35057c in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=18, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f3303c000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 20 (LWP 4278):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9d000000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9d000000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9d000000, run=run@entry=0x7e0e9d2d5000, bin=bin@entry=0x7e0f4b7a6990) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9d000000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35066b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=18, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0f21047000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 19 (LWP 4298):
#0 0x00007e0f4bc93607 in madvise () from /lib/libc.so.6
#1 0x00007e0f4c32f79e in je_pages_purge (addr=addr@entry=0x7e0ea7122000, length=<optimized out>) at src/chunk_mmap.c:134
#2 0x00007e0f4c328e31 in arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0ea7000000, arena=0x7e0f4b64d7c0) at src/arena.c:905
#3 arena_chunk_purge (all=<optimized out>, chunk=0x7e0ea7000000, arena=<optimized out>) at src/arena.c:995
#4 arena_purge (arena=arena@entry=0x7e0f4b64d7c0, all=all@entry=false) at src/arena.c:1075
#5 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b64d7c0) at src/arena.c:793
#6 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b64d7c0, chunk=chunk@entry=0x7e0eadc00000, run=run@entry=0x7e0eadd35000, bin=bin@entry=0x7e0f4b64de38) at src/arena.c:1842
#7 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b64d7c0, chunk=0x7e0eadc00000, ptr=ptr@entry=0x7e0eadd36b00, mapelm=<optimized out>) at src/arena.c:1896
#8 0x00007e0f4c3506af in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=9, tbin=<optimized out>) at src/tcache.c:127
#9 je_tcache_destroy (tcache=0x7e0f1b03d000) at src/tcache.c:325
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 18 (LWP 4137):
--Type <RET> for more, q to quit, c to continue without paging--
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c34fa2d in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 je_tcache_bin_flush_large (tbin=tbin@entry=0x7e0f3b83d3c8, binind=<optimized out>, rem=rem@entry=0, tcache=tcache@entry=0x7e0f3b83d000) at src/tcache.c:182
#4 0x00007e0f4c3508c3 in je_tcache_destroy (tcache=0x7e0f3b83d000) at src/tcache.c:338
#5 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#6 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#7 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#8 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 17 (LWP 4296):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea0c00000, run=run@entry=0x7e0ea0fd7000, bin=bin@entry=0x7e0f4b7a64d0) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea0c00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c3505eb in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=10, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f1b83c000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 16 (LWP 4095):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0ea8800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0ea8800000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0eab000000, run=run@entry=0x7e0eab3f9000, bin=bin@entry=0x7e0f4b7a63a0) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0eab000000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35066b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=8, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0eef02b000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
--Type <RET> for more, q to quit, c to continue without paging--
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 15 (LWP 4184):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9f000000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9f000000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c32b7d4 in je_arena_dalloc_large_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=<optimized out>, ptr=<optimized out>) at src/arena.c:1971
#8 0x00007e0f4c34fc8d in je_tcache_bin_flush_large (tbin=tbin@entry=0x7e0f3283c428, binind=<optimized out>, rem=rem@entry=0, tcache=tcache@entry=0x7e0f3283c000) at src/tcache.c:204
#9 0x00007e0f4c3508c3 in je_tcache_destroy (tcache=0x7e0f3283c000) at src/tcache.c:338
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 14 (LWP 4027):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9c800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9c800000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9cc00000, run=run@entry=0x7e0e9ccd5000, bin=bin@entry=0x7e0f4b7a6010) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9cc00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35057c in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=2, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0efbc2a000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
Thread 13 (LWP 4007):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9bc00000, run=run@entry=0x7e0e9be64000, bin=bin@entry=0x7e0f4b7a6600) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9bc00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c35062b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=12, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f01835000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 12 (LWP 3961):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e94400000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e94400000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9cc00000, run=run@entry=0x7e0e9cee3000, bin=bin@entry=0x7e0f4b7a6010) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9cc00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35066b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=2, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0f09c2a000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 11 (LWP 3807):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e99800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
--Type <RET> for more, q to quit, c to continue without paging--
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e99800000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9a800000, run=run@entry=0x7e0e9ab34000, bin=bin@entry=0x7e0f4b7a6bf0) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=0x7e0e9a800000, ptr=ptr@entry=0x7e0e9ab39200, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c3506af in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=22, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0f3302a000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 10 (LWP 3899):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9d800000, run=run@entry=0x7e0e9d8bc000, bin=bin@entry=0x7e0f4b7a6a28) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=0x7e0e9d800000, ptr=ptr@entry=0x7e0e9d8bfa80, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c3506af in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=19, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f1802a000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 9 (LWP 3680):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9c800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9c800000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea3800000, run=run@entry=0x7e0ea3b1f000, bin=bin@entry=0x7e0f4b7a6270) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea3800000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35066b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=6, tbin=<optimized out>) at src/tcache.c:127
--Type <RET> for more, q to quit, c to continue without paging--
#10 je_tcache_destroy (tcache=0x7e0f18c21000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 8 (LWP 3732):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0eaa800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0eaa800000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c32b7d4 in je_arena_dalloc_large_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=<optimized out>, ptr=<optimized out>) at src/arena.c:1971
#8 0x00007e0f4c34fc8d in je_tcache_bin_flush_large (tbin=tbin@entry=0x7e0f00818428, binind=<optimized out>, rem=rem@entry=0, tcache=tcache@entry=0x7e0f00818000) at src/tcache.c:204
#9 0x00007e0f4c3508c3 in je_tcache_destroy (tcache=0x7e0f00818000) at src/tcache.c:338
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 7 (LWP 3951):
#0 0x00007e0f4c328d8a in je_arena_bin_index (bin=<optimized out>, arena=<optimized out>) at include/jemalloc/internal/arena.h:791
#1 arena_chunk_stash_dirty (mapelms=<synthetic pointer>, all=false, chunk=0x7e0eaec00000, arena=0x7e0f4b7a5dc0) at src/arena.c:874
#2 arena_chunk_purge (all=false, chunk=0x7e0eaec00000, arena=0x7e0f4b7a5dc0) at src/arena.c:994
#3 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#4 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#5 0x00007e0f4c32b7d4 in je_arena_dalloc_large_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=<optimized out>, ptr=<optimized out>) at src/arena.c:1971
#6 0x00007e0f4c34fc8d in je_tcache_bin_flush_large (tbin=tbin@entry=0x7e0f0d02a428, binind=<optimized out>, rem=rem@entry=0, tcache=tcache@entry=0x7e0f0d02a000) at src/tcache.c:204
#7 0x00007e0f4c3508c3 in je_tcache_destroy (tcache=0x7e0f0d02a000) at src/tcache.c:338
#8 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#9 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#10 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#11 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
Thread 6 (LWP 3805):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9ac00000, run=run@entry=0x7e0e9ae9b000, bin=bin@entry=0x7e0f4b7a6bf0) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9ac00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c3505eb in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=22, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f3402a000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 5 (LWP 3709):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9c000000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9c000000, arena=<optimized out>) at src/arena.c:995
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c329759 in arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea8c00000, run=run@entry=0x7e0ea8c90000, bin=bin@entry=0x7e0f4b7a6990) at src/arena.c:1842
#8 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0ea8c00000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#9 0x00007e0f4c35062b in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=18, tbin=<optimized out>) at src/tcache.c:127
#10 je_tcache_destroy (tcache=0x7e0f0ac19000) at src/tcache.c:325
#11 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#12 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#13 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#14 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 4 (LWP 3690):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c328ffa in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_chunk_purge_stashed (mapelms=<synthetic pointer>, chunk=0x7e0e9d800000, arena=0x7e0f4b7a5dc0) at src/arena.c:926
#4 arena_chunk_purge (all=<optimized out>, chunk=0x7e0e9d800000, arena=<optimized out>) at src/arena.c:995
--Type <RET> for more, q to quit, c to continue without paging--
#5 arena_purge (arena=arena@entry=0x7e0f4b7a5dc0, all=all@entry=false) at src/arena.c:1075
#6 0x00007e0f4c329505 in arena_maybe_purge (arena=0x7e0f4b7a5dc0) at src/arena.c:793
#7 0x00007e0f4c32b7d4 in je_arena_dalloc_large_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=<optimized out>, ptr=<optimized out>) at src/arena.c:1971
#8 0x00007e0f4c34fc8d in je_tcache_bin_flush_large (tbin=tbin@entry=0x7e0f14023428, binind=<optimized out>, rem=rem@entry=0, tcache=tcache@entry=0x7e0f14023000) at src/tcache.c:204
#9 0x00007e0f4c3508c3 in je_tcache_destroy (tcache=0x7e0f14023000) at src/tcache.c:338
#10 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#11 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#12 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#13 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 3 (LWP 3475):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9c800000, run=run@entry=0x7e0e9ca3a000, bin=bin@entry=0x7e0f4b7a6600) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=0x7e0e9c800000, ptr=<optimized out>, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c3509f6 in je_tcache_bin_flush_small (rem=0, tcache=<optimized out>, binind=12, tbin=<optimized out>) at src/tcache.c:127
#6 je_tcache_destroy (tcache=0x7e0f22406000) at src/tcache.c:325
#7 0x00007e0f4c350bf3 in je_tcache_thread_cleanup (arg=<optimized out>) at src/tcache.c:396
#8 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#9 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#10 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Thread 2 (LWP 27):
#0 0x00007e0f4bf155eb in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007e0f4bf0e1b3 in pthread_mutex_lock () from /lib/libpthread.so.0
#2 0x00007e0f4c3296e8 in je_malloc_mutex_lock (mutex=0x7e0f4b7a5dc8) at include/jemalloc/internal/mutex.h:77
#3 arena_dalloc_bin_run (arena=arena@entry=0x7e0f4b7a5dc0, chunk=chunk@entry=0x7e0e9f400000, run=run@entry=0x7e0e9f773000, bin=bin@entry=0x7e0f4b7a6438) at src/arena.c:1823
#4 0x00007e0f4c32b5e3 in je_arena_dalloc_bin_locked (arena=arena@entry=0x7e0f4b7a5dc0, chunk=0x7e0e9f400000, ptr=ptr@entry=0x7e0e9f773340, mapelm=<optimized out>) at src/arena.c:1896
#5 0x00007e0f4c34f6af in je_tcache_bin_flush_small (tbin=0x7e0f4b006148, binind=9, rem=25, tcache=0x7e0f4b006000) at src/tcache.c:127
#6 0x00007e0f4c34fe25 in je_tcache_event_hard (tcache=0x7e0f4b006000) at src/tcache.c:39
#7 0x00007e0f47a5c4e4 in OPENSSL_LH_free () from /usr/local/openssl/lib/libcrypto.so.1.1
#8 0x00007e0f47a33441 in err_cleanup () from /usr/local/openssl/lib/libcrypto.so.1.1
#9 0x00007e0f47a59cca in OPENSSL_cleanup () from /usr/local/openssl/lib/libcrypto.so.1.1
#10 0x00007e0f4bbd9697 in __run_exit_handlers () from /lib/libc.so.6
#11 0x00007e0f4bbd983a in exit () from /lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#12 0x00007e0f4bbc1f11 in __libc_start_main () from /lib/libc.so.6
#13 0x000000000040114a in _start ()
Thread 1 (LWP 4159):
#0 0x00007e0f4bbd6fd1 in raise () from /lib/libc.so.6
#1 0x00007e0f4bbc0546 in abort () from /lib/libc.so.6
#2 0x00007e0f4bf7d269 in ruby_abort () at signal.c:979
#3 check_reserved_signal_ (name=name@entry=0x7e0f4c233526 "SEGV", name_len=name_len@entry=4) at signal.c:1012
#4 0x00007e0f4c15003d in sigsegv (sig=11, info=0x7e0f38c484b0, ctx=0x7e0f38c48380) at signal.c:944
#5 <signal handler called>
#6 0x00007e0f4c03904a in ruby_sized_xfree (size=<optimized out>, x=0x7e0f38c49060) at gc.c:10178
#7 ruby_sized_xfree (size=0, x=0x7e0f38c49060) at gc.c:10175
#8 ruby_xfree (x=0x7e0f38c49060) at gc.c:10185
#9 0x00007e0f4bf0bcd1 in __nptl_deallocate_tsd () from /lib/libpthread.so.0
#10 0x00007e0f4bf0bec3 in start_thread () from /lib/libpthread.so.0
#11 0x00007e0f4bc99adf in clone () from /lib/libc.so.6
Following is my observation:
There are 24 threads and all are doing the thread local memory cleanup as part of garbage collection process.
Thread 23 especially is reporting sigsegv during the GC process.
Thread 1 is SIGABORT due to the corruption at Thread 23
root [ / ]# ruby --version
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]
root [ / ]# gem --version
3.1.6
Updated by 
Updated by
Actions
Like0
Like0Like0