What to do if Sidekiq job is stuck

If your Sidekiq job is stuck, your first intention may be to try to stop this job. But Sidekiq doesn't have such mechanism, and it may make your worker stuck even futher, and to clean current processing job.

So the first thing that should be - to get logs about what is happening. To do that, send the TTIN signal to dump backtraces to the log, with command kill -TTIN [Sidekiq pid here. In docker container it most probably 1]

$ ps ax | grep sidekiq
PID TTY      STAT   TIME COMMAND                                                                                                                                            
  1 ?        Ssl   94:02 sidekiq 5.2.9 app [1 of 20 busy] 
$ kill -TTIN 1

It will dump the logs to stdout. Depending on your application settings, you may want to check logs on your logs collector service, e.g. in Kibana dashboard.

If the Sidekiq process is not responding, you can use GDB to dump backtraces for all threads with command gdb [ruby exec] [process number], like this:

$ which ruby                                                                                                                                                               
#=> /usr/local/bin/ruby
$ gdb /usr/local/bin/ruby 1

(gdb) info threads
  Id   Target Id         Frame 
  37   Thread 0x7f8b289d8700 (LWP 7994) "ruby-timer-thr" 0x00007f8b27a20d13 in *__GI___poll (fds=<optimized out>, fds@entry=0x7f8b289d7ec0, nfds=<optimized out>, 
    nfds@entry=1, timeout=timeout@entry=100) at ../sysdeps/unix/sysv/linux/poll.c:87
  36   Thread 0x7f8b23eb0700 (LWP 7995) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  35   Thread 0x7f8b23c2e700 (LWP 7996) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  34   Thread 0x7f8b239ac700 (LWP 7997) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  33   Thread 0x7f8b237aa700 (LWP 7998) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  32   Thread 0x7f8b28844700 (LWP 8002) "SignalSender" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  31   Thread 0x7f8b1e1bf700 (LWP 8003) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  30   Thread 0x7f8b1e0be700 (LWP 8006) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  29   Thread 0x7f8b1dd81700 (LWP 8009) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  28   Thread 0x7f8b1dc80700 (LWP 8010) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  27   Thread 0x7f8b1db7f700 (LWP 8011) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
...

(gdb) set logging file gdb_output.txt
(gdb) set logging on
(gdb) set height 10000
(gdb) t a a bt
(gdb) quit

It will save logs to gdb_output.txt.

You also can puts ruby logs to stdout, like so

(gdb) call (void)rb_backtrace()

After that, you can try to stop job in Sidekiq Admin with 'Stop' button, or with signals(TSTP to ask it to not pick up any new jobs, and TERM next).

Links

1 Sidekiq FAQ about frozen processes
2 Another guide about frozen processes
3 How to use GDB with ruby processes
4 How to stop Sidekiq jobs(as your application mechanic)

Blog

What to do if Sidekiq job is stuck

Dmitry Daw

Links

Join Our Newsletter. No Spam, Only the good stuff.

Related