Hello 👋 Welcome to my corner of the internet. I write here about the different challenges I encounter, and the projects I work on. Find out more about me.
Recently I needed to run a set of headless browsers in parallel. While tuning this up to 75 instances running next to each other, after about 15.000 sessions having completed, I started seeing a variety of errors:
Thread: can't start new thread
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
ThreadError: can't create Thread: Resource temporarily unavailable
Some of these were raised in the python runtime initiating the selenium sessions, some in other background jobs (ruby) and some in postgres workers.
The common aspect was: there seemed to be no threads available. You can roughly check how may threads are occupied by doing ps x | wc -l
I saw that when restarting the containers, the count would be low, but it would steadily grow into the many thousands. This indicated threads were not being released.
Inspecting the outcome of ps aux, it became clear many of the processes were in fact zombie processes (identifiable by the <defunct>). To see a list of all zombie processes, do:
bash
ps axo stat,ppid,pid,comm | grep -w defunct
A zombie process is a process that has terminated (= is dead) but it still occupies an entry in the process table. When all entries are occupied, none can obviously be given out to new processes, and thus I started seeing errors.
To view the maximum amount of pids/threads on your system or in your container, do:
bash
sysctl kernel.pid_max kernel.threads-max
For me this was ±32.000, and with each browser session leaving behind two zombie processes, the figures added up.
To understand this, we must first understand how zombie processes should get cleaned up. Usually, when a process finishes, it sends a SIGCHLD signal to its parent. The parent process should then clean up those children, i.e. remove them from the process table. Learn more about these POSIX signals here.
Again inspecting the output of ps axo stat,ppid,pid,comm | grep -w defunct you should be able to see the parent process for the defunct processes.
Looking through similar bug reports (of processes not getting reaped), it became clear that quite often processes are simply not implemented to respond to all signals, or specific signals such as SIGCHLD.
In my case, the process was initiated within a buildpack, where PID 1 was a general /start command.
So my hypothesis was that this process did not handle the SIGCHLD signal properly.
This stackoverflow answer made clear that there are general “process supervisor” systems that do handle these process signals properly, and can be prefixed to any other command. I ended up going with dumb-init.
To make it work, we moved from a buildpack based deploy to a dockerfile based deploy, where we could apt-install dumb-init into the container, and then simply do:
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
This makes dumb-init the first process, PID 1, and makes sure the signal can appropriately be handled.
Now, when going into the container at runtime and doing a watch ps axo stat,ppid,pid,comm, no zombie processes are visible anymore, meaning they are being effectively reaped.