24 Matching Annotations
  1. Nov 2022
    1. The second situation would be zombie reaping. If the process spawns child processes and does not properly reap them it will lead to a full process table
    2. For both of these concerns we recommend tini. It is incredibly small, has minimal external dependencies, fills each of these roles, and does only the necessary parts of reaping and signal forwarding.
    3. The first being signal handling. If the process launched does not handle SIGTERM by exiting, it will not be killed since it is PID 1 in the container
    4. There are two situations where an init-like process would be helpful for the container.
    5. highly recommended that the resulting image be just one concern per container; predominantly this means just one process per container, so there is no need for a full init system

      container images: whether to use full init process: implied here: don't need to if only using for single process (which doesn't fork, etc.)

    1. Good question! This is going to be a bit long, so bear with me
    2. Is whatever process you exec in your entrypoint registering signal handlers? A good way to figure this out might be to check whether your process responds properly to e.g. docker stop (or if it waits for 10 seconds before exiting)
    3. Tini does install explicit signal handlers (to forward them, incidentally), so those signals no longer get dropped. Instead, they're sent to Jenkins, which is not running as PID 1 (Tini is), and therefore has default signal handlers
    4. Second, if Jenkins runs as PID 1, then it may not receive the signals you send it! That's a subtlety in PID 1. Unlike other unlike processes, PID 1 does not have default signal handlers, which means that if Jenkins hasn't explicitly installed a signal handler for SIGTERM, then that signal is going to be discarded when it's sent (whereas the default behavior would have been to terminate the process).
    5. Tini fixes by "forwarding signals": if you send a signal to Tini, then it sends that same signal to your child process (Jenkins in your case).
    6. A second problem is that once your process has exited, Bash will proceed to exit as well. If you're not being careful, Bash might exit with exit code 0, whereas your process actually crashed (0 means "all fine"; this would cause Docker restart policies to not do what you expect). What you actually want is for Bash to return the same exit code your process had.
    7. When a zombie is created (i.e. which happens when its parent exits, and therefore all chances of it ever being waited by it are gone), it is reparent to init, which is expected to reap it (which means calling wait on it).
    8. In other words, someone has to clean up after "irresponsible" parents that leave their children un-wait'ed, and that's PID 1's job.
    9. Now, unlike other processes, PID 1 has a unique responsibility, which is to reap zombie processes.
    1. Unfortunately most init systems don't do this correctly within Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed with SIGKILL, which doesn't give them a chance to correctly deinitialize things.
    2. According to the Unix process model, the init process -- PID 1 -- inherits all orphaned child processes and must reap them. Most Docker containers do not have an init process that does this correctly. As a result, their containers become filled with zombie processes over time.
    1. When a process loses its parent, init becomes its new parent. init periodically executes the wait system call to reap any zombies with init as parent.
    2. Zombie processes should not be confused with orphan processes: an orphan process is a process that is still executing, but whose parent has died. When the parent dies, the orphaned child process is adopted by init (process ID 1). When orphan processes die, they do not remain as zombie processes; instead, they are waited on by init.
    3. The result is that a process that is both a zombie and an orphan will be reaped automatically.
    1. A low-level approach is to fork twice, running the desired process in the grandchild, and immediately terminating the child. The grandchild process is now orphaned, and is not adopted by its grandparent, but rather by init.
    2. In a Unix-like operating system any orphaned process will be immediately adopted by an implementation-defined system process: the kernel sets the parent to this process
    1. Let's look at a concrete example. Suppose that your container contains a web server that runs a CGI script that's written in bash. The CGI script calls grep. Then the web server decides that the CGI script is taking too long and kills the script, but grep is not affected and keeps running. When grep finishes, it becomes a zombie and is adopted by the PID 1 (the web server). The web server doesn't know about grep, so it doesn't reap it, and the grep zombie stays in the system.
    2. That is, instead of properly reaping adopted processes, it's probably expecting another init process to do that job, and rightly so.