#8-2020-Mar-“Signals for clean stops”

In what cases is a signal relevant?

  1. You run your application from the command line until KeyboardInterrupt

  2. You launch a subprocess and want it to shutdown gracefully

Whenever you run a process, there is a chance of someone killing your process. Be prepared and create a signal handler:

import signal
import logging
logger = logging.getLogger(__name__)
def _on_stop(signum, frame):
    logger.info(f"received signum: {signum}, frame={frame}")
    # Stop/cleanup my resources
            
signal.signal(signal.SIGINT, _on_stop)
signal.signal(signal.SIGTERM, _on_stop)

Why not wrap it into a register_shutdown_callback?

import logging

import signal
from threading import Lock
from typing import Callable

logger = logging.getLogger(__name__)
_callbacks = []
_registered = False
_lock = Lock()

def _on_stop(signum, frame):
    logger.info(f"received signum: {signum}, frame={frame}")
    for c in _callbacks:
        c()

def register_shutdown_callback(callback: Callable[[], None]):
    global _registered
    with _lock:
        _callbacks.append(callback)
        if not _registered:
            signal.signal(signal.SIGINT, _on_stop)
            signal.signal(signal.SIGTERM, _on_stop)
            _registered = True

# rabbitmq.my_safe_connection.py
class Connection:
    ...
    def disconnect(self):
        pass

connection = Connection()     
register_shutdown_callback(lambda: connection.disconnect())

Lessons learned and discussion

  1. Other frameworks might override your signal handler, e.g., faust.Worker

    1. Hook into their callbacks or execute the cleanup in the code parts you know are reached.

  2. I run my test in my IDEA. After stopping the process by clicking stop it continues to run. I have to use kill to stop it completely

    1. This probably means that some thread is still running. Somewhere there is some un-tracked resource. What can it be?

      1. A thread from a thread pool?

      2. An io-loop that is never told to stop?

      3. Your MainThread without signal handling?

Working with many threads and the scarier, many processes, is not easy. Know how to stop/kill can be essential to avoid weird behavior, memory leak, and race conditions.

How to stop/kill?

  1. Block on a conditional variable in a while loop

  2. Use a queue

  3. Offer an API for closing

  4. Give up after a timeout

  5. Know the PID and kill it

  6. Use a context manager