Python Server Monitor Stops Running: Keep It Alive
- Check interval
- 60 seconds
- You entered
- Failure threshold
- 3 count
- You entered
A Python server monitor stops running when its process is terminated, crashes on an unhandled exception, or is killed by the host system (OOM killer, reboot, or user logout). To keep it alive, you need to run it as a managed service with automatic restart, handle exceptions gracefully, and monitor the monitor itself.
Formula
The restart delay is the product of the check interval and the failure threshold:
Restart delay (seconds) = Check interval (seconds) × Failure threshold (count)
- Check interval (t) – how often the monitor checks the target service (in seconds).
- Failure threshold (f) – how many consecutive failed checks before a restart is triggered.
- Restart delay (rt) – the time that passes between the first failed check and the restart action (in seconds).
Worked Example
Using the default values: check interval = 60 seconds, failure threshold = 3.
- Multiply the interval by the threshold: 60 × 3 = 180 seconds.
- The monitor will restart the service after 180 seconds (3 minutes) of consecutive failures.
This delay gives the service time to recover on its own before a forced restart, avoiding unnecessary disruption.
Why Python Monitors Die
Python scripts are not inherently persistent. They run as a single process that ends when the script finishes, raises an unhandled exception, or is terminated by the operating system. Common causes include:
- Unhandled exceptions – network timeouts, file permission errors, or malformed data can crash the script.
- Memory leaks – long-running Python processes may gradually consume more RAM until the OS kills them.
- System reboot or logout – if the monitor runs in a terminal session, closing the session ends the process.
- OOM killer – on Linux, the Out-Of-Memory killer may terminate processes that use too much memory.
- Dependency failures – a library update or missing module can cause an import error on restart.
How to Keep It Alive: Basic Approaches
The simplest way is to run the monitor inside a loop that catches exceptions and sleeps between checks. For example:
import time
while True:
try:
check_service()
except Exception as e:
log_error(e)
time.sleep(60)
This prevents crashes from unhandled exceptions, but it does not protect against system reboots or process termination. For true resilience, you need a process manager.
Using systemd (Linux)
systemd is the standard init system on most Linux distributions. Create a service file at /etc/systemd/system/monitor.service:
[Unit]
Description=Python Server Monitor
After=network.target
[Service]
ExecStart=/usr/bin/python3 /path/to/monitor.py
Restart=always
RestartSec=10
User=monitor
[Install]
WantedBy=multi-user.target
Then enable and start it:
sudo systemctl enable monitor
sudo systemctl start monitor
The Restart=always directive tells systemd to restart the process if it exits for any reason. RestartSec=10 adds a 10-second delay before restarting, preventing a tight crash loop.
Using Supervisor (Cross-Platform)
Supervisor is a Python-based process manager that works on Linux and macOS. Install it with pip install supervisor, then create a configuration file:
[program:monitor]
command=python3 /path/to/monitor.py
autostart=true
autorestart=true
stderr_logfile=/var/log/monitor.err.log
stdout_logfile=/var/log/monitor.out.log
Start Supervisor with supervisord -c /path/to/supervisord.conf. It will keep the monitor running and restart it if it dies.
Using Docker with Restart Policies
Containerizing your monitor adds isolation and portability. A simple Dockerfile:
FROM python:3.12-slim
COPY monitor.py /app/monitor.py
CMD ["python", "/app/monitor.py"]
Run it with a restart policy:
docker run -d --restart unless-stopped --name monitor my-monitor-image
The unless-stopped policy restarts the container unless it was explicitly stopped by the user. This survives daemon restarts and reboots (if Docker is set to start on boot).
Common Pitfalls and How to Avoid Them
- Crash loops – if the monitor crashes immediately after start, a restart policy will cause a tight loop. Add a delay (
RestartSecin systemd,startsecsin Supervisor) and log the exception. - Resource leaks – close file handles, database connections, and network sockets. Use context managers (
withstatements) to ensure cleanup. - Time zone issues – if your monitor logs timestamps, use UTC or set the timezone explicitly to avoid confusion.
- Dependency drift – pin your Python dependencies in a
requirements.txtand rebuild the environment regularly. - Monitoring the monitor – set up a separate watchdog (e.g., a cron job that checks if the process is running) to alert you if the monitor itself dies.
Frequently Asked Questions
Why does my Python script stop running after I close the terminal?
When you close the terminal, the shell sends a SIGHUP signal to the foreground process, which terminates it. To keep the script running, use nohup or better, run it as a background service with systemd or Supervisor.
What is the difference between Restart=always and Restart=on-failure in systemd?
Restart=always restarts the service regardless of the exit status, even if it exits cleanly. Restart=on-failure only restarts if the process exits with a non-zero status or is terminated by a signal. For a monitor that should never stop, always is safer.
How do I check if my Python monitor is still running?
Use ps aux | grep monitor.py to see the process. For systemd, use systemctl status monitor. For Docker, docker ps. You can also add a heartbeat log that writes a line every minute and check the log file.
Can I use a cron job to keep the monitor alive?
Yes, you can schedule a cron job every minute that checks if the process is running and starts it if not. However, this is less reliable than a process manager because cron does not handle crashes between runs and can overlap if the script takes longer than a minute.
What should I do if the monitor keeps crashing due to an unhandled exception?
Add a top-level exception handler that logs the full traceback to a file and then sleeps before retrying. Use traceback.format_exc() to capture the error. Also, use a retry library like tenacity to handle transient network errors.
Is it better to run the monitor as a daemon or a container?
It depends on your environment. A daemon (systemd/Supervisor) is lighter and integrates with the host OS. A container provides isolation and easier deployment across machines. For a single server, a daemon is simpler; for a cluster, containers are easier to orchestrate.
Related
Formula used
in t | Check interval | seconds | 60 | 1 in f | Failure threshold | count | 3 | 1 out rt | Restart delay | seconds | t * f | 0
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.