Cron Job Monitoring: How to Verify a Run Truly Succeeded?
Cron job monitoring is the practice of confirming that a scheduled task not only executed but completed its intended work without errors. To verify a run truly succeeded, you must check three things: the exit code, the output or logs, and the side effects or expected results of the job. This article explains each layer of verification, common pitfalls, and how to build a reliable monitoring setup.
Understanding Cron Exit Codes and Output
When cron runs a command, it captures two things: the exit code (0 for success, non-zero for failure) and any output sent to stdout or stderr. By default, cron emails the output to the user who owns the crontab, but this email is often unmonitored or goes to a spam folder. To verify success, you must explicitly capture and inspect both the exit code and the output.
The exit code alone is insufficient because many commands return 0 even when their internal logic fails. For example, a backup script that copies files but encounters a permission error on one directory may still exit 0 if the error is not fatal. Similarly, a Python script that catches exceptions and prints a message may exit 0 while the intended data processing never happened.
Logging: The First Line of Defense
Every cron job should write its own log file with timestamps and status messages. Redirect stdout and stderr to a file using >> /var/log/myjob.log 2>&1 in the cron command. Then, after the command runs, append a line that records the exit code: echo "Exit code: $?" >> /var/log/myjob.log.
For jobs that produce a result file (e.g., a database dump, a report, or a processed dataset), verify that the file exists, is non-empty, and has a recent modification time. A simple shell test can do this: test -s /path/to/output && echo "Output OK" || echo "Output missing". This catches cases where the script ran but produced nothing.
Using External Monitoring Tools
Manual log checking is not scalable. External monitoring services (e.g., Cronitor, Healthchecks.io, or Dead Man’s Snitch) work by having your cron job send an HTTP request to a unique URL after completing. If the service does not receive the ping within a scheduled time, it alerts you via email, SMS, or webhook.
This approach verifies that the job ran to completion, but it does not verify correctness. To verify that the job did the right thing, you can have the script send a second ping with a status flag (e.g., curl -d "status=ok" https://hc-ping.com/your-uuid). Some services allow you to include a payload that contains the exit code or a custom success message, which you can later inspect.
Monitoring Side Effects and Expected Results
The most reliable way to verify a cron job succeeded is to check what it was supposed to change. For a database backup, verify the backup file size is within a normal range and that it can be restored (e.g., run a quick integrity check). For a data sync job, compare record counts between source and destination. For a report generator, check that the output file contains a known header and a reasonable number of rows.
This can be automated with a separate verification script that runs after the main job. For example, a cron entry could be: 0 2 * * * /usr/local/bin/backup.sh && /usr/local/bin/verify_backup.sh. If the backup fails, the second script never runs, and you get an alert from cron’s mail or your monitoring tool.
Common Pitfalls in Cron Monitoring
- Environment differences: Cron runs with a minimal PATH. Always use absolute paths for commands and scripts, and set environment variables explicitly (e.g.,
HOME=/home/user) inside the script. - Timezone and DST: Cron uses the system timezone. If your job depends on a specific time, verify the timezone setting and test around daylight saving changes.
- Overlapping runs: If a job takes longer than the interval, two instances may run concurrently. Use a lock file (e.g.,
flock) to prevent this, and monitor for stale locks. - Silent failures: Some commands fail silently (e.g.,
curlwith-sflag hides errors). Always check the exit code of each command in a chain usingset -eor explicitifstatements. - Email loss: Relying on cron’s default email is unreliable. Configure a proper mail relay or use a monitoring service.
How to Choose the Right Monitoring Approach
For low-risk jobs (e.g., a daily cleanup), a simple log check might suffice. For critical jobs (e.g., production database backups), use a combination of external heartbeat monitoring and side-effect verification. Consider the cost of a missed failure: if a failed backup goes unnoticed for a week, the cost of restoring from an old backup may be high.
Start with a free monitoring service that offers a generous ping interval (e.g., Healthchecks.io free tier). As your needs grow, you can add more sophisticated checks, such as integrating with your existing alerting system (PagerDuty, Slack) via webhooks.
Frequently Asked Questions
Why does my cron job appear to run but nothing happens?
This is often due to a missing PATH or environment variable. Cron does not load your shell’s profile. Add PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top of your crontab or inside the script. Also check that the script has execute permissions and the correct shebang line.
How can I see the output of a cron job?
By default, cron sends output to the user’s email. To see it, add MAILTO=your@email.com to your crontab or redirect output to a file: 0 2 * * * /path/to/script >> /var/log/myjob.log 2>&1. Then inspect that log file.
What does a cron exit code 1 mean?
Exit code 1 is a generic error code. It means the command failed, but the specific reason is in the output or log. Check the log for error messages. Common causes include missing files, permission issues, or a command not found.
Can I verify a cron job without using external services?
Yes. You can write a wrapper script that checks the exit code and sends an alert via local tools like mail or sendmail. For more advanced checks, use a system monitoring tool like Nagios or Zabbix that can execute scripts on a schedule and alert on failure.
How do I prevent cron jobs from overlapping?
Use a lock file with flock. For example: 0 * * * * flock -n /tmp/myjob.lock -c '/path/to/script'. If the lock is already held, the job exits immediately. This prevents concurrent runs.
What is the best way to monitor a cron job that runs only once a day?
For daily jobs, a heartbeat monitor with a ping interval of 24 hours is ideal. Services like Healthchecks.io allow you to set a grace period of several hours. If the ping does not arrive within the grace period, you get an alert. This is more reliable than checking logs manually.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.