Debug Failed Cron Jobs: Complete Guide to Crontab Logs

Yesterday at 2 AM, your database backup script failed silently. You discovered this during the morning standup when the primary database was corrupted. The backup hadn’t run in three days, and there was no indication of the failure until it was too late.

This scenario happens more often than you’d think. Cron jobs fail silently, and without proper logging, you’re left assuming what went wrong.

Why Cron Jobs Fail

Cron jobs are designed to run quietly in the background. By default, they produce no output unless something goes wrong, and even then, the output often disappears into the void. This creates several common failure scenarios:

Silent permission failures - Your script works perfectly when you run it manually, but fails in cron due to different user permissions or environment variables.

Path resolution issues - Commands that work in your interactive shell can’t be found by cron because of its limited PATH environment.

Resource constraints - Jobs fail due to insufficient disk space, memory limits, or file handle constraints that only manifest during automated execution.

Dependency failures - Database connections, network resources, or external services that are unavailable during the scheduled execution time.

Without logging, these failures remain invisible until they cause visible problems in your application or infrastructure.

What You’ll Find in Crontab Logs

Crontab logs show cron daemon activity, not the actual output of your script. They tell you when a job was triggered, who ran it, and what command was executed. They also capture system-level errors like permission issues or missing commands.

But they don’t log what happened inside your script. If the job fails due to a bug in your code or a missing dependency, you won’t see those details unless you’ve set up explicit logging.

Example: A Typical Crontab Log Entry

Jul 10 02:00:01 server CRON[12345]: (root) CMD (/usr/local/bin/backup.sh)

This means the cron daemon triggered backup.sh at 2:00 AM under the root user. If the script failed due to an internal error, that wouldn’t show up here unless it caused a system-level failure (like missing permissions or an invalid command).

To get visibility into script-level issues, always redirect your output to a log file:

0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

This captures both stdout and stderr, giving you the full picture.

Turn On System-Level Cron Logging

Most Linux distributions don’t log cron jobs in detail by default. To get full visibility into when jobs run (or don’t), you’ll need to enable system-level logging.

For most Linux systems (using rsyslog):

# Add cron logging to rsyslog
echo "cron.* /var/log/cron.log" | sudo tee -a /etc/rsyslog.d/50-default.conf

# Restart rsyslog to apply changes
sudo systemctl restart rsyslog

# Make sure the cron service is running
sudo systemctl status cron

On RHEL-based systems:

Cron logging is usually enabled out of the box, but it’s worth confirming:

# Check if cron logging is configured
grep -i cron /etc/rsyslog.conf
grep -i cron /etc/rsyslog.d/*

# Verify the log file exists and is active
ls -la /var/log/cron

Capture Script Output, Not Just Execution

System logs show that a cron job ran, but they won’t capture what your script actually did or any errors it threw. To get full context, you’ll want to redirect your script’s output manually.

Examples:

# Basic output redirection
0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

# Separate error logs for debugging
0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>> /var/log/backup-errors.log

# Include environment details
0 2 * * * env > /tmp/cron-env.log && /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

The 2>&1 part ensures that standard error (stderr) is captured along with standard output (stdout). Without it, silent failures can slip past unnoticed.

Recognize Common Cron Failure Patterns

Not all cron failures look the same. Different issues leave different traces in your logs. Knowing what to look for can save you a ton of time.

1. Permission Denied

You’ll usually see something like:

Dec 15 02:00:01 server CRON[12345]: (root) CMD (/usr/local/bin/backup.sh)
Dec 15 02:00:01 server CRON[12345]: (CRON) error (grandchild #12346 failed with exit status 1)

And in your script’s output log:

/usr/local/bin/backup.sh: line 15: /var/log/backup.log: Permission denied

This typically means your script is trying to write somewhere it doesn’t have access.

2. Command Not Found

Dec 15 03:00:01 server CRON[12347]: (root) CMD (python3 /opt/scripts/cleanup.py)
Dec 15 03:00:01 server CRON[12347]: (CRON) error (/bin/sh: python3: command not found)

This happens when the cron environment doesn’t have the same PATH as your shell. Either use absolute paths (e.g., /usr/bin/python3) or define the PATH explicitly at the top of your crontab.

3. Script Not Executable

Dec 15 04:00:01 server CRON[12348]: (root) CMD (/opt/scripts/deploy.sh)
Dec 15 04:00:01 server CRON[12348]: (CRON) error (grandchild #12349 failed with exit status 126)

Exit status 126 means the file was found, but cron couldn’t execute it—usually because it’s missing the +x (execute) permission. Fix it with:

chmod +x /opt/scripts/deploy.sh

How to Search, Filter, and Debug Cron Logs

Once cron logging is enabled, knowing how to sift through the logs is key. These commands help you find patterns, catch silent failures, and confirm that jobs actually ran.

Basic Log Checks

# View recent cron job executions
grep CRON /var/log/syslog | tail -20

# Stream cron logs in real-time
tail -f /var/log/cron.log

# Search for failed jobs or error messages
grep -i "failed\|error" /var/log/cron.log

# Filter logs by job name or user
grep "backup.sh" /var/log/cron.log
grep "(root)" /var/log/cron.log

# Check if a job didn’t run today
grep "$(date +%Y-%m-%d)" /var/log/cron.log | grep -v "your-expected-job"

For systemd-based systems (journalctl)

# View logs for the cron service
journalctl -u cron

# Look at a specific time window
journalctl -u cron --since "2024-01-15 00:00:00" --until "2024-01-15 23:59:59"

# Stream cron logs in real-time
journalctl -f -u cron

# Search for specific errors
journalctl -u cron | grep -i "error\|failed"

These logs tell you what ran and when—but they don’t always capture why it failed. That’s where structured job output and tools like Last9 help fill in the gaps by correlating logs, metrics, and alerts.

Step-by-Step Cron Job Troubleshooting

When a cron job fails, a structured approach saves time and helps you zero in on the issue. Here’s a basic sequence to follow:

1. Verify the Job is Scheduled

# Check if the job exists in crontab
crontab -l | grep -i backup

# Ensure the cron service is running
systemctl status cron

# Confirm system time and timezone
date
timedatectl status

2. Check for Execution Attempts

# Look for any attempt to run the job
grep "backup.sh" /var/log/cron.log

# View surrounding log context
grep -A5 -B5 "backup.sh" /var/log/cron.log

3. Test the Command Manually

# Run the script as the cron user
sudo -u root /usr/local/bin/backup.sh

# Mimic cron’s minimal environment
env -i /bin/bash -c '/usr/local/bin/backup.sh'

# Check for missing paths
which python3
which mysqldump

4. Examine Script Output

# Tail the job’s output logs
tail -50 /var/log/backup.log

# Check for system resource limits
df -h       # Disk space
free -h     # Memory usage
ulimit -a   # User-level limits

Why Scripts Work in Shell but Fail in Cron

Cron jobs don’t run with the same environment as your interactive shell. They inherit a limited set of environment variables, which often leads to jobs failing silently or behaving unpredictably, especially when they rely on PATH, HOME, or a working directory.

1. PATH Differences

# Add a full PATH at the top of your crontab
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Or use absolute paths in the command
0 2 * * * /usr/bin/python3 /opt/scripts/backup.py

Without this, cron may not know where to find binaries like python3, mysqldump, or curl.

2. Missing Environment Variables

# Define required environment variables directly in the crontab
HOME=/root
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin

0 2 * * * /usr/local/bin/backup.sh

If your script depends on certain variables being present, define them explicitly in the crontab or source an env file.

3. Wrong Working Directory

Cron jobs start in the user’s home directory unless told otherwise. If your script assumes a specific working directory, it’ll likely fail.

# Set the working directory explicitly
0 2 * * * cd /opt/backups && /usr/local/bin/backup.sh

# Or better: use absolute paths throughout your script

You can debug environment-related issues by dumping the cron environment like this:

0 2 * * * env > /tmp/cron-env.log

Then compare it to your shell’s env output. The differences often explain what went wrong.

Setup Proactive Monitoring for Cron Jobs

Don’t wait for a failed job to find out something’s broken. Add basic monitoring to catch problems early, before they lead to data loss or 2 AM debugging.

Monitor Job Activity with a Simple Script

Here’s an example script to check if a backup job ran recently and alert if it didn’t:

#!/bin/bash
# /usr/local/bin/check-backup-job.sh

LOGFILE="/var/log/backup.log"
WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# If the log file is missing, alert
if [ ! -f "$LOGFILE" ]; then
    curl -X POST -H 'Content-type: application/json' \
         --data '{"text":"Backup log file missing"}' \
         "$WEBHOOK_URL"
    exit 1
fi

# Check time since last update
LAST_BACKUP=$(stat -c %Y "$LOGFILE")
CURRENT_TIME=$(date +%s)
HOURS_SINCE=$(( ($CURRENT_TIME - $LAST_BACKUP) / 3600 ))

if [ $HOURS_SINCE -gt 25 ]; then
    curl -X POST -H 'Content-type: application/json' \
         --data '{"text":"Backup hasn’t run in '"$HOURS_SINCE"' hours"}' \
         "$WEBHOOK_URL"
fi

Schedule the Monitoring Script

Run it every few hours using cron:

# Run every 4 hours
0 */4 * * * /usr/local/bin/check-backup-job.sh

Add Alerts for Failures in Your Crontab

You can also update your existing cron jobs to alert on failure:

0 2 * * * /usr/local/bin/backup.sh || curl -X POST -H 'Content-type: application/json' \
--data '{"text":"Backup failed"}' https://hooks.slack.com/services/YOUR/WEBHOOK/URL

This sends a Slack notification if the script exits with a non-zero status.

Manage Cron Log File Growth

Cron logs and job output logs can quietly eat up disk space if you don’t rotate them. Use logrotate to keep things under control and avoid surprises.

Rotate System Cron Logs

Create a logrotate config for cron system logs:

sudo tee /etc/logrotate.d/cron << EOF
/var/log/cron.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
    postrotate
        systemctl reload rsyslog
    endscript
}
EOF

This keeps two weeks of daily logs, compressed after the first day, and reloads rsyslog after rotation.

Rotate Job Output Logs

To manage logs generated by your own cron jobs:

sudo tee /etc/logrotate.d/cron-jobs << EOF
/var/log/backup.log /var/log/backup-errors.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
}
EOF

This retains 30 days of logs for job output, compressing them to save space.

You can also test your config with:

logrotate -d /etc/logrotate.d/cron

Advanced Cron Debugging Techniques

For intermittent failures or jobs that mysteriously fail under cron but work manually, deeper debugging can help you pinpoint the issue.

Trace System Calls with strace

Use strace to watch what your script is actually doing at the system level—file accesses, network calls, permission issues, etc.

# Capture a trace log during cron execution
0 2 * * * strace -o /tmp/backup-trace.log /usr/local/bin/backup.sh

This creates a detailed trace at /tmp/backup-trace.log that you can inspect later.

Enable Bash Debugging Flags

Add these flags to your script for better visibility into how it runs:

#!/bin/bash
set -x  # Print each command before running it
set -e  # Exit on any error
set -u  # Exit on use of undefined variables

# Redirect debug output to a log file
exec 2>> /var/log/backup-debug.log

You’ll get a full command-by-command trace of the script’s execution.

Monitor Resource Usage with time

If you suspect resource constraints (e.g. memory, CPU), track them during job execution:

# Log resource usage for the job
0 2 * * * /usr/bin/time -v /usr/local/bin/backup.sh 2>> /var/log/backup-resources.log

This logs memory usage, CPU time, and I/O stats.

Reproduce the Cron Environment

The cron environment can differ drastically from your shell. To debug accurately:

# Dump cron’s environment
* * * * * env > /tmp/cron-env.log && /usr/local/bin/backup.sh

# Run your script with that same environment
env -i $(cat /tmp/cron-env.log | xargs) /usr/local/bin/backup.sh

This helps catch issues caused by missing environment variables or different runtime contexts.

Make Cron Jobs Observable, Secure, and Measurable

Cron jobs often run critical workflows, backups, data syncs, cleanup tasks. But if you’re not tracking performance, logging securely, or integrating with your observability stack, you’re flying blind when things go wrong or slow down over time. Here’s how to level up your cron jobs.

Track Performance Over Time

Monitor how long your jobs take, how they affect system resources, and whether they’re degrading over time.

# Log job duration
0 2 * * * START=$(date +%s) && /usr/local/bin/backup.sh && END=$(date +%s) && \
echo "Backup took $((END-START)) seconds" >> /var/log/backup-timing.log

# Monitor disk I/O during execution
0 2 * * * iostat -x 1 60 > /var/log/backup-iostat.log & /usr/local/bin/backup.sh; wait

# Capture disk usage before and after
0 2 * * * df -h > /var/log/disk-usage-before.log && /usr/local/bin/backup.sh && df -h > /var/log/disk-usage-after.log

Secure Your Logging Setup

Since cron jobs often touch sensitive systems or credentials, make sure logs don’t become a liability.

# Lock down log file permissions
chmod 600 /var/log/cron.log
chmod 600 /var/log/backup.log

# Run jobs as a non-login, dedicated system user
useradd --system --no-create-home --shell /bin/false cronuser
sudo -u cronuser crontab -e

And always sanitize your logs:

# Don't log sensitive data
# Avoid: echo "Connecting to database with password $DB_PASS"
# Use:   echo "Connecting to database"

Integrate with Your Observability Stack

Make cron jobs first-class citizens in your monitoring and alerting workflows.

Prometheus Metrics Export:

#!/bin/bash
# /usr/local/bin/backup-with-metrics.sh

METRICS_FILE="/var/lib/prometheus/node-exporter/cron_job_success.prom"

if /usr/local/bin/backup.sh; then
    echo "cron_job_success{job=\"backup\"} 1" > $METRICS_FILE
    echo "cron_job_last_success{job=\"backup\"} $(date +%s)" >> $METRICS_FILE
else
    echo "cron_job_success{job=\"backup\"} 0" > $METRICS_FILE
fi

This lets you monitor cron job success/failure from your existing Prometheus + Grafana dashboards.

Structured Logging for Log Aggregation:

log_structured() {
    local level=$1
    local message=$2
    local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
    echo "{\"timestamp\":\"$timestamp\",\"level\":\"$level\",\"job\":\"backup\",\"message\":\"$message\"}" >> /var/log/backup.json
}

# Example usage
log_structured "INFO" "Backup started"

Structured logs are easier to parse and ship to tools like Last9, Loki, or Elasticsearch for long-term analysis and alerting.

Common Cron Job Pitfalls (and How to Avoid Them)

Here are some common mistakes and how to prevent them:

1. Assuming the Cron Environment Matches Your Shell

Just because a script works in your terminal doesn’t mean it’ll work in cron. Cron runs in a stripped-down environment with minimal variables.

Fix: Always test your script using cron’s environment. Use env -i to simulate it or log the environment with env > /tmp/cron-env.log.

2. Ignoring Exit Codes

Cron treats any non-zero exit code as a failure, but it won’t alert you unless you’ve set up notifications.

Fix: Make sure your scripts return the right exit codes, and pipe failures to Slack, email, or an observability platform like Last9.

3. Not Accounting for Partial Failures

A script might succeed in one part and silently fail in another. Without detailed logging, you’ll miss these edge cases.

Fix: Log key steps and error branches explicitly. Don’t assume “no output” means success.

4. Skipping Log Rotation

Output logs that aren’t rotated will grow unchecked. Over time, they can fill up disk space and break other jobs, or the system itself.

Fix: Use logrotate to manage both system and job-specific logs. Test your config regularly.

5. Logging Sensitive Data

It’s easy to accidentally log secrets, especially in debugging mode.

Fix: Never log passwords, tokens, or API keys. Sanitize log messages before writing them.

Conclusion

Cron jobs will fail. The real issue is visibility, did the job run? Did it error out? Did it even start?

Last9 helps you answer those questions fast. By linking logs, metrics, and traces, you get a clear view of what the job touched, databases, APIs, queues, and where it broke. No jumping between tools, no assumption.

Get started for free with us today!

FAQs

Why do my cron jobs work when run manually but fail in cron?

The most common cause is environment differences. Cron runs with a minimal environment that often lacks the PATH, environment variables, and working directory assumptions your script relies on. Test your script with: env -i /bin/bash -c 'your-command' to simulate the cron environment.

How can I tell if my cron job is even running?

Check the cron logs for execution attempts: grep "your-job-name" /var/log/cron.log. If you see no entries, the job isn’t running, which could indicate syntax errors in your crontab or a stopped cron service.

What’s the difference between cron daemon logs and job output logs?

Cron daemon logs (/var/log/cron.log) show when jobs start and basic system-level errors like “command not found”. Job output logs capture what your script actually prints, including application errors and success messages. You need both for complete visibility.

How do I get notified immediately when a cron job fails?

Add notification to your crontab entry: 0 2 * * * /your/script.sh || curl -X POST -d '{"text":"Job failed"}' your-webhook-url. This sends an alert only on failure. For more sophisticated monitoring, use tools like Prometheus with alerting rules.

My log files are getting huge. How do I manage them?

Configure logrotate to automatically compress and remove old logs. Create a configuration file in /etc/logrotate.d/ that specifies rotation frequency, compression, and retention policies. This prevents logs from consuming excessive disk space.