Creating & Monitoring Cron Jobs in Python

Most Python scheduling problems start the same way: a developer writes a script, tests it locally, adds it to crontab, and walks away. Two weeks later someone notices the nightly database export hasn't run since Thursday. The script itself is fine. Cron ran it. But cron ran it with the system Python, which doesn't have pandas installed, and the import error went to /dev/null because nobody redirected stderr.

This guide covers every practical way to schedule Python code on a repeating timer: system crontab, in-process schedulers like APScheduler and the schedule library, Celery Beat for distributed setups, and container-friendly approaches for Docker. Each has a use case where it wins. Each has a way to silently fail that you won't notice until it costs you something.

System crontab: the 80% solution

For a single script on a single server, crontab is still the right answer. No daemons to babysit, no broker to configure, no library version to track. You write one line and the OS handles the rest. If you need a refresher on cron expression syntax, we have a full guide on that.

Here's a backup script that dumps a Postgres database every night at 3 AM:

0 3 * * * /opt/app/venv/bin/python /opt/app/backup_db.py >> /var/log/backup.log 2>&1

Notice the full path to the Python binary inside the virtualenv. Not python3, not ~/venv/bin/python. The absolute path. That single detail prevents about 60% of the "it works in my terminal but not in cron" tickets I've seen.

Why your script works manually but fails in cron

Cron doesn't load your shell profile. No .bashrc, no .zshrc, no conda activation, no pyenv shims. The PATH cron sees is usually just /usr/bin:/bin. Your script might depend on environment variables set in .env files that your shell sources automatically on login. Cron doesn't know those files exist.

The fix is blunt: spell everything out.

# Set env vars directly in crontab
DB_HOST=10.0.1.5
DB_NAME=production
PYTHONPATH=/opt/app

0 3 * * * /opt/app/venv/bin/python /opt/app/backup_db.py >> /var/log/backup.log 2>&1

Or source your .env file before running:

0 3 * * * . /opt/app/.env && /opt/app/venv/bin/python /opt/app/backup_db.py >> /var/log/backup.log 2>&1

Both work. The inline approach is easier to debug because crontab -l shows everything in one place.

Redirect stderr or lose the error

Cron captures stdout and stderr separately, and by default it tries to email the output to the crontab owner. On most servers, local mail delivery isn't configured. So the output vanishes. The 2>&1 at the end merges stderr into the log file. Without it, your Python tracebacks disappear into the void.

Quick test to confirm your cron entry works:

crontab -l

0 3 * * * /opt/app/venv/bin/python /opt/app/backup_db.py >> /var/log/backup.log 2>&1

Then check the log after the next scheduled run. If the file is empty or missing, cron either didn't fire (check grep CRON /var/log/syslog) or the path is wrong.

APScheduler: cron-style scheduling inside a Python app

Sometimes the scheduled task lives inside a running application: a Flask API that regenerates a cache every hour, a Django app that sends digest emails at 9 AM. You don't want a separate crontab entry because the task needs the app's database connection, configuration, and imports.

APScheduler (version 3.11.3 as of mid-2026) embeds a scheduler directly in your Python process. It supports cron-style triggers, interval triggers, and one-off date triggers.

from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger

scheduler = BackgroundScheduler()

def expire_stale_sessions():
    # ... your cleanup logic
    print(f"Cleaned up sessions at {datetime.now()}")

scheduler.add_job(
    expire_stale_sessions,
    CronTrigger(hour=2, minute=30),  # daily at 02:30
    id='session-cleanup',
    replace_existing=True,
)

scheduler.start()
# Your app continues running — scheduler runs in a background thread

BackgroundScheduler spawns a thread alongside your main application. BlockingScheduler is the alternative when the scheduler IS your application (a standalone worker script).

One gotcha that burned an afternoon

The weekday numbering doesn't match standard cron. In POSIX cron, 0 is Sunday. In APScheduler 3.x, 0 is Monday. Wasted a good chunk of time debugging why a Sunday job ran on Monday before I caught this. If you're porting a crontab expression like 0 3 * * 0 (Sunday at 3 AM), the equivalent is CronTrigger(hour=3, day_of_week='sun'). Use the string name to avoid confusion.

Job persistence is available through database stores (SQLAlchemy, Redis, MongoDB) so schedules survive process restarts. For most single-process apps, the default in-memory store is enough. Add a persistent store when you're running multiple app instances behind a load balancer and need to ensure the job runs exactly once.

One note: the 4.0 rewrite has been in alpha since 2024 (currently 4.0.0a6). Stick with the 3.x line for production. The API will change significantly when 4.0 stabilizes.

The schedule library: five lines and a while loop

Dan Bader's schedule library is the simplest in-process scheduler you'll find. Zero dependencies. Readable syntax. It does one thing.

import schedule
import time

def check_disk_space():
    # ... check and alert if low
    pass

def run_backup():
    # ... dump database to S3
    pass

schedule.every(30).minutes.do(check_disk_space)
schedule.every().day.at("03:00").do(run_backup)
schedule.every().wednesday.at("13:15").do(send_weekly_report)

while True:
    schedule.run_pending()
    time.sleep(1)

That while True loop is the whole runtime. If the process dies, the schedules are gone. No persistence, no catch-up on missed runs, no concurrency (a slow job blocks everything behind it). No timezone support either.

Use this for prototypes, dev-environment tasks, or scripts you'll supervise with systemd or supervisord. Don't use it when you need reliability guarantees. A missed run at 3 AM because the server rebooted won't be retried.

Celery Beat: for teams already running Celery

If your application already uses Celery for background tasks (email sending, image processing, webhook delivery), Celery Beat adds periodic scheduling without introducing another tool. You define schedules in the same config, and Beat pushes task messages onto the broker at the right times. Workers execute them.

# celery_app.py
from celery import Celery
from celery.schedules import crontab

app = Celery('myapp', broker='redis://localhost:6379/0')

app.conf.beat_schedule = {
    'nightly-cleanup': {
        'task': 'tasks.cleanup_old_records',
        'schedule': crontab(hour=3, minute=0),
    },
    'health-ping-every-5m': {
        'task': 'tasks.health_check',
        'schedule': 300.0,  # every 5 minutes
    },
}
app.conf.timezone = 'UTC'

# Two separate processes — always
celery -A celery_app beat --loglevel=info    # scheduler (exactly one)
celery -A celery_app worker --loglevel=info  # can run multiple

The critical constraint: exactly one Beat process per schedule. Two Beat instances means every task fires twice. In production, use a process manager (systemd, supervisord, Docker entrypoint) that guarantees a single Beat instance. django-celery-beat adds database-backed schedules if you need to change intervals at runtime without redeploying.

Celery Beat is the right tool when you need retry logic, task routing, result backends, or distributed execution across multiple workers. For a single Python script that runs nightly, it's overkill: you'd be running Redis, a Beat process, and at least one worker just to replace a crontab line.

Managing crontab entries from Python

The python-crontab library (v3.2.0) lets you read, create, and modify crontab entries programmatically. It doesn't execute anything itself. It writes to the same crontab file that the system cron daemon reads.

from crontab import CronTab

cron = CronTab(user=True)

# Create a new job
job = cron.new(command='/opt/app/venv/bin/python /opt/app/sync.py')
job.setall('*/15 * * * *')  # every 15 minutes
job.set_comment('Data sync')
cron.write()

# List all jobs
for j in cron:
    print(f"{j.slices} {j.command}  # {j.comment}")

# Remove by comment
cron.remove_all(comment='Data sync')
cron.write()

Handy for deployment scripts, CLI admin tools, or installers that need to register cron entries as part of setup. The library handles the crontab file format so you don't have to parse it yourself.

Running Python cron jobs in Docker

Standard cron inside a container is a mess. It strips environment variables, swallows output, and doesn't respond to SIGTERM (so docker stop hangs for 10 seconds before the kill signal). The container logs show nothing because cron writes to syslog internally. Spent a frustrating evening figuring out why a perfectly working script produced zero log output in a container before discovering this.

Supercronic fixes all of this. It's a Go binary that reads standard crontab syntax, runs in the foreground as PID 1, preserves environment variables from docker run -e, and sends all output to stdout/stderr where Docker's logging driver picks it up.

FROM python:3.12-slim

# Install supercronic (check github.com/aptible/supercronic/releases for latest)
ARG SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.33/supercronic-linux-amd64
RUN curl -fsSL "$SUPERCRONIC_URL" -o /usr/local/bin/supercronic \
    && chmod +x /usr/local/bin/supercronic

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ /opt/app/
COPY crontab /etc/crontab

CMD ["supercronic", "/etc/crontab"]

The crontab file:

0 3 * * * cd /opt/app && python backup_db.py
*/5 * * * * python /opt/app/health_check.py

No special setup for the Python path because the container only has one Python. Environment variables from Docker Compose or -e flags arrive intact. And when you run docker logs, you see the actual script output.

Serverless alternative: AWS Lambda + EventBridge

For tasks under 15 minutes that don't need a running server, Lambda with an EventBridge Scheduler trigger replaces both the server and the cron daemon. You deploy a function, set a schedule, and AWS runs it.

def lambda_handler(event, context):
    result = run_daily_export()
    return {'statusCode': 200, 'records_exported': result}

EventBridge cron expressions use six fields (they add a year) and require ? for unused day fields:

# Daily at 03:00 UTC
cron(0 3 * * ? *)

The cost for a daily job is effectively zero (free tier covers 1M invocations/month). The tradeoff: cold starts add 1-3 seconds, no persistent filesystem, and the 15-minute execution limit is hard. Long-running ETL jobs need a different approach (ECS tasks, Step Functions, or a plain server with crontab).

Monitoring your Python cron jobs for silent failures

Whichever scheduling method you pick, the failure mode is the same: the job stops running and nobody notices. A heartbeat monitor watches for the ping that should arrive after each successful run. If the ping doesn't come, you get an alert.

With urllib (zero dependencies, works everywhere):

import urllib.request

PING_URL = "https://monitoring.example.com/ping/YOUR-CHECK-ID"

def ping_on_success():
    try:
        urllib.request.urlopen(PING_URL, timeout=10)
    except Exception:
        pass  # monitoring failure shouldn't break the actual job

if __name__ == '__main__':
    run_backup()
    ping_on_success()

Or from crontab directly, chained with && so the ping only fires on success:

0 3 * * * /opt/app/venv/bin/python /opt/app/backup_db.py && curl -fsS --retry 3 https://monitoring.example.com/ping/YOUR-CHECK-ID > /dev/null

For jobs where you also want to track duration and catch failures explicitly:

import urllib.request

PING_URL = "https://monitoring.example.com/ping/YOUR-CHECK-ID"

# Signal start
urllib.request.urlopen(f"{PING_URL}/start", timeout=10)

try:
    run_backup()
    # Signal success
    urllib.request.urlopen(PING_URL, timeout=10)
except Exception:
    # Signal failure
    urllib.request.urlopen(f"{PING_URL}/fail", timeout=10)
    raise

WatchCron's heartbeat monitoring works exactly this way: you point your script at a ping URL, set an expected schedule, and if the ping doesn't arrive within the grace period, alerts fire to Slack, email, or whatever channels you've configured. Setup takes about two minutes.

Choosing the right Python scheduled task approach

A single Python script on a VPS that runs once a day: system crontab. Nothing else comes close for simplicity, and there's nothing to install.

Scheduling inside a running Flask or Django app: APScheduler with BackgroundScheduler. It shares the app's process and configuration without needing an external cron entry.

Quick prototype or development task: the schedule library. Accept the limitations (no persistence, no concurrency) and move on.

Distributed system with existing Celery workers: Celery Beat. Don't introduce it just for scheduling though; the infrastructure overhead (broker + workers + beat process) only makes sense if you're already running Celery for other things.

Docker containers: Supercronic as a drop-in cron replacement. It handles environment variables, logging, and graceful shutdown the way containers expect.

Serverless, short tasks: Lambda + EventBridge. Zero infrastructure to maintain, but the 15-minute limit and cold starts mean it's not for everything.

Whatever you choose, add monitoring. A dead man's switch catches the failures that your scheduling tool won't tell you about: the server that rebooted, the container that didn't restart, the Lambda that timed out. The scheduling part is the easy problem. Knowing when it breaks is the hard one.