How to Schedule Web Scrapers: cron to Airflow

A scraper you run by hand has a human in the loop. You watch it finish, you notice the 403s, you rerun it when something looks off. A scheduled scraper has none of that. It runs at 3 a.m., and whatever it does, it does alone.

That changes what "working" means. Four failure modes show up the moment you automate:

• Silent failure. The job stops firing, or it fires and returns empty pages, and nobody finds out until a dashboard downstream goes flat a week later. A crashed cron job emits no error to a human.
• Overlap. A run takes longer than its interval, so the next run starts while the first is still going. Now two copies fight over the same files, the same rows, and the same target IP.
• Redundant work. Every run re-scrapes the entire site from scratch, burning time and request budget on pages that have not changed since yesterday.
• The thundering herd. You add a second scraper, then a tenth, and they all fire at the top of the hour. The target sees a spike from your address range every 60 minutes, and starts blocking.

None of these appear in a "run this script" tutorial because a human standing at the terminal papers over all four. Automation removes the human, so the schedule has to do that job instead. The rest of this guide is about building those guarantees in.

There is no single best scheduler. There is the lightest tool that covers what you actually need. Match the row to your situation and do not reach further up the ladder than the work demands.

The progression is deliberate. cron and systemd cover a huge share of real scraping work, and plenty of teams never need more. You climb to Celery beat when scraping is already part of an application, and to Airflow or Prefect when one scrape feeds another and the dependency graph matters.

For a single machine, cron is still the right answer more often than the internet admits. The mistake is treating a raw cron line as finished. A production cron web scraper line does four things at once: it offsets the minute so it is not on the hour, it locks against overlap, it runs inside the project's virtualenv, and it captures logs.

# /etc/cron.d/scrapers   (user field is required in /etc/cron.d)
# Run the price scraper every 6 hours at :17, never on the hour.
17 */6 * * * scraper /usr/bin/flock -n /tmp/price.lock /opt/scrapers/venv/bin/python /opt/scrapers/price.py >> /var/log/scrapers/price.log 2>&1

flock -n grabs an exclusive lock and exits immediately if the previous run still holds it, which kills overlap for free. The 2>&1 sends stderr into the same log, so a stack trace is not lost. The :17 minute is the cheapest thundering-herd fix there is.

cron has real gaps, though. It never retries, it does not alert, and if the box was asleep at the scheduled time, that run is simply gone. systemd timers close those gaps. You write two files, a service and a timer:

# /etc/systemd/system/price-scraper.service
[Unit]
Description=Price scraper

[Service]
Type=oneshot
EnvironmentFile=/opt/scrapers/.env
ExecStart=/opt/scrapers/venv/bin/python /opt/scrapers/price.py

# /etc/systemd/system/price-scraper.timer
[Unit]
Description=Run the price scraper every 6 hours

[Timer]
OnCalendar=*-*-* 00/6:00:00
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now price-scraper.timer. Three lines earn their keep here. RandomizedDelaySec=900 spreads the real start across a 15-minute window, so a fleet of timers never fires in lockstep. Persistent=true runs a missed job as soon as the machine is back, which cron cannot do. EnvironmentFile keeps secrets out of the unit and out of your shell history. Logs land in journald, queryable with journalctl -u price-scraper.service.

When you have no server to babysit and the code already lives in a repo, GitHub Actions runs scheduled scraping on someone else's machine. Secrets, runners, and logs come included.

# .github/workflows/scrape.yml
name: scheduled-scrape
on:
  schedule:
    - cron: "23 */6 * * *"   # every 6h at :23; cron here is UTC only
  workflow_dispatch: {}        # lets you trigger a run by hand too

jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - name: Run scraper
        env:
          SPARKPROXY_API_KEY: ${{ secrets.SPARKPROXY_API_KEY }}
        run: python price.py

Three quirks are worth knowing before you rely on this. The cron schedule is UTC with no timezone option, so account for that when a job needs to line up with a local business day. Scheduled runs are best-effort and get delayed when GitHub's queue is busy, sometimes by many minutes, so do not use Actions for second-precise timing. And a schedule on a repository with no pushes for 60 days is automatically disabled, which has silently killed more than one long-running collector. Add workflow_dispatch so you always have a manual trigger, and commit the state file the scraper writes so incremental runs survive across ephemeral runners.

If your scraping already lives inside a Python app with a task queue, standing up a second scheduler is wasted effort. Celery beat schedules tasks onto the workers you already run.

# celery_app.py
from celery import Celery
from celery.schedules import crontab

app = Celery("scrapers", broker="redis://localhost:6379/0")

app.conf.beat_schedule = {
    "scrape-prices-6h": {
        "task": "tasks.scrape_prices",
        "schedule": crontab(minute=41, hour="*/6"),
        "options": {"expires": 60 * 30},  # drop the job if it hasn't started in 30 min
    },
}

# tasks.py
from celery_app import app

@app.task(bind=True, acks_late=True, max_retries=5,
          default_retry_delay=60, autoretry_for=(Exception,),
          retry_backoff=True, retry_jitter=True)
def scrape_prices(self):
    rows = run_scrape()
    upsert(rows)

Run the two processes side by side: celery -A celery_app worker and celery -A celery_app beat. The task decorator carries the reliability story. acks_late=True means the message is only acknowledged after the task finishes, so a worker that dies mid-scrape hands the job to another worker instead of dropping it. retry_backoff with retry_jitter spaces out retries and adds randomness so a batch of failures does not retry in a synchronized wave. The one operational rule: run exactly one beat process. Two beats double-schedule everything, so for high availability you guard it with a lock or use a single-leader deployment.

For the async patterns that pair well with Celery workers, our guide on using proxies with Python requests and aiohttp for async scraping covers the fetch side in depth.

Once a scrape is one step in a longer chain, discover URLs, fetch them, parse, then load into a warehouse, you want a DAG. Airflow and Prefect give you dependencies, retries per task, backfills over a date range, and a UI that shows exactly which task failed and why. This is where airflow scraping earns the extra operational weight.

Here is a real Airflow 2.x pipeline using the TaskFlow API and dynamic task mapping:

# dags/price_pipeline.py
from datetime import timedelta
import pendulum
from airflow.decorators import dag, task

@dag(
    schedule="17 */6 * * *",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,             # don't backfill every missed interval on first deploy
    max_active_runs=1,         # never let two runs of this DAG overlap
    default_args={
        "retries": 4,
        "retry_delay": timedelta(minutes=5),
        "retry_exponential_backoff": True,
        "max_retry_delay": timedelta(minutes=30),
    },
    tags=["scraping"],
)
def price_pipeline():

    @task
    def discover(data_interval_start=None) -> list[str]:
        # Only the URLs that changed since the last interval: incremental by design.
        return urls_changed_since(data_interval_start)

    @task(max_active_tis_per_dag=8)   # cap concurrent fetches so you don't flood the target
    def fetch(url: str) -> dict:
        return fetch_via_sparkproxy(url)

    @task
    def load(rows: list[dict]) -> None:
        upsert(rows)

    urls = discover()
    rows = fetch.expand(url=urls)     # one mapped task per URL, fanned out automatically
    load(rows.map(lambda r: r))

price_pipeline()

Two settings do the heavy lifting. max_active_runs=1 stops overlap at the pipeline level, and max_active_tis_per_dag=8 caps how many fetch tasks run at once, which is your rate lever against the target. catchup=False matters on the first deploy: without it, Airflow tries to backfill every interval since start_date in one burst, which is a self-inflicted thundering herd.

Prefect expresses the same idea with less ceremony, and its schedule lives with the deployment rather than in a separate daemon:

# flow.py
from prefect import flow, task

@task(retries=4, retry_delay_seconds=[30, 120, 300, 600])
def fetch(url: str) -> dict:
    return fetch_via_sparkproxy(url)

@flow(log_prints=True)
def price_pipeline(urls: list[str]):
    results = fetch.map(urls)
    upsert([r.result() for r in results])

if __name__ == "__main__":
    # Serve the flow with a cron schedule; no separate scheduler process to babysit.
    price_pipeline.serve(name="prices", cron="17 */6 * * *")

The retry_delay_seconds list gives explicit backoff steps, and serve(cron=...) registers the schedule in one line. Pick Airflow when you already run it or need its ecosystem, and Prefect when you want DAG features without operating a metadata database and scheduler yourself.

Idempotency is the property that running a job twice leaves the same result as running it once. It is the single most important habit for scheduled scraping, because retries, overlaps, and catch-up runs all mean your job will run more than once on the same data.

Two mechanics get you there. The first is a watermark: store the timestamp (or the last ID) you reached, and next run start from there instead of from zero.

# state.py
import json, pathlib

STATE = pathlib.Path("state/price.json")

def load_watermark() -> str:
    if STATE.exists():
        return json.loads(STATE.read_text())["last_run"]
    return "1970-01-01T00:00:00Z"

def save_watermark(ts: str) -> None:
    STATE.parent.mkdir(parents=True, exist_ok=True)
    STATE.write_text(json.dumps({"last_run": ts}))

The watermark turns a full re-scrape into an incremental one. Instead of pulling 40,000 product pages every six hours, you pull the few hundred that changed since load_watermark(), then advance the watermark only after the load succeeds. If the run dies halfway, the watermark stays put and the next run retries the same slice cleanly.

The second mechanic is an upsert at the storage layer, so a re-run overwrites instead of duplicating:

INSERT INTO prices (sku, price, currency, scraped_at)
VALUES (%(sku)s, %(price)s, %(currency)s, %(scraped_at)s)
ON CONFLICT (sku) DO UPDATE
  SET price = EXCLUDED.price,
      currency = EXCLUDED.currency,
      scraped_at = EXCLUDED.scraped_at;

With a unique key and ON CONFLICT DO UPDATE, running the same batch twice produces one row per SKU, not two. Combine the watermark and the upsert and your scraper becomes safe to retry, safe to overlap, and cheap to run, because it only ever touches new work.

Networks fail, targets return 429s, and pages occasionally load half-empty. A scheduled scraper has to expect transient failure and recover without a human. Wrap the fetch in retries with exponential backoff and jitter using tenacity:

import requests
from tenacity import (retry, stop_after_attempt,
                      wait_exponential_jitter, retry_if_exception_type)

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=2, max=60),
    retry=retry_if_exception_type(requests.RequestException),
    reraise=True,
)
def fetch(url: str) -> requests.Response:
    r = requests.get(url, timeout=60)
    r.raise_for_status()
    return r

wait_exponential_jitter backs off 2s, then roughly 4s, 8s, and so on, with random jitter added so a burst of failures does not retry in a synchronized wave. Cap the attempts so a genuinely dead target does not stall the whole run.

Retries handle the failures you can see. The dangerous failures are the ones you cannot: the job that stops running entirely. A scraper that never fires produces no error and no log line, so ordinary alerting never triggers. The fix is a dead man's switch, a heartbeat you ping on every success. If the ping stops arriving, the monitoring service alerts you.

import requests

def heartbeat(ok: bool) -> None:
    # Ping a monitor (Healthchecks.io, Cronitor, etc.) after each run.
    suffix = "" if ok else "/fail"
    try:
        requests.get(f"https://hc-ping.com/{PING_KEY}{suffix}", timeout=10)
    except requests.RequestException:
        pass  # never let the heartbeat crash the scraper

Beyond the heartbeat, track a handful of signals per run so you catch slow decay, not just hard failure:

The "rows toward zero" signal is the one people forget. A site redesign does not throw an error. Your selectors just stop matching, the scraper writes zero rows, and it reports success the whole time. Alert on output volume, not only on exit codes.

The thundering herd is what happens when many jobs wake at the same instant. Schedule ten scrapers at 0 and the target sees ten simultaneous bursts from your IP range at the top of every hour. That pattern is trivial to rate-limit and looks nothing like human traffic.

Jitter breaks the synchronization. You have already seen three ways to add it:

• Offset the minute. Use :17 or :23, never :00, and give each job a different offset.
• Let the scheduler splay it. systemd's RandomizedDelaySec and Airflow's max_active_tis_per_dag both spread real start times.
• Splay inside the job. Add a short random sleep before the first request so even jobs on the same cron minute do not fire together.

import random, time

# At the top of the scraper, before the first request.
time.sleep(random.uniform(0, 90))   # spread the real start across a 90-second window

Jitter matters between requests too, not just at start-up. Fixed delays are as detectable as no delay, because human clicks are never metronomic. For steady-state pacing across a large run, our guide on how to scrape high-volume data without rate limiting goes deep on request budgets and concurrency, which is the other half of not looking like a herd.

An unattended job cannot type a password, so its credentials sit somewhere on disk or in the environment. Get this wrong and you leak an API key into a public repo or a world-readable crontab. A few rules keep scheduled jobs safe:

• Never hardcode keys in the script, the crontab line, or a committed file. A crontab is readable, and git remembers everything.
• Inject secrets through the environment. Use systemd's EnvironmentFile, GitHub Actions secrets, or a .env loaded with python-dotenv and listed in .gitignore.
• Reach for a secret manager at scale. Vault, AWS Secrets Manager, Doppler, or your platform's equivalent handle rotation and audit for you.
• Give each job its own credential so you can revoke one scraper without breaking the rest, and rotate on a schedule.

import os

API_KEY = os.environ["SPARKPROXY_API_KEY"]   # fail loudly if the secret is missing

Proxy configuration follows the same logic. A scheduled job that manages its own rotating proxy list has to persist and update that list across runs, which is state you do not want on an ephemeral runner. The cleaner pattern is to push proxy selection, rotation, and geo-targeting behind a single fetch layer, so the job carries one secret and stays stateless. If your scrapers manage raw proxies directly, our notes on how to avoid getting your proxy blocked cover the hygiene that keeps an automated job off blocklists.

The SparkProxy Scraping API is a good fit for scheduled work precisely because it removes state from the job. It handles proxy rotation, JavaScript rendering, and geo-targeting on the server side, so your cron line, timer, or DAG task holds one secret (the API key) and never touches a proxy list. Every fetch helper in this guide, fetch_via_sparkproxy, is this function:

import os
import requests
from tenacity import (retry, stop_after_attempt,
                      wait_exponential_jitter, retry_if_exception_type)

API = "https://scrape.sparkproxy.io/api/v1"
KEY = os.environ["SPARKPROXY_API_KEY"]

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential_jitter(initial=2, max=60),
    retry=retry_if_exception_type(requests.RequestException),
    reraise=True,
)
def fetch_via_sparkproxy(url: str, country: str = "us") -> dict:
    resp = requests.get(
        API,
        headers={"X-API-Key": KEY},
        params={
            "url": url,
            "render_js": "true",        # server-side headless Chromium
            "country_code": country,     # geo-target without your own regional proxies
            "json_response": "true",     # wrap the result so we can read status + credits
        },
        timeout=90,
    )
    resp.raise_for_status()
    data = resp.json()
    if data["status_code"] >= 400:
        # A 429 or 403 from the target is retryable; raise so tenacity backs off.
        raise requests.HTTPError(f"target returned {data['status_code']}")
    return data

With json_response=true, the response envelope carries credits_used, duration_ms, and status_code, which feed directly into the monitoring table above: log credits_used per run and you have your cost-and-retry-storm signal for free.

Long renders are the other scheduling headache. If a page takes 30 seconds to load, a synchronous request ties up the scheduler slot for the whole time. The API's webhook mode fixes that. Pass a callback_url and the request returns 202 immediately, then POSTs the finished result to your endpoint later. That decouples slow fetches from a tight cron window, which is exactly what you want when a batch runs every few minutes.

Whether to hand proxy management to an API at all is a real decision with tradeoffs, and our comparison of a web scraping API versus self-managed proxies lays out when each one wins. For scheduled, unattended jobs, the stateless fetch layer usually pays off, because the fewer moving parts a 3 a.m. job has, the fewer ways it fails while you sleep.