−50%

50% off your first month on any paid plan · code WELCOME50 · until Nov 30

Start free →
Guide

Schedule data sync between APIs with cloud cron

Schedule data sync between APIs with cloud cron

Syncing data between APIs sounds simple until a silent failure leaves 15 months of records missing. This guide covers scheduling reliable sync scripts with Cloud Cron and catching partial failures before they pile up.

This is what a broken sync looks like in the API logs:

GET /api/orders?updated_after=2026-10-19T00:00:00Z
200 OK — 0 records

GET /api/orders?updated_after=2025-07-03T00:00:00Z
200 OK — 4,218 records

That gap between the two dates is the problem. The sync script ran every night for over a year, exited cleanly, logged "sync complete." But a rate limit change on the source API started returning 429s after the first 100 records. The script treated the partial response as a full sync. Fifteen months of missed data, sitting in one system but not the other. That one stung.

This guide covers how to sync data between two systems on a schedule, keep it reliable, and use Cloud Cron to trigger and monitor the whole thing so silent failures don't pile up for months.

Why scheduled data sync matters#

Most apps don't live in isolation. Orders come in through Shopify but reporting lives in your own database. Users sign up in your app but the sales team needs them in HubSpot. Inventory updates in a warehouse system but your storefront needs to reflect it.

The usual fix: a script that pulls data from one API and pushes it to another on a schedule. Every hour, every 15 minutes, once a night. Simple idea, but surprisingly easy to break without noticing.

The three common sync patterns:

  • One-way pull — read from a source API, write to your database. Your copy is read-only. Most common pattern.
  • One-way push — your system sends updates to an external API. New users going from your app to a CRM, for example.
  • Two-way sync — both systems create and update records. Conflict resolution makes this an order of magnitude harder. Avoid if you can.

There's also event-driven sync via webhooks, where the source pushes changes to you in real time. Faster, but not every API supports it. When it's available, use it for time-sensitive data and keep a scheduled poll as a safety net.

What makes a sync script reliable#

A sync that works once isn't the hard part. A sync that works every hour for six months without anyone touching it — that's the challenge. Two things keep it reliable:

Upsert, not insert. If the record exists (matched by source ID), update it. If not, create it. This way, rerunning the script after a crash doesn't duplicate data. PostgreSQL has INSERT ... ON CONFLICT UPDATE, MySQL has ON DUPLICATE KEY UPDATE, MongoDB has updateOne({ upsert: true }).

Cursor-based progress. Store the timestamp of the last record you processed. Next run picks up from there. If the script crashes mid-batch, the cursor hasn't moved, so the next run reprocesses the failed batch. No data lost, no data doubled.

Here's the basic flow in any language:

1. Read last sync timestamp from your database
2. Fetch records from source API where updated_after > timestamp
3. Upsert each record into target (by source ID)
4. Save the latest updated_at as your new cursor
5. Return count of synced records + exit

The exit code matters. If the script exits 0, whatever triggers it (crontab, Cloud Cron, CI) thinks it succeeded. If something fails partway through, exit 1 so the next layer knows to log the failure and alert you.

Rate limits and why syncs break in production#

This is where most sync scripts fail. They work fine in dev with 50 records. Then production has 10,000 records and the API allows 100 requests per minute.

Three things to handle:

The Retry-After header. When an API returns 429 (too many requests), it usually tells you exactly how long to wait. Don't guess — read it and wait.

Exponential backoff. First retry after 5 seconds, second after 10, third after 20. If you hammer a rate-limited API with instant retries, providers like Stripe will block your key entirely. We've seen it happen.

Pagination. Never fetch all records in one request. Use the API's cursor or page-based pagination and process one page at a time. Lower memory, fewer rate limit hits, easier to resume after failure.

How Cloud Cron triggers and monitors your sync#

If you have a server with crontab, you can schedule the sync there and add a heartbeat ping at the end. But many apps run on platforms without shell access: Vercel, Railway, managed hosting, serverless setups.

The approach that works everywhere: wrap your sync in an HTTP endpoint and let Cloud Cron call it on a schedule.

POST https://yourapp.com/api/sync/orders
Headers: X-Api-Key: your-secret-key
Schedule: */15 * * * * (every 15 minutes)

Cloud Cron sends the POST request on schedule. Your endpoint runs the sync and returns a response. That's the trigger side. But the real value is what happens next.

If the endpoint returns 200, the sync ran. If it returns 500 or times out, you get an alert through Slack, email, or Discord. You don't need to set up a server or check logs manually. Cloud Cron watches the response for you.

For the sync endpoint itself, protect it with an API key in the header and return structured JSON so you can track what happened:

{
  "status": "ok",
  "records_synced": 47,
  "records_skipped": 3,
  "duration_seconds": 12
}

Use a cron expression builder if the schedule syntax trips you up.

Monitoring sync health, not just "did it run"#

A sync that returns 200 every hour but processes zero records for three weeks is technically running. It's also useless. That's why the response body matters as much as the status code.

We track a few signals on our own syncs. Record count per run is the obvious one. If it drops to zero several times in a row, something changed: expired API key, broken filter, source system down. Duration matters too — a sync that goes from 10 seconds to 10 minutes is either hitting rate limits or processing way more data than expected. Worth looking into before it times out.

The less obvious signal is staleness. Compare the latest updated_at in your local copy against the source. If the gap grows beyond your sync interval, data is falling behind and nobody told you. Set up a dead man's switch on the job itself. Even successful syncs can have one record out of 200 fail validation quietly, so log partial failures separately.

Picking the right sync interval#

For operational data (orders, inventory, tickets), every 15 minutes is a good default. Fresh enough for dashboards, cheap enough on API quota to sustain long-term. Hourly is fine for analytics and CRM sync where nobody needs live data.

Going down to every minute rarely justifies the cost. Unless your users genuinely need data fresher than 5 minutes, you're paying for freshness nobody notices.

On top of the incremental sync, run a nightly full reconciliation at 2 AM UTC. It catches anything the incremental missed: a skipped page, a rate limit that silently truncated a batch, a record that updated twice between polls. We've caught gaps this way that would have gone unnoticed for weeks.

# Every 15 minutes (incremental)
*/15 * * * *

# Daily at 2 AM UTC (full reconciliation)
0 2 * * *

When sync breaks and how to recover#

APIs change without warning. I've had a CRM remove a field from their API response without a changelog entry. The sync kept running, kept reporting success. Found out three days later when a report came up empty. Frustrating, because the fix was a five-line schema check we should have had from the start.

When you catch a gap, reset the cursor to the start of the gap and rerun. If the gap is large (weeks or months), break it into smaller date ranges so you don't hit rate limits on the source API. The nightly full reconciliation catches most small gaps before they become big ones.

Schema validation on every response helps too. Check that expected fields exist before processing. Pin to versioned API endpoints (v2, not unversioned URLs). But the real safety net is the monitoring: if record count suddenly drops or duration spikes, something changed on the source side.

When you don't need a custom script#

If you're syncing between two well-known SaaS tools (Shopify to Google Sheets, HubSpot to Slack, Stripe to Airtable), check Zapier, Make, or n8n first. They handle the plumbing and you skip writing code entirely. We moved two of our own internal syncs to n8n last year and haven't touched them since.

Custom scripts make sense when one or both systems have a custom API, you need transformation logic that no-code tools can't express, or volume is high enough that per-task pricing on Zapier gets expensive.

For everything else, a no-code tool with a scheduled trigger is faster to set up and easier to maintain. Whatever you choose, the sync that silently breaks for fifteen months is the one nobody monitored. Don't be that sync.

Frequently asked#

Every 15 minutes works for most operational data (orders, users, tickets). Hourly for analytics and reporting. Run a daily full reconciliation on top of incremental syncs to catch anything missed. Syncing every minute is rarely worth the API cost.

Use upsert operations (INSERT ON CONFLICT UPDATE) with the source system's ID as the unique key. Combined with cursor-based progress tracking, this makes the sync safe to rerun after any failure without creating duplicates.

Read the Retry-After header and wait. Use exponential backoff on retries. Paginate requests instead of fetching everything at once. If rate limits are tight, reduce sync frequency or request a higher quota from the API provider.

Yes. Wrap the sync logic in an HTTP endpoint in your existing app. Use WatchCron Cloud Cron to POST to that endpoint on schedule. Cloud Cron handles timing and monitors the response. Your app handles the sync. No separate server needed.

Include record counts and duration in the sync response. If records_synced drops to zero for multiple runs, the source API likely changed or credentials expired. Cloud Cron monitors the response code and body, so a 500 or timeout triggers an alert automatically.
Written by Vitalii Holben

Founder of WatchCron. Full-stack developer building monitoring tools that help teams keep cron jobs, uptime, and SSL certificates under control. Based in Europe, shipping code since 2014.

Start monitoring in under 2 minutes.

Free plan includes 20 checks · No credit card required
Pricing Docs
Log in Start free →