Scheduled jobs

So far every job has been started by something a user did: placing an order, uploading a photo, rejecting an order. Some work is started by the clock instead. Every Monday, restaurants get paid. Every night, carts nobody touched for a week are cleared out. Every morning, someone checks last night's backup actually restores (Module 5 made the case for that).

On one server, this is a one-line cron entry. On eight servers, the same line runs eight times, and if that line pays restaurants, you've paid them eight times. This lesson is about running clock-driven jobs exactly where and when you mean to.

Cron in one line

Cron is the classic Unix scheduler. Each line is a schedule and a command:

# minute  hour  day-of-month  month  day-of-week   command
  0       2     *             *      *             node jobs/clean-carts.js
  30      20    *             *      0             node jobs/weekly-payouts.js
  */5     *     *             *      *             node jobs/cancel-stale-orders.js

Most schedulers, from Kubernetes to cloud schedulers to job libraries, use this five-field format or a close variant (some add a seconds or year field), so it's worth being able to read.

The problem with server two

Back in Module 4, Tiffin went from one server to several identical ones behind a load balancer, and it now runs eight at dinner. Identical means the same code, the same config and, if you put cron in the server image, the same crontab.

Some jobs survive this. Clearing old carts eight times is wasteful but harmless, because deleting is idempotent. Payouts, statements and "your weekly summary" emails are not. You need each scheduled run to happen once.

Option 1: one special server

The obvious fix is to run cron on only one machine. Plenty of companies do this, and for a while it's fine. The problems show up later:

• It's a single point of failure. If that server is down at 20:30 on Sunday, payouts don't run, and nothing tells you.
• It's a snowflake. One server that's different from the rest gets forgotten during deploys, migrations and scaling changes. One day someone replaces it with a fresh server and the crontab doesn't come along.

It's a reasonable start. Just know the failure it brings is silent: work that quietly doesn't happen.

Option 2: everyone tries, one wins

A sturdier approach is to let every server run the schedule, and have them race for a lock. Only the winner does the work.

A Postgres advisory lock

Postgres has advisory locks: named locks that mean nothing to the database itself, only to your code.

SELECT pg_try_advisory_lock(7001);  -- 7001 = "weekly payouts"
-- true for exactly one session, false for everyone else

The server that gets true runs the job, then calls pg_advisory_unlock(7001). If it crashes, its database connection closes and the lock is released automatically.

One catch, since Module 5 added a connection pooler: a session-level lock belongs to one database connection. With a pooler in transaction mode, your next statement may run on a different connection, and the lock is held by a connection you no longer control. Either use a direct connection for the lock, or use pg_try_advisory_xact_lock, which holds the lock only until the current transaction ends (so the job has to run inside that one transaction).

A lock in Redis

With Redis (from Module 3), the same idea is one command:

SET lock:weekly-payouts server-3 NX PX 600000

NX means "only set it if it doesn't exist", so only one server succeeds. PX 600000 makes the lock expire after 10 minutes, so a crashed server doesn't hold it forever.

That expiry is also the weak spot. If the job takes 12 minutes, the lock expires at minute 10 and another server can start a second run while the first is still going. A long garbage-collection pause can do the same thing to a short job. Locks that expire, and the fencing tokens that make them safe, are covered properly in Module 12. For now, the lesson is: a lock reduces duplicates, it doesn't guarantee there are none.

Better: claim the run, not the lock

Here's a sturdier version that uses a tool you already trust, a unique constraint:

CREATE TABLE job_runs (
  job_name      text        NOT NULL,
  scheduled_for timestamptz NOT NULL,
  started_at    timestamptz NOT NULL DEFAULT now(),
  finished_at   timestamptz,
  PRIMARY KEY (job_name, scheduled_for)
);

INSERT INTO job_runs (job_name, scheduled_for)
VALUES ('weekly-payouts', '2026-10-18 20:30:00+00')
ON CONFLICT DO NOTHING
RETURNING job_name;

All eight servers try the insert. One gets a row back and runs the job. The others get nothing and stop. Unlike a lock, the claim never expires, so a slow run can't be duplicated by a timer. And you get a history for free: you can see which runs started, which finished, and which never did.

Option 3: the scheduler only enqueues

The cleanest setup splits "when" from "how":

The scheduler's only job is to drop "run weekly payouts for the week ending 18 Oct" into the queue at the right time. Everything from the earlier lessons then applies: retries with backoff, a dead-letter queue, alerting on stuck jobs.

The scheduler can be a managed service: a Kubernetes CronJob (set concurrencyPolicy: Forbid to stop overlapping runs), Amazon EventBridge Scheduler or Google Cloud Scheduler. Even these don't promise exactly once. The Kubernetes documentation, for example, warns that a CronJob can occasionally start a job twice or not at all, and says jobs should be idempotent. So pair the scheduler with the job_runs claim, or make the job itself idempotent, or both.

Missed runs and overlapping runs

Two timing problems trip up almost everyone.

Missed runs. The database was being upgraded from 20:25 to 20:45 on Sunday. The 20:30 payout never started. Cron won't run it later. It just waits for next Sunday.

The fix is to write jobs that catch up instead of jobs that assume they ran last time. A payout job shouldn't mean "pay for this week". It should mean "pay for every finished week that hasn't been paid yet". If one run is missed, the next one (or a manual run) pays both weeks. A cheap safety net: every hour, check job_runs for scheduled runs that never started, or started and never finished, and alert. The claim never expires, so a run whose worker died stays claimed until a human (or the next catch-up run) deals with it.

Overlapping runs. A job scheduled every 5 minutes usually takes 1 minute. At dinner time it takes 7. Now a second copy starts while the first is still running, and both work on the same rows.

Either prevent the overlap (a lock, or concurrencyPolicy: Forbid), or make each run handle a clear slice that can't clash, like "orders placed between 20:00 and 20:05", claimed with SKIP LOCKED so two runs never grab the same row.

Time zones and daylight saving

"Pay restaurants at 2am on Monday" hides a question: 2am where?

• Run schedulers in UTC. Servers in different regions, or a server whose time zone setting changes, will otherwise disagree about when 02:00 is. Tiffin's 02:00 Monday in India (UTC+5:30) is 20:30 UTC on Sunday. That's why the crontab above says 30 20 * * 0.
• Watch out for daylight saving. India doesn't use it, but when Tiffin expands to a city that does, a job at 02:30 local time may be skipped on the night clocks jump forward and run twice on the night they fall back, depending on the scheduler. Scheduling in UTC avoids this.
• Define business periods in local time, store them in UTC. "Week ending Sunday midnight in Mumbai" is a business rule. Convert it once, store the UTC boundaries, and use scheduled_for, not now(), inside the job. A job that runs 20 minutes late should still pay for the same week.

Delayed one-off jobs

Not every clock-driven job repeats. "Cancel this order if the restaurant hasn't accepted it in 10 minutes" is a single job with a delay. You could run a cron every minute that finds stale orders, and that works well. Or you could enqueue a job with run_at = now() + 10 minutes when the order is placed. The Postgres queue from earlier already supports that, since workers only claim jobs whose run_at has passed. The job checks the order's status when it runs and does nothing if the restaurant already accepted, which also makes it idempotent.