Every company has a crontab somewhere that everybody is afraid of. It sends the invoices, rotates the keys, and refreshes the reports. It runs on one machine called something like jobs-01, and when that machine dies, nobody notices until a customer asks why they didn't get their invoice.
A distributed scheduler replaces that machine with a service: you tell it "run this at 9:00 every weekday" or "run this once at 14:32 tomorrow", and it does, on time, once, even when servers die. "Once" is the hard word in that sentence. You'll design it from a blank page. Try each step before reading on.
The brief
You're building a hosted job scheduler, used by many internal teams (or, as a product, by many customers). A job says when to run and what to run. "What" is usually "call this HTTP endpoint with this body" or "put this message on this queue". The scheduler doesn't run the business logic itself. It makes sure it gets triggered.
Who uses it?
- Engineers creating, pausing and editing jobs through an API.
- Target services, which receive the triggers and do the actual work.
- On-call engineers, who want to know why a job didn't run.
In scope: one-off and recurring (cron) jobs, firing them on time, retries, timeouts, run history, and not running a job twice.
Out of scope: workflows with dependencies between jobs (a DAG engine like Airflow is a bigger lab), running arbitrary containers (we trigger, we don't host), and billing.
Requirements
Functional
- Create a job with a cron expression and time zone, or a single run time.
- Each job has a target (HTTP endpoint or queue), a payload, a timeout and a retry policy.
- Pause, resume, edit and delete jobs.
- Show each job's run history: when it was due, when it started, how it ended.
- Let a job choose what happens if runs overlap or are missed.
Non-functional
- On time: a run starts within 5 seconds of its due time at p99. A job that accepts a jitter window (the default, see "The midnight problem" below) is on time if it starts inside its window.
- Once: each scheduled run is triggered once. If a duplicate can't be ruled out, the target can detect it.
- Never silently skipped: a run that can't be triggered is retried, then reported, never just lost.
- Highly available: losing any single server, or a whole zone, doesn't stop jobs.
- Scales: millions of jobs, and big spikes at popular times.