A metrics system looks boring from the outside. Servers send numbers, a dashboard draws lines, and a pager goes off when a line crosses a threshold. Inside, it's one of the hardest write-heavy systems you can build. Millions of tiny numbers arrive every second, almost none of them are ever read individually, and the one moment everybody needs the system is the moment everything else is on fire.
You'll design it from a blank page. Try to answer each question before you read on.
The brief
You're building a hosted monitoring product, in the spirit of Datadog. Customers install an agent on their servers and containers. The agent reports numbers like cpu.user, http.requests and queue.depth every 10 seconds, each tagged with things like host, service and region. Customers build dashboards and set alerts like "page me if p99 latency on checkout is above 800 ms for 5 minutes".
Who uses it?
- Agents on customer machines. They write constantly and never read.
- Engineers looking at dashboards. Mostly idle, then very busy during an incident.
- The alert engine, which is your own internal reader, running all day.
In scope: ingesting metrics, storing them, querying them for dashboards, and evaluating alerts.
Out of scope: logs and traces (related products with different storage), the notification delivery itself (paging, email, Slack; see the notifications lab), billing, and the agent's internals.
Requirements
Functional
- Accept data points:
metric name, tags, timestamp, value. - Query a metric over a time range, filtered and grouped by tags, with functions like
sum,avg,maxand percentiles. - Keep 10-second data for 15 days, and coarser data for 13 months.
- Define alert rules on a query and a threshold, and fire when the rule holds for a set duration.
Non-functional
- Ingest never falls over. Losing data during an incident is the worst possible failure, because that's when people look.
- Fresh data: a point should be queryable within about 10 seconds of arriving.
- Alerts fire fast: within about 60 seconds of the condition becoming true.
- Dashboards are quick: a 1-hour graph in under a second, a 30-day graph in a few seconds.
- Alerting is more available than dashboards. If something must degrade, the graphs go slow before the pager goes quiet.
- Small loss is tolerable: if one point in a million is dropped, nobody notices. Losing a whole minute for a customer is noticed.