Dashboards and alerts

The first lesson of this module named the four vitals: traffic, latency, errors and saturation. The deploys lesson leaned on them too: a canary is only as good as the numbers you compare. But "watch the vitals" isn't something you can do at 8:15pm on a Friday by staring at a server. You need the numbers collected, drawn, and turned into an alarm that wakes someone up when they go wrong, and stays quiet when they don't.

This lesson builds that for Tiffin, concretely: which metrics, which graphs, which alerts, and the exact queries, on both AWS CloudWatch and Prometheus.

Where the numbers come from

Tiffin runs on AWS: app servers as ECS tasks on Fargate, behind an Application Load Balancer (ALB), with RDS for PostgreSQL. A lot of the vitals come for free, because each managed service publishes its own metrics to CloudWatch every minute:

The ALB is a good place to measure because it sees every request from the outside, including the ones your app never answered. TargetResponseTime is the time from the ALB sending a request to the app starting its response, so it's your app's latency as a customer feels it, minus the network.

What the free metrics can't tell you is anything about the business. Nothing in AWS knows that a request was an order. So Tiffin publishes one custom metric of its own: OrdersPlaced, counted by the app. You'll see shortly why that one matters most.

Many teams use Prometheus instead, or as well: the app exposes counters and histograms on a /metrics page, Prometheus scrapes it every 15 seconds or so, and Grafana draws the graphs. The ideas below are the same either way, so each one is shown in both.

One dashboard, four rows

Resist the urge to graph everything. A dashboard is for answering one question fast: is Tiffin OK right now, and if not, where is it hurting? Tiffin's main dashboard has four rows, one per vital, plus the business row on top:

Row 0  Orders a minute, vs same time last week
Row 1  Requests a second, total and per endpoint
Row 2  Latency p50 and p99, menu and checkout
Row 3  Error rate: 5xx as % of requests
Row 4  Saturation: app CPU, RDS CPU,
       RDS connections, healthy tasks

In CloudWatch, a few of these need metric math, which combines metrics with an expression. The error rate is the important one:

m1 = HTTPCode_Target_5XX_Count   Sum, 1 minute
m2 = HTTPCode_ELB_5XX_Count      Sum, 1 minute
m3 = RequestCount                Sum, 1 minute
e1 = 100 * (FILL(m1, 0) + FILL(m2, 0)) / m3

Why the FILL? The ALB only publishes a 5xx count when it's above zero. In a good minute there's no data point at all, and without FILL the expression has nothing to compute. Why both 5xx metrics? Target counts errors your app returned. ELB counts errors the load balancer returned on its own, like a 502 when a task crashed mid-request or a 503 when no task was healthy. In a real outage, it's often only the second one that moves.

In Prometheus, the app records http_requests_total with a status label and a latency histogram, and the same rows are:

# traffic
sum(rate(http_requests_total[5m]))

# error rate
  sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))

# p99 latency from histogram buckets
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Notice that latency comes from a histogram: the app counts requests into buckets (under 50 ms, under 100 ms, under 250 ms and so on), and the percentile is estimated from the buckets. You can't average per-server p99s into a fleet p99; percentiles don't add up that way. Buckets do.

Alerts: what deserves a 3am phone call

A dashboard waits for you to look. An alert comes and finds you. Get alerts wrong in one direction and outages run for an hour before anyone notices. Get them wrong in the other and the team gets paged five times a night for nothing, learns to ignore the pager, and then ignores the real one.

Tiffin sorts every alert into two kinds:

• Page: customers are hurting now, and a human must act within minutes. Phones ring, any hour.
• Ticket: something needs attention this week. It goes to a queue, never to a phone.

And a rule for which kind: page on symptoms customers feel, not on causes. "CPU is at 85%" is a cause. Maybe it's fine; the load test showed the server still answers quickly at 74%. "2% of checkouts are failing" is a symptom. Somebody needs to look now, whatever the cause.

Here are Tiffin's alerts, with the numbers and where each number comes from:

Three details make these work.

"For N minutes in a row." One bad minute is noise: a slow deploy, a retry storm that clears itself. CloudWatch calls this datapoints to alarm: "5 out of 5" one-minute periods. In Prometheus it's the for: clause. Without it, the error alert would fire every time one task restarts.

A minimum traffic guard. At 4am Tiffin gets a few requests a minute. One failure out of three is a 33% error rate, and a page for nothing. So the error alert only counts when there are at least 120 requests a minute. CloudWatch's metric math has an IF for this; Prometheus uses and.

The business alert. One night, a payment provider changed an API field. Checkout returned 200, showed "Order placed!", and silently didn't place the order. Errors: 0%. Latency: perfect. Orders: down 90%. Only the orders alert could catch it. Comparing with the same time last week, rather than a fixed number, handles the daily and weekly rhythm: 50 orders a minute is a disaster at 8pm on a Friday and a busy minute at 4pm on a Tuesday. In CloudWatch, an anomaly detection alarm learns that rhythm from two weeks of history. In Prometheus, offset 1w compares directly.

Here's the Prometheus version of the three paging alerts that come from the app's metrics (checked with promtool check rules):

groups:
  - name: tiffin-api
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.02
          and sum(rate(http_requests_total[5m])) > 2
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "More than 2% of API requests are failing"
          runbook: "https://wiki.tiffin.internal/runbooks/api-errors"

      - alert: SlowRequests
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 10m
        labels:
          severity: page

      - alert: OrdersDropped
        expr: |
          sum(rate(orders_placed_total[10m]))
            < 0.5 * sum(rate(orders_placed_total[10m] offset 1w))
          and hour() >= 5 and hour() < 18
        for: 10m
        labels:
          severity: page

(rate(...) is per second, so > 2 is 120 requests a minute. hour() is in UTC, so 5 to 18 is 10:30am to 11:30pm in India.) Every paging alert links to a runbook: a short page saying what the alert means, what to check first, and how to roll back. The person woken at 3am shouldn't have to think from scratch.

What it costs

On CloudWatch, the free metrics are free, and alarms are cheap. Mumbai prices, late 2026:

Standard alarms     7 alarm metrics x $0.10  $0.70
Anomaly detection   3 alarm metrics x $0.10  $0.30
Custom metric       OrdersPlaced             $0.30
Total, a month                              ~$1.30

The error-rate alarm reads three metrics, so it counts as three alarm metrics.

The trap is custom metrics. Each unique combination of a metric's name and its labels (CloudWatch calls them dimensions) is billed as a separate metric. Someone adds restaurant_id to OrdersPlaced to see orders per restaurant. Tiffin has about 3,000 restaurants, so that's 3,000 metrics: $900 a month, for a graph nobody can read anyway. Add an endpoint label with 20 values to a latency metric per restaurant and you're at 60,000 metrics. Prometheus doesn't bill per metric, but each combination is a separate time series in memory, and the same mistake makes it slow or kills it. Per-customer and per-restaurant questions belong in logs or the analytics database, not in metrics.