It's Friday, 8:20pm. A restaurant owner calls support: "orders stopped coming in ten minutes ago." Support checks Twitter and finds forty angry customers. That's how you find out about an outage when you have no observability. Your customers are your monitoring system, and they're slow, noisy and very unhappy about the job.
The goal of this lesson is the opposite: Tiffin's on-call engineer gets paged at 8:12pm, before most customers notice, and can see what broke within a few minutes. You get there with three kinds of data and a careful approach to alerts.
Three signals, three jobs
| Signal | What it is | Answers | Cost driver |
|---|---|---|---|
| Metrics | Numbers over time: counters, gauges, histograms | Is something wrong? How much? Since when? | Number of distinct series |
| Logs | One record per event, with details | What exactly happened to this request? | Volume of bytes |
| Traces | The path of one request across services, with timings | Where did the time or the error come from? | Number of traces kept |
You need all three, but they work as a funnel. A metric tells you checkout errors jumped at 8:11. A trace of a failing checkout shows the payment call timing out. The payment service's logs for that trace ID show "connection pool exhausted". Each step narrows the search.