Module 23 / Security and operations

Module 23 · lesson 8 of 9 · Read 9 min

Incidents and postmortems

What to do when it breaks anyway.

You can do everything in this module right and things will still break. A cloud zone has a bad night. A config change looks harmless. A payment provider slows down on the busiest Friday of the year. The difference between a good team and a bad one isn't whether they have incidents. It's how fast they notice, how calmly they fix it, and whether the same thing ever happens again.

This lesson follows one Tiffin incident from the first page to the last action item.

Detectan alert fires
→
Responddeclare, assign roles
→
Mitigatestop the harm
→
Learnpostmortem and fixes
The life of an incident. The highlighted step is the one that matters most while customers are hurting.

What counts as an incident

An incident is any unplanned event that hurts users or puts them at risk, and needs a coordinated response. Agree on severity levels before you need them, so nobody argues about it at 8pm on a Friday.

Severity Meaning at Tiffin Response
SEV1 Customers can't order or pay in one or more cities, or data is lost or exposed Page on-call now, incident commander, status page update within 15 minutes
SEV2 A major feature is broken or badly slow for many users (tracking down, orders delayed) Page on-call, incident channel, status page if customers notice
SEV3 Minor or partial impact, a workaround exists Handle in working hours

When in doubt, declare the higher severity. Downgrading later costs nothing. Under-reacting for the first 30 minutes costs a lot.