Module 18 / High availability and multi-region

Module 18 · lesson 3 of 7 · Read 7 min

Degrading gracefully

Recommendations down, ordering still works.

At 8:10pm on a Saturday, Tiffin's recommendations service starts to struggle. It's not down. It's just slow: 4 seconds per response instead of 80 milliseconds. The home screen calls it before showing anything, so the home screen takes 4 seconds. App servers fill up with requests waiting on recommendations. Soon there are no free threads left for checkout, and customers who already have a full cart can't pay.

A feature nobody would miss for an hour took down the one that pays the bills. This lesson is about making sure that can't happen: when something breaks, the app should get a little worse, not stop.

Graceful degradation means designing a system so that when a part fails, the rest keeps working with reduced features, instead of failing completely.

Sort features by what really matters

Before you can protect the important features, you have to say which ones they are. Write it down, with the product team, before the incident. During an outage is too late to have that argument.

Tier Tiffin features If it fails
Critical Place order, pay, kitchen receives order, rider assignment Tiffin is down. Every minute costs money and trust
Important Search, menus, live tracking, order history Painful, but people can still order with effort
Nice to have Recommendations, reviews, offers banner, loyalty points, "people near you also ordered" Most customers won't even notice

Now there's a clear rule for every design decision: a nice-to-have feature must never be able to break a critical one. Each of the following techniques is a way to enforce that rule.