In this lab you'll design Tiffin's checkout across services, end to end. The order, payment, promotions and restaurant services each own their own data, and the card is charged by an outside payment gateway. Your job is to make sure that no matter which call fails, or when, every order ends in a clear, correct state.
Decide each step yourself before opening the answer.
The brief
Checkout today is a chain of HTTP calls from the order service. Last month's incident review listed:
- 212 customers charged for orders that were never sent to a kitchen.
- 61 customers who used a "first order" coupon twice by double-tapping.
- Card holds that stayed on customers' cards for a week after the restaurant rejected the order.
- A 40-minute outage where the promotions service was down and nobody could check out, including customers with no coupon.
The services:
| Service | Owns | Notes |
|---|---|---|
| Order | Orders and their status | Shows the customer their order |
| Promotions | Coupons and reservations | Optional: most orders have no coupon |
| Payment | Payment records | Talks to the gateway, which supports authorise, capture, void, refund, and idempotency keys |
| Restaurant | Kitchen tablets, menus, specials | Kitchen has up to 3 minutes to accept |
The numbers:
- 4 million orders a day, 60% of them between 7pm and 10pm. At the dinner peak, that's 2.4 million / 10,800 s, about 220 checkouts a second.
- The gateway's authorise call takes 300 ms to 2 s, and times out about 1 in 5,000 calls.
- About 3% of orders are rejected by the kitchen, mostly for items running out.
Requirements: a customer is never charged for an order that isn't cooked; holds are released within minutes of a rejection; a coupon is used at most once; and an outage in an optional service doesn't stop checkout.