Module 12 / Distributed system pitfalls

Module 12 · lesson 1 of 8 · Read 8 min

The network is unreliable

Lost, slow and duplicated messages, and why you cannot tell them apart.

What you'll learn

  1. A function call versus a network call
  2. Everything that can go wrong
  3. You cannot tell them apart
  4. Doesn't TCP fix this?
  5. Slow is the common case, not lost
  6. Some numbers to keep in your head
  7. The toolkit for living with it

In Module 11 we split Tiffin into services: orders, payments, restaurants, riders. On a whiteboard that looks like a tidy refactor. In production it changes something deep. A call that used to be a function call inside one process is now a message sent over a network to another machine.

A function call either returns or throws. A network call can do something much worse: nothing. No answer, no error, just silence. This module is about the problems that silence creates, starting with the most basic one.

A function call versus a network call

Inside the old Tiffin monolith, placing an order did this:

const payment = chargeCard(order.customerId, order.total) // same process
saveOrder(order, payment.id)

If chargeCard returned, the card was charged. If it threw, it wasn't. There were only two outcomes, and you always knew which one you got.

Now chargeCard lives in the payments service, on another machine:

const payment = await http.post('http://payments/charges', { customerId, amount })

That one line now crosses network cards, cables, switches, maybe a load balancer, and two operating systems' network stacks. Any of them can drop, delay or duplicate the message. And the payments service itself can crash, restart or pause halfway through.