Hashing promises an even spread, and it delivers one for keys. Real traffic isn't spread evenly across keys. A few keys are always far busier than the rest: the famous account everyone follows, the product on sale, the restaurant with a queue out the door.
When one key gets a big share of the traffic, its partition gets that share too, and no amount of hashing or adding machines changes that. This lesson is about spotting those keys and the handful of techniques that deal with them.
A Friday night at Tiffin
Say Tiffin partitions each region's orders across 16 database partitions by hashing restaurant_id. It looks sensible: the kitchen tablet asks "what are my open orders?" all evening, and with this key that's a one-partition query.
At 7pm, Biryani Bros, a chain with one central kitchen per city, launches "50% off, tonight only" with a push notification to 5 million customers. In Hyderabad, that one kitchen ID suddenly gets 25% of the region's orders, and the tablets, status checks and rider assignments that go with them.
Every one of those requests hashes to the same partition. To keep the arithmetic simple, say each partition comfortably takes 1,000 writes a second, so the Hyderabad cluster can take 16,000. Tonight it's taking 4,000 writes a second (orders plus every status change, tablet update and assignment that goes with them). If a quarter of those go to one restaurant, that's 1,000 for Biryani Bros plus the partition's normal share of the rest, around 190. That partition is over its limit while the other 15 are at a fifth of capacity.