Module 19 / Algorithms behind big systems

Module 19 · lesson 6 of 22 · Read 8 min

Count-min sketch

Approximate counts for millions of keys.

A Bloom filter answers "have I seen this?". The next question is usually "how many times?". How many requests has this IP sent in the last minute? How many times was each dish ordered today? How often was each search term typed?

With a few thousand keys, a hash map answers that exactly. With tens of millions of keys, most of which appear once or twice, the hash map becomes the biggest thing in memory. A count-min sketch gives you approximate counts for any number of keys in a fixed, small table, with an error you choose in advance.

The problem: counting a long tail

Tiffin's gateway sees about 40 million distinct client IPs a day. It wants a per-IP request count to spot abusive clients (Module 8's rate limiting, but for detection rather than enforcement). An exact map from IP to count, with per-entry overhead, costs a few gigabytes per gateway node, and it grows without bound during a bot attack that uses millions of spoofed addresses. That's the worst possible time to run out of memory.

What the team actually needs is: "for any IP, roughly how many requests?", with the guarantee that a heavy IP is never under-reported.