Module 19 / Algorithms behind big systems

Module 19 · lesson 5 of 22 · Read 8 min

Bloom filters

"Definitely not" or "maybe", in very little memory.

Some questions are expensive to answer but usually have the answer "no". Is this coupon code one we ever issued? Does this SSTable on disk contain key order:99812? Has this user already seen this notification? Asking the database each time costs a network hop or a disk read, for a "no" you could have predicted.

A Bloom filter answers "is X in the set?" from a small bit array in memory. It never says "no" when the answer is yes. It occasionally says "maybe" when the answer is no. In exchange, it uses about 10 bits per item (for a 1% false positive rate), no matter how long the items are.

The problem: lookups for things that don't exist

In Module 3 you met cache penetration: requests for keys that don't exist sail past the cache (there's nothing to cache) and hit the database every time. A bot hammering Tiffin with random promo codes like FREEFOOD7731 does exactly that. Every guess is a miss in Redis and a query to Postgres.

If Tiffin keeps a Bloom filter of every promo code ever issued, the API can reject most fake codes without touching the database. Only "maybe" answers go to Postgres, which then gives the exact answer.