Module 19 / Algorithms behind big systems

Module 19 · lesson 20 of 22 · Read 10 min

CRDTs

Data types that merge without conflicts.

Tiffin now runs active-active in three regions (Module 18): Mumbai, Delhi and Bengaluru. Each region accepts writes locally, so customers get fast responses, and regions sync with each other in the background. That design has a recurring headache, which you met in Module 13: two regions change the same thing at the same time, and when they sync, something has to give.

Last-write-wins throws one change away. Vector clocks (earlier in this module) detect the conflict but hand it to your code to resolve. CRDTs take a third route: use data types whose merge rule is designed so that conflicts simply can't happen. Every replica can apply updates in any order, receive them twice, and still end up in exactly the same state.

The idea: merges that can't disagree

CRDT stands for conflict-free replicated data type. The best-known family (state-based CRDTs) works like this: each replica keeps a local state and changes it freely. Replicas periodically send each other their whole state, and the receiver merges it in. The merge function must be:

Property Meaning Why it matters
Commutative merge(A, B) = merge(B, A) Order of syncs doesn't matter
Associative merge(merge(A, B), C) = merge(A, merge(B, C)) Grouping of syncs doesn't matter
Idempotent merge(A, A) = A Receiving the same state twice does no harm

With those three properties, the network can delay, reorder and duplicate messages as much as it likes. Once every replica has received every update, by any path, they're all identical. That guarantee is called strong eventual consistency.

A CRDT is a data type whose replicas can be updated independently and merged in any order, any number of times, always reaching the same result without coordination.

The hard part is designing a data type with a merge like that which still means something useful. Let's look at the common ones.