Partitions don't stay where you put them. Data grows unevenly, traffic shifts, machines die, and you add new machines to keep up. Every one of those means moving partitions, and you have to do it while customers are placing orders.
Moving data between machines is called rebalancing. It's where many partitioned systems actually hurt: the scheme looked fine on day one, and then the first rebalance took eleven hours and slowed everything down.
Rebalancing is moving partitions between machines so that data and load stay evenly spread as machines join, leave or fill up unevenly.
Why you rebalance
- You add machines because load or data grew. They start empty and need a share.
- A machine fails or is retired. Its partitions need a new home.
- Partitions become uneven. One range grew faster, or one partition has the busy keys.
Whatever the reason, you want three things from a rebalance:
- Afterwards, load and data are spread fairly again.
- During it, reads and writes keep working.
- It moves only as much data as it has to, because moving data costs network, disk and time.