For its first years, Slack sharded MySQL the way many startups do: by customer. Each workspace's data lived on one shard, and the application knew which. In 2017 Slack began moving to Vitess, a sharding layer for MySQL, and by December 2020 Vitess was serving 99% of Slack's query load, peaking at 2.3 million queries a second.
This note follows Slack's own account. It's worth studying because the original design wasn't wrong; it was a good design for the product Slack used to be. The product changed, and the sharding key stopped fitting.
What they had
The 2020 post by Arka Ganguli, Guido Iaquinti, Maggie Zhou and Rafael Chacón describes the starting point:
- Data was split across three kinds of cluster: shards, partitioned by workspace ID; a metadata cluster; and a "kitchen sink" cluster for everything else.
- The application did the routing: given a workspace, it knew which shard to query.
- Each shard ran on at least two MySQL instances in different data centres with asynchronous replication, in an active-active setup.
Sharding by workspace is natural for a team-chat product: almost every query is "for this workspace", so it hits exactly one shard. It's the course's sharding lesson choosing a shard key from the dominant access pattern.