Cloud

Field note · 8 min

Dropbox leaves S3 for its own storage

Why an exabyte-scale customer built Magic Pocket, and what it took to move the data.

In 2016 Dropbox announced something most companies never do: it had moved the bulk of its users' files out of Amazon S3 and into storage it built and ran itself. By the time of the announcement it held about 500 petabytes of user data for over 500 million signups, and over 90% of it was now stored and served from Dropbox's own machines.

This note follows the story as Dropbox's engineers and its 2018 IPO filing told it. It's the clearest public example of a company outgrowing object storage from a cloud provider, and it shows exactly which conditions made building your own worth it.

What they had

Before the move, Dropbox ran a split setup. In Akhil Gupta's account, file content lived on Amazon S3, while metadata and the web servers ran in Dropbox's own data centres.

That's a sensible starting shape for a file-sync company, and it's close to what the Dropbox lab designs: a metadata service that knows which blocks make up which file, and an object store that holds the blocks. The object store is the part that grows with every byte users save. When Gupta joined in 2012 it held about 40 petabytes; four years later it held over twelve times that.