Labs

System lab · hard · 27 min

S3: Object storage

Huge numbers of objects, stored durably and served from anywhere.

Object storage is the system every other system leans on. Video platforms put their files there, databases put their backups there, and analytics teams put whole data lakes there. The promise is simple: give it bytes and a name, and you get the same bytes back years later. "Eleven nines of durability" means that if you store 10 million objects, you'd expect to lose one about every 10,000 years.

Keeping that promise across hundreds of thousands of disks that fail every day is the design problem. You'll build it from a blank page. Try each step yourself before reading on.

The brief

You're building an object store in the style of Amazon S3, for one region. Customers create buckets and put objects in them. An object is a key like photos/2026/10/beach.jpg, some bytes (from 0 bytes to 5 TB in this design, the limit S3 had for years; S3 itself now allows 48.8 TiB, which AWS announces as 50 TB), and a little metadata (content type, size, checksum, custom headers).

Who uses it?

  • Applications writing and reading files through an HTTP API, often through presigned URLs.
  • Batch jobs reading huge datasets in parallel.
  • Backup tools writing large objects once and almost never reading them.

In scope: create and delete buckets, put, get, delete and list objects, multipart upload for large objects, and durability.

Out of scope: cross-region replication, lifecycle rules and storage tiers, versioning, access policies beyond "this key may access this bucket", and billing. Module 21 covers how S3 looks from the user's side.

Requirements

Functional

  • PUT an object under a bucket and key. A second PUT to the same key replaces it.
  • GET an object, including byte ranges (Range: bytes=0-1048575).
  • DELETE an object.
  • LIST keys in a bucket by prefix, in sorted order, paginated.
  • Upload large objects in parts, in parallel, and resume after a failure.

Non-functional

  • Durability: 99.999999999% a year per object. Losing data is the one unforgivable failure.
  • Availability: 99.99% for reads and writes. Briefly unavailable is bad, lost is far worse.
  • Strong read-after-write consistency. After a successful PUT, every GET and LIST sees the new object.
  • Throughput scales out: thousands of clients reading one dataset in parallel should get aggregate bandwidth in the hundreds of Gbps.
  • Low cost per byte. Customers store petabytes. Every extra copy is real money.