In the early 2000s, Google had a storage problem that no off-the-shelf file system was built for. The web crawler wrote enormous files, often many gigabytes each. Indexing jobs read them start to finish, again and again. Hundreds of machines appended results to the same file at the same time. And all of this ran on thousands of cheap, ordinary machines, so on any given day some disks died, some machines hung and some power supplies failed. Failure wasn't an exception to plan for. It was the normal state of the cluster.
Existing distributed file systems, like NFS, assumed a different world: small files, edits in the middle of files, and a single server you could trust. So Google built its own, and in 2003 described it in "The Google File System" (Ghemawat, Gobioff and Leung, SOSP 2003). It became one of the most influential systems papers ever written: Hadoop's file system, HDFS, is a close copy, and its ideas run through almost every storage system in this course.
The assumptions that shaped it
The paper is unusually honest about its workload, and every design decision follows from it. In plain terms:
- Machines fail constantly. Detection, replication and recovery must be automatic and routine.
- Files are huge. Multi-gigabyte files are common. A design built around billions of tiny files is not the goal.
- Writes are appends. Files are mostly written once, front to back, or appended to by many writers. Random overwrites are rare.
- Reads are big and sequential. Jobs stream through files. Throughput matters far more than the latency of any single read.
- Applications are Google's own. The file system and its users can be designed together, so the file system doesn't need full POSIX behaviour if the applications don't need it.