Module 20 / Databases in depth

Module 20 · lesson 7 of 10 · Read 10 min

From memory to disk

Buffer pools, the OS page cache, and what fsync really promises.

What you'll learn

  1. The layers
  2. The buffer pool: the database's own cache
  3. The OS page cache
  4. What fsync promises
  5. When fsync fails
  6. Torn pages
  7. Where it breaks, and the knobs

When Postgres says COMMIT succeeded, where are your bytes? Not in one place. A change passes through the database's own memory, the operating system's memory, the drive's cache and finally the storage itself, and each layer has its own idea of "written". Most of the time you can ignore all of this. You can't when you size a database server's memory, choose a durability setting, write your own file format, or read a postmortem that says "fsync returned an error".

This lesson follows one write down through the layers, then looks at the two big caches (the buffer pool and the OS page cache), what fsync actually promises, and the failures that live in between.

The layers

Layer Owned by Survives a database crash? Survives a power cut?
Buffer pool The database process No No
OS page cache The kernel Yes No
Drive's write cache The drive's firmware Yes Only with power-loss protection
Storage media The drive Yes Yes

A database crash (the process dies, the machine keeps running) loses only the first layer. A kernel panic or power cut loses the first two, and the third unless the drive is built to protect it. Durability is about knowing which layer your data has reached when you tell the customer "order placed".