Module 18 / High availability and multi-region

Module 18 · lesson 4 of 7 · Read 9 min

Disaster recovery

When a whole data centre goes away.

The previous lesson spread Tiffin across three availability zones in one cloud region. That survives a broken switch, a dead server or even one data centre losing power. It doesn't survive the region itself going away.

That sounds far-fetched until you look at what has actually happened. In March 2021 a fire destroyed one of OVHcloud's data centres in Strasbourg, and customers who kept their servers and their backups in that building lost both. In December 2021 an automated scaling activity overloaded the internal network of AWS's oldest and busiest region, us-east-1, and broke a long list of well-known apps for about seven hours. And in January 2017 an engineer at GitLab, working late to fix a replication problem, deleted the data directory on the production database primary, then found that several of the backup methods they'd relied on hadn't been working. They restored from a copy that happened to be about six hours old, and lost roughly six hours of database changes (issues, comments, new accounts). The Git repositories themselves were safe.

Disaster recovery (DR) is the plan for these days: how you get back up when a whole site is gone, or when your own data has been destroyed.

Two kinds of disaster

It helps to split disasters into two groups, because they need different defences.

The place is gone. A region outage, a fire, a flood, a cut fibre, a cloud account suspended by mistake. Your data is fine, but you can't reach it, or the copy you can reach is somewhere else.