Systems Realities: Why Rollbacks Are Harder Than Deployments

Every deployment pipeline has a rollback button, and the existence of that button does a lot of quiet psychological work. It makes shipping feel reversible. If this goes wrong, we undo it.

The button is real. The reversibility mostly isn’t.

A deployment moves your system forward into a state you designed, tested and rehearsed. A rollback moves it into a state nobody designed at all — your old code, running against everything that changed while the new code was live. Those are not opposite operations of equal difficulty. One of them you have done a thousand times. The other you are attempting for the first time, usually while something is on fire.

A rollback is not a reversal. It is a deployment of old code into a world that has already seen the new code.

Your Data Already Moved On

Code is versioned and immutable. You can fetch any commit from any point in history and it will be exactly what it was. Data has none of those properties.

While v2 was running, it wrote rows. Rows with a column v1 doesn’t know about, enum values v1 doesn’t handle, a JSON shape v1’s parser will reject, timestamps in a format v1 reads differently. Roll the code back and v1 meets data from the future. It will do one of three things: crash loudly, which is the good outcome; silently ignore fields, which is survivable; or misinterpret them, which is how a rollback turns one incident into two.

Migrations make this sharper. Frameworks offer a down() method and it creates an impression of symmetry that does not exist. Adding a nullable column is reversible. Backfilling it and then dropping it destroys the backfill. Splitting one column into two and migrating the data is reversible only if you kept the original, which the migration usually deleted because leaving it felt untidy.

The honest position is that most schema changes are one-way doors wearing a two-way sign, and the rollback plan quietly assumes you will never need to walk back through.

The Path You Only Run When You’re Already Losing

You test deployments continuously, because every deployment is a test of deployment. Dozens of times a week, under normal conditions, with the pipeline exercised end to end.

Rollback gets tested approximately never. It gets exercised for the first time at 2am, during an outage, by whoever happens to be on call — statistically the person with the least context — while someone senior asks for an ETA every ninety seconds.

The procedure you run only when things are broken is the procedure most likely to be broken.

The failure modes are mundane and devastating. The old container image was pruned from the registry by a retention policy. The config format changed, so v1 starts and immediately crashes on a key it doesn’t recognise. The rollback script references an infrastructure resource that was renamed two sprints ago. None of these are hard problems. All of them take twenty minutes to diagnose, and you are discovering them at the worst possible moment with the worst possible information.

And the cognitive asymmetry compounds it. Deployments happen calmly, in business hours, with review and a checklist. Rollbacks happen with adrenaline, in the dark, with partial information and an audience. Same technical complexity, a fraction of the available judgement.

The World Adapted While You Weren’t Looking

Your service did not run in isolation while v2 was live. Other things noticed it and adjusted.

A downstream team saw the new field in your API response and started reading it. A consumer cached v2-shaped responses with a long TTL. There are v2-format messages sitting in a queue right now, and after the rollback v1 will pick them up and fail to parse them. A partner integration was updated on their side to match your new payload.

Roll your service back and none of that rolls back with it. You have not returned to a previous state of the world; you have created a new one where your service is old and its neighbours are not. That configuration has never been tested, because it never existed before.

Then there are the effects that left your boundary entirely. Emails sent. Payments captured. Webhooks delivered. Records created in a third-party system. Rollback restores your code. It does not unsend anything.

This is why “we can always roll back” is a reasonable thing to say about a self-contained service and a dangerous thing to say about anything that talks to other systems — which is everything you actually run.

Nobody Agrees What “Roll Back” Means

Under pressure, someone says “roll it back” and four people hear four different operations.

Redeploying the previous artifact. Reverting the commit and running the pipeline forward. Flipping the feature flag off. Restoring the database to a snapshot. Scaling the new version to zero and the old one back up.

These have wildly different blast radii and wildly different reversibility. Flipping a flag is near-instant and genuinely reversible. Restoring a database snapshot destroys every write since the snapshot was taken and is not reversible in any sense. One of those is a shrug and the other ends careers, and they are being proposed with the same two words at the same volume in the same channel.

Deciding in advance which of these your team means — and writing it where someone can read it at 2am — costs an afternoon and removes an entire category of incident.

Make Rollbacks Rare, Not Fast

Most teams respond to a painful rollback by trying to make rollbacks faster. That is optimising the wrong variable. The goal is to need them less.

Separate deploying from releasing. Ship the code dark behind a flag, turn it on for a fraction of traffic, and turn it off when it misbehaves. A flag flip has none of the problems above — no artifact to find, no data written in a new shape, no neighbour that adapted — because the old path never stopped existing.

Evolve schemas additively, in stages. Add the new column, write to both, migrate readers, and only drop the old one once nothing has touched it for a sensible period. Each step is independently safe to abandon, which is a much stronger property than a down() migration that has never been run.

And make the new code tolerant of old data and the old code tolerant of new data, at least for the window you intend to keep the rollback option open. That window is a real engineering commitment with a real cost, and the useful thing is to state it out loud — “we can roll this back safely for 48 hours, after which we can’t” — rather than letting everyone assume it’s indefinite.

The rollback button is worth having. It is just not the safety net people believe it is, and systems designed on the assumption that it is tend to find out during the incident rather than before it.