Skip to content

Independent marketplace for verified cloud accounts

Designing for Failure: Redundancy That Is Worth the Cost

Redundancy is insurance, and some of it is a luxury purchase. A method for deciding which failures to pay to survive and which ones to absorb.

Redundancy is one of the few engineering decisions where the correct answer is sometimes “no”. Every duplicated component costs money every hour, adds a failure mode of its own, and makes the system harder to reason about. The teams that get this right are not the ones who duplicate everything. They are the ones who can say out loud which failures they are buying protection against and which ones they have decided to absorb.

What follows is a way to make that decision deliberately, starting from the failures rather than from the architecture diagram.

Start by naming what you are protecting against

“High availability” is not a requirement. It is a vague feeling. The useful version is a list of specific events with specific consequences: a single virtual machine stops responding, a disk fails, a deploy ships a bug, a zone loses power, a region goes dark, a provider suspends the account, somebody deletes the production database.

Write that list out and put two numbers next to each entry: roughly how often you expect it, and what an hour of it costs you. The list immediately sorts itself. Single instance failure is common and usually cheap to survive. Regional loss is rare and extremely expensive to protect against. Account level problems are rarer still and are not solved by any amount of infrastructure inside that account, which is a point worth sitting with.

Most teams discover that two or three entries dominate the expected loss and the rest are rounding errors. Spend your redundancy budget there and nowhere else.

The cost curve of redundancy

Availability improvements do not cost linearly. Going from one server to two is close to a doubling of compute cost and removes the single largest category of outage. Going from two to three adds half as much again and mostly protects you during maintenance on one of the first two. Going from one region to two does not double the bill, it more than doubles it, because you now pay for duplicated data, cross region replication traffic, a second set of managed services and the engineering time to keep both halves identical.

The non obvious costs sit on the operational side rather than the invoice. Two of everything means two things to patch, two sets of configuration that can drift apart, and a deployment process that has to handle the case where one side updated and the other did not. Every one of those is a new way to have an outage, and a meaningful fraction of real incidents in redundant systems are caused by the redundancy mechanism rather than by the fault it was protecting against. A failover that triggers on a transient network blip and takes thirty seconds to complete has just created an outage that would not otherwise have happened.

The curve bends sharply, and the bend is usually somewhere between zone redundancy and region redundancy. Knowing where your own bend sits is the point of the exercise in the previous section. If an hour of downtime costs less than the monthly premium for multi region, the multi region design is a luxury purchase, and that is fine as long as you call it one.

Stateless tiers are cheap to duplicate, state is not

Application servers that hold nothing between requests are the easy case. Run two or more, put a load balancer in front, and failure of any one becomes a non event. The marginal cost is the extra instance, and on value oriented providers the extra instance is genuinely small money. This is the single highest return redundancy purchase available and almost everyone should make it.

State is the hard case. A database cannot simply be duplicated, because two copies that both accept writes will disagree, and resolving that disagreement is a distributed systems problem rather than a configuration setting. You get to choose between synchronous replication, which costs latency on every write and protects you fully, and asynchronous replication, which is cheap and fast and quietly accepts that a failover may lose the most recent writes.

Decide which of those you have chosen before an incident, not during one. An asynchronous replica presented to the business as a hot standby is a promise you cannot keep. If your write path can tolerate a few milliseconds extra, synchronous replication across zones is usually the right default, and it is one of the few places where paying more buys an unambiguous improvement.

Backups are not redundancy

These two words get used interchangeably and they solve different problems. Redundancy protects against a component disappearing. Backups protect against data being wrong. A replicated database will faithfully replicate a bad migration, a mistaken delete and a ransomware encryption to every copy you own, in milliseconds.

So you need both, and the backup side has its own requirements. Backups should live in a different failure domain from the thing they back up, ideally a different account or project so that a credential compromise cannot destroy them. They need a retention window long enough that you can go back past the point where corruption started, which is often days rather than hours. And they need to be restored on a schedule, because an untested backup is a hypothesis.

Running backups into an entirely separate account is one of the better uses of a cheap secondary provider. A low cost storage focused account such as Buy Contabo Account or Buy OVH Account alongside a primary hyperscaler gives you a copy that no single compromised credential can reach, and the OVHcloud and Contabo comparison covers the trade offs between those two in more depth.

Health checks decide whether failover works

A redundant system that cannot tell the difference between healthy and unhealthy will route traffic into a broken instance forever, or worse, will remove healthy instances during a transient blip and take itself down. The health check is the part that makes redundancy real, and it is usually an afterthought.

A good check exercises the actual dependency path: can this instance reach the database, is it serving correct responses, is it out of disk. A check that returns a static two hundred from a web server proves the web server process is running and nothing else. Equally, a check that hammers the database on every probe from every load balancer adds load that scales with your redundancy, which is an excellent way to cause the outage you were preventing.

Tune three things: what the check tests, how many consecutive failures count as down, and how long a recovered instance waits before receiving traffic again. Those three values are the difference between graceful failover and a flapping cluster.

Testing failure on purpose

You do not know whether your redundancy works. You know whether it has been exercised. Those are different states and only one of them is knowledge.

Start small and in daylight. Terminate one application instance during business hours and watch whether anything pages. Fail over the database replica deliberately and time how long writes are rejected. Detach a volume. Block a security group rule. Each of these takes under an hour and each has a reasonable chance of surfacing something embarrassing, such as a hardcoded instance address, a connection pool that never retries, or an alert that was silenced eight months ago.

Build this into a routine rather than treating it as an event. Teams that already run monitoring and alerting with real thresholds find these drills cheap to run, because the instrumentation to observe the result already exists.

Where redundancy is not worth buying

Internal tools used by a handful of people during office hours do not need multi zone anything. A nightly batch job that can simply run again tomorrow does not need a standby. Development and staging environments should be deliberately less reliable than production, both because the cost is real and because fragility in staging teaches the team things.

There is also a category where redundancy actively hurts: systems where a split brain is worse than an outage. Anything that issues payments, allocates unique identifiers or controls physical equipment may be better off refusing to operate than operating in two minds. For those, correct behaviour is to fail closed and alert loudly.

The honest summary is that redundancy is insurance, and like all insurance it is worth buying when the premium is smaller than the expected loss. Price the premium properly, including the engineering hours and the extra complexity, and a lot of the fashionable architecture turns out to be optional. If a cheap second account is part of your plan, compare the running costs across the full provider catalogue before committing, and keep the redundant side simple enough that you will actually maintain it.

Related comparisons

Hetzner vs Contabo
vs

Hetzner vs Contabo

Both are German hosts competing on low prices, and Contabo is usually the cheaper number on the page. The useful…

OVH vs Contabo
vs

OVH vs Contabo

Contabo sells generous specifications at low prices; OVHcloud sells a full infrastructure catalogue with enterprise options. The comparison is really…

Accounts discussed in this guide

Buy Contabo Account
Buy Contabo Account
Contabo

Buy Contabo Account

Contabo has built a reputation as one of the most affordable cloud hosting providers in the industry, offering virtual private servers and dedicated servers…

From$70 1 option
Buy OVH Account
Buy OVH Account
OVHcloud

Buy OVH Account

OVHcloud has built a formidable reputation as one of Europe's largest and most independent cloud providers, offering a compelling alternative to the hyperscalers for…

From$30 2 options

Keep reading

Need the account, not just the theory?

Browse documented configurations across every provider in the catalog, or ask us which one fits your workload.

Telegram