Platform Principles / Engineering
Why the Best Infrastructure Is the Infrastructure Nobody Notices
Good infrastructure earns trust quietly. Thoughts on reliability, platform engineering and recognising the engineers who prevent problems before anyone notices.
Nobody notices the network when it works.
Nobody gives much thought to DNS when a website loads, or to the database when an application responds. We open an app, send an email or make a payment, and expect it to work.
Then a certificate expires. A database fails. A small configuration change has a much bigger effect than anyone expected. Suddenly, everybody wants to know about the infrastructure.
It’s a strange part of this job. When we’re doing it well, most people don’t know we’re there.
I’ve come to think that’s a good outcome. The service should feel straightforward to use. Getting it to that point takes a lot of work, and the people doing that work deserve to be seen.
The work behind something that just works
I’ve spent much of my career in infrastructure, from traditional on-premises environments through to cloud engineering and architecture.
The tools have changed considerably. Physical servers, networks and virtualisation platforms have given way to cloud environments, automated deployments and platforms that let teams provision their own infrastructure.
But the person using the service still wants much the same thing: for it to work.
And why shouldn’t they? Someone using an application shouldn’t need to understand the routing, identity controls or recovery arrangements behind it. That complexity belongs with us.
Making it feel simple means making decisions about capacity, networking, security, monitoring and recovery long before anyone clicks a button. Some of those decisions won’t be tested until months or years later.
We tend to celebrate the visible moments: a migration, a new platform, the engineer who gets a critical service back online. Those things deserve recognition.
So does the less dramatic work. The engineer who removes a single point of failure. The team that automates a process everyone used to dread. The person who tests a restore and discovers the backup wasn’t doing what we thought it was.
There might be no exciting story afterwards. That’s often a sign the work paid off.
Good infrastructure should be boring
I like new technology. There’s always something worth exploring, particularly in cloud infrastructure, where keeping up can feel like a job in itself.
Production is where I want a bit less excitement.
A boring platform behaves predictably. Changes are repeatable. Monitoring tells us when something needs attention. Backups have been restored successfully, rather than simply showing a green tick. Recovery procedures make sense to someone other than the person who wrote them.
Nobody needs to ring one particular engineer to find out which order to restart three servers.
That doesn’t mean nothing ever breaks. Dependencies fail. People make mistakes. Carefully designed systems still surprise us.
The useful question is what happens next. Can the service keep running? Will we notice the fault? Does someone know how to recover it?
I’d rather have clear answers to those questions than an impressive architecture diagram.
Try it: the same failure, a different outcome
Imagine a service running on one server. Now imagine the same service behind a load balancer, with a second healthy server that can take the traffic. Take one server away and compare what the person using the service sees.
What happens when a server fails?
Same component failure. Different experience.
One server is handling the service. If it fails, there is no healthy instance to take over.
A simplified illustration, not an availability prediction. The failover example assumes a working load balancer, enough healthy capacity and no shared dependency failure. Detection and failover can still cause disruption.
The second design still needs investigation. A healthy user journey doesn’t mean there’s nothing for the engineering team to do. It means they have a chance to fix the fault without making it everybody else’s problem.
The complexity hasn’t disappeared
I often think of infrastructure as an iceberg.
Above the surface is the application or website. Below it are the networks, compute, storage, identity, DNS, certificates, monitoring, security, automation, backups and recovery arrangements.
Then there are the people, processes and operational knowledge needed to keep all of that working.
We can’t make the complexity disappear. We can decide who has to deal with it, and how often.
Take provisioning an environment for a development team. It can mean a string of requests, manual configuration, conversations about networking, approval queues and eventually somebody finding time to build it.
Or the team can choose a supported environment, provide the details that matter and let automation do the repeatable work.
Someone still had to design the network, agree the controls, write the templates and work out how to monitor and support the result. The difference is that the common problems have been dealt with once, instead of handed to every team that comes along.
That’s a better use of engineering time.
A platform has to make someone’s day easier
This is one of the reasons I’ve become increasingly interested in platform engineering.
When each project builds its own infrastructure, teams end up solving the same problems in slightly different ways. Deployment processes, security patterns, naming conventions, monitoring: all individually reasonable, all gradually adding up to something difficult to operate.
A useful internal platform gives teams a sensible starting point. A supported foundation, reusable templates and a way to get things done without raising a ticket for every step. Security and governance are part of that foundation, rather than a surprise at the end of delivery.
But abstraction has limits. If we’ve hidden so much that nobody can diagnose a failure, we’ve created another problem. Teams need a way to understand what’s underneath, and a clear route to support when the standard path doesn’t fit.
A platform also needs an owner, maintenance and feedback. Building something clever isn’t the finish line.
The test I’d use is fairly ordinary: does this make life easier for the people we expect to use it? If the answer is no, we need to listen before adding another feature.
The incidents that never happened
We usually know who helped restore a service during an outage. Their work is visible, urgent and easy to recognise.
Prevention is harder to point to.
Consider a failed component. In one environment, customers lose access and an incident bridge opens. In another, redundancy keeps the service running, monitoring alerts the team, and the component gets replaced without users noticing.
Neither environment can be judged from that one moment alone. Requirements, budgets and risks differ. Some services need stronger resilience than others. Redundancy costs money and brings its own operational work.
Even so, the second outcome shows the value of decisions made before the failure.
You can’t easily write an incident report about an outage that didn’t happen. But the avoided disruption matters to customers, and to the engineers who would otherwise spend their evening recovering the service.
A migration completes quietly. A certificate renews automatically. A failed server is replaced without interruption. A vulnerability is patched before anyone can exploit it.
These are achievements worth recognising, even when there’s very little drama to describe.
Invisible infrastructure shouldn’t mean overlooked engineers
Infrastructure can become an afterthought when it’s working. It’s expected to be available, secure, fast and increasingly cheap, as though those qualities look after themselves.
That’s a difficult basis for deciding where to invest.
When everything looks fine, it’s easy to question the cost of monitoring, resilience, automation or keeping enough engineering knowledge in the team. The value becomes much more obvious when those things are missing.
I think we have some responsibility for how we explain the work, too.
Not everyone needs the details of a routing design or a deployment pipeline. They do need to understand what it changes for them. Can a team deliver sooner? Can we recover with more confidence? Have we removed a risky manual step? Have we stopped three teams paying to solve the same problem?
Those are conversations worth having before the next outage gives us an audience.
Measure the experience, as well as the activity
Servers deployed, environments built, tickets closed. They’re easy numbers to collect, and sometimes useful ones.
On their own, they don’t tell us much about whether the infrastructure is any good.
A busy support queue could mean a team is working hard. It could also mean a recurring problem needs fixing. Deploying hundreds of servers isn’t much to celebrate if nobody needed them.
I’d rather ask:
- How long does it take a team to get a usable environment?
- How often do our changes cause failures?
- Can we restore the service, and when did we last prove it?
- How much repetitive manual work have we removed?
- Can the teams using our platform get on with their jobs?
If every deployment needs a conversation with the infrastructure team, there’s probably more work to do.
The aim is to give engineers time for problems that need their judgement, rather than keep them busy repeating the same tasks.
The best compliment is often silence
Earlier in my career, I was drawn to the technology itself. Servers, networks, virtualisation, and the satisfaction of finally getting something complicated working.
I still enjoy that part.
But increasingly, I’m interested in the experience we’re creating for the people who rely on it. Can they deploy without unnecessary friction? Can the service tolerate a failure? Can the next engineer understand what’s going on? Can we make a change without hoping for the best?
Infrastructure exists to enable something else. A business, a public service, an application, or something as ordinary as sending an email.
The person using it should be able to trust it without thinking about everything underneath.
Somewhere, an engineer is removing a failure point that would otherwise cause an outage. Another is automating a task nobody will have to do manually again. Someone else is quietly maintaining a service that thousands of people depend on.
Most of that work will go unnoticed by the people using the service.
The best infrastructure doesn’t demand attention. It earns trust. And behind that trust are the invisible engineers.
The engineering debrief
The principle
Good infrastructure absorbs complexity so the people depending on it can get on with their work. The engineering team still needs to understand and operate what sits underneath.
The question to take back to your team
What is one task that still depends on a particular person, a manual step or an untested assumption? Start there. Removing that dependency may be more valuable than adding another tool.
The leadership lesson
Recognise the work that prevents problems, alongside the work that fixes them. Explain it in terms of delivery, risk, recovery and the experience of the teams you support.