Most outages start with a change

Google’s SRE book has a sentence that every on-call engineer already knows from experience: “SRE has found that roughly 70% of outages are due to changes in a live system” (Google SRE Book, Introduction).

The same passage lists the practices that follow from it: progressive rollouts, “quickly and accurately detecting problems,” and “rolling back changes safely when problems arise.” Those three together limit how many users are exposed to a bad change.

Most teams have the first and third in some form. Canary deploys and kubectl rollout undo are not new. The middle one, detecting the problem quickly and accurately, is where things usually fall apart.

How a bad deploy usually gets found

Think about the last incident your team had that turned out to be a bad change. How was it found? For a lot of teams the honest answer is one of these:

• A threshold alert fired, eventually, because the error rate finally crossed a number someone set months ago.
• A downstream team noticed their own service degrading and started asking around.
• A customer reported it.

Then comes the slow part. Someone has to work out what changed. In a busy cluster that can mean scrolling through a dozen deploys, a Helm upgrade, a config edit, and an autoscaler event from the same half hour, and trying to line them up against a graph.

Thresholds are the wrong question

A static alert asks: is this number above a line? A change-aware detector asks a better question: is this service behaving differently than it did five minutes ago, right after something about it changed?

That second question has two advantages.

It catches regressions that never cross a threshold. A service whose error rate goes from 0.1 to 0.6 per second may sit well under any alert you would sensibly set. It is still a sixfold regression that started 90 seconds after a deploy, and it is worth knowing about.

It narrows the search immediately. If a regression starts right after a specific change to a specific service, the investigation starts with one diff instead of the whole system. That is the main reason change-aware detection tends to be more accurate than investigating from an alert alone: the search space is small.

Capture changes where they land

To do this you need two things side by side: a reliable record of every change, and a baseline of how each service normally behaves.

The record of changes is harder than it sounds. Changes come from CI pipelines, GitOps controllers, Helm, manual kubectl edits, autoscalers, and config maps someone updated at 4pm on a Friday. Each tool only knows about its own changes. The one place they all show up is the cluster itself, which is why sudoSRE captures changes there, with an agent that watches rollouts, config, scaling, and releases as they happen.

The baseline needs to be per service and aware of time of day, so the normal Monday morning traffic ramp does not look like an incident. A few hours of observation gets most services to a usable baseline.

Where this leaves alerts

Threshold alerts still have a job. They catch slow-burning problems and things that are not tied to a change at all. But if roughly 70 percent of outages begin with a change, the fastest detector you can build is one that watches every change and checks what it did.

That is the core of what we are building at sudoSRE. If you run Kubernetes and want to try it on your own cluster, we are onboarding design partners.