How to test an AI SRE's accuracy claims

If you are evaluating AI SRE tools this year, you have seen the numbers. Root cause accuracy of 82 percent, 90 percent, 94 percent. Time to root cause in under five minutes. Mean time to resolve cut by a third or more.

Some of these numbers are probably true for some customers. The problem is that almost all of them are measured by the vendor, on the vendor’s choice of incidents, with the vendor’s definition of “correct.”

What the independent evidence says

There is not much of it, which is itself telling. One of the more useful public experiments came from ClickHouse, which injected known faults and tested whether frontier models could find the root cause. Their conclusion was that no model reliably found the injected causes without human guidance, and that models tended to lock onto one line of reasoning and miss incidents with more than one cause (ClickHouse, AI SRE agents).

None of this means AI SREs do not work. It means you should measure one yourself.

A replay test you can run

The fairest test uses incidents you already understand.

1. Pick 20 to 50 resolved incidents from the last few months. Include easy ones, hard ones, and a few where the cause was not a change at all.
2. Write down the answer key first. For each incident, record the actual cause in one sentence, as your team understood it after the postmortem.
3. Give the tool only what it would have had at the time. Telemetry and changes up to the moment the incident started, not the postmortem.
4. Score three outcomes, not two. Right, wrong, and “not sure.” A tool that says it does not know is far more useful than one that is confidently wrong at 3am.
5. Count misleading answers separately. A wrong answer that sends the on-call engineer down the wrong path for twenty minutes is worse than no answer.
6. Time it. How long until the first useful finding?

What a good result looks like

You are not looking for perfection. A useful tool is right most of the time on change-caused incidents, says “not sure” on the strange ones, and almost never sends you confidently in the wrong direction. The ratio of right to misleading matters more than the headline accuracy.

Why we offer this up front

At sudoSRE we think every AI SRE should be judged this way, including ours. Design partners get a replay of their own recent incidents during onboarding, scored against their own answer key, so the decision rests on their production rather than our marketing.

If that sounds like the right way to evaluate a tool, we would like to run it on yours.