Docs / Why this method
Why evals from production failures
Every team shipping an LLM agent has the same unanswered question: did the last change make things better or worse? Answering it requires evals built from the failures your system actually produces.
The question nobody can answer
A team ships a prompt change, a model upgrade, a new tool, a retrieval tweak. Then someone asks whether it helped. The honest answers available today are usually one of:
- "Vibes are better." Somebody tried eight prompts by hand.
- "The benchmark went up." A benchmark that has nothing to do with your product.
- "No new complaints yet." Which means the feedback loop is your customers.
None of these is a measurement. The gap is not tooling for storing traces, which is a solved and commoditised problem. The gap is that nobody has written down what "wrong" means for this specific product, in a form precise enough to count.
The teams we work with keep arriving at the same point independently: monitoring is not the scarce thing. Everyone already stores traces and draws dashboards. What nobody can answer is what to fix.
Why generic metrics do not work
Off-the-shelf eval suites ship with a fixed set of dimensions: helpfulness, groundedness, toxicity, relevance, coherence. They are cheap to turn on, and that is the problem. They measure what a vendor imagined a generic assistant might get wrong, scored by a model that has never seen your product.
Three concrete failures of that approach:
| Problem | What it looks like in practice |
|---|---|
| Wrong vocabulary | Your real failure is "the agent proceeded after a validation gate returned failed". No generic metric has a name for that, so it scores 0.82 on helpfulness and moves on. |
| Unactionable output | A groundedness score of 0.71 does not tell an engineer what to change. A rate of 71% on "hallucinated external fact", with 198 linked traces, does. |
| No ownership | Nobody's job gets worse when a generic score drifts, so nobody watches it. A named failure mode has an owner because someone argued for its name. |
Where this method comes from
None of this is new, and that is the strongest thing we can say about it. Two older traditions arrive at the same instruction from opposite directions, and recent work has pointed both of them at LLM systems. Zoroval is the tooling around that instruction, not the instruction itself.
Error analysis, from machine learning
Long before LLMs, the standard advice for improving a model was to stop tuning and go read what it got wrong. Andrew Ng has taught the mechanical version of this for years: pull roughly a hundred misclassified examples into a spreadsheet, give each example a row and each kind of mistake a column, count how many examples fall under each kind, and spend your effort on the column with the largest count. He also adds a column for examples that turn out to be mislabelled, because some of them always are.
That procedure is the entire idea, and every part of Zoroval maps onto a part of it:
| Error analysis | Zoroval |
|---|---|
| A row per misclassified example | An open code on a trace, tied to the evidence |
| A column per kind of mistake | An axial code, a named failure mode |
| The count under each column | The error rate per failure mode, per day |
| Work the largest column first | Choosing which failure modes to monitor and fix |
| A column for mislabelled examples | Insufficient verdicts, kept out of the denominator |
What changes with LLM agents is the material, not the method. A misclassified image is one row. A failed agent trace is a hundred events deep, the failure is often one sentence inside one of them, and there are more traces arriving every day than anyone will ever paste into a spreadsheet. That is why the evidence for a finding has to point back into the trace itself rather than be described in prose, and why the counting has to be done by a judge rather than by hand.
Open and axial coding, from qualitative research
The second tradition is qualitative research, where analysts face a structurally identical problem: a large pile of unstructured material, and no reliable categories to count with until someone reads it.
In that tradition, open coding means reading the material and attaching short descriptive labels to what you actually observe, without deciding in advance what the categories are. Axial coding is the pass that follows, where those scattered labels are consolidated into a smaller set of named categories with defined boundaries. Those are the two words the product uses, and they are used in their original sense.
Pointed at LLM systems
Bringing the two together and applying them to LLM traces is work done in public by Hamel Husain and Shreya Shankar, whose teaching on error analysis for LLM systems is the clearest treatment of it we know of. Their talk How to Automate AI Evals (Correctly) is a good place to start if you want the method without the product. A fair amount of Zoroval's shape came from working through it.
What the combination buys you
Applied to agent traces, it gives you two properties nothing top-down can give you:
- The categories come from the data. You end up measuring the failures your agent has, not the ones a vendor anticipated.
- Every category is defensible. Each one is backed by specific traces a human labelled, so when someone challenges a number you can open the evidence rather than defend a methodology.
The cost is honest: somebody has to read traces. Zoroval reduces that cost, and then makes the investment durable by scaling the resulting labels with a judge. It does not pretend the reading step can be skipped.
Why a judge, and what makes one trustworthy
Human coding does not scale to every trace, every day. An LLM judge does. But a judge is only worth running if its verdicts agree with the people who defined the code, and a code that two people cannot apply consistently is not one worth scaling. That check happens before a code is monitored, and it is done by your team, not by the model. See Calibration.
Two constraints follow from that, and both are deliberate:
- A verdict has to show its work. A judge that is right about a trace but cannot point at what convinced it is not finished. Every verdict carries the evidence behind it, which is what you see when you drill into a number.
- Insufficient is a valid answer. Some traces genuinely do not contain enough to decide a given failure mode. Forcing those into "no" quietly deflates every rate you report, so they are counted separately and kept out of the denominator.
What you get out of it
- A short list of failure modes in your team's own words, that new engineers can learn in an afternoon.
- A rate per failure mode per day, with drill-down to individual traces and the exact evidence.
- A before and after for every change you ship, without waiting for customers to report it.
- A shared vocabulary that turns "the agent felt worse this week" into a claim somebody can check.