Blog / Engineering

Finding one failure mode in a 230,000 character trace

A reviewer open-codes one specific failure in an agent trace. Now you must find it again in thousands of traces that are too long to read. We tried three approaches. Here is what we measured.

By Chandan K S, Founder, Zoroval · 6 October 2026 · 6 min read

The problem

A domain expert reviews an agent trace and marks something wrong. It is not a general quality score. It is one specific problem. For example, the agent retried four times and never recovered. Or the image prompt it wrote carried the whole conversation history, when two sentences were enough.

The expert writes a note in their own words. They do this on a few more traces. Then the notes group into a named failure mode.

This is open coding. The method comes from qualitative research, and it is the part of evals that works. The failure modes come from what people actually see. Nobody has to guess the categories in advance.

Next, you want to know how often that failure mode happens in all your other traces. There are thousands of them. So you use an LLM as a judge.

Here is the difficulty. We measured 1,349 production traces from one of our design partners:

A median trace is about 58,000 tokens. Almost half of the traces are longer than 250,000 characters. The largest is more than one million. A median trace has 33 events, and some have more than 60. Each event is a model call or a chain step, with its own inputs and outputs.

Modern context windows can technically hold most of that. But that is not the question.

The real question is this. Can a judge, given all of that text, reliably find one specific failure that a reviewer already open-coded? And can it do this at a cost that lets you run it on every trace, every day, forever?

We tried three approaches.

Approach 1: hand the judge a large slice and let it look

This is the obvious first move, and it is the one we shipped first. Rank every event in the trace by how relevant it seems. Fill a 60,000 character budget with the top events. Then ask the judge if the failure mode is present.

This approach costs about $19 for every 10,000 verdicts. It was the most expensive approach we tested. Now run it for several failure modes, for each client, every day. At that point, the bill is the difference between a product and a science project.

There is a worse problem under the cost. A large budget is very good at hiding bad retrieval. (Retrieval is the step that picks what the judge reads.) If the ranking is weak, the budget still catches the evidence often enough. Nothing looks broken. You find out months later.

Approach 2: filter by what kind of activity it is

This is the natural next step. You do not need the whole trace. Send only the model generations, or only the user turns, or only the tool calls. Pick the layer where errors live, and drop the rest.

This is a real decision layer, and it does help. But it is not enough.

On the same dataset, we kept only the generation events. This cut the median trace almost in half, from 230,228 characters to 124,260. That is a good reduction. But it still leaves about 124,000 characters to judge. About seven in ten traces stay above 100,000 characters. The largest trace does not get smaller at all, because almost all of its text was already in generations.

You also pay for that reduction. When you drop a layer, you drop the evidence in it. Take the retry failure mode. You cannot see it in any single generation. You see it in the shape of the run: the same step appears six times under one parent. If you keep only generations, you delete the only place where that failure mode was visible.

Activity type is a rough guide to relevance. It loosely matches where errors live, and the match is different for each failure mode. The filter does not know what it is looking for.

Approach 3: let the open codes decide where to look

This approach uses something we had ignored.

When you are ready to scale a failure mode, you already have several human examples of it. During open coding, the reviewer highlighted specific text in specific places. They also explained, in their own words, what was wrong. Then the open codes were grouped into a named failure mode. That group kept all of this information.

Together, this information shows where the failure mode becomes visible. But it was not in a form that a retrieval system could use.

So we added a step between coding and judging. A model does this step once for each failure mode, not once for each trace. It reads:

• the grouped open codes
• the reviewer's notes
• the places in the trace that the reviewer was looking at
• a few sample trace structures

From this, the model writes a short, structured description of where to look for this failure. The description says:

• which parts of a trace matter
• what to take from those parts
• how much of each part to keep
• whether the signal is in the content or in the shape of the run

Then a deterministic step, which is plain code with no model, applies that description to every trace. It gives the judge a small, focused slice.

Three properties matter to us.

It is made once, not for each verdict. The expensive reasoning about where a failure mode lives happens one time. Everything after that is cheap. We also tested the alternative: a planner that decided again where to look on every trace. It cost more and took twice as long. It worked out the same answer again, thousands of times.

It is an artifact you can read. You can inspect and edit the description. If the judge starts to give wrong verdicts, you have a document to look at. Usually, you can see the mistake in it. The per-trace planner leaves nothing behind between calls. So every debugging session starts from zero.

The reviewer never writes it. They highlight text and type a note. That is open coding, and nothing more. The reviewer is a domain expert, not a retrieval engineer. If we asked them to write rules, we would lose the point. The point is to let the failure modes come from what they see.

Where this landed, on context and on cost

The generated approach uses about 16,000 characters of context for each verdict. The wide net from approach 1 uses 60,000. So the judge reads about a quarter of the text.

The generated approach is also about 30% faster. It costs about $5.50 for every 10,000 verdicts. The wide net costs $19.

We did not want retrieval rules that an engineer writes by hand for each failure mode. Hand-written rules stop working as soon as you have a second customer. The generated description comes from the open codes, so each new failure mode gets its own description with no extra engineering.

The whole study cost less than $2 in API calls. It is small: 52 traces and two failure modes. It shows a direction, but it is not a benchmark. We plan to repeat it on more traces with our design partners. We will publish the results, even if they disagree with this post.

Conclusion

Retrieval quality mattered much more than context budget. The two are easy to confuse, because a large budget makes weak retrieval look like it works.

The main lesson is about open coding. We used to treat it as a labelling step, a way to get examples of a failure mode. But it is also the most detailed description we have of where that failure mode appears inside a very large trace. For us, that made it more valuable as an input to retrieval than as training data. Most of the gain came from one discovery. We already had the information we needed, and we threw it away after we used it once.

Do you run judges over long agent traces? Have you measured something similar, or something that disagrees with this? We would like to hear about it.