Docs / The method / Open coding

Open coding

Open coding is the part where a human reads real traces and writes down what went wrong, in their own words, tied to the exact passage that shows it. Everything downstream inherits the quality of this step.

What an open code is

An open code is one observation about one trace. It has three parts:

A label. A short phrase in your team's language. "Cited a source that isn't in the trace." "Kept retrying the same failing tool." "Answered in prose when the contract says JSON."
The evidence. The passage you selected, kept tied to where it came from in the trace. This is what makes the code checkable later.
An optional note. Anything a colleague would need to understand why this is wrong, especially when the wrongness is domain knowledge rather than something visible in the text.

Open codes are deliberately cheap and deliberately messy. You are not designing a taxonomy at this stage. You are recording observations. Consolidation happens later, in axial coding.

How it works in the product

Open a traceThe Open coding view lists ingested traces with filters. Each one opens as a normalised timeline: the conversation transcript where that applies, and the underlying events with their inputs, outputs and tool calls.
Find the moment it goes wrongLong fields are truncated by default with a Show full event control, and nested structures expand in place. Read at the altitude where the failure is actually visible, which is usually a model output or a tool result, not a framework wrapper.
Select the evidenceHighlight the passage that shows the failure. Zoroval remembers where in the trace it came from, so the code stays tied to its source rather than to a copy of it.
Type the codeType the label you would use in a bug report. Existing codes autocomplete, so reusing a label you or a teammate already wrote is the path of least resistance, which is exactly what you want.
Move onOne trace often carries several codes, and plenty carry none. Both are useful signal.

Example trace 7c0a11d4

{"status": "failed", "reason": "schema validation failed on field `applicant_id`", "retryable": false}

Why the evidence matters

A label on its own is an opinion. A label attached to the passage that provoked it is a specimen, and specimens are what the rest of the method runs on: they are how codes get grouped, and they are what settles the argument when somebody disputes a number six weeks from now.

The altitude mistake

The most common and most damaging error is selecting evidence at the wrong level of the trace. Text inside a framework wrapper event, for example the serialised input of a chain runner, technically records something but points at a container rather than at the operation that failed. Select from the event that actually produced the bad behaviour.

What good open coding looks like

Do	Instead of
Describe the failure	Prescribing the fix. "Cited a nonexistent study" is a code. "Add a citation checker" is a ticket.
One observation per code	Compound labels. "Slow and hallucinated and wrong format" cannot be counted as anything.
Use your team's words	Inventing research vocabulary. If your team says "gate failure", the code says gate failure.
Record the near misses	Only coding disasters. Traces that were nearly wrong define the boundary of a failure mode more sharply than obvious ones.
Say when you are unsure	Guessing. Note the uncertainty. Ambiguous cases are the ones worth discussing with the person who knows the domain.

How many traces to code

Code until new traces stop producing new labels. In practice that point tends to arrive somewhere between thirty and sixty traces for a single agent surface, and it arrives sooner if you sample deliberately rather than taking the most recent traces.

A useful sampling mix:

A block of consecutive recent traces, so you see the ordinary case at its real frequency.
Traces that are unusually long, unusually short, or ended in an error, where failures concentrate.
Anything a customer or a colleague already complained about, so known problems get a name early.

You are not trying to estimate a rate by hand. That is the judge's job. You are trying to discover the categories.

AI proposals

Proposals run on their own. There is no threshold of human coding to cross first and nothing to switch on per trace: traces are queued for review automatically, and the results wait for you in the AI proposals queue alongside All traces. Working through the queue rolls it forward onto traces nobody has looked at.

Each proposal arrives as a label, a note, and a quote from the trace, and you approve it, edit it, or reject it. Approving one turns it into an open code on that trace, with its evidence already attached.

Proposals are grounded, or they are discarded

A proposal has to quote one exact, substantive passage copied verbatim out of your trace. Paraphrases are rejected, and so is citing a header or a piece of metadata as though it were evidence. A trace with nothing wrong in it produces no proposals rather than a filler observation.

That constraint is what makes a proposal reviewable in seconds. You are reading a quote from your own trace and deciding whether it is a problem, not evaluating an argument about a trace you would have to go and open.

What proposals do not know

The proposal model does not see your open codes, your axial codes, or any vocabulary your team has built. It looks for concrete moments that appear wrong, risky, unsupported, inefficient, incomplete, contradictory or off-task, and it names them in its own words. Two things follow from that:

Edit the label, do not just approve it. An approved proposal enters your open codes carrying the model's phrasing. Rewriting it into the words your team actually uses is what keeps the taxonomy legible three months from now, and it costs a few seconds.
Do not build a taxonomy out of proposals alone. A code list assembled entirely from approvals inherits whatever the model finds salient, which is precisely the top-down problem this method exists to avoid. Read traces yourself first, then let proposals extend the coverage.

Cost control

Proposals are generated for a bounded queue, never for your whole backlog at once, and they draw from a daily allowance. When that allowance runs out the job stalls rather than running up a bill, and the queue screen says so rather than failing quietly.