Docs / Overview

Overview

Zoroval turns "is our agent good enough?" into a set of named failure modes, each with a rate that is measured every day and is supposed to move when you ship a fix.

What Zoroval does

Zoroval reads the production traces your LLM agent already writes, and helps your team do four things with them, in order:

Name the failures. A human reads real traces and writes down, in plain language, what went wrong, tied to the exact passage that shows it.
Group them. Those individual observations get consolidated into a short list of named failure modes specific to your product.
Scale them. An LLM judge, calibrated against your team's own labels, scores every new trace against those failure modes.
Track them. Each failure mode gets a rate per day, so a regression shows up as a number that moved rather than a customer complaint.

The first-class object in Zoroval is a failure mode, not a metric. A failure mode is defined by a person who looked at real traces, it has a rate, and that rate is expected to visibly drop when your team ships a fix. Everything else in the product exists to make that number trustworthy.

The short version

Generic quality scores tell you that something is wrong on average. Zoroval tells you which failure is running at what rate, and hands you the exact span of the exact trace that proves each instance.

The loop

Zoroval runs one loop. Two stages are human work and take real time. The rest is automated and runs without you.

Ingestion Zoroval

Zoroval pulls traces from the observability stack you already run, on a schedule, and turns them into a timeline a human can read. Nothing is written back to your stack, and re-running an import never duplicates a trace.

Open coding You

You read individual traces and label what went wrong, selecting the exact evidence as you go. This is the input everything downstream depends on. See Open coding.

Axial coding You, with proposals

Zoroval takes each approved open code and tries to place it in the vocabulary you already have, reusing an existing failure mode wherever one fits and proposing a new one only when nothing does. You confirm, rename, re-scope or reject. The confirmed list is your team's failure taxonomy. See Axial coding.

Monitoring setup You

You choose which confirmed failure modes to watch. Only monitored codes cost anything to run. See Monitoring.

Daily judging Judge

For each completed trace day, every sampled trace is scored against every monitored failure mode. Each verdict is recorded with a confidence and the evidence behind it.

Dashboard, logs and report Zoroval

Rates per failure mode per day, with the day over day delta, every individual verdict, and a written summary you can hand to whoever owns the fix. See Reading the results.

You ship a fix, and the loop closes You

The next evaluation day scores the same failure mode against new traces. If the fix worked, the rate falls, and you can point at the day it fell.

Every number drills down to the trace

The difference that matters most in daily use is what happens when you click a number. Zoroval does not simply tell you that a trace failed. It opens the trace at the passage the verdict rests on, highlighted, so you are reading your own data rather than a summary of it.

Example trace 2b95e07f

… based on the vendor's own reliability data, a 2024 study by the Institute for Applied Systems found a 31% reduction in downtime across comparable deployments …

That is what makes a failure mode auditable. A rate you cannot drill into is a number your team will eventually stop trusting.

What Zoroval is not

It is not a replacement for your tracing stack. Zoroval reads from it. Keep Langfuse, LangSmith, Phoenix or whatever you already run.
It is not a generic quality score. There is no helpfulness number, no vendor checklist. If your team did not name a failure mode, Zoroval does not measure it.
It is not a fully automatic labeller. The judge scales your team's judgement. It does not replace the part where a human decides what counts as wrong.

Where to go next

Why this methodThe case for building evals out of your own production failures instead of adopting someone else's metrics. Open codingHow to read traces and label failures so the labels survive being scaled. Axial codingTurning scattered observations into a small, defensible list of failure modes. MonitoringPromoting a failure mode to a daily measurement, and the limits that keep it predictable. CollaborationWhy this is a team practice, and what goes wrong when one person owns it. OnboardingWhat we need from you, what happens in what order, and how long each part really takes.