Docs / Working as a team / Collaboration
Collaboration
The judge inherits your team's agreement about what counts as wrong. If the people on your team disagree, no model resolves that for you, so the disagreement has to be worked out in the open.
Why one person cannot do this
Open coding looks like solo work. It is not, and teams that treat it that way get a taxonomy that quietly encodes one person's blind spots.
Different people looking at the same trace see genuinely different failures:
- The engineer sees the tool that was retried four times and the schema that was violated.
- The domain expert sees the answer that is fluent, well formatted, structurally valid, and wrong in a way that would embarrass the company. This is the scarcest and most valuable label in the system, and it is invisible to anyone without the domain.
- The support or success person knows which failures customers actually complain about, which is rarely the same list as the failures that look worst in a trace.
- The product owner knows which failures matter for the thing the product promises.
A taxonomy built by one of those four measures one of those four perspectives. The rate will be real, and it will be a rate for the wrong thing.
The taxonomy becomes your vocabulary
The most underrated effect of this work is linguistic. Once a team has a confirmed list of failure modes, quality conversations change shape:
| Before | After |
|---|---|
| "The agent felt worse this week." | "Hallucinated external fact went from 44% to 71% on the 14th." |
| A bug report describing one bad output in three paragraphs. | "Another gate failure case, trace 2b95e07f." |
| "Did the prompt change help?" | "Wrong output format dropped nine points and nothing else moved." |
| Onboarding an engineer by having them read old incidents. | Onboarding them by having them read the code list and label ten traces. |
That vocabulary outlives individual people. It is the part of this work that keeps paying after the person who started it has moved to another team.
Calibration: the exercise worth doing
Calibration means two or more people coding the same traces independently and then comparing. Disagreement is not a failure of the exercise, it is the entire output of it.
- Pick ten traces nobody has codedInclude a few that are clearly fine. Agreement on negatives matters as much as agreement on positives.
- Code them independentlyNo discussion first. Discussion before coding produces agreement that is not real.
- Compare, one trace at a timeFor each disagreement, ask which of three things happened: different reading of the same evidence, different evidence entirely, or different definition of the code.
- Fix the definition, not the peopleAlmost every disagreement worth acting on is a scope problem. Rewrite the code name until both readings become impossible. Do not average two labels into a vaguer one.
- Re-code the disputed tracesOnce the name is fixed, go back and apply it to the traces you disagreed on. If it still does not settle them, the name is not finished.
If two people who know the product cannot apply a code to the same ten traces and agree, the code is not ready to monitor. Send it back to axial coding.
A workable division of labour
| Who | Does | Time |
|---|---|---|
| Domain expert | First open coding pass on a sample. Names what is wrong in domain terms. | One or two sittings, then occasional |
| Engineer | Second pass for mechanical failures, checks that the evidence points at real operations, owns fixes. | One sitting, then per release |
| Both together | Axial coding and calibration. This is the meeting that matters. | An hour, once, then monthly |
| Whoever owns quality | Chooses the monitored codes, watches the dashboard, calls regressions. | Minutes daily |
Everyone you nominate works in the same workspace, on the same traces, the same codes and the same verdicts. Nobody has a private copy of the taxonomy, because a private copy is how two people end up meaning different things by the same code name.
What goes wrong
- The hero. One person codes everything, and the taxonomy becomes unreadable to everyone else. When they leave, the numbers keep updating and nobody knows what they mean.
- Coding by committee. Five people arguing over a name before anyone has read twenty traces. Read first, name second.
- Silent re-scoping. Somebody widens a code's meaning during a review without saying so, and the chart shows a regression that is really a definition change.
- Monitoring what nobody owns. A code with no owner produces a line nobody looks at, and it uses one of a limited number of slots.
- Treating the judge as the authority. The judge is a scaled copy of your team's judgement. When it disagrees with a careful human read, the human is the ground truth and the code or its retrieval needs work.