Module 0.1 · the method

Find the failures that matter first

Imagine that your AI assistant gives a poor answer. Was the source missing, did its knowledge-base search fail, or did the assistant refuse a reasonable request? Before you build a check, look at the records your system already produced and find the patterns. Software can group similar failures; you decide which groups matter enough to act on.

what you'll be able to do By the end of this module you can map your system's real failure modes with a discovery judge and decide which ones need a release-blocking check (a gate).

In this course, a harness is the repeatable release-checking setup: it runs selected checks, keeps the evidence and applies the consequence. This first module supplies its starting map of failures.

The trap: shipping judges for failures you never mapped

A team may begin with the failure it has heard about most, such as a made-up answer. Its own records may show a different problem: perhaps tool calls fail silently much more often. If the team writes checks before looking at those records, it can spend time on the wrong problem. Start by finding the failure patterns that actually occur. Then compare how often they happen and how much harm they cause.

the move The recorded input, output and tool activity from one run form a trace. A discovery judge reviews a sample of these traces and groups similar failures. It does not approve or reject a release. It returns a short list of failure types, their counts and an example of each, so a person can choose what to investigate first.

This is where the human belongs

When there are more traces than one person can reasonably inspect, the discovery judge handles the first sorting pass. A person then reviews the much shorter list: which groups are real problems, which are labeling mistakes and which can wait? The software handles volume. The person remains responsible for deciding what should block a release.

people remain responsible for the standard The harness automates repetitive work: reading a defined sample of traces and running the selected checks after each change. You still decide which groups represent real failures, which ones matter, and what evidence should block a release. You also review uncertain or disputed cases.

our running example — the retrieval-grounded support bot Through modules 0.1→0.9 we'll use one fictional system so the method forms a single cumulative story: a support bot that searches a knowledge base before answering customer questions from a knowledge base. This is the system whose traces you're about to mine. Its failure modes — answers without citations, knowledge-base searches that fail without warning, and unnecessary refusals — are the ones the discovery judge below surfaces. In 0.3 you'll write judges for them; in 0.5 you'll gate on them; in 0.9 you'll wire the whole harness around them. (A pattern, not a benchmark — you build and measure your own.)

before you read on — which records should a discovery judge examine first?

A sample of production traces (input + output + any tool/error metadata), not a curated eval set. Discovery is about how the system actually fails in use; a hand-picked set can distort the apparent frequency of different failure types.

A discovery judge, concretely

It ingests traces, proposes failure-mode labels, assigns each trace to a mode (or "clean"), and returns the taxonomy with counts and one example per mode — the evidence you'll skim.

The ranking starts the decision: compare how often each failure occurs with the harm it can cause. A rare, severe failure may deserve a gate before a frequent, low-impact one.

def discover_failures(traces, discovery_judge):
    labeled = []
    for t in sample(traces, n=500):          # judge reads volume; you won't
        mode = discovery_judge(t)             # {"mode": "silent_tool_error", "evidence": span}
        labeled.append(mode)
    taxonomy = cluster(labeled)               # ranked: mode -> (count, example)
    return taxonomy                           # YOU review this — 12 rows, not 10k traces

Now — and only now — you pick the top modes and, in the next modules, turn each into a judge with a verdict, evidence and a threshold. Discovery ranks the work; the harness does the work.

counts are the start, not the answer A ranked taxonomy tells you how often each mode happens. The next step is stratified metrics, meaning one separate scorecard for each failure group instead of one blended average. Make the scorecards by (1) putting cases in their failure groups, (2) tallying the evaluator’s correct and incorrect decisions within each group, and (3) comparing those group-level results before choosing a gate. For example, 90% overall accuracy can conceal that the evaluator catches only 4 of 10 missing-citation cases. Precision asks, “when the evaluator flags a failure, how often is it right?”; recall asks, “of the real failures, how many did it catch?” A true positive is a caught real failure, a false positive is a false alarm, and a false negative is a missed real failure. Measure precision/recall per cluster, then perform an error analysis on the top clusters: are these the same root cause split into three labels, or three genuinely different failures? To do this, make one row per failure group, write the evaluator’s true positives, false positives and false negatives in that row, then compare the resulting precision and recall before choosing a gate. A frequent cluster can still be low-severity; a rare one can be the one that loses customers. Discovery gives you the map; stratification tells you which roads to pave first.

case in practice In a solo-run video pipeline the same move works: a discovery pass surfaces the failure modes that actually recur — lip-sync drift, persona breaks — and one operator decides which become hard gates. A person defines the standard; automated checks apply it repeatedly and send uncertain cases back for review. (This is a pattern, not a benchmark — you build and measure your own in the lab.)

why discovery is a recurring ritual, not a one-off — criteria drift A discovery judge is itself a large language model (LLM): a model prompted to classify or explain text. LLM judges do not hold their criteria fixed. Shankar et al., "Who Validates the Validators?" (arXiv 2404.12272, UIST '24), document criteria drift: LLM-as-judge evaluators inconsistently apply and shift their rubric while annotating, drifting out of alignment with the human preference they were meant to encode. The consequence is direct — you cannot run discovery once and trust the taxonomy forever. Re-run it periodically; treat the ranked clusters as a snapshot that can become stale.

why cluster failures before writing any judge?

Because observed examples show which failure types deserve investigation. Without that evidence, a team may build checks for a memorable but low-priority problem. Repeat discovery periodically because traffic and judge criteria can change.

a cluster has 40% of failures but is low-severity; a 3% cluster loses customers. Which do you gate first?

Neither decision follows from the count alone. Discovery gives you the map; you still pair frequency with severity (and with stratified precision/recall per cluster) to decide what earns a gate. Counts are the start, not the verdict.

no-code decision exercise A scheduling assistant gives a wrong appointment time in 8% of sampled traces and exposes a private note in 1%. Without code, choose the first failure to gate and state the evidence you would record.

compare your decision with the feedback

Gate the exposed private note first. Its lower frequency does not erase the higher harm; record the trace, the exposed span, and the release decision. Reject the plausible alternative “gate the wrong time because it is more frequent”: frequency ranks investigation, but it does not by itself set severity.

how to read the result A repeated failure cluster supports investigating that workflow first because it appears in several reviewed cases. It does not estimate prevalence or prove a root cause; confirm the pattern with broader cases before turning it into a release rule.

▶ optional technical extension · runnable lab

failure-discovery

Point a discovery judge at a folder of traces and get back a ranked failure taxonomy with counts and examples. Run it on the included sample, then swap in your own traces.

labs/failure-discovery/ · python discover.py

key takeaways

Map observed failures before building judges so the first checks address the problems that matter.
A discovery judge groups a defined sample of traces; you review the groups and their examples.
Counts are the start — add per-cluster (stratified) metrics and an error analysis before you decide what to gate.
The human sets the bar (what's real, what matters); the judges apply it to everything.