Primer · before you start

How to read the evidence behind an AI release

You do not need to know Python, statistics, or how to call a language model. Start with a familiar problem: an AI answer looks convincing, but you still need to decide whether it is safe and useful enough to release. This primer gives you the few plain-language ideas needed to examine that decision.

who this is for You have seen an AI assistant produce an answer, but you may never have written code or evaluated an AI system. If words such as "trace", "RAG", "judge", or "gate" are new, you are in the right place: each term is introduced only after the problem it helps solve.

1 · Define the workflow before the judge

Start with a familiar task: an assistant answers a customer's refund question. Before checking the answer, write down what the assistant is allowed to do, which policy it must use, what a useful result looks like and when a person must take over. This short description is the workflow contract. It keeps a well-written answer from being mistaken for a successful task.

what you'll be able to do Frame one AI workflow as an inspectable contract before you write its rubric, judge, or gate.

Work: ____________________

Actor: ____________________

Context: ____________________

Allowed action: ____________________

Observable outcome: ____________________

Failure / exception: ____________________

Human owner: ____________________

Evidence retained: ____________________

worked example For the refund assistant: the work is “answer a refund question”; the required context is the current refund policy; the allowed action is to explain the policy, not issue money; success means a cited answer that matches the policy; and a person takes over when the policy does not cover the case. Writing these fields first matters because the evaluator can now check a real outcome instead of judging whether the wording merely sounds helpful.

a green gate is not the outcome The gate proves that an output met a versioned policy. Only the observed result after delivery tells you whether the workflow worked. Keep both: gate evidence for control, outcome evidence for learning.

which workflow-contract field stops “good writing” from becoming the success criterion?

Observable outcome. It names the result outside the model response. The judge can test a proxy before release; the real outcome is observed after the action and feeds the next calibration.

2 · An LLM call, and what a "trace" is

An LLM call is a function: you send a prompt (the input messages), the model returns a completion (the output text). That's it at the simplest level. A trace is the recorded version of that call — the input, the output, plus the things you need to debug and measure it: how long it took (latency), how many tokens it used, what tool calls happened, what context was retrieved. Every eval in this course runs on traces. No trace → no eval.

# the simplest LLM call, and what gets recorded as a trace
out = llm.chat(messages=[{"role": "user", "content": "Summarize the meeting"}])

trace = {
    "input":  "Summarize the meeting",
    "output": out.text,
    "latency_ms": out.latency,
    "tokens":    out.usage,
}
# the course's judges read fields like these — that's why traces come first

3 · RAG in two lines

Suppose the assistant needs the current refund policy before it can answer. Your software first retrieves the relevant passage and includes it with the question. This pattern is called RAG (Retrieval-Augmented Generation). It gives the model a source to use, but does not guarantee that the source is relevant or that the answer follows it. That distinction matters at release time: a fluent answer must fail when it cites the wrong policy or contradicts the retrieved passage. Several modules show how to check those two failure points.

the one diagram to hold in your head First the system produces an answer. Then a repeatable check compares it with the recorded evidence. A required failure blocks the version; a repairable failure can trigger another attempt. Later sections name these parts judge, gate and retry. For now, remember the order: produce, inspect, decide, then act.

4 · Optional technical extension: the Python used in the labs

You can complete the core reasoning path without Python. If you choose to run the optional labs, the course uses a small, consistent subset:

Functions and arguments — def judge(case, llm): and keyword args like --threshold 0.7.
Dataclasses — a typed struct, e.g. @dataclass class Verdict: passed; evidence; confidence. It's just a bag of named fields.
Dicts and lists — the trace is a dict; a dataset is a list of dicts.
Running a script — python some_lab.py --input samples/. Every lab runs offline with no API key.

If these lines look unfamiliar, continue with the no-code exercises. Return to this technical extension only if you want to implement the method.

5 · Judge, gate, harness, eval — the words the course uses

Four terms recur and they're not interchangeable:

A judge is code that inspects an output and returns a verdict + evidence + confidence. (Module 0.3.)
A gate is a decision that blocks or allows a release based on a judge's verdict. (Module 0.5.)
A harness is the repeatable system that runs the cases, calls the judges, applies the gates, and records the evidence. (Module 0.9.)
An eval is the broader practice of measuring quality systematically — traces, golden sets, drift. (Track EVAL.)

That's the entire vocabulary. The rest of the course builds on these five ideas — workflow contract, call/trace, RAG, the Python subset, and judge/gate/eval — without re-explaining them.

start with a compact human-reviewed set One more thing, because it surprises people: this course never asks you to hand-label thousands of examples. From Module 0.4 onward you will use known-answer cases and compact human-reviewed reference sets to test a judge. A jury can add repeated independent readings and reveal disagreement, but agreement among judges is not automatically ground truth.

what is a "trace", and why does the course start there?

A trace is the recorded input, output, and intermediate steps of one LLM/agent run. You can't judge, gate, or eval what you didn't record — so every module downstream assumes you have traces to read.

"judge", "gate", "harness", "eval" — say each in one line

Judge: code that inspects an output and returns a verdict + evidence + confidence. Gate: a go/no-go decision that blocks or promotes based on judges. Harness: the repeatable system that runs the cases, calls the judges, applies the gates, and records the evidence. Eval: running judges over a dataset to measure quality. The course builds them in that order.

what role does the judge play in the workflow contract?

Between the candidate output and the gate. It checks the output against versioned context and policy before an action proceeds. Exceptions return control to the named human owner; observed outcomes remain a separate signal that recalibrates later cases and gates.

key takeaways

Define work, action, exception, human ownership, and observable outcome before you write a judge.
An LLM call is a function; a trace is the recorded call with the fields a judge needs. No trace → no eval.
RAG fetches relevant source material before the model answers. It can reduce unsupported answers when missing or poorly retrieved context is the cause, but it does not prevent every kind of error.
The no-code path teaches the decision; Python is an optional extension for implementing it.
Judge (inspects) · gate (blocks/allows) · eval (measures systematically). The whole course is a generate→judge→gate→retry loop.