LLM application release checklist: evidence before traffic

Why this matters

The model passed, but the release was still not ready#

A candidate can pass its model evaluations and still be unsafe to ship: the artifact is not identifiable, a secret entered the build, the critical journey was never exercised, alerts are absent, or nobody has rehearsed rollback. None of those failures appears in an answer-quality score.

A release checklist exists to close this wider operational gap. Unlike a quality gate, which decides whether behavior meets an evaluation policy, this article follows the whole candidate from defined contract to security, observability, ownership, recovery and traffic promotion.

The missing artifact, secret, journey, alert, and rollback path all point to the same first step: define what “ready” means.

01 · Behavior contract

Define success before running the model#

Write the task, allowed tools, output schema, refusal behavior, latency budget, and critical failure modes before selecting a metric. A vague requirement such as “be helpful” cannot produce an auditable gate.

Separate invariants from preferences. Schema validity, authorization and prohibited tool use are deterministic blockers. Tone, relevance and completeness may require rubrics or human review. Mixing both into one score hides severe failures.

A defined contract tells the team what to test; the evaluation set checks whether the candidate actually meets it.

02 · Evaluation evidence

Test known cases, boundaries, and recovery paths#

Build a versioned set with ordinary journeys, difficult boundaries, adversarial inputs and previously observed failures. Record dataset, prompt, model, tool and rubric versions with every run.

Do not promote on pass rate alone. Inspect failures by capability and risk slice, compare against the last approved release, and treat missing coverage as uncertainty rather than success.

Ordinary and boundary cases cover behavior, but high-impact security failures need their own stop condition.

03 · Critical blockers

Give security and authorization their own gate#

The OWASP Top 10 for LLM Applications describes risks that an average quality score cannot neutralize. Prompt injection, excessive agency, sensitive information disclosure and insecure output handling need explicit tests and owners.

A critical invariant should fail closed: the release stops even if every soft-quality indicator improves. Document the exception process so urgency cannot silently redefine the policy.

After critical blockers pass, the team still needs production signals that do not create a new privacy risk.

04 · Operations

Observe decisions without creating a data leak#

Trace model and tool operations, latency, errors, token use and evaluation verdicts with low-cardinality identifiers. Capture message content only when necessary, with redaction, access control and retention limits.

Connect the gate to a named release owner and a rollback signal. An alert without an owner is a log; a rollback without a tested path is a wish.

Those signals are useful only when they feed one explicit release outcome with a named owner.

05 · Release decision

End with PROMOTE, HOLD, or ROLLBACK#

Use the three outcomes as operational actions. They state what happens to traffic, not whether the model is “good” in the abstract.

1. P

   PROMOTE

   Required evidence is complete, critical invariants pass, and residual risk is accepted by the owner.

2. H

   HOLD

   Evidence is absent, conflicting, outside the validated range, or awaiting a human decision.

3. R

   ROLLBACK

   Production evidence crosses a predefined reliability, safety, or business boundary.

{
  "schema_version": "release-decision@1",
  "candidate": "release-2026-07-20.1",
  "artifact": "image@sha256:<digest>",
  "evidence": ["tests", "evals", "security", "operations"],
  "owner": "release-owner",
  "decided_at": "2026-07-20T14:30:00Z",
  "verdict": "HOLD",
  "reason": "candidate journey not verified",
  "rollback_target": "release-2026-07-19.3"
}

The checklist now brings the contract, evidence, blockers, operations, and owner into one pre-traffic review.

Pre-traffic checklist

Twelve questions before production traffic#

Answer each question for the exact artifact and revision being considered. A blank answer is missing evidence, not an implicit pass.

• ✓Is the behavior contract versioned?
• ✓Do schema, authorization, and policy invariants pass?
• ✓Does the set cover ordinary, boundary, adversarial, and recovery journeys?
• ✓Were previous production failures replayed?
• ✓Are judge rubric and model versions pinned?
• ✓Were high-risk slices inspected separately from averages?
• ✓Does each critical risk have a hard blocker?
• ✓Are prompts and responses redacted or capture-disabled by default?
• ✓Can a run be reproduced from stored versions and identifiers?
• ✓Is there a named release owner?
• ✓Is rollback defined and rehearsed?
• ✓Is the final decision and its evidence recorded?

The result is a release record whose green status can be explained and reversed.

Conclusion

A green pipeline is not the same as a ready release#

Release readiness is not the sum of checked boxes. It is a reviewable argument that the expected behavior is defined, critical risks are blocked, representative journeys were exercised, operations can observe the result, and a named owner accepts the remaining uncertainty.

Before sending traffic, ask one final question: can another operator identify the exact artifact, reproduce the decisive evidence, explain why the candidate was promoted, and return to a known-good revision? If any part is missing, record HOLD instead of allowing an empty field to become a silent pass.