Choose a statistical method without starting from the test

A concrete reason to pause

Thirty records, one vague question, several incompatible answers#

AURORA-30 is a wholly synthetic teaching dataset: 30 fictional records constructed for methods training. It contains no real patients, institutions, interventions, or clinical results, and nothing calculated from it is clinical evidence. In the scenario, adults are followed for 30 days after a standardized discharge program or usual care. A colleague sends baseline and day-30 symptom scores and asks, “Did the program help?” It is tempting to open a test menu and compare whichever two numeric columns are visible.

But the rows do not decide the question. A between-group contrast at day 30 compares program with usual care; a baseline-to-day-30 contrast asks whether the same people changed over time; a between-group comparison of change asks whether those trajectories differed. All three can produce correct arithmetic and still support different sentences in the report. Without an explicit trail, the analyst can begin with one question and publish the answer to another without any software error announcing the switch.

To prevent that silent switch, follow the decision in order: first preserve the question, then let the design restrict the method.

Why evidence comes before technique

A result must carry the path that makes it interpretable#

In health research, the practical risk is not limited to choosing the wrong test from a menu. The unit of analysis, time point, missing-data handling, target effect, or primary claim can shift while the work is underway. A precise p-value cannot repair that drift. It may only make the wrong scientific answer look more authoritative.

An evidence trail keeps the question, data decisions, estimate, assumptions, diagnostics, and limits connected. It lets a collaborator or reviewer reconstruct the route, challenge a mismatch, and distinguish a prespecified conclusion from an exploratory finding. Technical depth matters here because every number inherits meaning from those decisions—not because complexity is valuable by itself.

That path begins with the quantity the study is trying to estimate—not with the name of a test.

Start before the software

Write the question and target effect#

A question such as “does the intervention work?” is not yet an analysis contract. Name the population, intervention or exposure, comparison, outcome, time point, and the effect you intend to estimate. A mean difference at day 30, an odds ratio over follow-up, and a survival contrast are different targets even when they come from the same dataset.

The complete numerical target—population, comparison, outcome, time point, and effect—is called the estimand. It states exactly what the analysis should estimate. ICH E9(R1) formalizes this connection for clinical trials; outside that scope, the same discipline remains useful, but it does not turn an observational comparison into a causal effect.

Once the target is explicit, the next question is who contributes each measurement and which observations are linked.

The design changes the method

Count units, not rows#

In AURORA-30, people in different care groups are independent at day 30, while baseline and day-30 scores from the same person are paired. The spreadsheet contains both structures.

1. 01

   Independent groups

   Different people or independent units contribute to each group.

2. 02

   Paired observations

   The simplest repeated structure: commonly two linked measurements from the same person, or one measurement from each member of a matched pair.

3. 03

   Repeated measures

   The broader case: multiple times or conditions belong to the same unit and must be analyzed with their correlation preserved.

4. 04

   Clusters

   People may also be nested in clinics, households, wards, or sites.

With the unit and dependence identified, these choices can be recorded before the results make one route look more attractive.

The decision record

Turn the hidden choices into a one-page analysis contract#

Now the contract has a reason to exist: it is a compact, verifiable memory of the decisions that prevents the original request from changing silently while the analysis is built. Record the outcome scale, number of groups or times, dependence, target effect, sample-size justification, missing-data plan, family of claims that may require multiplicity control, diagnostics, and reporting package. Separate primary, prespecified claims from exploratory comparisons. Empty fields are useful because they expose decisions that have not been made.

The analysis contract is not a legal document, does not choose a method automatically, and is not paperwork to complete after the result. It narrows the defensible options before execution and gives a reviewer enough context to challenge a mismatch—for example, an independent-samples test applied to repeated measurements from the same participant.

{
  "study_id": "AURORA-30",
  "question": {
    "population_id": "eligible_participants",
    "outcome_id": "symptom_score_day_30",
    "comparison_ids": ["discharge_program", "usual_care"],
    "estimand_id": "mean_difference_day_30_program_minus_usual_care"
  },
  "structure": {
    "unit": "participant",
    "between_groups": "independent",
    "baseline_followup": "paired",
    "missingness_reviewed": true
  },
  "planning": {
    "sample_size_justification_id": "precision_for_target_effect",
    "primary_claim_ids": ["day_30_mean_difference"],
    "exploratory_comparisons_labeled": true
  },
  "report": {
    "required": ["estimate", "confidence_interval", "diagnostics", "limitation"],
    "causal_claim_allowed": false
  }
}

import copy
import json
import sys

REQUIRED_REPORT = {"estimate", "confidence_interval", "diagnostics", "limitation"}

def validate(contract):
    problems = []
    question = contract.get("question", {})
    structure = contract.get("structure", {})
    planning = contract.get("planning", {})
    report = contract.get("report", {})

    for key in ("population_id", "outcome_id", "comparison_ids", "estimand_id"):
        if not question.get(key):
            problems.append(f"question.{key} is required")
    if structure.get("unit") != "participant":
        problems.append("structure.unit must preserve the participant")
    if structure.get("between_groups") != "independent":
        problems.append("the day-30 group contrast must record independent groups")
    if structure.get("baseline_followup") != "paired":
        problems.append("baseline and follow-up must remain paired within participant")
    if structure.get("missingness_reviewed") is not True:
        problems.append("missingness review is required")
    if not planning.get("sample_size_justification_id"):
        problems.append("planning.sample_size_justification_id is required")
    if not planning.get("primary_claim_ids"):
        problems.append("planning.primary_claim_ids must name at least one primary claim")
    if planning.get("exploratory_comparisons_labeled") is not True:
        problems.append("exploratory comparisons must be labeled")
    missing_report = REQUIRED_REPORT - set(report.get("required", []))
    if missing_report:
        problems.append("report.required lacks: " + ", ".join(sorted(missing_report)))
    if report.get("causal_claim_allowed") is not False:
        problems.append("report.causal_claim_allowed must remain false for this teaching case")
    return problems

def show(label, contract):
    problems = validate(contract)
    status = "HOLD" if problems else "READY_FOR_METHOD_REVIEW"
    print(f"{label}: {status}")
    for problem in problems:
        print(f"  - {problem}")

with open(sys.argv[1], encoding="utf-8") as source:
    supplied = json.load(source)

print(f"{supplied.get('study_id', 'UNNAMED')} | structural analysis-contract check")
show("supplied_contract", supplied)

pairing_probe = copy.deepcopy(supplied)
pairing_probe["structure"]["baseline_followup"] = "independent"
show("pairing_regression_probe", pairing_probe)

planning_probe = copy.deepcopy(supplied)
planning_probe.pop("planning", None)
show("missing_planning_probe", planning_probe)

causal_probe = copy.deepcopy(supplied)
causal_probe["report"]["causal_claim_allowed"] = True
show("causal_claim_regression_probe", causal_probe)
print("scope: structural planning aid; no method selection or clinical authorization")

AURORA-30 | structural analysis-contract check
supplied_contract: READY_FOR_METHOD_REVIEW
pairing_regression_probe: HOLD
  - baseline and follow-up must remain paired within participant
missing_planning_probe: HOLD
  - planning.sample_size_justification_id is required
  - planning.primary_claim_ids must name at least one primary claim
  - exploratory comparisons must be labeled
causal_claim_regression_probe: HOLD
  - report.causal_claim_allowed must remain false for this teaching case
scope: structural planning aid; no method selection or clinical authorization

The plan narrows the candidate methods; diagnostics then show whether the selected model is a reasonable description of the data.

Assumptions are design claims

Diagnose the model, not normality by ritual#

A large p-value from a normality test does not prove normality, and a small p-value does not automatically require a rank test. Consider the design, residual structure, sample size, outliers, skew, variance pattern, measurement scale, and what parameter the method actually estimates. When relevant to the selected model, use visual checks such as Q–Q and residual-versus-fitted plots as diagnostics—not as mechanical proof that assumptions hold.

Rank-based methods are not generic “tests of medians.” Depending on the method and assumptions, they may address distributional ordering, probability of superiority, or a location shift; a median contrast requires additional conditions. When assumptions are doubtful, consider a method that targets the effect more directly, a robust analysis, or a prespecified sensitivity analysis.

After checking those assumptions, the result still needs enough context to be interpreted without guesswork.

Interpretation package

Do not let the p-value travel alone#

For the chosen AURORA-30 question, a reader needs five pieces together. Removing any one of them changes what can be understood from the result.

1. 01

   Estimate

   State the effect on a scale the question can defend.

2. 02

   Interval

   Show the uncertainty around that estimate.

3. 03

   Diagnostics

   Record the checks that matter for the selected method.

4. 04

   p-value

   Use it only when a prespecified hypothesis test is relevant.

5. 05

   Limitation

   Name what the design, missingness, bias, or data cannot establish.

This package makes the analysis reviewable, but it does not give the analysis powers that the study design never had.

Reusable checklist

Twelve questions before naming the method#

Read these questions from top to bottom. Each answer becomes an input to the next, so the checklist is a path through the analysis rather than twelve independent boxes.

• ✓What is the question and estimand?
• ✓What population and unit of analysis are represented?
• ✓What is the outcome type and scale?
• ✓How many groups, times, or conditions exist?
• ✓Are observations independent, paired, repeated, or clustered?
• ✓How was the sample obtained, how was its size justified, and which biases are plausible?
• ✓Are missingness, outliers, units, dates, and coding reviewed?
• ✓Which assumptions are reasonable from design and data?
• ✓Was the analysis prespecified, which claims are primary or exploratory, and how many comparisons are planned?
• ✓Which method estimates the target effect directly?
• ✓Which effect, interval, and diagnostics accompany the result?
• ✓Which limitations prevent causal or broader claims?

If the twelve answers point to one coherent analysis, the method can finally be named and its result interpreted within the declared limits.

Conclusion

Preserve the question, not only the result#

The central lesson is not that every health analysis needs a more sophisticated method. It is that the same data can support several mathematically correct analyses while only one matches the question the study promised to answer. A defensible result therefore carries its route: question, unit, target effect, assumptions, decisions about the data, estimate, uncertainty, diagnostics, and limits.

Before naming a method, write one sentence stating who is compared, on which outcome, at what time, and through which effect. If another qualified person cannot reconstruct the same analysis from that sentence and its decision record, the work is not ready for a test menu. The contract earns its value by making that gap visible early—when the question can still be repaired without rewriting the result.