EXP-003: The ANCOVA Bias Audit

In EXP-001, ANCOVA estimated a treatment coefficient of ~1.07 (cumulative change score) while the LMM estimated ~0.11/month (slope rate) for the same simulated data. The true treatment effect was a 50% slowing of progression in slow progressors. These coefficients are on different scales — ANCOVA measures total change while LMM measures per-month rate — and the apparent discrepancy demanded proper explanation.

The Board Room identified two competing hypotheses. First: ANCOVA's bias comes from informative dropout — sicker patients drop out, biasing the survivors upward. Under this hypothesis, removing MNAR dropout should eliminate the bias. Second: ANCOVA targets a fundamentally different estimand — the survivor average treatment effect — and the bias is structural regardless of the dropout mechanism.

This experiment disambiguates between these two hypotheses by sweeping the MNAR severity from 0 (pure MAR) to 1 (fully informative) and observing whether the bias persists, grows, or vanishes.

Data-Generating Process. Same three-class ALS trajectory model as EXP-001 and EXP-002: slow, fast, and stable-then-crash progressors with informative dropout. N=200 per arm, 200 simulations per MNAR level.

MNAR gradient: Six levels — 0.0, 0.2, 0.4, 0.6, 0.8, 1.0 — controlling the degree to which dropout depends on unobserved disease severity. At 0.0, dropout is purely MAR (depends only on observed covariates). At 1.0, dropout is fully informative (sicker patients are much more likely to drop out).

Two scenarios: Null (no treatment effect) and class-specific (50% slowing in slow progressors only — same as EXP-001).

Fig. 1 — Treatment effect coefficients across the MNAR gradient. ANCOVA (orange) and ANCOVA-12mo (red) report larger coefficients because they measure cumulative change scores, while LMM (blue) measures per-month slope rate. These are different scales — the gap reflects estimand differences, not a multiplicative bias. MNAR increases all estimates modestly.

Fig. 2 — Statistical power (treatment scenario) and Type I error (null scenario) across the MNAR gradient. All methods control Type I error under the null. Power differences reflect estimand differences, not error rate inflation.

Fig. 3 — Dropout rates across the MNAR gradient. Dropout increases modestly from ~40% to ~43% as MNAR severity increases. The collider bias from conditioning on survival grows with MNAR severity, as expected from the analytical derivation (R ≈ 1.36 at full MNAR).

Click each scenario below to see detailed results tables.

Finding 1: ANCOVA and LMM target different estimands. ANCOVA estimates a cumulative change score (1.07) while LMM estimates a per-month slope rate (0.11/month). These are on fundamentally different measurement scales. On comparable scales (LMM × 12 months ≈ 1.29 cumulative), the methods give similar magnitude estimates under MAR. The apparent "discrepancy" from EXP-001 was a scale comparison error, which we correct here.

Finding 2: Collider bias inflates ANCOVA under MNAR by ~36%. ANCOVA-12mo (survivors only) gives 1.32 under MAR — nearly unbiased against the true cumulative effect of 1.35. Under full MNAR, this rises to 1.89 (~40% inflation). The more aggressively you condition on survival under informative dropout, the worse the collider bias.

Finding 3: MNAR drives the real bias. Under MAR (MNAR=0.0), ANCOVA-12mo is essentially unbiased (0.97× truth). Under full MNAR, it inflates to 1.40× truth. The collider bias is a direct consequence of conditioning on a post-treatment outcome (survival) when dropout is informative.

Finding 4: LMM stays robust across the entire gradient. The LMM coefficient ranges from 0.107 to 0.124 across all MNAR levels — essentially flat. Converted to cumulative scale (×12), this gives 1.29–1.49, showing ~10% inflation at full MNAR. LMM is more robust to informative dropout but not immune.

Finding 5: Under the null, all methods are unbiased. Type I error is controlled near 5% for all methods at all MNAR levels. The estimand mismatch only manifests when there's a real treatment effect — meaning ANCOVA doesn't produce false positives, it produces estimates on a different scale and with collider bias under MNAR.

Finding 6: The oracle confirms the true effect size. The oracle's coefficient (~0.25/month) is consistent across all MNAR levels, representing the actual within-class treatment effect. The LMM's lower coefficient (0.11/month) reflects the diluted population-level estimand. ANCOVA's coefficient (1.07 cumulative) is on a different scale and should not be directly compared to the per-month slope estimates.

The standard primary analysis in most ALS clinical trials — some form of ANCOVA on change from baseline — targets a different estimand than longitudinal models when there's differential survival between treatment arms. ANCOVA estimates the survivor average treatment effect; LMM estimates the population average effect. Under informative dropout, conditioning on survival introduces collider bias of approximately 36%, inflating the treatment effect estimate relative to the population-level truth.

This matters for interpretation. A trial using ANCOVA under MNAR conditions will overestimate the treatment effect by ~36% compared to the population-level estimand — enough to influence regulatory decisions and set misleading expectations for future trials. The issue isn't that ANCOVA is "wrong" — it answers a different question — but researchers must be explicit about which estimand they're targeting.

The fix is straightforward: use longitudinal models that don't condition on post-randomization outcomes. The LMM does this naturally by using all available timepoints under the MAR assumption. More sophisticated approaches (pattern-mixture models, joint models for longitudinal and survival data) can handle MNAR explicitly.

The deeper lesson: the estimand defines the analysis, not the other way around. If you want the population-level treatment effect, don't use a method that conditions on a post-treatment outcome (survival). If you genuinely want the survivor average effect, use ANCOVA — but know what you're estimating and report it honestly.

Builds on: EXP-001: The Cost of Linearity — first identified the ANCOVA–LMM coefficient discrepancy (now understood as a scale difference). EXP-002: The Oracle Haircut — established the two-stage LCMM pipeline as a viable alternative.

Requested by: Board Room Session 004 — "Is the ANCOVA discrepancy a genuine estimand mismatch or an artifact of MNAR?"

Answers: It's an estimand mismatch compounded by collider bias. ANCOVA targets the survivor average treatment effect. Under MNAR, conditioning on survival inflates this by ~36%. Under MAR, ANCOVA-12mo is nearly unbiased on its own scale.

Next: Formal estimand framework (ICH E9 R1 addendum). Sensitivity analyses under explicit MNAR assumptions. Joint modeling of longitudinal outcomes and survival.