EXP-004: K-Selection — BIC vs ICL

In EXP-002, the two-stage LCMM pipeline recovered most of the oracle's power advantage — but BIC consistently selected K=4 instead of the true K=3. Session 005 replaced BIC with ICL (BIC plus an entropy penalty that rewards well-separated classes), reasoning that the entropy correction would prevent spurious class splitting.

This experiment tests that hypothesis directly: does ICL outperform BIC for K-selection, and if not, what's actually driving the over-selection?

Data-Generating Process. Same three-class ALS trajectory model: slow, fast, and stable-then-crash progressors. N per arm of 100, 200, and 400. 200 simulations per scenario per sample size.

K-selection: Fit LCMM for K=1 through K=5. Select K by BIC (lowest) and ICL (lowest). Quality filters: minimum class size >5%, mean posterior probability >0.70. EM: 50 iterations, 3 random restarts, linear trajectories.

Two scenarios: Null (no treatment effect) and class-specific treatment (50% slowing in slow progressors — same as EXP-001 through EXP-003).

Finding 1: Under null conditions, both BIC and ICL recover K=3 perfectly. 100% recovery rate at all sample sizes (N=100, 200, 400 per arm). The EM algorithm, quality filters, and model selection machinery all work correctly. The three-class structure is identifiable.

Finding 2: Under class-specific treatment, both criteria fail similarly. K=3 recovery drops to 38–44%. K=4 is the most common selection. ICL's entropy penalty provides marginal benefit at N=100 (44% vs 38%) but zero benefit at larger N. The problem is not criterion sensitivity — it's that the treatment creates a genuine 4th trajectory pattern.

Finding 3: The K=4 selection is not an error. This is the critical insight. When slow progressors receive treatment, their trajectories diverge from untreated slow progressors. The mixture model correctly identifies this as a distinct class. It's doing exactly what it should — detecting real structure in the data. The "over-selection" in EXP-002 was the model responding faithfully to treatment-induced heterogeneity.

Finding 4: ICL is not the fix. Session 005 replaced BIC with ICL on the premise that entropy correction would prevent spurious splitting. It doesn't — because the splitting isn't spurious. The entropy of a 4-class model that captures a real treatment × class interaction can be just as clean as a 3-class model.

The two-stage LCMM pipeline from EXP-002 needs a structural correction. Class enumeration must happen on pooled data — both arms combined — without treatment as a covariate. Otherwise, the model will detect the treatment effect as additional trajectory heterogeneity and inflate K.

This is not a flaw in BIC, ICL, or the EM algorithm. It's a design principle: discover structure first, then test hypotheses within that structure. If you let the hypothesis (treatment) contaminate the discovery step (class enumeration), you get circular reasoning — the treatment effect bootstraps itself into the class structure.

The practical implication for the ALS pipeline: Step 1 fits LCMM on all patients ignoring treatment assignment. Step 2 assigns patients to trajectory classes. Step 3 tests treatment effects within each class. This clean separation is what makes the two-stage approach valid.

Builds on: EXP-002: The Oracle Haircut — identified K over-selection as a problem. Board Room Session 005 — locked ICL as the replacement for BIC.

Answers: Does ICL fix BIC's K over-selection? No — the over-selection is treatment-induced class splitting, not an information criterion flaw.

Pipeline update: LCMM class enumeration should be performed on pooled data without treatment covariates, then treatment effects estimated within discovered classes.