---
name: proving-two-ci-arms-populations-are-comparable
description: A base-vs-head CI pair means nothing until the arms' selected test populations are proven comparable — and five different cheap checks each fail at it in their own way.
metadata:
  type: feedback
---

releases#294, 2026-09-10. A base-vs-head arm pair produced a 35-test gap on Linux Phase A, and
"is that just added tests, or did something RELOCATE between phases?" took five attempts because
every cheap check is wrong in its own way. All of it is recoverable from RETAINED CI LOGS with no
build — the logs enumerate every executed test, which is better than a listing because it is what
actually ran (todlando's pointer; check artifacts before scheduling a build).

**The five traps, in the order I hit them:**

1. **A static `#[test]` count is not the selected population.** cfg gates, `#[ignore]`, doc-tests
   and module-level gates are approximated at best. It supports direction and magnitude, never the
   figure.
2. **A NET count conceals offsetting changes.** My +38 static hid 6 REMOVALS. hertz: net counts
   cannot exclude relocation, full stop. Diff IDENTITIES, not totals.
3. **VALIDATE THE EXTRACTOR AGAINST THE SUMMARY COUNT BEFORE TRUSTING ONE DIFF FROM IT.** Mine
   produced 2361 of 3360 because nextest space-pads the counter (`( 626/3360)`) and my regex
   rejected it. A second pass was still 2 short because 2 tests print only a `LEAK` verdict line
   and no `PASS` — a `PASS|FAIL` extractor drops them from ONE arm and invents that many phantom
   head-only rows. deployah's independent extractor had exactly this bug; his HEAD number was
   right, which is what made it invisible.
4. **A name search is CIRCULAR for the relocation question** (todlando). "This arrival's name is
   absent at base" proves a new IDENTITY, not new BEHAVIOUR — and a renamed-and-moved test also has
   a name absent at base. It cannot exclude the case it is aimed at.
5. **Disjoint commits do not exclude a move SPLIT ACROSS commits** (hertz), and worse — I asserted
   disjointness after checking ONE pairing of six. Two of the six actually shared a commit with a
   PhaseB arrival. Verifying one instance and generalising to the population is the recurring shape;
   see [[an-identical-treatment-controls-the-input-not-the-effect]] and
   [[a-detector-must-pass-a-control-built-from-what-motivated-it]].

**What actually works, cheapest first.** nextest partitions by **BINARY**, not by test — the HEAVY
filter names binaries. So: (a) diff BINARY-level phase membership across arms; a rename cannot hide
a phase change because the check never reads a name. (b) Diff test IDENTITIES per phase. (c) For
each arrival into the phase in question, read its own commit's full REMOVAL set. (d) For removals
still unmatched across the range, compare BODIES/assertions. Step (a) alone closes every test whose
binary stayed put; the rest closes the renamed-and-moved residue.

**Know when to stop.** hertz offered the off-ramp explicitly: keep the narrower proven claim and
stop, because the census gates no RCA. Take it once the claim is established past what the decision
needs — the temptation to close the last epistemic 1% is where this burned the most rounds.