# RCA — golden r2 33273378238 RED 2 (Linux test, kitsubito) — doyle ruling, 2026-08-29 ~21:30Z Run at `e4f52047` (v0.66.0 shape), ref golden/semaphore-242-r2. 3157 run: 3153 pass (2 leaky), 4 fail, 1 skip. RED 1 (Windows n1-gate floor, 26.5→87.55 GB self-recovered) is deployah's, no RCA owed, cleared without intervention. ## Verdict: TEST DEFECT (readiness-detection), product not implicated. Close at named mechanism; r3 same-sha is the release leg, not the closer. ## The four (one mechanism) activity_link_push_e2e / endpoint_autostart_e2e / idle_edge_seal_e2e / idle_edge_drain_e2e — each panicked `PRECONDITION: brain never came up` after burning an identical, untuned 30s `wait_for_ready_pid` budget (50ms file poll for a `pid` key in `brain.ready`; knows nothing about brain-up). ## Named mechanism 1. **The panic message is false by construction.** Every cell's captured daemon stderr shows `BRAIN_UP` — the brain connected to its broker in 27–65 ms and latched its first write (`CONN_LIFECYCLE conn=1 role=brain event=write-start`). What timed out is the `brain.ready` FILE, not the brain. 2. **The silent window.** Between `BRAIN_UP` and the ready file, `run_brain` (brainproc.rs:194–274 at the measured sha) executes: `announce_coordinator_image` (sync broker round-trip, silent on success) → `resume_session_cursors` (sync round-trip, silent on the always-taken empty arm — daemon hosts no PTY sessions) → `write_ready`, whose `current_exe_hash()` first call `std::fs::read`s the entire debug-profile binary and SHA-256s it before the atomic write. Nothing in this window emits on the happy path. 3. **Stall point within the window: UNDISCRIMINATED, and bounded so.** Panels exclude the loud arms (CONN_WRITE_POISONED=0, CONN_WRITE_RETIRED=0, no transport close, no error match arms) — so the stall is (a) blocked on a broker reply READ (ms bound is write-side only) or (b) inside the exe-hash read. Log surface exhausted (deployah, grep of R2_lin_test.log); the two are not separable without instrumentation. Both are product code running SLOW under load, neither is a wedge; the split does not change the classification, and the fixup lane makes the window observable so the next occurrence answers in one grep. 4. **The load is in the record.** Three of the four brains issued first writes within 365 ms of each other (wall_ms 1788035430340/430667/430705) — three concurrent daemon boots, Phase A full-parallel, distinct brokers ⇒ BOX contention. The r2 shape bumped every workspace crate version (Cargo.lock), forcing a full workspace recompile where r1 rode a warm ec6da9b0 cache — cold page cache under recompile load is the plausible tail-widener (deployah arm 4, offered as lead; consistent, not separately proven). 5. **Code cause excluded by construction.** e4f52047 − ec6da9b0 = Cargo.toml, Cargo.lock, CHANGELOG; zero `.rs`. Same 4 cells GREEN at parent, same box, same job (r1 33268589586: 3157/3157; durations 21.5–43.2s vs 30s budget — deadline-adjacent even when passing). ## Rulings - **Structural presumption DISCHARGED** for these cells (pre-declared never-executed-cells tension): green-at-parent on byte-identical source is the "shown otherwise" the declaration anticipated. They execute and assert for real; these are load-tail flakes, not structural vacancies. - **r3 same-sha (`e4f52047`) on both boxes** — release leg. Warm e4f52047 cache now on kitsubito removes the unique load source; Windows floor self-recovered. - **hertz fixup lane (test-only, POST-cut, not blocking):** (1) correct the false precondition message (name the ready FILE and quote observed BRAIN_UP); (2) harden the 4 cells' precondition budget (untuned 30s under full-parallel Phase A is inside its own passing distribution). - **todlando thin-lane scope +1 (product, POST-cut):** breadcrumb(s) making the BRAIN_UP→ready window discriminable in one grep (round-trips vs exe-hash vs post-ready), via `emit_line_err!`. Rides with his delivery.rs:149,313 conversion. ## ADDENDUM — r3 outcome splits the ruling (2026-08-29 ~22:30Z) r3 (attempt 2, same sha, warm cache) = 3156/3157: idle_edge_seal, idle_edge_drain, endpoint_autostart PASS — **load-tail ruling CONFIRMED for those three** (recovered exactly as the self-discharging argument predicted). **activity_link_push_e2e FAILED AGAIN** — identical signature (BRAIN_UP 26ms, no poison/retire/error arms, full 30.062s budget burned), now cold AND warm, 4 recorded occurrences. A tail that survives removal of its supposed load is not a tail: this cell is CARVED OUT of the load-tail ruling, reclassified deterministic-and-unexplained, and the undiscriminated stall point (reply-read vs first-call exe-hash) is now gate-blocking, not a footnote. Ruling: the BRAIN_PHASE breadcrumb lane is PULLED AHEAD of the cut (was post-cut); local kitsubito repro with the instrumented build before any further golden attempt. Also superseded by r3: "RED 1 self-recovered" — FALSE. The Windows floor red is DETERMINISTIC adjacency: the Win test job's ~57 GB of artifacts are not yet reclaimed (async cleanup) when n1-gate's preflight fires 2s after it; two low-water readings 64 minutes apart agree within 10 MB (32229498880 / 32219230208 vs 34359738368 floor, ~2.1 GiB short every adjacent placement; r1-attempt-2 passed only by accidental time separation). Fix belongs in the workflow (bounded-wait floor preflight), not in disk clearing. Meter correction (deployah, self-caught): his r2/r3 watcher exit files read 0 on red runs — `gh run watch` without `--exit-status` exits 0 regardless. Control and treatment identical ⇒ instrument indicted. All session verdicts came from API `conclusion` JSON, so no verdict changes; the RULE is restated — an exit file is only as good as its producer's exit semantics; authority for gh runs is the API conclusion. ## ADDENDUM 2 — repro ANSWERS it: the stall is the exe-hash, deterministic (2026-08-29 ~22:55Z) Kitsubito repro, breadcrumbed build (60a12056), 12 iterations × the 4 cells full-parallel: 12/12 iterations 4/4 PASS (~21s each) — no repro at 4-cell parallelism — but the harvest (stderr sinks copied out of live tempdirs, 48 files with BRAIN_PHASE) yields the passing-run distribution, and it is not a tail: BRAIN_PHASE:announce done in 0ms (60 lines) BRAIN_PHASE:resume done in 0–7ms (60 lines) BRAIN_PHASE:ready written in ~10,100–10,200ms (exe-hash ≈ ALL of it, computed=true, EVERY brain) **`current_exe_hash()` costs ~10.1s per brain process on an IDLE box** — SHA-256 of the 439,868,944-byte debug `spt` binary through debug-compiled sha2 at ~43 MB/s (system `sha256sum` does the same file in 0.25s; the cost is unoptimized hash code, CPU-bound, hence the tight distribution). It sits ON the ready path, first-call per process, so EVERY test-spawned daemon pays it before `brain.ready` exists. Mechanism, complete: 30s budget − 10.1s deterministic hash = ~20s idle margin; golden Phase A runs dozens of daemon boots concurrently and each boot is itself a 10s full-core hash burn, so the boots compound each other and the CPU-bound hash stretches multiplicatively — past 30s in the tail cells. The load-tail and deterministic readings unify: a deterministic 10s floor whose multiplier is load. Release builds hash ~40× faster over a smaller binary (sub-second) — field impact negligible; the burden is debug/CI. IR-17 scope, CORRECTED BY REPLACEMENT (first written "unifies IR-17 entirely" — over-reach, caught by deployah's magnitude challenge): the KITSUBITO member (activity_link) IS this mechanism. The WINDOWS resident_service specimen is NOT: deployah measured the Windows debug binary at 67,485,696 bytes ⇒ ~1.5s/boot at the measured hash rate (local dev build; right order of magnitude, not CI's exact number), and a source read shows resident_service boots ONE daemon against a 45s ready budget (resident_service_e2e.rs:207) with the assert deferred to :453 — its 123.99s is accumulated post-miss bounded waits, not one wait. A ~1.5s hash cannot blow a 45s budget at any plausible load multiplier. The Windows specimen stays OPEN under IR-17 with its existing tree-kill candidate; the hash mechanism also explains why the Windows test job passed its 30s budgets in r2 AND r3 while Linux failed (~10.1s vs ~1.5s — the DWARF-vs-PDB binary-size split). Margins against the 120s budget: Linux ~12×, Windows ~80×. Gate-time consequence (deployah, owned): a genuinely wedged brain now costs up to 5×120s before its red — slow red ≠ new hang. Dispositions: - TEST-SIDE (rides respin): the 4 cells' `wait_for_ready_pid` budget 30s → 120s — an early-exit poll makes a generous bound FREE (banked craft) — plus the honest message. POPULATION (todlando review, measured at 71f82025): **24 brain.ready wait sites total; the 5 converted are the 5 that happened to red — 19 files still carry 30s against the 10.1s floor** (availability-selected fix, the emission-census shape, named knowingly). RULED to the hertz follow-on, NOT the respin: the 19 held green through r2's full-Phase-A worst load, and the fix's right form is ONE named constant in tests/common (iolog doctrine — 24 literals is 24 places to be wrong), which also gives the product fix a single number to walk back down. Follow-on also carries: daemon-B None-flows-to-later-assert message symmetry, and the unmeasured second 30s replay poll (daemon B pays the hash twice). - PRODUCT (board request, operator triages, NOT this cut): move `current_exe_hash()` off the ready path — todlando's flagged design fix, now carrying these figures. - AUTHORSHIP DEVIATION, declared: hertz is offline; the gater (doyle) authors the test-only budget fix, todlando reviews, deployah's golden gates. Operator's hertz-delegation rule is a dispatch split, not a prohibition; deviation recorded here and at close. ## ADDENDUM 3 — r4 (run 33280058910 @ d931dd63): both fixes FIELD-VALIDATED; one NEW red (2026-08-30 ~00:05Z) 8 of 9 jobs green. Linux test: 3158/3158 (+ todlando's computed-flag unit), Phase B 195/195. **The four cells passed at 42.396 / 44.558 / 45.137 / 58.342s — every one ABOVE the old 30s budget.** The pre-registered falsification did not fire; the mechanism math held, and the figures prove the old budget was the defect: under it, the ready-wait share of each cell was what timed out. BRAIN_PHASE absent from the green log is structural (panels build inside the panic), not silence. Floor fix validated: both n1-gate jobs green; Windows test job's post-checkout floor read 65.1 GiB honest-healthy where the pre-checkout read took the low-water mark. NEW red: Windows test job step 36 (docs-drift gate) died COMPILING its own tool — `LNK1318: Unexpected PDB error; LIMIT (12)` on `xtask.pdb`; the drift comparison never ran, so the red is upstream of the cell's subject (pre-declared structural rule applies). Control: same gate green on the same box at e4f52047 in r2 AND r3. Deployah excluded by measurement: PDB size ceiling (max 0.38 GB), stale toolchain processes (none), floor at job start (healthy). NOT established: free space at link time (job ended at 38.91 GB vs 34.4 GiB floor — 4.5 GB headroom, trajectory unreconstructable); transient-vs-deterministic (nothing varied yet). Primary candidate: the 2.92 GB `incremental/` dir (incremental+MSVC PDB = known LNK1318 source, accumulates ACROSS runs — compatible with two greens then a red without the head's 129 lines meaning anything; also why a bare same-sha rerun discriminates nothing). Secondary: disk-transient (the banked LNK-at-the-floor face). RULED sequence (doyle → deployah, local, evidence-first): (1) snapshot incremental/ structure; (2) live repro of the exact xtask compile, unvaried; (3) vary CARGO_INCREMENTAL=0 once — green-after-red names the mechanism by variation; (4) clear incremental (snapshotted), confirm clean green, r4 ATTEMPT 2 same-sha — parity intact at d931dd63, red closed as box-state defect at named mechanism, not by rerun; (5) durable rider (CARGO_INCREMENTAL=0 on Windows CI / scheduled hygiene) rides the next workflow commit, NOT an r5 respin. If (2) greens unvaried: stop, nothing cleared, rethink with the negative on record. ## ADDENDUM 4 — CLOSED. r4 attempt 2 GREEN, 9/9 at d931dd63 (2026-08-30 ~01:25Z) Trajectory flat (11 samples/11 min, 21.6 MB spread; deployah's earlier observed drift self-attributed to his own repro's incremental write — suspicion corrected, not left standing). Attempt 2 fired: API conclusion=success, exit file 0 with --exit-status, the two agreeing. **Disk CONFIRMED PREDICTIVELY, the pre-registered arm: attempt 1 floor 65.1 GiB → LNK1318 at 23:35; attempt 2 floor 85.85 GB → docs-drift SUCCESS. Same sha, same job, same step; only variable the reclaim.** 9/9 at ONE sha across two attempts, each job verified non-vacuous with real workload (stated plainly: not one clean sweep). All four budget cells green at 42.4/44.6/45.1/58.3s in the carried Linux job. main ff-pushed ec6da9b0..d931dd63; tag v0.66.0 == main == tested, verified at SOURCE (ls-remote both refs). Near-miss recorded: a name-classified reap would have destroyed todlando's live ir57 lane (203 GB, stale NAME, live REF) — identity is the checked-out ref, never the directory name; executor's classification caught it. Publish + verification pending (quiet window); the register riders (mid-job floor re-read, incremental hygiene, box pool-weight audit, 24-site budget population, hash-off-ready-path board request) carry everything durable. ## Artifacts deployah's box, scratchpad `dfdf64db-*/scratchpad`: R2_lin_test.log (4306 lines), R1_lin_test.log (control, 5097), R2_win_n1.log, R2B_verdict.json, R2B_alljobs.json. Restore cost: ship-on-request from deployah; quote-bearing excerpts reproduced above.