=== 9834545a docs(flake-ledger): the five rows the #272 r2/r3 arc produced, and a fourth occurrence Five rows by test name, all from the v0.68.0 golden arc: spt-daemon::mesh_recovery roster_route_survives_a_transient_dial_failure_with_discovery_disabled r2 run 34262154550 att3, cell 15.715 s against a 15.0 s converge() budget spt-daemon::registry_lifecycle oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing r2 att4, a flag-stopped-never-joined pump delivering one straggler past the sample spt-daemon::twohost two_host_ladder_role_a r2 att4, the A-3 rung asserting a wire discriminant the contract does not promise spt::webserve_attachment_e2e ... ARM 12, plus its ARM 11 sibling r2 att2 (ttl race), and the count-delta arm fixed test-only by rider 5 at f6110c2a spt::resident_service_e2e teardown LEAK UPDATED IN PLACE, not appended as a new row The first four are doyle's, verbatim from his preserved row files (manifest 4/4 OK, verified from inside the directory its paths were generated in). Two shaping decisions worth stating, because both could have been done the damaging way: The resident_service row was UPDATED, never duplicated. Rows here are keyed by test name and the file says so; a fourth occurrence of an existing signature is that row getting bigger. Its occurrence count goes 3 -> 4 and the r3 run id joins the list. The assert also MOVED, :664 -> :670 (8d10b280, text byte-identical), so the row's title now names both positions and repeats that the key is the name — otherwise the next reader greps :664, finds nothing, and opens a fifth row for the same defect. That is precisely how this file's `FLAKE-LEDGER #15` citations came to point at the wrong entry. The twohost row's CORRECTION note (todlando's split, 03:22Z, which REFUTED and withdrew the fixed-period phase-offset story) is folded into that row's own status column rather than left as a paragraph between rows. A prose line between table rows ends the table. CONSUMER CENSUS of the flake REGISTRY, as ordered — recorded here, no file changed. `.github/ci/flake-registry.json` has exactly one automated consumer: golden.yml:54 runs `flake-registry.py` with NO arguments, which is the validate-only path (`FLAKE_REGISTRY_OK: N entries` / exit 1 on `FLAKE_REGISTRY_INVALID`). The script's `--match TEST_ID [--platform]` query mode has NO caller anywhere in the tree. So an entry annotates nothing, reruns nothing, and gates nothing by itself. Its real consumer is a process rule read by people: GOLDEN-CI.md step 3 — a failure may use the flake avenue ONLY with an active entry matching the exact test id and platform — and step 4 — a match permits exactly one workflow_dispatch rerun of the unchanged sha. An entry is therefore an AUTHORIZATION to rerun a red, not a note. That is a real consequence, which is why resident_service's fourth occurrence is left a ledger row only: it meets the same-sha-rerun-confirmed shape on paper (34310511612 att1 red -> att2 Phase B green), but candidate is not entry, and the registry is untouched by this lane. Ledger-only. No registry, no workflow, no code, no REQ tag. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WAw9XAcTR8oSwzYPsqZoY2 Co-authored by: hertz diff --git a/docs/FLAKE-LEDGER.md b/docs/FLAKE-LEDGER.md index 3373e613..ee98261c 100644 --- a/docs/FLAKE-LEDGER.md +++ b/docs/FLAKE-LEDGER.md @@ -39,7 +39,7 @@ contemporaneous one is a row that will be cited as if it were. | `spt-daemon::servicehost_supervision_e2e bits_swap_under_the_hold_and_the_new_ones_come_up_on_release` | 1× hfenduleam Windows golden (2026-08-01 run 30720655534, sha 13e94d5) | **TEST DEFECT, not a load flake — the only entry here whose mechanism is the assert itself.** The new-process leg asserted identity by PID INEQUALITY (`assert_ne!(second, Some(first))`); Windows recycles pids aggressively enough under suite churn to hand the SAME pid back to the replacement, and the run failed with both sides `Some(47100)` while the log SELF-PROVED the swap (SERVICE_STARTED 47100 → SERVICE_QUIESCED cooperative exit under hold → SERVICE_STARTED 47100). The immediately-preceding asserts — `ServiceOutcome::Started` on release and the swapped-in binary's beacon — had already passed, so the functional swap was never in doubt: a pid was standing in for an identity it cannot carry past any exit (the KNOWN-HAZARDS `pid_started_at` class). File untouched by DOORBELL (last touch 1488e39, already on main) — pre-existing surface | **HARDENED (2026-08-01, releases#93):** the leg now compares the original's BIRTH STAMP (`proc::process_identity`, the `(pid, start-time)` pair) captured while it is alive against the same pid re-read after the ceremony — `Absent` when nothing took the pid, a DIFFERENT `Present` when something did, and only the original still running reads equal. Guarded by an INSTRUMENT assert that the stamp was readable at capture, because `Unproven != Unproven` is false and would have passed the leg by having nothing to compare. Mutation-proved both ways: an original that survives the ceremony REDS the identity assert, an unreadable stamp REDS the instrument assert. **No flake-registry.json entry** — its `retire_when` is "after the owning defect is fixed", the fix ships in the same commit, so an entry would be born retired; and registry entries require a same-SHA-rerun confirmation, which this never had (doyle's RCA proved it from the log, not from a rerun) | | `spt::psyche_real_bound_kill_soft_budget_e2e` | assert-FAIL 1× hfenduleam Windows golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :309 — expected `Some(Ok)`, got `Timeout` 1s×11 on summarizer.bat); LEAK 7/7 across recent golden Windows legs (7th on the att3 GREEN leg, 17.497s) | **two defects, not one signal** (hertz 2026-08-04): (1) assert arm — the positive control (:309) fires the FAST body through the SAME 1s kill bound; `wait_bounded` starts its deadline AFTER spawn (runtime.rs:941) so 1s covers the child's whole startup, and a trivial `@echo` bat can exceed it under load (mechanism, not yet a measurement — the control leg's wall time is the instrument); (2) leak arm is STRUCTURAL and distinct — `wait_bounded` kills the DIRECT child only (child.kill() :949 on cmd.exe); the sleeper is a GRANDCHILD (`cmd /C summarizer.bat` → ping), so ten kills leave ten ~5s survivors against nextest's 100ms leak timeout; no bound-shortening can fix it. USHER-chain delta-eliminated by dependency direction (spt-daemon below spt, deployah source-verified). Kin: releases#90 echo-commune spawn-killed-at-bound | DISPATCHED hertz 2026-08-04 (pkg item 1): control-leg bound via `BrainLifecycle::refresh_manifest` (lifecycle.rs:681) SAME host — budgets are host state (`Arc>` :672) so the standing latch and the `stamp()==None` clear row stay load-bearing; a second host would make that row vacuous. Leak fix lane-split doyle-ruled: TEST-ONLY sleeper fixture bin in spt-daemon = hertz now; PROD `wait_bounded` tree-kill-on-timeout = EVAL request to board, not dispatched | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_comes_back_when_knocked_awake` | 1× kitsubito Linux golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :1198 — erhost not up in 30s; brain stderr EMPTY vs named `ENGINE_ROOM_SPAWN_FAIL` emitter) | test file brand-new on the USHER branch. The chain's own suspect commit d449ce5 (exit-in-thread) source-eliminated on two strong legs: TEMPORAL (rc blocked in the bring_up RPC the whole window, rig-killed after — teardown never reached) + ARTIFACT OWNERSHIP (the awaited artifact is the broker's in-memory `sessions.insert`, broker.rs:6138-6153 — daemon-side chain, no CLI participant). Offline arm differs from the passing control arm ONLY by prior warm bring-up + `endpoint stop` | observed 1×; did NOT repeat at att3 same-sha (heavy-leg population 177 matched, the test RAN and passed — non-vacuous green). Declared read: next occurrence gets the stop-survivor broker-state RCA (wake_inflight gate broker.rs:5782-5852, `WAKE_DEDUP_WAIT=2s`; sessions reap; ceremony/perch re-read :5600-5604), not a rerun | -| `spt::resident_service_e2e` teardown LEAK at :664 (second svcmock outlived the sweep) | 3 classified hfenduleam Windows golden occurrences: 2026-08-04 run 30940180764 att2 @17f95a7; 2026-08-30 run 33296634901 att1 @da71b785; 2026-09-06 run 34017906638 att3 @04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64. First: second `svcmock.exe` pid 14452 outlived teardown; functional half all-TRUE, 3 reap verdicts Killed | teardown/zombie class (IR-34 kin). DISTINCT from this test's :382 PRECONDITION signature (ir15 att1, daemon never came up in 124s — that one is IR-17's family and carries no defect evidence): 2-of-11 recent Windows victim legs by test, 1-of-11 by this assert | DISPATCHED hertz 2026-08-04 (pkg item 4). Did not recur at att3 (passed, no leak row). +1 UNCLASSIFIED sighting 2026-08-17: `a_declared_service_rises…` FAILED 25.7s in a 745-full-parallel local sweep (nameplate-asm @2564f93, agent env scrubbed) — failure body NOT captured (`tail` ate it; labelled hole, no signature to compare), targeted rerun ×3 green. Counted as a sighting of SOME row of this test, attributable to none. **+1 CLASSIFIED 2026-08-30 (v0.67.0 golden r1, run 33296634901 att1 @da71b785): SECOND occurrence of THIS row** — same assert :664, same box, functional half all-TRUE, 3 reap verdicts Killed; NEW data the 08-04 occurrence lacked: the survivor is specifically the svcBOOT mock (pid 54000) and `went_clean=false after 60.31s of a 60s budget` — the settle ran to FULL budget, i.e. the leak never drained rather than appearing post-sweep. Signature predates the head's entire diff (row classified at 17f95a7); ruled ledgered-class at triage, same-sha rerun att2 = cell re-executed PASS 5.160s, 9/9. Gate closed, ROW STAYS OPEN; artifact: spt-preserve\v0670-golden-r1\ (deployah, log capture + SHA256SUMS). **+1 CLASSIFIED 2026-09-06: THIRD occurrence**, [run 34017906638 att3, job 101463810585](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101463810585), sha `04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64`: same :664 assertion, functional half green, three named reap verdicts Killed; survivors **svcboot pid 50484 + relshell pid 41892**, `went_clean=false after 60.1700792s of a 60s budget`. QUIET box per doyle: zero builder load, todlando's last activity 09:08Z; Phase A 197.840s versus att2 251s. The run's final orphan cleanup still named both pids at 09:44:19Z; temp-sandbox cleanup had failed on the relshell executable. This is the Windows face of hertz's daemon-leak cluster; the W2 rig-fixups PR must name this keyed row. ROW STAYS OPEN — recording a recurrence is not a repair. | +| `spt::resident_service_e2e` teardown LEAK, the assert rendered :664 then **:670** on main (moved by 8d10b280, text byte-identical — the row is keyed by TEST NAME, never position; grep both) (second svcmock outlived the sweep) | 4 classified hfenduleam Windows golden occurrences: 2026-08-04 run 30940180764 att2 @17f95a7; 2026-08-30 run 33296634901 att1 @da71b785; 2026-09-06 run 34017906638 att3 @04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64; 2026-09-09 run 34310511612 att1 @f6110c2a12df0dd50b87dfb60a2ec4120b5cf98d. First: second `svcmock.exe` pid 14452 outlived teardown; functional half all-TRUE, 3 reap verdicts Killed | teardown/zombie class (IR-34 kin). DISTINCT from this test's :382 PRECONDITION signature (ir15 att1, daemon never came up in 124s — that one is IR-17's family and carries no defect evidence): 2-of-11 recent Windows victim legs by test, 1-of-11 by this assert | DISPATCHED hertz 2026-08-04 (pkg item 4). Did not recur at att3 (passed, no leak row). +1 UNCLASSIFIED sighting 2026-08-17: `a_declared_service_rises…` FAILED 25.7s in a 745-full-parallel local sweep (nameplate-asm @2564f93, agent env scrubbed) — failure body NOT captured (`tail` ate it; labelled hole, no signature to compare), targeted rerun ×3 green. Counted as a sighting of SOME row of this test, attributable to none. **+1 CLASSIFIED 2026-08-30 (v0.67.0 golden r1, run 33296634901 att1 @da71b785): SECOND occurrence of THIS row** — same assert :664, same box, functional half all-TRUE, 3 reap verdicts Killed; NEW data the 08-04 occurrence lacked: the survivor is specifically the svcBOOT mock (pid 54000) and `went_clean=false after 60.31s of a 60s budget` — the settle ran to FULL budget, i.e. the leak never drained rather than appearing post-sweep. Signature predates the head's entire diff (row classified at 17f95a7); ruled ledgered-class at triage, same-sha rerun att2 = cell re-executed PASS 5.160s, 9/9. Gate closed, ROW STAYS OPEN; artifact: spt-preserve\v0670-golden-r1\ (deployah, log capture + SHA256SUMS). **+1 CLASSIFIED 2026-09-06: THIRD occurrence**, [run 34017906638 att3, job 101463810585](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101463810585), sha `04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64`: same :664 assertion, functional half green, three named reap verdicts Killed; survivors **svcboot pid 50484 + relshell pid 41892**, `went_clean=false after 60.1700792s of a 60s budget`. QUIET box per doyle: zero builder load, todlando's last activity 09:08Z; Phase A 197.840s versus att2 251s. The run's final orphan cleanup still named both pids at 09:44:19Z; temp-sandbox cleanup had failed on the relshell executable. This is the Windows face of hertz's daemon-leak cluster; the W2 rig-fixups PR must name this keyed row. ROW STAYS OPEN — recording a recurrence is not a repair. **+1 CLASSIFIED 2026-09-09 (v0.68.0 / #272 golden r3, run 34310511612 att1, job 102336053348, sha `f6110c2a`, Phase B 69/234, FAIL 71.881 s): FOURTH occurrence of THIS row.** Same assertion, now rendered `:670` (moved from `:664` by 8d10b280, text byte-identical; test file identical to main `e4444413`, zero riders touch it). Functional half all TRUE; three named reap verdicts Killed (boot 53100, rel 25012, brain 42716). Survivor: svcboot `svcmock.exe` pid **25596 — a DIFFERENT pid from the 53100 the test started and killed**, `went_clean=false after 60.6070532s of a 60s budget` (full budget, never drained). NEW evidence this face adds: (a) the brain — host of the service supervisors — was ALIVE when the boot service was killed (its own REAP afterwards reports SUCCESS on 42716 AND on a child 50480, identity not captured); (b) the test DISCARDS the `daemon stop --force` result (`let _ =`, :389) and kills children before the supervisor host; (c) `servicehost.rs` relaunches an un-asked exit (`on_exit -> ExitAction::Relaunch`, backoff base 1 s). So a supervisor-relaunched svcboot in the boot-kill..brain-kill window fits every fact — but the SERVICE_EXIT line that would PROVE it lived in the brain's stderr sink inside the temp sandbox, which the job's cleanup removed, so that link is UNPROVEN and is what the dispatched lane exists to close. Ruled LEDGERED-CLASS at triage (doyle); same-sha rerun att2 ordered and #272's own Windows cells were green in the same leg. Preserved: `.spt/preserved/golden-272-r3-drive/r3-run-34310511612.log`. DISPATCHED hertz 2026-09-09, test-only: observe the daemon-stop result, kill the supervisor host BEFORE the supervised children, stamp the survivor's start time and parent pid, preserve the brain sink on the leak path, identify child 50480. **ROW STAYS OPEN until that lane lands and a Windows golden passes through it.** | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1212 MINTED-NOT-RESUMED (sid identity) | 1× hfenduleam local assembly gate (2026-08-17, worktree nameplate-asm @2564f93, QUIET targeted round 1-of-3, 6.85s; rounds 2-3 green; +1 earlier FAIL 53.9s in the same tree's 745-full-parallel sweep, body tail-eaten — that occurrence carries no signature). Hot reproducer: **4–5/20** in quiet single-cell isolation at `f2d215a`; baseline arm at `60d74ea`: **5/20**. The cross-sha rate proves the family pre-existed the milestone head. | **TEST DEFECT — assert-side sampling race, not stopped-seat re-adoption.** Failure captures held `sid_after == sid_before` while broker truth crossed `None → Some(2)`, warm bring-up spawned, and stop returned `STOPPED`. The engine-room path unconditionally mints a fresh label (`broker.rs:5643`) and launches it with `is_resume=false`; the broker inserts its numeric PTY session into `sessions` before replying `Spawned`, while the child stamps the minted label only on its later self-bind. The test waited only for broker-table visibility and immediately sampled `info.json`, so it intermittently read the stopped life’s deliberately retained sid before the fresh bind rotated it. The alternate wake/reconcile path is ineligible: `endpoint stop` terminal-normalizes to `offline` + `suspended`, while `resume_woken_endpoint` requires active rest intent. Probe runs ended less than one second after admission and showed zero seat-gate involvement, matching the boundary. NAMEPLATE remains source-eliminated: no broker/session-mint/erhost file in that delta. | **CLOSED — GATE PASS** on `test/keystone-182-family-b` @`541e56f` (code @`a58a559`): after broker truth appears, poll under a hard 10s bound for the observable sid rotation before killing the controller. A fresh self-bind passes; unreadable, never-bound, or genuinely re-adopted records return the still-equal observed sid and remain red. Focused cell 1/1 green (6.97s), focused Clippy `-D warnings` green, traceability 776/776. The pre-registered signature-classified gate on the lane tip ran the exact cell ×20: **family-B sid signature 0/20** versus the hot 4–5/20 baseline, proving the fix effective. All four failures were the independent pre-#197 family-A throttle signature at 61–63s — expected on this `f2d215a`-based lane and not counted against this fix. Rides the next assembly as the test lane. | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1150 PRECONDITION (warm bring-up never spawns) | 3×/3 hfenduleam main-checkout pool (2026-08-17, main @27d40b9, agent env scrubbed, 186s each — consistent, not intermittent) | **TEST INFRASTRUCTURE DEFECT, not product evidence (IR-21/IR-39 class).** The clean main target had no `target/debug/mock-session.exe`: `cargo test -p spt` builds the `spt` test target but not another package's `mock-adapter` binary. `engine_room_bringup_e2e` derived that absent sibling path without asserting it, registered it as the harness command, then spent the full launch bound waiting for a program that never existed. This fully explains `warm bring-up spawned=false`, `endpoint stop` → `NO_SUCH_ENDPOINT`, the empty brain stderr, the 186s duration, and why a warm assembly pool passed or reached a later assertion. Explicitly building `cargo build -p mock-adapter --bin mock-session` restored the real loop; the earlier environment-sensitivity hypothesis is retired. Distinct from the genuine offline/control-projection race and from row 41's Linux occurrence. | CLOSED by test-only hardening: fail immediately with the missing path plus actionable build command, and prebuild the cross-package fixture before the golden suite. No product policy changed. RCA independently source-confirmed by doyle 2026-08-17 | | `spt-daemon daemon::tests::a_tree_teardown_reaches_a_grandchild_the_service_spawned` | 2× hfenduleam Windows golden: 10.176s run 30776330383 @184f2ac (2026-08-03); 10.225s run 30873007187 att1 @4b37512 (2026-08-04) | **TWO mechanisms wearing one assert string — do not count as a cross-sha repeat** (doyle source verification 2026-08-04): red 1 was the bare-pid `provably_gone(grandchild)` poller reading a pid-reuse stranger — it is the cited motivating incident in the pin comment (daemon.rs:2726-2737) and was RETIRED by a5042ec ("poll the process we pinned, not the number it happens to hold"); red 2 is POST-fix (pin present at 4b37512:2371) — a pinned, stamp-checked identity stayed not-provably-gone for the FULL 10s bound (daemon.rs:2757), mechanism OPEN (H1 unauthenticated SELECTION / H2 enrollment gap — see INFRA-REGISTER IR-17 sub-observation). Both reds sit AT the bound; todlando's off-CI discriminator ran 0/200 under live-fleet load with every pass 1-4s (never near the bound; filter positive-controlled via `nextest list` = exactly one test). The lesson this row carries: assert-body diffing across shas carried the string across a mechanism change — identity must include the polled predicate's SUBJECT, not just the message | INSTRUMENTED-AWAITING-FIRE (hertz selection probe, `SELECTION_PROBE` line rides main; a green retires nothing — doyle-ruled). Green at att3 17f95a7. The bound question (is 10s right for a loaded runner) parked with hertz pkg item 4; kill path and spawn flags untouched until the probe fires (#131 stays todlando's, operator-triage-gated). **GREEN-CAPTURE BASELINE (hertz item 3, 2026-08-04, doyle-scoped: run-local flags, N=20, this row only).** Repro: `cargo nextest run -p spt-daemon --lib --success-output immediate --test-threads 1 -E 'test(a_tree_teardown_reaches_a_grandchild_the_service_spawned)'`, ×20, on a released box (no CI worker, no release build), agent env stripped, lane `test/probe-green-capture` @`60ca8f8` base `6ec5237` — source-identical to the v0.54.0 tag `86f0d84` for this row (the range is the release commit only: Cargo.toml/Cargo.lock/CHANGELOG.md, **zero `.rs`**). **field 4 (ppid-match count) = 2 on 20 of 20 runs. min 2 / median 2 (lower-middle, convention pre-registered BEFORE the capture) / max 2. Raw: 2×20. ZERO variance.** All 20 exits 0; durations 1.86–2.79s (consistent with todlando's 1–4s passes, nowhere near the 10s bound). Population asserted per run (`Starting 1 test` ×20) and the 20 probe lines carry 20 UNIQUE candidate pids, so these are 20 distinct runs, not one line re-read. Both sweep populations 0 after: pool-scoped and the System32 `cmd`/`ping` victims. **What this establishes:** a second process in the child's parentage is STRUCTURAL here, not an excursion — hertz's prediction (pre-registered with doyle before the capture, falsifier `min=median=1`) SURVIVED. Mechanism named in the same lane's comment fix: this row spawns WINDOWLESS, which masks `DETACHED_PROCESS` back off (daemon.rs:1218-1219) so the child owns a console. **What it does NOT establish, and must not be read as:** (a) the companion's IDENTITY is UNMEASURED — the probe renders only the CHOSEN candidate and the reject census prints only on the panic path, so "the second match is a conhost" remains inference, not data; a red is what would print it. (b) The floor here is **2**, so the incident's `3` is an excursion above 2, **not** above the `1` the comment and register carry — on this box, at this sha, "it had always read 1" is not what the scan reads. That figure needs re-deriving on the hfenduleam runner before anything is built on it; a 20/20 zero-variance local read cannot speak for a different box. (c) A green still retires NOTHING (doyle's standing rule) — this is a baseline SHAPE, not a fix, and the row stays INSTRUMENTED-AWAITING-FIRE | @@ -49,3 +49,8 @@ contemporaneous one is a row that will be cited as if it were. | `spt-daemon::twohost_web two_host_web_role_b` — the A_CELLS completion witness (:257-284) | 1x, the FIRST-EVER real pair run (one box, 2026-09-07 02:56Z, `.worktrees/gate-w1-26a96d58/.spt/twohost-web/none2/`). No pair had ever run role B before — kitsubito's leg was the env-gated no-op, so this cell had never once executed for real. | **NOT A FLAKE — a DETERMINISTIC completion-witness defect, and it would have failed every pair run forever.** Filed here because doyle routed it here; the distinction matters to whoever reads this row next, because a reruns-sometimes-passes reading would be wrong. B's witness polled `brain.net_streams()` every 250 ms for peer-initiated rows and timed out at `role B saw only 0 of 3 requester streams within 240s` — while THE PRODUCT WAS GREEN: A's three cells all passed (403 naming WEB / byte-equal / 206 with Content-Range, 0.14-0.23 s each) and B's own breadcrumbs proved it served all three (`WEB_STREAM stream=1 refused ACCESS_DENIED`, `stream=2 sent 200 35 bytes`, `stream=3 sent 206 4 bytes`). MECHANISM, source-confirmed at 8d980fdf, TWO removal paths not one: a served stream leaves the table either by `retire_stream` (the `retired` flag, filtered out in `stream_infos_filtered`, nethost.rs:1956-1961) or by `retire_stream_terminal` (nethost.rs:2053, `streams.remove`) — and the dispatcher's own worker performs that retirement in its `DispatchOutcome::Served` arm the moment the serve completes. On a loopback pair a cell finishes far inside the 250 ms gap, so the row is gone before the next sample. SAME CLASS AS ROW 46 (a witness reading an INSTANTANEOUS level the product is entitled to lower), reached by a different removal path — second instance in one day. | **FIXED** on `build/ws272-w1` as a fixup: new `run_dispatch_loop_observed` seam exposes a MONOTONIC `served` counter (`AtomicU64`, incremented in the `Served` arm BEFORE the retirement that removes the row), `run_dispatch_loop` delegates to it so all ~8 existing call sites are untouched; role B waits on the counter and no longer calls `net_streams`. Counts every answer the owner produced — dispatch.rs's `StreamFamily::Web` arm maps `Sent`, `Refused` (the 403) and `Failed` (the 502) alike to `Served`, reserving `DispatchOutcome::Failed` for a transport error, so the deny cell counts and a retryable transport failure cannot inflate the total. NOT a shorter poll: a race with a smaller window is the same race. Re-gate is the pair only (doyle). | | `spt::resident_service_e2e a_declared_service_rises_with_the_daemon_and_reaches_the_cli` — the **:453 PRECONDITION** assert (a THIRD distinct signature for this test: distinct from its :664 teardown-LEAK row above, and the direct successor of that row's noted ":382 PRECONDITION" signature, which is this assert at its old line) | 1× kitsubito Linux (2026-09-07, the W1 #249 builder battery AT the gated sha `8d980fdf`, gate dir `~/spt-w1/.worktrees/ws272-w1/.spt/ws272-w1-gate/`; nextest exit 100, ONE Summary `2999 tests run: 2997 passed (8 slow, 1 leaky), 2 failed, 1 skipped`, `panicked at` = 2, FAIL at 53s). Raw preserved at `.spt/preserved/w1-kitsubito-8d980fdf/nextest.raw`, sha256 `9c456e21a6a0f75d…3b51e`, verified against the remote by hash | **The witness expired; the daemon did not fail to come up — the assert's own words are false as rendered.** `daemon_up = wait_until(45s, brain_ready(&ready_path).is_some())` (:207) went false, and the panel that same assert prints shows the tree UP: broker generation 0 pid 3300168, `BRAIN_UP` pid 3300200, `BRAIN_PHASE:announce done in 1ms`, `BRAIN_PHASE:resume done in 0ms`, `SERVICE_BOOT:svcboot: Started`, `SERVICE_STARTED` for BOTH svcboot and relshell, and both services reaped `verdict=KILLED` with an empty survivor set. Load: a 2999-test Phase-A pool with 8 slow siblings, on a binary that spawns a REAL `spt daemon run` tree and was NEVER in the HEAVY class — the 2026-07-15 rotating-victim mechanism verbatim, same 45s-deadline family as the `resume_no_control_steal_e2e` row below (its co-victim in this very Summary). **UNMEASURED, not concluded:** where the 45s went. Nothing stamps the interval between the daemon child's spawn and the brain's first log line, so the ~10.1s exe-hash on the ready path (measured v0.66.0, a different lane) is a CANDIDATE here and nothing more. **Named and EXCLUDED:** the panel carries `DOCS_SERVER_BIND_FAIL: port 5474: Address already in use (os error 98)` twice — the rig daemon lost the well-known docs port to kitsubito's resident perch daemon. `EADDRINUSE` returns immediately and the daemon continues by design, so that is a rig-hygiene defect (fixed below), NOT this red's cause; it is recorded here because it is the loudest line in the panel and would otherwise be re-derived by the next reader | **HARDENED AT THE RECIPE (2026-09-07), not at the test.** Added to both `` strings under the HEAVY-AT-BIRTH ruling, together with 10 other never-swept daemon-tree binaries — the census (method, all eleven names, and the byte-equality assertion on the two copies) is the stanza in `.config/nextest.toml`. The 45s budget is deliberately UNTOUCHED: retuning a deadline is the same race with a different number. Separately and independently of this red, every rig `spt daemon run` spawn in `crates/spt/tests` now sets `SPT_TEST_EPHEMERAL_ADVISORY_PORTS=1` (37 sites across 31 files; 2 sites already had it) so no test tree competes with a resident fleet daemon for 5474 again. **ROW STAYS OPEN** — the harden is a prediction until this binary is seen executing in Phase B; a red on the quiet serialized box would refute the pool mechanism and re-open the box/product question | | `spt::resume_no_control_steal_e2e brain_respawn_keeps_every_session_controller_and_still_promotes` — keyed on the MESSAGE `the trial candidate never stamped brain.ready`, NOT on the rendered line. The panic renders at `:358`, which is `teardown_panic`'s funnel `panic!` shared by every failing arm in the file (:397 and :490 today); a row keyed on :358 would silently absorb an unrelated future red | 1× kitsubito Linux (2026-09-07, same battery, same Summary and same preserved raw as the `resident_service_e2e` :453 row above — the two are co-victims of ONE window; FAIL at 46s) | The `:490` call site: `wait_ready_pid(&ready_path, 45s)` returned `None` — the trial brain candidate never stamped `brain.ready` inside the budget. Panel: `CONN_WRITE_POISONED conn=3 … wall_ms=1788745673787 mono_ms=2784` (the 800ms brain-write bound this test itself sets via `SPT_BRAIN_WRITE_DEADLINE_MS=800`), then `BRAIN_SUBSCRIBER_STALL_EVICT` 1/2/3 at mono 7327-7335 and all three controller conns `event=writer-exit … reason=write-failed kind=TimedOut` at 7343 — every controller conn retired long before the 45s expired. Same 45s-ready-deadline family and same never-swept-HEAVY recipe as the row above: a real `supervise_brain` + `spt daemon brain` tree at full Phase-A parallelism. **DISCRIMINATOR, and the reason this is not filed as a bare rerun-clear:** doyle's control leg re-ran BOTH reds isolated in the same pool — exit 0, ONE Summary `2 run: 2 passed`, at 10.47s and 11.92s against budgets they had just consumed 46s and 53s of. At-budget under load vs. 4x under budget alone is the load signature; a logic defect does not respect pool occupancy. **NOT claimed:** that the 800ms write bound CAUSED the missing ready stamp — the poisoned conns belong to the OLD generation's controllers and are equally readable as a co-symptom of the same starvation; distinguishing them needs a stamp this rig does not have | **HARDENED AT THE RECIPE (2026-09-07)** with the row above and the same nine other binaries — see the `.config/nextest.toml` census stanza. Neither the 45s wait nor the 800ms knob was retuned. **ROW STAYS OPEN**, same pre-registered discriminator: serialized in Phase B this binary runs alone, and a red there refutes the pool mechanism and re-opens the product question | +| `spt-daemon::mesh_recovery roster_route_survives_a_transient_dial_failure_with_discovery_disabled` — "never converged: roster route converged after the transient" (`mesh_recovery.rs:97` at `25e60015`; `converge()` = 600 × 25 ms = 15.0 s wall-clock budget) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 3 (2026-09-09 00:43:52Z, job 102283894969, Phase B serialized, cell 15.715 s, 233/234). Same cell PASSED at the same sha on attempts 1 and 2 (9.801 s, 7.172 s). | **BOX CONTENTION, not product — budget sat inside the box's own variance:** the pump/dial path is untouched by #272 (diff v0.67.0..25e60015 on `crates/spt-daemon/src/pump` + `crates/spt-net` = webmsg/xfer only). Stderr: cache-leg `PUMP_PEER_FAIL` at the 1.5 s test dial bound as designed, roster-leg `PUMP_DIAL_SUBMIT`, then no line for ~13.8 s until the panic — a burst ate the 5-8 s of headroom the cell had. Box evidence: Phase A (pure unit) slowed monotonically 448.7 → 495.1 → 542.6 s across the three attempts at ONE sha; per-cell attempt3/attempt2 over 73 Phase B cells ≥ 1 s: median 1.05×, 19 cells ≥ 1.5×, worst 5.2×; cargo/rustc/nextest 0 and ~1.1 of 16 cores busy at census; no other CI run on either runner. Mechanism candidate (todlando, measured by doyle 01:17Z): Windows Defender real-time ON, MsMpEng at 68 % of a core on the idle box, a fresh 35 MB exe pays 1.0-2.1 s on first execution vs 20-260 ms warm; exclusion list unreadable unelevated. Rotating single victim across attempts (ttl 5 s cell on a2, this 15 s cell on a3) = RANDOM-VICTIM signature: one env cause, N apparent flaky tests. Same-sha rerun-failed ruled once more (attempt 4, last), stop on any Phase B red. | OPEN — hertz rider 5 ordered 2026-09-09 01:20Z (post-v0.68.0 thin PR, base e4444413): the `for _ in 0..600 { sleep 25 ms }` shape is 31 sites across 4 test files (mesh.rs 9, mesh_recovery.rs 6, pump.rs 7, registry_lifecycle.rs 9; closed family repo-wide); derive every budget from the named bound it races (test-local const shared with the rig's `set_quic_op_timeout`), keep it under the 60 s nextest SLOW line, print elapsed / samples / last predicate state at the panic; predicates untouched. Runner-desktop contention registered as an INFRA entry (Defender exclusion = operator). | +| `spt-daemon::registry_lifecycle oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing` — "the refresh must subscribe NO historical rows (zero replay churn): held 0 -> 1, seats 1" (`registry_lifecycle.rs:514` at `25e60015`; the directional held/seats assertion after the gen-2 refresh) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:36Z, job 102306494097, Phase B serialized, cell 11.594 s, 233/234). Same cell PASSED at the same sha on attempts 1-3 (10.817 s, 10.518 s, 23.344 s). The two earlier victims of this arc (arm 12 ttl race; mesh_recovery converge budget) both PASSED on this attempt. | **RIG SAMPLE RACE, load-widened — not product:** A's pump is a bare `thread::spawn` (:307) stopped by the `pump_stop` flag (:386) and NEVER JOINED; the drains converge on B's gauges, gen-1's dispatcher is joined (:414), `b_held_before` is sampled, gen-2 starts — and the pump's last 100 ms-cadence round can still deliver ONE feed to B after that sample, which B then holds as one row with one in-flight seat. One row, not a replay storm: the re-apply bound one assertion earlier (:497) PASSED, and a replay regression re-subscribes the whole history. The test's own comment (:500-506) records this single-straggler face ("the extra held row was ONE straggler feed … mis-attributed to gen-2"). Product path untouched by the milestone (`registryhost.rs`/`pump` no diff v0.67.0..25e60015; `dispatch.rs` changes are the Web stream family; `broker.rs` a test accessor). Stderr carries only `CONN_LIFECYCLE` lines: a stream-8 subscription attached at mono 8.1 s and a serve on it at 8.7 s, neither released before the panic at ~11.45 s; conn ids are not attributable to A vs B from the log. Box: per-cell attempt4/attempt2 median 1.00× but 12 of 72 cells ≥ 1.5×, worst 4.75× — bursty. Third distinct single victim in four attempts at one sha = one environment cause (random-victim family); STOP ruled, sha retired. | OPEN — hertz rider ordered 2026-09-09 02:47Z (test-only, stacked on b359e40e, rides r3's head): `spawn_pump` returns its JoinHandle and the test JOINS it after `pump_stop` before the drains/sample (verify `run_peer_pump` exits at a round boundary on the flag); the :514 panic prints the gauges AND B's stream table (ids/families); audit the file for other flag-stopped-never-joined actors sampled by gauges (:417 documents the gen-1 worker case); predicates and both directional assertions untouched. | +| `spt-daemon::twohost two_host_ladder_role_a` — "A-3 setup suspend applied an edge at B: NoEdge" (`twohost.rs:2496` at `25e60015`; the rung asserts `matches!(out, RestRequestOutcome::Edge(_))` on the wire reply) | 1× golden twohost-a (hfenduleam role A / kitsubito role B), #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:44:22Z, job 102316461906, 19.95 s; twohost-b job 102316461873 then burned its 900 s on the wake anchor A never sent — collateral, one transaction). The same rung PASSED at attempts 1-3 (attempt 3: A-3 reached B ~4.5 s after ID_B's wake; attempt 4: 10.3 s, the toast rung's replication wait ran 7.8 s vs 1.0 s). | **RIG ASSERTS THE WIRE DISCRIMINANT AGAINST THE DOCUMENTED CONTRACT; the double delivery is a pre-existing dispatcher overlap, not this milestone.** Contract (`resthost.rs:21-27`, `:198-202`): exactly-once covers the rest STREAM OPEN only; the request line is unjournaled and a redelivered rest request is a natural `no_edge`, "report, don't dedup". Evidence: B's stored intent for ID_B (`read_rest` = info.json `rest_state`, no derivation) flipped Active → Suspended inside [22.348, 22.460] (B polled its anchor at 250 ms from 14.593Z, passed 22.598Z), the window of A's single request (`request_rest` is one-shot), which was answered `no_edge` at 22.4617Z and by the transition guard wrote nothing — so a second Suspend was applied at B in that instant. B's `CONN_LIFECYCLE` shows the Rest worker's fresh-query-then-subscribe pair TWICE for stream 85 (conn 119/120 @22.4327-.4330; conn 121/122 @22.4579-.4586, conn 122's close = A's reply; conn 120 closes 22.4973). One dispatcher generation cannot re-serve at 25 ms (InFlight claim held until the worker returns, retry floor 500 ms), so two dispatcher instances = two brain processes against B's broker; the claim path is untouched by #272 (one counter line, a thin wrapper, a comment word in the Rest arm). Auto-suspend (node knob off, no daemon.json on the box, tick fires only from Dormant, per-endpoint override never set) and the liveness-derived state (never persisted) are ruled out from source. Dispatcher generation is not logged on a clean serve. | OPEN — hertz rider 4 ordered 2026-09-09 03:11Z (test-only, rides r3's head): the four wire-Edge assertions in `twohost.rs` (:2223, :2242, :2497, :2541) accept Edge or NoEdge, fail only on Refused/Failed/NoReply/BadRequest, print which reply arrived, and witness the DURABLE observable each rung already waits on (registry row at A; `read_rest` at B). Two post-publish product lanes seeded for todlando: a served-path `dispatch_event` naming its dispatcher generation; the two-dispatcher overlap hazard (a3's B log: 27 streams attached by 2+ connections; a4's: 3 — the overlap may be constant and normally masked by retire-before-second-poll). **CORRECTION 2026-09-09T03:22Z (todlando split, manifest-verified a3 B log): lane-2 data = a3 25 non-controller query+subscribe pairs (2 of the 27 were controller handovers: streams 9, 82), gaps 0-21 ms median 4 + one 127 s long-lived-row outlier; a4 2 pairs (stream 12 was a handover), gaps 25.3/25.5 ms. Scatter => fixed-period phase-offset story REFUTED and withdrawn. Standing read: two dispatcher workers, constant overlap, normally masked by the pre-serve 'row already gone' arm (dispatch.rs:1053-1060, unlogged); load stretches the mask window. Lane-2 instrument = generation named on the Served path, count distinct generations.** | +| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` — ARM 12 "the attachment this arm just registered is not in the registry" (`webserve_attachment_e2e.rs:612` at `25e60015`) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 2 (2026-09-08 20:33:40Z, job 102216542171, Phase B serialized, cell 11.733 s, 233/234). The emitter's OWN comment records an earlier occurrence at ttl 1 s during lane development (one red then a pass), widened to 5 s — this is the same signature one size up. | **RIG WALL-CLOCK RACE, not product — two product mechanisms EXCLUDED at the sha before the word:** (1) an in-daemon clobber between the register write and the reaper tick: `livehost.rs:1244 reap_expired_attachments` runs under `servehost::with_registry_write` and `servehost::apply_at` takes `REGISTRY_WRITE`; the CLI registers THROUGH the daemon socket (`attach.rs:94`), so one writer process, lock holds; (2) a torn read: `ServingRegistry::save_at` is `atomic_write_bytes_durable`. POSITIVE EVIDENCE (deployah 20:52Z): the SAME cell PASSED at the SAME sha on attempt 1 of the same run (Phase B 234/234, 18:47-19:31Z), nothing between the two attempts but a pool reap and a queue — a pass-then-fail at one sha is the race signature, not a deterministic defect. The cell's own envelope was 11.733 s against a 5 s ttl, so the register-to-read gap had room to cross it. Mechanism: the arm mints a 5 s attachment, the reaper ticks every 5 s, and the spawned `spt send` child plus the read landed past expiry on the loaded golden runner; the daemon's own `ATTACHED … ttl 5000ms` line is in the panic, so the serve path DID register it. Elapsed time UNMEASURED: the Windows sandbox cleanup step removed the rig home at 20:41:09Z before anyone could read a `SERVE_REAP` stamp. Same-sha rerun-failed ruled once (rate rerun, said so), non-vacuity = the cell re-executes and passes. | OPEN — hertz rider 3 BUILT 88625fa0 (green Linux + Windows, unpushed, post-v0.68.0 thin PR; ordered 2026-09-08 20:50Z): make ARM 12 deterministic (long ttl + expire under the daemon's own writer path, or capture the snapshot path from the send's stderr) and print elapsed ms at the panic so the next occurrence carries its own number. | +| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` ARM 11 — `webserve_attachment_e2e.rs:551` "each attachment registered its OWN entry" left 4 right 5 | Windows at c4919243/tree be2184af: 4 reds in 15 runs (x5: FAIL 14.591 s; x10: FAIL runs 4/8/9 at 17.707/22.834/24.498 s), every red exactly -1; fastest run of 15 is a red, slowest a pass — no elapsed correlation. Linux 5/5 at 9.9 s flat (window proportionally smaller; not evidence of absence). | **COUNT-DELTA ARM RACING A DELIBERATE CORPSE AGAINST THE 5 s PULSE.** ARM 10 mints a real `--ttl 1s` attachment and sleeps 1.5 s; `ServeRequest::List` (servehost.rs:224) returns entries unfiltered so the corpse is in `before_count`; `reap_expired_attachments()` rides the 5 s reconcile pulse (livehost.rs); a pulse inside the before..after window makes the delta +2-1. Sibling of the arm-12 race afb711c9 retired, one arm up. Census at the sha: ARM 11 is the only count-delta arm after ARM 10. | FIXED test-only by rider 5 f6110c2a (parent c4919243): ARM 11 asserts identity (each ATTACHED url's served name present in `serve list --json`), before_count and the +2 delta removed, no Reconcile between arms. _SIBLING ROW (arm 11) 2026-09-09 03:5xZ, found by hertz's r3 rider proof legs, verified by doyle from source at c4919243:_ | == ledger table column census (HEAD) == pipes=5 rows=35 == non-table lines between table rows? == == IR token audit on HEAD register == headings: 4020:### IR-80 — five leaky cells on ONE module (`brainproc` / `supervise_brai 4300:### IR-81 — a process kill can be written scoped or machine-wide, and not 4425:### IR-82 — a daemon death leaves NO record of who died: autostart decide 4531:### IR-83 — a two-host rig addressed over the tailnet cannot receive INBO 4664:### IR-85 — the Windows self-hosted box runs its fs-heavy tests 3-4x slow 4787:### IR-86 — golden's 32 GiB floor is BELOW the measured 67.4 GiB Windows 4826:### IR-87 — two-host ceremony halves are INDEPENDENT jobs on DIFFERENT ru 4855:### IR-88 — `pool-release` via `cargo run -p xtask` REBUILDS xtask INTO t 4884:### IR-89 — hfenduleam's Windows Firewall drops cold inbound UDP to the r IR-79..84 tokens in the ADDED lines: 2 IR-82 2 IR-83 1 IR-84 refs to IR-8[5-9] resolve? (each referenced number has a heading): IR-85 heading=1 refs=1 IR-86 heading=1 refs=1 IR-87 heading=1 refs=3 IR-88 heading=1 refs=1 IR-89 heading=1 refs=1 kin refs to landed entries in added lines: 1 IR-31 2 IR-46 1 IR-56 1 IR-59 1 IR-62 3 IR-64 1 IR-73 3 IR-76 2 IR-82 2 IR-83 1 IR-84 1 IR-85 1 IR-86 3 IR-87 1 IR-88 1 IR-89