# Flake ledger

Tests that have failed nondeterministically, with mechanism notes and hardening
status. Discipline: **any flake gets an entry, not a shrug** — a second
occurrence upgrades the entry from "observed" to "harden it." Rig flakes from
the two-host ladder (docs/TWO-HOST-RUNBOOK.md) land here too.

<!-- [doc->REQ-FLAKE-LEDGER-KEYED-ROWS] -->
**Rows are keyed by the test (or CI step) name, never by position.** There is no
row number: a hand-numbered first column made two lanes appending "row N+1"
conflict textually AND mint the same number, and a forward-referenced number can
be squatted by a later, unrelated row — that is exactly how every in-tree
`FLAKE-LEDGER #15` citation came to point at the wrong entry, and how the
daemon-tree bring-up row below went a year unwritten while 20 sites cited it.
Cite an entry from elsewhere in the tree as `FLAKE-LEDGER: <test name>`.

Append new sightings at the end. A BACKFILL of a historical flake goes in date
order instead, and says in its own status column that it is reconstructed and
what evidence did not survive — a row that cannot be told apart from a
contemporaneous one is a row that will be cited as if it were.

| test | occurrences | mechanism | status |
|------|-------------|-----------|--------|
| `spt-daemon brain_restart_survives_gaplessly` | 1× hosted Windows (2026-06-03, hung) | ConPTY stall under hosted-runner load | job `timeout-minutes` bounds it; not seen self-hosted |
| `spt-term digest_parses_a_real_pty_stream` | 3× gravity (2026-06-03; 2026-06-04 runs 26957386850, 26966738231) | sprint-collapse assert (`digest.rs`) — PTY input echo and the child's write are **concurrent writers** into the master-read stream; back-to-back `send_line`s let a later line's echo interleave mid-`Write(…)`, breaking the tool regex for that turn | HARDENED (2026-06-04): sends serialized — each line pumped back (adaptive echo-copy count, 2s/line bound) before the next, so no concurrent writer races an earlier line; failure asserts now print the raw transcript |
| `spt-daemon sync.rs torn_pull` + `concurrent_writes` | 1× (2026-06-03) | `wait_for_stream` deadline too tight under load | HARDENED: `wait_for_stream_except` 2s→10s; no recurrence |
| `spt-live context::write_context_suppresses_stale_llm` | 1× gravity (2026-06-04 run 26967808321) | test calls `write_context` (reads SPT_HOME node identity + epoch stamp) WITHOUT the `with_home` lock — a parallel `with_home` test's env swap + temp-dir teardown races the stamp's epoch write into a deleted home (`NotFound`) | HARDENED (2026-06-04): test wrapped in `with_home` — serializes on the home lock like every other SPT_HOME-touching test |
| `spt-daemon attach.rs:600` re-serve (`output gap: got seq 1 want 0`) | 1× kitsubito Linux (2026-06-15 run 27545857429, PR #14 stale-base CI) | re-serve replays the buffered output stream on re-attach; under load the new subscriber observed `seq 1` before `seq 0` — an ordering/timing gap in the re-serve sequence assert, not a content error. Subsystem disjoint from the triggering change (PR #14 = digest-proof CLI key-fill, cannot touch attach sequencing) | OBSERVED 1×: did NOT reproduce on the clean-base rerun (`gh pr update-branch` onto post-#15 main → green both runners). Not hardened; a 2nd occurrence upgrades to harden |
| `spt-term tests/stream.rs:49 bounded_backpressure_stalls_an_unconsumed_reader` | 1× hfenduleam Windows (2026-06-15 run 27545857429, PR #14 stale-base CI) | asserts an unconsumed bounded stream stalls the writer-side reader; got `16 -> 82` (reader advanced past the bound) — a timing-sensitive backpressure assert racing the bound check under hosted-runner load. Subsystem disjoint from PR #14 (digest-proof CLI) | OBSERVED 1×: did NOT reproduce on the clean-base rerun (green both runners). Not hardened; a 2nd occurrence upgrades to harden |
| `spt-daemon::attach attach_survives_target_brain_restart_exactly_once` | 1× kitsubito (2026-06-16 run 27595180782 attempt 1, M11-W3) | **TIMEOUT, not an assertion fail** — SLOW>60>120>180>TERMINATING, nextest TIMEOUT at the 240s cap (`attach.rs:600`). A heavy multi-process brain-restart-exactly-once test on the slow shared kitsubito box under concurrent load (n1-gate on the same runner + a parallel `ci.yml` docs-publish run firing simultaneously). W3-INDEPENDENT: Windows green end-to-end; ALL tunnel tests passed fast on Linux too (tunnel_e2e 0.941s, loopback_tunnel_backpressures_without_loss 1.091s) — a logic bug fails both runners deterministically, this timed out on one under load | observed; cleared by `gh run rerun --failed` (builds reused, no re-tag). Harden if it recurs — candidate: raise this test's per-test nextest timeout, or serialize it off the concurrent n1-gate slot |
| CI build step `Build notify-shell` (kitsubito Linux) | 1× kitsubito (2026-06-16 run 27652755792 attempt 1, v0.8.1 PR #17) | crates.io dep-download blip during the notify-shell build — `download of config.json failed, curl failed` fetching `serde_json`: a transient registry/network fetch failure on the runner, not a code or test fault (a real dep break fails both runners deterministically; this hit one runner once) | observed; cleared by `gh run rerun --failed` (attempt 2 green, builds reused). Harden if it recurs — candidate: a cargo fetch retry / registry cache warm on the runner |
| `spt dummy_harness_e2e` (BOTH tests: `endpoint_run_brings_up_a_long_lived_dummy_harness_and_rc_attaches` + `endpoint_run_attach_awaits_online_before_attaching`) | 1× kitsubito (2026-07-05, REMOTE-TRUTH Batch-2 re-gate @f2a799e; roles swap run-to-run) | **process-global `set_var` race in a multi-test e2e binary.** Both tests `std::env::set_var("SPT_HOME", own_tempdir)` then do IN-PROC reads (`perch::spt_home`/`adapters_dir`, `registry::register`, psychebin `fs::copy` staging). One file = one test binary → the two run on parallel threads; whichever set_var last wins for BOTH → `register`/staging land in the wrong home (`ENDPOINT_RUN_ADAPTER_UNREGISTERED`; os-32 sharing collision). Child procs were safe (explicit `.env`) — only the in-proc reads leaked. File untouched since F-030; Batch-2 broker-reap timing shift woke it. `--test-threads=1` → 2/2 green (proves serialize) | HARDENED (2026-07-05): file-local `static E2E_LOCK: Mutex<()>` held whole-body in each test (contract_e2e.rs:26 pattern); green at default parallelism. **CLASS = process-global `set_var("SPT_HOME")` + in-proc read in a multi-`#[test]` e2e binary.** doyle's gate2 (`--no-fail-fast`) proved it beyond dummy_harness: `live_adapt_translation_swap_e2e` (3/4 red) + `multi_subnet_bringup_e2e` (3/3 red), same signature. **SWEPT (anchored `grep -cE '^#\[test\]'` census):** locked `dummy_harness_e2e` (2), `live_adapt_translation_swap_e2e` (4), `multi_subnet_bringup_e2e` (3), `translate_proof` (2); `contract_e2e` (26) + `gateway_e2e` (35) already locked. EXCLUDED `brain_survive` + `n1_pairing` — each a SINGLE `#[test]` (no intra-binary parallelism → cannot race); both already carry an in-file "a 2nd `#[test]` must bring the env-lock" deferred contract (a never-contended Mutex contradicting its own doc is worse than nothing). NB: doyle's first census overcounted these two — an UNANCHORED `grep -c '#\[test\]'` counted the attribute mentioned in their comment prose; anchor the count |
| `spt-store registry::tests::concurrent_registration_never_locks` | 1× hfenduleam local gate (2026-07-05, REMOTE-TRUTH Batch-2 bless matrix @3f6f327, fresh worktree, `nextest --workspace --no-fail-fast`) | 16-concurrent-writer SQLite registration gate FAILED at 5.63s — the SQLITE_BUSY load-contention class ALREADY documented at `db.rs:60`: box saturated by the 1588-test full-parallel suite meant a writer couldn't schedule its retry inside the default 5s `registry_busy_timeout`. Isolated ×5: green, 0.25–0.35s each. NOT a product defect — the exact scenario the `SPT_REGISTRY_BUSY_TIMEOUT_MS` env knob (default 5s) was added for | observed 1× (local gate, not CI); in-code remedy pre-exists: set `SPT_REGISTRY_BUSY_TIMEOUT_MS=30000` on saturated gate rigs. Harden candidate on recurrence: bake the raised knob into gate-matrix/CI test invocations |
| `spt-daemon::resume resume_mode_brain_spawning_new_sessions_delivers_each` | 1× kitsubito (2026-07-05 run 28768663321 attempt 1, REMOTE-TRUTH [twohost] PR #52) | Phase-B heavy-class timing EOF — "session exited before the expected output arrived" (UnexpectedEof) at 62s under the serialized heavy pool. SAME FAMILY as this file's prior Linux-CI flake (7940318, D4-2b: sequential spawn/read hardening) — a spawned session's exit raced the expected output read on the slow shared box. File untouched by the wave; Windows leg green same run; the wave's spt-daemon changes (C1 broker lifecycle) are cross-platform and every local Win gate was green — but note this run was the wave's FIRST full Linux suite | observed 1×: cleared by `gh run rerun --failed` (attempt 2 green, builds reused, no re-tag — the `attach_survives_target_brain_restart_exactly_once` / `Build notify-shell` rerun-clears-it pattern). Harden if it recurs — candidate: extend the 7940318 sequential-spawn/read pattern to this test's spawn loop, or a per-test nextest timeout raise |
| `spt-daemon::input_ack_deadlock input_flood_through_serve_attach_does_not_deadlock_broker` | 1× hfenduleam local gate (2026-07-06, REMOTE-TRUTH F-2 matrix @a21bc6b, `nextest --workspace --no-fail-fast`) | deadline-margin under full-suite load: the flood exchange's `recv_timeout(30s)` (input_ack_deadlock.rs:512) blew under the 1608-test parallel pool + 7 leaky daemons — FAIL at 34.5s vs ~32s natural runtime isolated (the margin is ~3s by design). Subsystem disjoint from the triggering commit (F-2 = api/reporting.rs soft-end guard; cannot touch broker input-ack). Isolated ×5: green, 31.9–32.2s | observed 1× (local gate, not CI); harden candidate on recurrence: widen the :512 recv bound or serialize the flood test into the Phase-B heavy class (CI already runs it serialized — this class only bites full-parallel local matrices) |
| `spt::bin cli::tests::probe_all_cap_batches_into_windows` | 1× hfenduleam local gate (2026-07-06, stack matrix @0f08fa9 leg-3 rerun) | fast unit (0.52s natural) FAILED at 1.7s under the 1614-test full-parallel pool — probe-batching window assert with a timing component starved under load; subsystem disjoint from the triggering stack (F-2 reporting / E-2 grid title / picker rider). Isolated ×5: green, 0.52–0.55s. Same night as `input_flood_through_serve_attach_does_not_deadlock_broker` — the full-parallel local matrix under a busy box is the common factor, not the tests | observed 1× (local gate, not CI); no harden yet. NIGHT-PATTERN NOTE: `concurrent_registration_never_locks`, `input_flood_through_serve_attach_does_not_deadlock_broker` and this entry are all first-sightings from the same gate recipe (full-parallel nextest on a loaded shared box) — if a FOURTH distinct test trips this way, harden the RECIPE (nextest profile with heavy-class serialization locally, mirroring CI's Phase split) rather than the tests |
| `spt::bin rc::tests::attach_viewport_reconnects_across_a_broker_bounce` | 1× hfenduleam local gate (2026-07-06, WORKER-TRUTH W-3 build @worker-truth, `nextest -p spt --bin spt` full-parallel 360-test run) | **TIMEOUT, not an assertion fail** — SLOW>60…>TERMINATING at the 240s cap under the 360-way `--bin spt` pool. A REAL-broker + broker-bounce reconnect UNIT test (rc.rs:2071) that escaped the HEAVY-class serialization: the Phase-A/B split + `heavy-broker-pty` test-group target integration test BINARIES (`kind(test)`), but this is a `kind(bin)` unit test inside the `spt` binary, so it ran in the LIGHT pool at full parallelism and starved. Isolated ×1: green ~1.3s. Subsystem (rc attach/broker-bounce) disjoint from the W-3 worker-reap wave that triggered the run | **THE FOURTH DISTINCT FULL-PARALLEL TRIP** the NIGHT-PATTERN note on `probe_all_cap_batches_into_windows` called for — so HARDENED at the RECIPE, not the test: the rc real-broker/bounce unit class is folded into the `heavy-broker-pty` nextest test-group (`.config/nextest.toml`, a `kind(bin) & test(/^rc::tests::(…)$/)` override) so it serializes with the rest of the heavy class and can never be starved by the light pool again. EXTEND the name set on each new rc-broker sighting |
| `spt` daemon-tree e2e bring-up, PRECONDITION "brain never came up" — rotating across `bind_honest_cross_perch_e2e`, `live_adapt_translation_swap_e2e`, `multi_subnet_bringup_e2e`, `run_no_dup_session_e2e` | 6× kitsubito Linux (2026-07-15, across the FORKENING W3 + W4 gates — two independent runs), rotating membership run to run | **RECONSTRUCTED ENTRY — see the provenance note in the status column; this row was never written at the time.** Four `spt` e2e binaries that spawn a REAL `spt daemon run` tree (`CARGO_BIN_EXE` daemon run + ~10s bringup) sat in the LIGHT Phase-A pool, so a 16-way daemon storm on kitsubito blew the 30s brain-readiness deadline probabilistically. ROTATING membership across the two gates is the load-flake tell (a logic bug picks the same victim); `run_no_dup_session` SOLO on the warm tree PASSED in 10.7s. Exonerated by absence of the competing causes: no OOM / fork / TasksMax events on the box. Noted but not charged: v6-first DNS with no default v6 route, the broken-IPv6-iroh tell, recorded as a boot pad rather than this mechanism. All four QUALIFIED as HEAVY under `.config/nextest.toml`'s own sweep criterion and postdated the last sweep | **HARDENED AT THE RECIPE (2026-07-15, `5199464`)** — the four binaries added alphabetically to all three `<HEAVY>` strings (ci.yml Phase A exclude, Phase B include, nextest.toml `heavy-broker-pty` override; verified byte-identical), plus `daemon_refresh_e2e`, new on that branch, classed heavy AT BIRTH rather than after its first Phase-A flake. `composite_e2e` stayed LIGHT (daemonless apply, no daemon tree); `translate_proof` stayed Phase A by ruling (one sighting, single-child spawn, watching). **This row is the origin of the HEAVY-AT-BIRTH ruling** — classify a real-daemon-tree/real-broker test at file creation, not after its first Phase-A timeout — whose canonical text is the `<HEAVY>` stanza in `.config/nextest.toml`. **PROVENANCE (reconstructed 2026-08-02, releases#95, hertz; doyle-ruled after confirming the primary RCA log is no longer retained):** `5199464`'s commit message ends "Ledger entry appended." and its diff touches only `.github/workflows/ci.yml` and `.config/nextest.toml` — the entry was never written, while the citation `FLAKE-LEDGER #15` it minted spread to 20 sites and was later squatted by the unrelated servicehost row below (`e1a3338`, 2026-08-01). Reconstructed from that commit message plus the surviving citation sites; no run ids, no per-sighting timestamps and no raw gate logs survive, so this row carries the mechanism and the remedy but NOT the primary evidence, and must not be cited as if it did |
| `spt-daemon::servicehost_supervision_e2e bits_swap_under_the_hold_and_the_new_ones_come_up_on_release` | 1× hfenduleam Windows golden (2026-08-01 run 30720655534, sha 13e94d5) | **TEST DEFECT, not a load flake — the only entry here whose mechanism is the assert itself.** The new-process leg asserted identity by PID INEQUALITY (`assert_ne!(second, Some(first))`); Windows recycles pids aggressively enough under suite churn to hand the SAME pid back to the replacement, and the run failed with both sides `Some(47100)` while the log SELF-PROVED the swap (SERVICE_STARTED 47100 → SERVICE_QUIESCED cooperative exit under hold → SERVICE_STARTED 47100). The immediately-preceding asserts — `ServiceOutcome::Started` on release and the swapped-in binary's beacon — had already passed, so the functional swap was never in doubt: a pid was standing in for an identity it cannot carry past any exit (the KNOWN-HAZARDS `pid_started_at` class). File untouched by DOORBELL (last touch 1488e39, already on main) — pre-existing surface | **HARDENED (2026-08-01, releases#93):** the leg now compares the original's BIRTH STAMP (`proc::process_identity`, the `(pid, start-time)` pair) captured while it is alive against the same pid re-read after the ceremony — `Absent` when nothing took the pid, a DIFFERENT `Present` when something did, and only the original still running reads equal. Guarded by an INSTRUMENT assert that the stamp was readable at capture, because `Unproven != Unproven` is false and would have passed the leg by having nothing to compare. Mutation-proved both ways: an original that survives the ceremony REDS the identity assert, an unreadable stamp REDS the instrument assert. **No flake-registry.json entry** — its `retire_when` is "after the owning defect is fixed", the fix ships in the same commit, so an entry would be born retired; and registry entries require a same-SHA-rerun confirmation, which this never had (doyle's RCA proved it from the log, not from a rerun) |
| `spt::psyche_real_bound_kill_soft_budget_e2e` | assert-FAIL 1× hfenduleam Windows golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :309 — expected `Some(Ok)`, got `Timeout` 1s×11 on summarizer.bat); LEAK 7/7 across recent golden Windows legs (7th on the att3 GREEN leg, 17.497s) | **two defects, not one signal** (hertz 2026-08-04): (1) assert arm — the positive control (:309) fires the FAST body through the SAME 1s kill bound; `wait_bounded` starts its deadline AFTER spawn (runtime.rs:941) so 1s covers the child's whole startup, and a trivial `@echo` bat can exceed it under load (mechanism, not yet a measurement — the control leg's wall time is the instrument); (2) leak arm is STRUCTURAL and distinct — `wait_bounded` kills the DIRECT child only (child.kill() :949 on cmd.exe); the sleeper is a GRANDCHILD (`cmd /C summarizer.bat` → ping), so ten kills leave ten ~5s survivors against nextest's 100ms leak timeout; no bound-shortening can fix it. USHER-chain delta-eliminated by dependency direction (spt-daemon below spt, deployah source-verified). Kin: releases#90 echo-commune spawn-killed-at-bound | DISPATCHED hertz 2026-08-04 (pkg item 1): control-leg bound via `BrainLifecycle::refresh_manifest` (lifecycle.rs:681) SAME host — budgets are host state (`Arc<Mutex<PsycheBudgets>>` :672) so the standing latch and the `stamp()==None` clear row stay load-bearing; a second host would make that row vacuous. Leak fix lane-split doyle-ruled: TEST-ONLY sleeper fixture bin in spt-daemon = hertz now; PROD `wait_bounded` tree-kill-on-timeout = EVAL request to board, not dispatched |
| `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_comes_back_when_knocked_awake` | 1× kitsubito Linux golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :1198 — erhost not up in 30s; brain stderr EMPTY vs named `ENGINE_ROOM_SPAWN_FAIL` emitter) | test file brand-new on the USHER branch. The chain's own suspect commit d449ce5 (exit-in-thread) source-eliminated on two strong legs: TEMPORAL (rc blocked in the bring_up RPC the whole window, rig-killed after — teardown never reached) + ARTIFACT OWNERSHIP (the awaited artifact is the broker's in-memory `sessions.insert`, broker.rs:6138-6153 — daemon-side chain, no CLI participant). Offline arm differs from the passing control arm ONLY by prior warm bring-up + `endpoint stop` | observed 1×; did NOT repeat at att3 same-sha (heavy-leg population 177 matched, the test RAN and passed — non-vacuous green). Declared read: next occurrence gets the stop-survivor broker-state RCA (wake_inflight gate broker.rs:5782-5852, `WAKE_DEDUP_WAIT=2s`; sessions reap; ceremony/perch re-read :5600-5604), not a rerun |
| `spt::resident_service_e2e` teardown LEAK, the assert rendered :664 then **:670** on main (moved by 8d10b280, text byte-identical — the row is keyed by TEST NAME, never position; grep both) (second svcmock outlived the sweep) | 4 classified hfenduleam Windows golden occurrences: 2026-08-04 run 30940180764 att2 @17f95a7; 2026-08-30 run 33296634901 att1 @da71b785; 2026-09-06 run 34017906638 att3 @04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64; 2026-09-09 run 34310511612 att1 @f6110c2a12df0dd50b87dfb60a2ec4120b5cf98d. First: second `svcmock.exe` pid 14452 outlived teardown; functional half all-TRUE, 3 reap verdicts Killed | teardown/zombie class (IR-34 kin). DISTINCT from this test's :382 PRECONDITION signature (ir15 att1, daemon never came up in 124s — that one is IR-17's family and carries no defect evidence): 2-of-11 recent Windows victim legs by test, 1-of-11 by this assert | DISPATCHED hertz 2026-08-04 (pkg item 4). Did not recur at att3 (passed, no leak row). +1 UNCLASSIFIED sighting 2026-08-17: `a_declared_service_rises…` FAILED 25.7s in a 745-full-parallel local sweep (nameplate-asm @2564f93, agent env scrubbed) — failure body NOT captured (`tail` ate it; labelled hole, no signature to compare), targeted rerun ×3 green. Counted as a sighting of SOME row of this test, attributable to none. **+1 CLASSIFIED 2026-08-30 (v0.67.0 golden r1, run 33296634901 att1 @da71b785): SECOND occurrence of THIS row** — same assert :664, same box, functional half all-TRUE, 3 reap verdicts Killed; NEW data the 08-04 occurrence lacked: the survivor is specifically the svcBOOT mock (pid 54000) and `went_clean=false after 60.31s of a 60s budget` — the settle ran to FULL budget, i.e. the leak never drained rather than appearing post-sweep. Signature predates the head's entire diff (row classified at 17f95a7); ruled ledgered-class at triage, same-sha rerun att2 = cell re-executed PASS 5.160s, 9/9. Gate closed, ROW STAYS OPEN; artifact: spt-preserve\v0670-golden-r1\ (deployah, log capture + SHA256SUMS). **+1 CLASSIFIED 2026-09-06: THIRD occurrence**, [run 34017906638 att3, job 101463810585](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101463810585), sha `04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64`: same :664 assertion, functional half green, three named reap verdicts Killed; survivors **svcboot pid 50484 + relshell pid 41892**, `went_clean=false after 60.1700792s of a 60s budget`. QUIET box per doyle: zero builder load, todlando's last activity 09:08Z; Phase A 197.840s versus att2 251s. The run's final orphan cleanup still named both pids at 09:44:19Z; temp-sandbox cleanup had failed on the relshell executable. This is the Windows face of hertz's daemon-leak cluster; the W2 rig-fixups PR must name this keyed row. ROW STAYS OPEN — recording a recurrence is not a repair. **+1 CLASSIFIED 2026-09-09 (v0.68.0 / #272 golden r3, run 34310511612 att1, job 102336053348, sha `f6110c2a`, Phase B 69/234, FAIL 71.881 s): FOURTH occurrence of THIS row.** Same assertion, now rendered `:670` (moved from `:664` by 8d10b280, text byte-identical; test file identical to main `e4444413`, zero riders touch it). Functional half all TRUE; three named reap verdicts Killed (boot 53100, rel 25012, brain 42716). Survivor: svcboot `svcmock.exe` pid **25596 — a DIFFERENT pid from the 53100 the test started and killed**, `went_clean=false after 60.6070532s of a 60s budget` (full budget, never drained). NEW evidence this face adds: (a) the brain — host of the service supervisors — was ALIVE when the boot service was killed (its own REAP afterwards reports SUCCESS on 42716 AND on a child 50480, identity not captured); (b) the test DISCARDS the `daemon stop --force` result (`let _ =`, :389) and kills children before the supervisor host; (c) `servicehost.rs` relaunches an un-asked exit (`on_exit -> ExitAction::Relaunch`, backoff base 1 s). So a supervisor-relaunched svcboot in the boot-kill..brain-kill window fits every fact — but the SERVICE_EXIT line that would PROVE it lived in the brain's stderr sink inside the temp sandbox, which the job's cleanup removed, so that link is UNPROVEN and is what the dispatched lane exists to close. Ruled LEDGERED-CLASS at triage (doyle); same-sha rerun att2 ordered and #272's own Windows cells were green in the same leg. Preserved: `.spt/preserved/golden-272-r3-drive/r3-run-34310511612.log`. DISPATCHED hertz 2026-09-09, test-only: observe the daemon-stop result, kill the supervisor host BEFORE the supervised children, stamp the survivor's start time and parent pid, preserve the brain sink on the leak path, identify child 50480. **ROW STAYS OPEN until that lane lands and a Windows golden passes through it.** |
| `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1212 MINTED-NOT-RESUMED (sid identity) | 1× hfenduleam local assembly gate (2026-08-17, worktree nameplate-asm @2564f93, QUIET targeted round 1-of-3, 6.85s; rounds 2-3 green; +1 earlier FAIL 53.9s in the same tree's 745-full-parallel sweep, body tail-eaten — that occurrence carries no signature). Hot reproducer: **4–5/20** in quiet single-cell isolation at `f2d215a`; baseline arm at `60d74ea`: **5/20**. The cross-sha rate proves the family pre-existed the milestone head. | **TEST DEFECT — assert-side sampling race, not stopped-seat re-adoption.** Failure captures held `sid_after == sid_before` while broker truth crossed `None → Some(2)`, warm bring-up spawned, and stop returned `STOPPED`. The engine-room path unconditionally mints a fresh label (`broker.rs:5643`) and launches it with `is_resume=false`; the broker inserts its numeric PTY session into `sessions` before replying `Spawned`, while the child stamps the minted label only on its later self-bind. The test waited only for broker-table visibility and immediately sampled `info.json`, so it intermittently read the stopped life’s deliberately retained sid before the fresh bind rotated it. The alternate wake/reconcile path is ineligible: `endpoint stop` terminal-normalizes to `offline` + `suspended`, while `resume_woken_endpoint` requires active rest intent. Probe runs ended less than one second after admission and showed zero seat-gate involvement, matching the boundary. NAMEPLATE remains source-eliminated: no broker/session-mint/erhost file in that delta. | **CLOSED — GATE PASS** on `test/keystone-182-family-b` @`541e56f` (code @`a58a559`): after broker truth appears, poll under a hard 10s bound for the observable sid rotation before killing the controller. A fresh self-bind passes; unreadable, never-bound, or genuinely re-adopted records return the still-equal observed sid and remain red. Focused cell 1/1 green (6.97s), focused Clippy `-D warnings` green, traceability 776/776. The pre-registered signature-classified gate on the lane tip ran the exact cell ×20: **family-B sid signature 0/20** versus the hot 4–5/20 baseline, proving the fix effective. All four failures were the independent pre-#197 family-A throttle signature at 61–63s — expected on this `f2d215a`-based lane and not counted against this fix. Rides the next assembly as the test lane. |
| `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1150 PRECONDITION (warm bring-up never spawns) | 3×/3 hfenduleam main-checkout pool (2026-08-17, main @27d40b9, agent env scrubbed, 186s each — consistent, not intermittent) | **TEST INFRASTRUCTURE DEFECT, not product evidence (IR-21/IR-39 class).** The clean main target had no `target/debug/mock-session.exe`: `cargo test -p spt` builds the `spt` test target but not another package's `mock-adapter` binary. `engine_room_bringup_e2e` derived that absent sibling path without asserting it, registered it as the harness command, then spent the full launch bound waiting for a program that never existed. This fully explains `warm bring-up spawned=false`, `endpoint stop` → `NO_SUCH_ENDPOINT`, the empty brain stderr, the 186s duration, and why a warm assembly pool passed or reached a later assertion. Explicitly building `cargo build -p mock-adapter --bin mock-session` restored the real loop; the earlier environment-sensitivity hypothesis is retired. Distinct from the genuine offline/control-projection race and from row 41's Linux occurrence. | CLOSED by test-only hardening: fail immediately with the missing path plus actionable build command, and prebuild the cross-package fixture before the golden suite. No product policy changed. RCA independently source-confirmed by doyle 2026-08-17 |
| `spt-daemon daemon::tests::a_tree_teardown_reaches_a_grandchild_the_service_spawned` | 2× hfenduleam Windows golden: 10.176s run 30776330383 @184f2ac (2026-08-03); 10.225s run 30873007187 att1 @4b37512 (2026-08-04) | **TWO mechanisms wearing one assert string — do not count as a cross-sha repeat** (doyle source verification 2026-08-04): red 1 was the bare-pid `provably_gone(grandchild)` poller reading a pid-reuse stranger — it is the cited motivating incident in the pin comment (daemon.rs:2726-2737) and was RETIRED by a5042ec ("poll the process we pinned, not the number it happens to hold"); red 2 is POST-fix (pin present at 4b37512:2371) — a pinned, stamp-checked identity stayed not-provably-gone for the FULL 10s bound (daemon.rs:2757), mechanism OPEN (H1 unauthenticated SELECTION / H2 enrollment gap — see INFRA-REGISTER IR-17 sub-observation). Both reds sit AT the bound; todlando's off-CI discriminator ran 0/200 under live-fleet load with every pass 1-4s (never near the bound; filter positive-controlled via `nextest list` = exactly one test). The lesson this row carries: assert-body diffing across shas carried the string across a mechanism change — identity must include the polled predicate's SUBJECT, not just the message | INSTRUMENTED-AWAITING-FIRE (hertz selection probe, `SELECTION_PROBE` line rides main; a green retires nothing — doyle-ruled). Green at att3 17f95a7. The bound question (is 10s right for a loaded runner) parked with hertz pkg item 4; kill path and spawn flags untouched until the probe fires (#131 stays todlando's, operator-triage-gated). **GREEN-CAPTURE BASELINE (hertz item 3, 2026-08-04, doyle-scoped: run-local flags, N=20, this row only).** Repro: `cargo nextest run -p spt-daemon --lib --success-output immediate --test-threads 1 -E 'test(a_tree_teardown_reaches_a_grandchild_the_service_spawned)'`, ×20, on a released box (no CI worker, no release build), agent env stripped, lane `test/probe-green-capture` @`60ca8f8` base `6ec5237` — source-identical to the v0.54.0 tag `86f0d84` for this row (the range is the release commit only: Cargo.toml/Cargo.lock/CHANGELOG.md, **zero `.rs`**). **field 4 (ppid-match count) = 2 on 20 of 20 runs. min 2 / median 2 (lower-middle, convention pre-registered BEFORE the capture) / max 2. Raw: 2×20. ZERO variance.** All 20 exits 0; durations 1.86–2.79s (consistent with todlando's 1–4s passes, nowhere near the 10s bound). Population asserted per run (`Starting 1 test` ×20) and the 20 probe lines carry 20 UNIQUE candidate pids, so these are 20 distinct runs, not one line re-read. Both sweep populations 0 after: pool-scoped and the System32 `cmd`/`ping` victims. **What this establishes:** a second process in the child's parentage is STRUCTURAL here, not an excursion — hertz's prediction (pre-registered with doyle before the capture, falsifier `min=median=1`) SURVIVED. Mechanism named in the same lane's comment fix: this row spawns WINDOWLESS, which masks `DETACHED_PROCESS` back off (daemon.rs:1218-1219) so the child owns a console. **What it does NOT establish, and must not be read as:** (a) the companion's IDENTITY is UNMEASURED — the probe renders only the CHOSEN candidate and the reject census prints only on the panic path, so "the second match is a conhost" remains inference, not data; a red is what would print it. (b) The floor here is **2**, so the incident's `3` is an excursion above 2, **not** above the `1` the comment and register carry — on this box, at this sha, "it had always read 1" is not what the scan reads. That figure needs re-deriving on the hfenduleam runner before anything is built on it; a 20/20 zero-variance local read cannot speak for a different box. (c) A green still retires NOTHING (doyle's standing rule) — this is a baseline SHAPE, not a fix, and the row stays INSTRUMENTED-AWAITING-FIRE |
| `spt-daemon::brain_resume_conn_deadlock daemon_cursor_only_resume_keeps_heartbeat_live_respawn_interleave` (+ twin `…_steady_state` on att2) | 2× kitsubito Linux golden 34014574926 @34fdb848 (v0.67.1 docs-only patch #274): att1 `respawn_interleave` FAIL 6.822s 97 ticks/6s, twin PASS 6.809s in the same window; att2 (same sha, `rerun --failed`) BOTH twins FAIL — 88 and 97 ticks/6s, ordinals 1652/1655 | **LIGHT-POOL STARVATION OF A HEAVY-QUALIFYING BINARY — a classification miss, not the assert's mechanism.** The file spins a REAL broker + SIX real PTY `yes` floods per test and its two timing cells are byte-identical rigs, so Phase A runs them CONCURRENTLY (12 floods + 2 brokers) inside a 3186-tests-in-48s full-parallel window. It was in NEITHER `<HEAVY>` string at the sha and `git log -S` says it never was: born `03c71093` 2026-07-09, six days BEFORE the HEAVY-AT-BIRTH ruling (2026-07-15), which was applied forward and never back over pre-ruling files; sibling flood rig `input_ack_deadlock` IS heavy. The `>= 100` floor guards the SharedSend self-deadlock whose signature is ~0 ticks (header: healthy = hundreds); 88–97 is ~100× above it — a load-margin miss, and the deterministic `daemon_resume_leaves_zero_brain_subscribers` guard PASSED both attempts. Delta test: 5 files, ZERO `.rs` (CHANGELOG, Cargo version material, two docs-site pages) — the diff cannot reach the cell; Windows leg + n1 + twohost GREEN at the same sha (platform split = load tell). The `CONN_WRITE_RETIRED` BrokenPipe lines in the failure block are stamped mono_ms 6693+ — the rig's own `kill_pid` sweep after the 600ms settle + 6s window, teardown not cause. First sighting in ~2 months of goldens = the margin was thin the whole time. RCA: `RCA-274-R1-LINUX.md` (doyle, 2026-09-06); logs preserved `Documents\spt-preserve\v0671-golden-r1` | **HARDENED AT THE RECIPE (2026-09-06, this commit, config-only, folded on top of `34fdb848` for the v0.67.1 respin):** binary added to BOTH `<HEAVY>` strings (`.config/nextest.toml` override + `golden.yml` job-level `HEAVY` env; the two copies extracted and asserted byte-equal before and after by the edit script — the comment claiming "exactly two places" was MEASURED, ci.yml carries none). Same-sha rerun was tried ONCE under the named mechanism and its second red fired the pre-committed respin, never a third rerun. PRE-REGISTERED DISCRIMINATOR: in Phase B (serialized, quiet box) the twins run one at a time — a red THERE refutes the pool mechanism. Riders for hertz's next test lane (NOT this commit): re-sweep spt-daemon tests for other real-PTY/real-broker binaries outside `<HEAVY>` (report the population); `eprintln!` the tick count on green so the margin is visible before it reds **⚠ AMENDED 2026-09-07 (hertz) — A SECOND CELL IN THIS FILE, SAME FAMILY, DIFFERENT MECHANISM, AND IT FALSIFIES A CLAUSE ABOVE.** This row says "the deterministic `daemon_resume_leaves_zero_brain_subscribers` guard PASSED both attempts". True at `34fdb848` — but it must NOT be read as immunity: that cell RED at `8d980fdf` (W1 Windows battery, :352, 0.869s), its SEAM-SENSITIVITY arm reading `Some(0)` viewers for session 1 while the broker log showed `SUBSCRIBE_DECISION decision=viewer` for sessions 1/3/2 already landed. **MECHANISM, source-verified and product-INNOCENT:** the arm asserted an INSTANTANEOUS level the product is entitled to lower. `viewers.len()` is not monotonic after an attach — `push_frame_to_viewers` (broker.rs:3436-3448) evicts and `viewers.remove(&vid)`s a seat whose bounded channel fills, on the PRODUCER thread. The test's barrier orders the INSERT (KIND_SUBSCRIBE and KIND_SESSIONS are arms of one per-conn dispatch loop, and `add_viewer` inserts synchronously — both stated conjuncts HOLD) but has no ordering relationship with that evict, so the comment's unstated third conjunct — "no viewer writer blocks/evicts" — is the false one, and its "no timing window" claim is what sent two readers to the wrong place. Ties to this row's family exactly: six PTY floods per cell IS the pressure that fills the channel, light-pool starvation IS when the writer is not scheduled to drain it, and never-in-HEAVY follows because more CPU means the writer keeps up. **FIXED** by asserting the monotonic `next_viewer_id` (increments per attach, never decrements) via a new `test_session_viewer_attaches` seam, dropping the level assert from the seam arm only; the positive arm's `Some(0)` is the product's contract and is UNCHANGED. doyle-ruled TEST DEFECT, closed at named mechanism. todlando's ×3 same-pool control rides with this row and a GREEN control does NOT close it — the mechanism is load-dependent by construction, so a quiet box is the condition under which it cannot reproduce. |
| `spt::attach_link_push_e2e attachment_frames_reach_a_linked_shell_through_the_real_daemon` — clause 5 re-link snapshot | 1× Windows Phase B, golden [34017906638](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638), job [101447073928](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101447073928), sha `04e32c8c`: `changed=Some(node)` where the re-link snapshot requires `None` | **TEST ORDERING RACE:** spawning `rc --view` did not prove attachment before `bind(token_b)` and shell launch. The observer could remember the new link with zero viewers, then correctly emit a State frame when the viewer arrived. The premise “viewer arrived while the link was down” was unenforced. | HARDENED, test-only: observe the broker-owned `info.json.viewer_count` at zero after the prior viewer, then positive after the new spawn, before rebinding; missing/unreadable records never satisfy the barrier (an absent count in a readable record means zero). Snapshot assertion unchanged. Windows deterministic mutation removed the barrier, bound/activated and observed the empty Link snapshot before spawning the viewer: RED at the same `changed=None` assertion (`Some(node)`, 13.90s). Restored fix: 3/3 Windows passes, 13.14s / 14.21s / 16.01s; measured spawn-to-attachment barrier waits 305.5237ms / 253.4742ms / 134.4401ms. No product change. <!-- [doc->REQ-ATTACH-LINK-PUSH] --> |
| `spt-store::wtlock_two_process_int two_processes_commit_into_one_worktree_without_failing` — ARM1 contention | Loaded golden Windows observation (doyle report, run `34017906638`): a waiter caught no acquisition gap within the 10s production bound while the other child continued committing | **UNFAIR SENTINEL, WAITER LOSES GAPS UNDER LOAD; BOUND 10s SIZED FOR ONE OP.** Starvation is probabilistic: the owner can reacquire between its back-to-back commits, but the waiter can also catch a gap before that batch ends. A batch longer than 10s does NOT force the timeout. | HARDENED, test-only: each ARM1 child gets `SPT_TEST_WT_LOCK_WAIT_MS=120000`, derived from the existing `CHILD_BOUND`; no parent/global override, no change to lock code, fairness, polling, or backoff. ARM1's WAIT-never-fail assertion and A/B/wall reporting remain; ARM2 stays unchanged as the deterministic lock proof. Honest Windows non-reproductions with env UNSET and A observed holding the sentinel before B spawned: A24/B6 GREEN, A=13.1621871s; A96/B6 GREEN, A=59.2755278s. These correct the proposed deterministic-starvation premise; they do not disprove the loaded-box observation. Corrected normal rig: 2/2 GREEN; ARM1 A=2.3879168s, B=2.2405768s, wall=4.8639264s (A/B are the existing sequential wait measurements); ARM2 waited=1.5035324s for a 1.5s hold. No reaped cmdline matches or image-name matches in either arm. <!-- [doc->REQ-PSYCHE-INGEST-SERIALIZED] --> |
| `spt-daemon::twohost_web two_host_web_role_b` — the A_CELLS completion witness (:257-284) | 1x, the FIRST-EVER real pair run (one box, 2026-09-07 02:56Z, `.worktrees/gate-w1-26a96d58/.spt/twohost-web/none2/`). No pair had ever run role B before — kitsubito's leg was the env-gated no-op, so this cell had never once executed for real. | **NOT A FLAKE — a DETERMINISTIC completion-witness defect, and it would have failed every pair run forever.** Filed here because doyle routed it here; the distinction matters to whoever reads this row next, because a reruns-sometimes-passes reading would be wrong. B's witness polled `brain.net_streams()` every 250 ms for peer-initiated rows and timed out at `role B saw only 0 of 3 requester streams within 240s` — while THE PRODUCT WAS GREEN: A's three cells all passed (403 naming WEB / byte-equal / 206 with Content-Range, 0.14-0.23 s each) and B's own breadcrumbs proved it served all three (`WEB_STREAM stream=1 refused ACCESS_DENIED`, `stream=2 sent 200 35 bytes`, `stream=3 sent 206 4 bytes`). MECHANISM, source-confirmed at 8d980fdf, TWO removal paths not one: a served stream leaves the table either by `retire_stream` (the `retired` flag, filtered out in `stream_infos_filtered`, nethost.rs:1956-1961) or by `retire_stream_terminal` (nethost.rs:2053, `streams.remove`) — and the dispatcher's own worker performs that retirement in its `DispatchOutcome::Served` arm the moment the serve completes. On a loopback pair a cell finishes far inside the 250 ms gap, so the row is gone before the next sample. SAME CLASS AS ROW 46 (a witness reading an INSTANTANEOUS level the product is entitled to lower), reached by a different removal path — second instance in one day. | **FIXED** on `build/ws272-w1` as a fixup: new `run_dispatch_loop_observed` seam exposes a MONOTONIC `served` counter (`AtomicU64`, incremented in the `Served` arm BEFORE the retirement that removes the row), `run_dispatch_loop` delegates to it so all ~8 existing call sites are untouched; role B waits on the counter and no longer calls `net_streams`. Counts every answer the owner produced — dispatch.rs's `StreamFamily::Web` arm maps `Sent`, `Refused` (the 403) and `Failed` (the 502) alike to `Served`, reserving `DispatchOutcome::Failed` for a transport error, so the deny cell counts and a retryable transport failure cannot inflate the total. NOT a shorter poll: a race with a smaller window is the same race. Re-gate is the pair only (doyle). |
| `spt::resident_service_e2e a_declared_service_rises_with_the_daemon_and_reaches_the_cli` — the **:453 PRECONDITION** assert (a THIRD distinct signature for this test: distinct from its :664 teardown-LEAK row above, and the direct successor of that row's noted ":382 PRECONDITION" signature, which is this assert at its old line) | 1× kitsubito Linux (2026-09-07, the W1 #249 builder battery AT the gated sha `8d980fdf`, gate dir `~/spt-w1/.worktrees/ws272-w1/.spt/ws272-w1-gate/`; nextest exit 100, ONE Summary `2999 tests run: 2997 passed (8 slow, 1 leaky), 2 failed, 1 skipped`, `panicked at` = 2, FAIL at 53s). Raw preserved at `.spt/preserved/w1-kitsubito-8d980fdf/nextest.raw`, sha256 `9c456e21a6a0f75d…3b51e`, verified against the remote by hash | **The witness expired; the daemon did not fail to come up — the assert's own words are false as rendered.** `daemon_up = wait_until(45s, brain_ready(&ready_path).is_some())` (:207) went false, and the panel that same assert prints shows the tree UP: broker generation 0 pid 3300168, `BRAIN_UP` pid 3300200, `BRAIN_PHASE:announce done in 1ms`, `BRAIN_PHASE:resume done in 0ms`, `SERVICE_BOOT:svcboot: Started`, `SERVICE_STARTED` for BOTH svcboot and relshell, and both services reaped `verdict=KILLED` with an empty survivor set. Load: a 2999-test Phase-A pool with 8 slow siblings, on a binary that spawns a REAL `spt daemon run` tree and was NEVER in the HEAVY class — the 2026-07-15 rotating-victim mechanism verbatim, same 45s-deadline family as the `resume_no_control_steal_e2e` row below (its co-victim in this very Summary). **UNMEASURED, not concluded:** where the 45s went. Nothing stamps the interval between the daemon child's spawn and the brain's first log line, so the ~10.1s exe-hash on the ready path (measured v0.66.0, a different lane) is a CANDIDATE here and nothing more. **Named and EXCLUDED:** the panel carries `DOCS_SERVER_BIND_FAIL: port 5474: Address already in use (os error 98)` twice — the rig daemon lost the well-known docs port to kitsubito's resident perch daemon. `EADDRINUSE` returns immediately and the daemon continues by design, so that is a rig-hygiene defect (fixed below), NOT this red's cause; it is recorded here because it is the loudest line in the panel and would otherwise be re-derived by the next reader | **HARDENED AT THE RECIPE (2026-09-07), not at the test.** Added to both `<HEAVY>` strings under the HEAVY-AT-BIRTH ruling, together with 10 other never-swept daemon-tree binaries — the census (method, all eleven names, and the byte-equality assertion on the two copies) is the stanza in `.config/nextest.toml`. The 45s budget is deliberately UNTOUCHED: retuning a deadline is the same race with a different number. Separately and independently of this red, every rig `spt daemon run` spawn in `crates/spt/tests` now sets `SPT_TEST_EPHEMERAL_ADVISORY_PORTS=1` (37 sites across 31 files; 2 sites already had it) so no test tree competes with a resident fleet daemon for 5474 again. **ROW STAYS OPEN** — the harden is a prediction until this binary is seen executing in Phase B; a red on the quiet serialized box would refute the pool mechanism and re-open the box/product question |
| `spt::resume_no_control_steal_e2e brain_respawn_keeps_every_session_controller_and_still_promotes` — keyed on the MESSAGE `the trial candidate never stamped brain.ready`, NOT on the rendered line. The panic renders at `:358`, which is `teardown_panic`'s funnel `panic!` shared by every failing arm in the file (:397 and :490 today); a row keyed on :358 would silently absorb an unrelated future red | 1× kitsubito Linux (2026-09-07, same battery, same Summary and same preserved raw as the `resident_service_e2e` :453 row above — the two are co-victims of ONE window; FAIL at 46s) | The `:490` call site: `wait_ready_pid(&ready_path, 45s)` returned `None` — the trial brain candidate never stamped `brain.ready` inside the budget. Panel: `CONN_WRITE_POISONED conn=3 … wall_ms=1788745673787 mono_ms=2784` (the 800ms brain-write bound this test itself sets via `SPT_BRAIN_WRITE_DEADLINE_MS=800`), then `BRAIN_SUBSCRIBER_STALL_EVICT` 1/2/3 at mono 7327-7335 and all three controller conns `event=writer-exit … reason=write-failed kind=TimedOut` at 7343 — every controller conn retired long before the 45s expired. Same 45s-ready-deadline family and same never-swept-HEAVY recipe as the row above: a real `supervise_brain` + `spt daemon brain` tree at full Phase-A parallelism. **DISCRIMINATOR, and the reason this is not filed as a bare rerun-clear:** doyle's control leg re-ran BOTH reds isolated in the same pool — exit 0, ONE Summary `2 run: 2 passed`, at 10.47s and 11.92s against budgets they had just consumed 46s and 53s of. At-budget under load vs. 4x under budget alone is the load signature; a logic defect does not respect pool occupancy. **NOT claimed:** that the 800ms write bound CAUSED the missing ready stamp — the poisoned conns belong to the OLD generation's controllers and are equally readable as a co-symptom of the same starvation; distinguishing them needs a stamp this rig does not have | **HARDENED AT THE RECIPE (2026-09-07)** with the row above and the same nine other binaries — see the `.config/nextest.toml` census stanza. Neither the 45s wait nor the 800ms knob was retuned. **ROW STAYS OPEN**, same pre-registered discriminator: serialized in Phase B this binary runs alone, and a red there refutes the pool mechanism and re-opens the product question |
| `spt-daemon::mesh_recovery roster_route_survives_a_transient_dial_failure_with_discovery_disabled` — "never converged: roster route converged after the transient" (`mesh_recovery.rs:97` at `25e60015`; `converge()` = 600 × 25 ms = 15.0 s wall-clock budget) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 3 (2026-09-09 00:43:52Z, job 102283894969, Phase B serialized, cell 15.715 s, 233/234). Same cell PASSED at the same sha on attempts 1 and 2 (9.801 s, 7.172 s). | **BOX CONTENTION, not product — budget sat inside the box's own variance:** the pump/dial path is untouched by #272 (diff v0.67.0..25e60015 on `crates/spt-daemon/src/pump` + `crates/spt-net` = webmsg/xfer only). Stderr: cache-leg `PUMP_PEER_FAIL` at the 1.5 s test dial bound as designed, roster-leg `PUMP_DIAL_SUBMIT`, then no line for ~13.8 s until the panic — a burst ate the 5-8 s of headroom the cell had. Box evidence: Phase A (pure unit) slowed monotonically 448.7 → 495.1 → 542.6 s across the three attempts at ONE sha; per-cell attempt3/attempt2 over 73 Phase B cells ≥ 1 s: median 1.05×, 19 cells ≥ 1.5×, worst 5.2×; cargo/rustc/nextest 0 and ~1.1 of 16 cores busy at census; no other CI run on either runner. Mechanism candidate (todlando, measured by doyle 01:17Z): Windows Defender real-time ON, MsMpEng at 68 % of a core on the idle box, a fresh 35 MB exe pays 1.0-2.1 s on first execution vs 20-260 ms warm; exclusion list unreadable unelevated. Rotating single victim across attempts (ttl 5 s cell on a2, this 15 s cell on a3) = RANDOM-VICTIM signature: one env cause, N apparent flaky tests. Same-sha rerun-failed ruled once more (attempt 4, last), stop on any Phase B red. | OPEN — hertz rider 5 ordered 2026-09-09 01:20Z (post-v0.68.0 thin PR, base e4444413): the `for _ in 0..600 { sleep 25 ms }` shape is 31 sites across 4 test files (mesh.rs 9, mesh_recovery.rs 6, pump.rs 7, registry_lifecycle.rs 9; closed family repo-wide); derive every budget from the named bound it races (test-local const shared with the rig's `set_quic_op_timeout`), keep it under the 60 s nextest SLOW line, print elapsed / samples / last predicate state at the panic; predicates untouched. Runner-desktop contention registered as an INFRA entry (Defender exclusion = operator). |
| `spt-daemon::registry_lifecycle oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing` — "the refresh must subscribe NO historical rows (zero replay churn): held 0 -> 1, seats 1" (`registry_lifecycle.rs:514` at `25e60015`; the directional held/seats assertion after the gen-2 refresh) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:36Z, job 102306494097, Phase B serialized, cell 11.594 s, 233/234). Same cell PASSED at the same sha on attempts 1-3 (10.817 s, 10.518 s, 23.344 s). The two earlier victims of this arc (arm 12 ttl race; mesh_recovery converge budget) both PASSED on this attempt. | **RIG SAMPLE RACE, load-widened — not product:** A's pump is a bare `thread::spawn` (:307) stopped by the `pump_stop` flag (:386) and NEVER JOINED; the drains converge on B's gauges, gen-1's dispatcher is joined (:414), `b_held_before` is sampled, gen-2 starts — and the pump's last 100 ms-cadence round can still deliver ONE feed to B after that sample, which B then holds as one row with one in-flight seat. One row, not a replay storm: the re-apply bound one assertion earlier (:497) PASSED, and a replay regression re-subscribes the whole history. The test's own comment (:500-506) records this single-straggler face ("the extra held row was ONE straggler feed … mis-attributed to gen-2"). Product path untouched by the milestone (`registryhost.rs`/`pump` no diff v0.67.0..25e60015; `dispatch.rs` changes are the Web stream family; `broker.rs` a test accessor). Stderr carries only `CONN_LIFECYCLE` lines: a stream-8 subscription attached at mono 8.1 s and a serve on it at 8.7 s, neither released before the panic at ~11.45 s; conn ids are not attributable to A vs B from the log. Box: per-cell attempt4/attempt2 median 1.00× but 12 of 72 cells ≥ 1.5×, worst 4.75× — bursty. Third distinct single victim in four attempts at one sha = one environment cause (random-victim family); STOP ruled, sha retired. | OPEN — hertz rider ordered 2026-09-09 02:47Z (test-only, stacked on b359e40e, rides r3's head): `spawn_pump` returns its JoinHandle and the test JOINS it after `pump_stop` before the drains/sample (verify `run_peer_pump` exits at a round boundary on the flag); the :514 panic prints the gauges AND B's stream table (ids/families); audit the file for other flag-stopped-never-joined actors sampled by gauges (:417 documents the gen-1 worker case); predicates and both directional assertions untouched. |
| `spt-daemon::twohost two_host_ladder_role_a` — "A-3 setup suspend applied an edge at B: NoEdge" (`twohost.rs:2496` at `25e60015`; the rung asserts `matches!(out, RestRequestOutcome::Edge(_))` on the wire reply) | 1× golden twohost-a (hfenduleam role A / kitsubito role B), #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:44:22Z, job 102316461906, 19.95 s; twohost-b job 102316461873 then burned its 900 s on the wake anchor A never sent — collateral, one transaction). The same rung PASSED at attempts 1-3 (attempt 3: A-3 reached B ~4.5 s after ID_B's wake; attempt 4: 10.3 s, the toast rung's replication wait ran 7.8 s vs 1.0 s). | **RIG ASSERTS THE WIRE DISCRIMINANT AGAINST THE DOCUMENTED CONTRACT; the double delivery is a pre-existing dispatcher overlap, not this milestone.** Contract (`resthost.rs:21-27`, `:198-202`): exactly-once covers the rest STREAM OPEN only; the request line is unjournaled and a redelivered rest request is a natural `no_edge`, "report, don't dedup". Evidence: B's stored intent for ID_B (`read_rest` = info.json `rest_state`, no derivation) flipped Active → Suspended inside [22.348, 22.460] (B polled its anchor at 250 ms from 14.593Z, passed 22.598Z), the window of A's single request (`request_rest` is one-shot), which was answered `no_edge` at 22.4617Z and by the transition guard wrote nothing — so a second Suspend was applied at B in that instant. B's `CONN_LIFECYCLE` shows the Rest worker's fresh-query-then-subscribe pair TWICE for stream 85 (conn 119/120 @22.4327-.4330; conn 121/122 @22.4579-.4586, conn 122's close = A's reply; conn 120 closes 22.4973). One dispatcher generation cannot re-serve at 25 ms (InFlight claim held until the worker returns, retry floor 500 ms), so two dispatcher instances = two brain processes against B's broker; the claim path is untouched by #272 (one counter line, a thin wrapper, a comment word in the Rest arm). Auto-suspend (node knob off, no daemon.json on the box, tick fires only from Dormant, per-endpoint override never set) and the liveness-derived state (never persisted) are ruled out from source. Dispatcher generation is not logged on a clean serve. | OPEN — hertz rider 4 ordered 2026-09-09 03:11Z (test-only, rides r3's head): the four wire-Edge assertions in `twohost.rs` (:2223, :2242, :2497, :2541) accept Edge or NoEdge, fail only on Refused/Failed/NoReply/BadRequest, print which reply arrived, and witness the DURABLE observable each rung already waits on (registry row at A; `read_rest` at B). Two post-publish product lanes seeded for todlando: a served-path `dispatch_event` naming its dispatcher generation; the two-dispatcher overlap hazard (a3's B log: 27 streams attached by 2+ connections; a4's: 3 — the overlap may be constant and normally masked by retire-before-second-poll). **CORRECTION 2026-09-09T03:22Z (todlando split, manifest-verified a3 B log): lane-2 data = a3 25 non-controller query+subscribe pairs (2 of the 27 were controller handovers: streams 9, 82), gaps 0-21 ms median 4 + one 127 s long-lived-row outlier; a4 2 pairs (stream 12 was a handover), gaps 25.3/25.5 ms. Scatter => fixed-period phase-offset story REFUTED and withdrawn. Standing read: two dispatcher workers, constant overlap, normally masked by the pre-serve 'row already gone' arm (dispatch.rs:1053-1060, unlogged); load stretches the mask window. Lane-2 instrument = generation named on the Served path, count distinct generations.** |
| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` — ARM 12 "the attachment this arm just registered is not in the registry" (`webserve_attachment_e2e.rs:612` at `25e60015`) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 2 (2026-09-08 20:33:40Z, job 102216542171, Phase B serialized, cell 11.733 s, 233/234). The emitter's OWN comment records an earlier occurrence at ttl 1 s during lane development (one red then a pass), widened to 5 s — this is the same signature one size up. | **RIG WALL-CLOCK RACE, not product — two product mechanisms EXCLUDED at the sha before the word:** (1) an in-daemon clobber between the register write and the reaper tick: `livehost.rs:1244 reap_expired_attachments` runs under `servehost::with_registry_write` and `servehost::apply_at` takes `REGISTRY_WRITE`; the CLI registers THROUGH the daemon socket (`attach.rs:94`), so one writer process, lock holds; (2) a torn read: `ServingRegistry::save_at` is `atomic_write_bytes_durable`. POSITIVE EVIDENCE (deployah 20:52Z): the SAME cell PASSED at the SAME sha on attempt 1 of the same run (Phase B 234/234, 18:47-19:31Z), nothing between the two attempts but a pool reap and a queue — a pass-then-fail at one sha is the race signature, not a deterministic defect. The cell's own envelope was 11.733 s against a 5 s ttl, so the register-to-read gap had room to cross it. Mechanism: the arm mints a 5 s attachment, the reaper ticks every 5 s, and the spawned `spt send` child plus the read landed past expiry on the loaded golden runner; the daemon's own `ATTACHED … ttl 5000ms` line is in the panic, so the serve path DID register it. Elapsed time UNMEASURED: the Windows sandbox cleanup step removed the rig home at 20:41:09Z before anyone could read a `SERVE_REAP` stamp. Same-sha rerun-failed ruled once (rate rerun, said so), non-vacuity = the cell re-executes and passes. | OPEN — hertz rider 3 BUILT 88625fa0 (green Linux + Windows, unpushed, post-v0.68.0 thin PR; ordered 2026-09-08 20:50Z): make ARM 12 deterministic (long ttl + expire under the daemon's own writer path, or capture the snapshot path from the send's stderr) and print elapsed ms at the panic so the next occurrence carries its own number. |
| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` ARM 11 — `webserve_attachment_e2e.rs:551` "each attachment registered its OWN entry" left 4 right 5 | Windows at c4919243/tree be2184af: 4 reds in 15 runs (x5: FAIL 14.591 s; x10: FAIL runs 4/8/9 at 17.707/22.834/24.498 s), every red exactly -1; fastest run of 15 is a red, slowest a pass — no elapsed correlation. Linux 5/5 at 9.9 s flat (window proportionally smaller; not evidence of absence). | **COUNT-DELTA ARM RACING A DELIBERATE CORPSE AGAINST THE 5 s PULSE.** ARM 10 mints a real `--ttl 1s` attachment and sleeps 1.5 s; `ServeRequest::List` (servehost.rs:224) returns entries unfiltered so the corpse is in `before_count`; `reap_expired_attachments()` rides the 5 s reconcile pulse (livehost.rs); a pulse inside the before..after window makes the delta +2-1. Sibling of the arm-12 race afb711c9 retired, one arm up. Census at the sha: ARM 11 is the only count-delta arm after ARM 10. | FIXED test-only by rider 5 f6110c2a (parent c4919243): ARM 11 asserts identity (each ATTACHED url's served name present in `serve list --json`), before_count and the +2 delta removed, no Reconcile between arms. _SIBLING ROW (arm 11) 2026-09-09 03:5xZ, found by hertz's r3 rider proof legs, verified by doyle from source at c4919243:_ |