# Flake ledger

Tests that have failed nondeterministically, with mechanism notes and hardening
status. Discipline: **any flake gets an entry, not a shrug** — a second
occurrence upgrades the entry from "observed" to "harden it." Rig flakes from
the two-host ladder (docs/TWO-HOST-RUNBOOK.md) land here too.

| # | test | occurrences | mechanism | status |
|---|------|-------------|-----------|--------|
| 1 | `spt-daemon brain_restart_survives_gaplessly` | 1× hosted Windows (2026-06-03, hung) | ConPTY stall under hosted-runner load | job `timeout-minutes` bounds it; not seen self-hosted |
| 2 | `spt-term digest_parses_a_real_pty_stream` | 3× gravity (2026-06-03; 2026-06-04 runs 26957386850, 26966738231) | sprint-collapse assert (`digest.rs`) — PTY input echo and the child's write are **concurrent writers** into the master-read stream; back-to-back `send_line`s let a later line's echo interleave mid-`Write(…)`, breaking the tool regex for that turn | HARDENED (2026-06-04): sends serialized — each line pumped back (adaptive echo-copy count, 2s/line bound) before the next, so no concurrent writer races an earlier line; failure asserts now print the raw transcript |
| 3 | `spt-daemon sync.rs torn_pull` + `concurrent_writes` | 1× (2026-06-03) | `wait_for_stream` deadline too tight under load | HARDENED: `wait_for_stream_except` 2s→10s; no recurrence |
| 4 | `spt-live context::write_context_suppresses_stale_llm` | 1× gravity (2026-06-04 run 26967808321) | test calls `write_context` (reads SPT_HOME node identity + epoch stamp) WITHOUT the `with_home` lock — a parallel `with_home` test's env swap + temp-dir teardown races the stamp's epoch write into a deleted home (`NotFound`) | HARDENED (2026-06-04): test wrapped in `with_home` — serializes on the home lock like every other SPT_HOME-touching test |
| 5 | `spt-daemon attach.rs:600` re-serve (`output gap: got seq 1 want 0`) | 1× kitsubito Linux (2026-06-15 run 27545857429, PR #14 stale-base CI) | re-serve replays the buffered output stream on re-attach; under load the new subscriber observed `seq 1` before `seq 0` — an ordering/timing gap in the re-serve sequence assert, not a content error. Subsystem disjoint from the triggering change (PR #14 = digest-proof CLI key-fill, cannot touch attach sequencing) | OBSERVED 1×: did NOT reproduce on the clean-base rerun (`gh pr update-branch` onto post-#15 main → green both runners). Not hardened; a 2nd occurrence upgrades to harden |
| 6 | `spt-term tests/stream.rs:49 bounded_backpressure_stalls_an_unconsumed_reader` | 1× hfenduleam Windows (2026-06-15 run 27545857429, PR #14 stale-base CI) | asserts an unconsumed bounded stream stalls the writer-side reader; got `16 -> 82` (reader advanced past the bound) — a timing-sensitive backpressure assert racing the bound check under hosted-runner load. Subsystem disjoint from PR #14 (digest-proof CLI) | OBSERVED 1×: did NOT reproduce on the clean-base rerun (green both runners). Not hardened; a 2nd occurrence upgrades to harden |
| 7 | `spt-daemon::attach attach_survives_target_brain_restart_exactly_once` | 1× kitsubito (2026-06-16 run 27595180782 attempt 1, M11-W3) | **TIMEOUT, not an assertion fail** — SLOW>60>120>180>TERMINATING, nextest TIMEOUT at the 240s cap (`attach.rs:600`). A heavy multi-process brain-restart-exactly-once test on the slow shared kitsubito box under concurrent load (n1-gate on the same runner + a parallel `ci.yml` docs-publish run firing simultaneously). W3-INDEPENDENT: Windows green end-to-end; ALL tunnel tests passed fast on Linux too (tunnel_e2e 0.941s, loopback_tunnel_backpressures_without_loss 1.091s) — a logic bug fails both runners deterministically, this timed out on one under load | observed; cleared by `gh run rerun --failed` (builds reused, no re-tag). Harden if it recurs — candidate: raise this test's per-test nextest timeout, or serialize it off the concurrent n1-gate slot |
| 8 | CI build step `Build notify-shell` (kitsubito Linux) | 1× kitsubito (2026-06-16 run 27652755792 attempt 1, v0.8.1 PR #17) | crates.io dep-download blip during the notify-shell build — `download of config.json failed, curl failed` fetching `serde_json`: a transient registry/network fetch failure on the runner, not a code or test fault (a real dep break fails both runners deterministically; this hit one runner once) | observed; cleared by `gh run rerun --failed` (attempt 2 green, builds reused). Harden if it recurs — candidate: a cargo fetch retry / registry cache warm on the runner |
| 9 | `spt dummy_harness_e2e` (BOTH tests: `endpoint_run_brings_up_a_long_lived_dummy_harness_and_rc_attaches` + `endpoint_run_attach_awaits_online_before_attaching`) | 1× kitsubito (2026-07-05, REMOTE-TRUTH Batch-2 re-gate @f2a799e; roles swap run-to-run) | **process-global `set_var` race in a multi-test e2e binary.** Both tests `std::env::set_var("SPT_HOME", own_tempdir)` then do IN-PROC reads (`perch::spt_home`/`adapters_dir`, `registry::register`, psychebin `fs::copy` staging). One file = one test binary → the two run on parallel threads; whichever set_var last wins for BOTH → `register`/staging land in the wrong home (`ENDPOINT_RUN_ADAPTER_UNREGISTERED`; os-32 sharing collision). Child procs were safe (explicit `.env`) — only the in-proc reads leaked. File untouched since F-030; Batch-2 broker-reap timing shift woke it. `--test-threads=1` → 2/2 green (proves serialize) | HARDENED (2026-07-05): file-local `static E2E_LOCK: Mutex<()>` held whole-body in each test (contract_e2e.rs:26 pattern); green at default parallelism. **CLASS = process-global `set_var("SPT_HOME")` + in-proc read in a multi-`#[test]` e2e binary.** doyle's gate2 (`--no-fail-fast`) proved it beyond dummy_harness: `live_adapt_translation_swap_e2e` (3/4 red) + `multi_subnet_bringup_e2e` (3/3 red), same signature. **SWEPT (anchored `grep -cE '^#\[test\]'` census):** locked `dummy_harness_e2e` (2), `live_adapt_translation_swap_e2e` (4), `multi_subnet_bringup_e2e` (3), `translate_proof` (2); `contract_e2e` (26) + `gateway_e2e` (35) already locked. EXCLUDED `brain_survive` + `n1_pairing` — each a SINGLE `#[test]` (no intra-binary parallelism → cannot race); both already carry an in-file "a 2nd `#[test]` must bring the env-lock" deferred contract (a never-contended Mutex contradicting its own doc is worse than nothing). NB: doyle's first census overcounted these two — an UNANCHORED `grep -c '#\[test\]'` counted the attribute mentioned in their comment prose; anchor the count |
| 10 | `spt-store registry::tests::concurrent_registration_never_locks` | 1× hfenduleam local gate (2026-07-05, REMOTE-TRUTH Batch-2 bless matrix @3f6f327, fresh worktree, `nextest --workspace --no-fail-fast`) | 16-concurrent-writer SQLite registration gate FAILED at 5.63s — the SQLITE_BUSY load-contention class ALREADY documented at `db.rs:60`: box saturated by the 1588-test full-parallel suite meant a writer couldn't schedule its retry inside the default 5s `registry_busy_timeout`. Isolated ×5: green, 0.25–0.35s each. NOT a product defect — the exact scenario the `SPT_REGISTRY_BUSY_TIMEOUT_MS` env knob (default 5s) was added for | observed 1× (local gate, not CI); in-code remedy pre-exists: set `SPT_REGISTRY_BUSY_TIMEOUT_MS=30000` on saturated gate rigs. Harden candidate on recurrence: bake the raised knob into gate-matrix/CI test invocations |
| 11 | `spt-daemon::resume resume_mode_brain_spawning_new_sessions_delivers_each` | 1× kitsubito (2026-07-05 run 28768663321 attempt 1, REMOTE-TRUTH [twohost] PR #52) | Phase-B heavy-class timing EOF — "session exited before the expected output arrived" (UnexpectedEof) at 62s under the serialized heavy pool. SAME FAMILY as this file's prior Linux-CI flake (7940318, D4-2b: sequential spawn/read hardening) — a spawned session's exit raced the expected output read on the slow shared box. File untouched by the wave; Windows leg green same run; the wave's spt-daemon changes (C1 broker lifecycle) are cross-platform and every local Win gate was green — but note this run was the wave's FIRST full Linux suite | observed 1×: cleared by `gh run rerun --failed` (attempt 2 green, builds reused, no re-tag — ledger #7/#8 pattern). Harden if it recurs — candidate: extend the 7940318 sequential-spawn/read pattern to this test's spawn loop, or a per-test nextest timeout raise |
| 12 | `spt-daemon::input_ack_deadlock input_flood_through_serve_attach_does_not_deadlock_broker` | 1× hfenduleam local gate (2026-07-06, REMOTE-TRUTH F-2 matrix @a21bc6b, `nextest --workspace --no-fail-fast`) | deadline-margin under full-suite load: the flood exchange's `recv_timeout(30s)` (input_ack_deadlock.rs:512) blew under the 1608-test parallel pool + 7 leaky daemons — FAIL at 34.5s vs ~32s natural runtime isolated (the margin is ~3s by design). Subsystem disjoint from the triggering commit (F-2 = api/reporting.rs soft-end guard; cannot touch broker input-ack). Isolated ×5: green, 31.9–32.2s | observed 1× (local gate, not CI); harden candidate on recurrence: widen the :512 recv bound or serialize the flood test into the Phase-B heavy class (CI already runs it serialized — this class only bites full-parallel local matrices) |
| 13 | `spt::bin cli::tests::probe_all_cap_batches_into_windows` | 1× hfenduleam local gate (2026-07-06, stack matrix @0f08fa9 leg-3 rerun) | fast unit (0.52s natural) FAILED at 1.7s under the 1614-test full-parallel pool — probe-batching window assert with a timing component starved under load; subsystem disjoint from the triggering stack (F-2 reporting / E-2 grid title / picker rider). Isolated ×5: green, 0.52–0.55s. Same night as #12 — the full-parallel local matrix under a busy box is the common factor, not the tests | observed 1× (local gate, not CI); no harden yet. NIGHT-PATTERN NOTE: #10/#12/#13 are all first-sightings from the same gate recipe (full-parallel nextest on a loaded shared box) — if a FOURTH distinct test trips this way, harden the RECIPE (nextest profile with heavy-class serialization locally, mirroring CI's Phase split) rather than the tests |
| 14 | `spt::bin rc::tests::attach_viewport_reconnects_across_a_broker_bounce` | 1× hfenduleam local gate (2026-07-06, WORKER-TRUTH W-3 build @worker-truth, `nextest -p spt --bin spt` full-parallel 360-test run) | **TIMEOUT, not an assertion fail** — SLOW>60…>TERMINATING at the 240s cap under the 360-way `--bin spt` pool. A REAL-broker + broker-bounce reconnect UNIT test (rc.rs:2071) that escaped the HEAVY-class serialization: the Phase-A/B split + `heavy-broker-pty` test-group target integration test BINARIES (`kind(test)`), but this is a `kind(bin)` unit test inside the `spt` binary, so it ran in the LIGHT pool at full parallelism and starved. Isolated ×1: green ~1.3s. Subsystem (rc attach/broker-bounce) disjoint from the W-3 worker-reap wave that triggered the run | **THE FOURTH DISTINCT FULL-PARALLEL TRIP** the #13 NIGHT-PATTERN note called for — so HARDENED at the RECIPE, not the test: the rc real-broker/bounce unit class is folded into the `heavy-broker-pty` nextest test-group (`.config/nextest.toml`, a `kind(bin) & test(/^rc::tests::(…)$/)` override) so it serializes with the rest of the heavy class and can never be starved by the light pool again. EXTEND the name set on each new rc-broker sighting |