---
name: update-wedge-rca-v53
description: RCA of the fetch--apply wedge — v53 (0.30.2) brain fails the update trial against the live v51 broker; 30s brainless freeze then rollback
metadata: 
  node_type: memory
  type: project
  originSessionId: 2ac55eb7-e705-4c6b-8665-2fcab1cdb922
---

**Incident (2026-07-09, hfenduleam):** `spt update fetch --apply` (v51/0.30.0 → v53/0.30.2) FROZE all 7 live session PTYs. Operator observation that cracked it: the `spt rc` 30s reconnect countdown did NOT fire during the freeze (only after `daemon stop --force`) → the broker did NOT crash, it stayed alive holding every connection while the BRAIN was gone → sessions unprocessed = frozen, no disconnect. NOT a broker crash.

**ROOT (ground truth, daemon.stderr.log lines ~24277-24290):**
- `fetch --apply` → `apply_staged` → `request_brain_restart` → broker cycles the brain child onto the swapped binary (`BRAIN_UPDATE_RESTART` / `BRAIN_RESTART: generation 1 start-reason update`). Broker (v51) stays up holding endpoints; only the brain restarts (the seamless-update design).
- New v53 brain: `BRAIN_UP … generation 1 … start-reason update` (connected) → net setup (`NET_FAMILY_GATE`, `PAIR_MEET_UP`) → **NO `BRAIN_RESUMED` line** (the working old brain logged "re-attached 7 session(s)") → **`BRAIN_TRIAL_TIMEOUT: candidate alive but never ready`** after the 30s trial window (`SUPERVISE_HEALTHY_RUN=30s`, brainproc.rs) → **`BRAIN_ROLLBACK: update v53 failed readiness — rolling back to v51 (spt.exe.old-53), quarantining v53`**.
- **The freeze = the 30s trial window**: old brain killed, new v53 brain never reaches ready, 7 sessions brainless. The v53 brain fails the resume/ready handshake against the still-running **v51 broker** (newer-brain↔older-broker). Sub-root (resume_sessions hang vs ready-stamp gate-reject vs net-consumer block) NOT yet pinned — needs an isolated repro (v53 brain vs v51 broker trial).

**Second bug (recovery also broken):** after rollback, the gen-2 (crash-reason) brain came up but `BRAIN_SUBSCRIBER_STALL_EVICT:1 — controller writer blocked >15000ms on a suspended/black-holed brain conn` fired (REQ-HAZARD-BROKER-VIEWER-BRAIN-DECOUPLE) → even the rolled-back brain left a session black-holed → continued freeze → operator `stop --force` + `start`. So the "auto-rollback → healthy" promise did NOT hold cleanly.

**Classification:** LIFECYCLE-TRUTH (v0.29.0 seamless brain-swap) REGRESSION, introduced v0.30.0→v0.30.2. Prime suspects = the PUMP-TRUTH W1/W2 diffs: pump/mod.rs +714 (W2 2-pump-brain event-drain split + D4c presence seam), brain.rs +161 (resume_sessions lives here @brain.rs:985), nethost.rs +236. The auto-rollback (D6) DID fire correctly; the trial-readiness + clean-rollback-restore are what regressed.

**FORENSIC LESSON (cost me ~an hour of wrong-home red herrings):**
- The LIVE modern daemon home is **`C:\Users\decid\AppData\Local\spt-core`** (binary product name `spt-core`). The legacy **`AppData\Local\spt`** is the pre-migration owl home, FROZEN at 2026-07-06 (last-seen-version 1.11.25 = old claude_skill_owl) — do NOT RCA against it.
- **`SPT_HOME/logs/daemon.stderr.log`** (REQ-DAEMON-STDERR-PERSIST, landed v0.29.0 @32c789e) is THE daemon/brain RCA source — brain stderr is otherwise /dev/null (detached). Grep it for `BRAIN_UP|BRAIN_RESUMED|BRAIN_TRIAL_TIMEOUT|BRAIN_ROLLBACK|BRAIN_UPDATE_RESTART|STALL_EVICT`. Interleaves broker + all brain generations; `=== spt brain stderr — generation N — pid … ===` headers delimit.
- RCA-discipline lesson: do NOT infer mechanism from ambient state (running procs, `spt --version`, version files) — anchor to the stderr log + the operator's exact action sequence. I was corrected 3× (version, --finish framing, action order) before finding the log.

**Next:** isolated repro (v53 brain trial vs a v51 broker, capture the trial-brain stderr) to pin the exact never-ready sub-step → dispatch todlando to fix (trial-readiness handshake + clean-rollback session restore) → gate with a real brain-swap-under-live-sessions rig. BLOCKS re-running `fetch --apply` on any live-session box (quarantine currently protects). Likely higher priority than the counter-54 followup cascade. See [[isolated-resident-broker-field-repro]], [[lifecycle-truth-build-progress]].

---

**doyle DRIVE UPDATE (2026-07-09, gate prep before todlando's repro report):**

- **RELEASE BASE = `origin/main` @3ea09bf** (NOT local main). Local `main` HEAD (5555648) is STALE — behind by the v0.30.1/v0.30.2 release merges + #70/#71, and it carries LEFTOVER pump-truth diag mods in the working tree (`pumpdiag.rs` + `pump_diag!` in pump/mod.rs, broker.rs, nethost.rs, sync.rs, lib.rs) — those are stale W1/W2 RCA instrumentation, NOT the fix; do NOT build/gate on this dirty local checkout. `origin/main` already carries v0.30.2 (pump W1/W2 shipped code) + #70 GATEWAY + #71 F-035 guard. **Counter 54 cuts from origin/main = update-wedge fix ONLY on top.**
- **Refined RCA (code-read, origin/main): the dispatch's 3 candidates likely MISS the real one.** brain.rs +161 in v0.30.0→v0.30.2 is ALL pump-mode methods (peer_reply_deadline / read_peer_reply_until / submit_dial / net_presence_subscribe_tail) — the **boot-to-ready path is UNCHANGED**: `run_brain` (brainproc.rs:159) does connect → `resume_sessions()` (near-noop, no daemon-hosted PTYs) → `write_ready(gen)` at **brainproc.rs:211, BEFORE** net-consumers/livehost. So the brain stamps ready EARLY regardless of pump. **CANDIDATE 4 (mine):** promotion needs BOTH `ready_seen && old_gen_drained()` (run_trial brainproc.rs:660). `old_gen_drained()` = `!broker.any_local_controller_wedged()` (brainproc.rs:534, REQ-UPDATE-PROMOTE-DRAINED, landed v0.29.0 e75e856 — gate itself NOT the regression). `WindowElapsedAlive` (:820) fires even when ready_seen=true, so **"candidate alive but never ready" is MISLEADING — it can mean ready-stamped-but-never-DRAINED.** A W1/W2 pump/nethost change that WEDGES a local controller conn ⇒ `any_local_controller_wedged()` stays true ⇒ never promotes ⇒ timeout+rollback ⇒ AND the SAME wedged conn fires `STALL_EVICT` (the 2nd bug). **ONE root explains BOTH legs.** todlando's repro must check: did the v53 brain WRITE its ready stamp (gen match) but never promote? If yes → root = wedged local controller from W1/W2, and the fix locus (brain-side conn behavior vs broker-side wedge detection) decides brain-vs-broker fix (the operator-flag pivot).

---

**todlando CONFIRMED verdict (2026-07-09, report-before-fix — pinned, reported to doyle, awaiting his design ruling + REQ registration):** doyle's CANDIDATE 4 is CORRECT — root-locked. Answer to the key question = **YES, the v53 brain stamped `brain.ready` with generation==1 yet still timed out** (empirical from log order, not inference): `write_ready` (brainproc.rs:211) runs UNCONDITIONALLY before the heartbeat loop; the loop is the ONLY path to `spawn_net_consumers` (:253), the ONLY source of `NET_FAMILY_GATE`/`PAIR_MEET_UP`; the v53 log (daemon.stderr.log 24280-24287) shows those between BRAIN_UP and TRIAL_TIMEOUT ⇒ main thread passed 211 ⇒ ready stamped. Refutes candidate #1 (resume_sessions hang never reaches 253). **Mechanism of the never-drain:** `any_local_controller_wedged()` (broker.rs:2703) is a PURE READ — never drives the evict. The stall-evict that CLEARS the wedge (`stall_evict_controller` broker.rs:1039) only runs via `reap_dead_controller` on a **KIND_SESSIONS poll** (broker.rs:2879). Nothing polls KIND_SESSIONS during the isolated trial window. Old brain HARD-killed (`child.kill()` brainproc.rs:851) ⇒ 7 live PTYs keep writing to a dead Windows named pipe ⇒ writer BLOCKS not EOF ⇒ `write_blocked_since=Some` ⇒ wedged=true for the full 30s ⇒ gate false ⇒ rollback. Post-rollback gen-2 KIND_SESSIONS finally reaps ⇒ STALL_EVICT (line 24303). **Both legs = ONE root: a wedged old-gen controller conn nothing DRIVES to eviction during the trial.** **Proposed fix (BRAIN-SIDE, self-applying — recommended):** incoming candidate brain issues a KIND_SESSIONS poll each heartbeat in its boot/trial loop (currently only `net_status`, brainproc.rs:244) ⇒ drives the LIVE older broker's ALREADY-SHIPPED reap→evict (≥v0.29.0; stuck broker is v0.30.0 so it has it) ⇒ evicts the black-holed conn at ~15s < 30s ⇒ promote; same poll on the rollback brain kills the STALL_EVICT recovery leg. ONE fix BOTH legs, avoids the operator STOP+flag (broker-side self-drive would NOT self-apply). Build FRESH off origin/main @3ea09bf under `.worktrees/`, not the dirty local tree. Gate = brain-swap-under-live-sessions int rig (promote-with-wedged-old-conn + clean-rollback-no-STALL_EVICT).

---

**FIX X SHIPPED (2026-07-09, todlando) — gate SHA `ce053dc`, counter-54. GATE PASS @ce053dc (doyle: clippy 0 | nextest 4/4 | traceable 0 | design-PASS | no leaks) → PUBLISHED v0.30.3 counter 54 @3136702 (deployah PR#72 bump-in-PR, rides #70+#71). Field-verify PENDING (not blocking): perri operator-gated on HFENDULEAM's 7 live agents (apply is operator-timed), flynn (non-critical) queued/suspended. See [[v0303-published]].** Branch `update-wedge-drain-drive` in `.worktrees/update-wedge`. impl = the brain.sessions() KIND_SESSIONS drive in run_brain's heartbeat (brainproc.rs, [impl->REQ-UPDATE-TRIAL-DRAIN-DRIVE]); doc = CONTEXT.md:572; int = crates/spt/tests/trial_drain_drive_e2e.rs. 3 traceable edits: TRIAL-DRAIN-DRIVE [doc,impl,int] + 2 deferred seeds (REQ-BRAIN-UPDATE-RESTART-CLEAN-CLOSE = graceful clean-close of the outgoing brain's conns on planned restart, the hitch-free root; REQ-LIVEHOST-RECONCILE-TRIAL-SILENT = why field livehost was silent). Local gate: nextest GREEN+RED-bites, clippy --workspace 0, traceable-reqs check 0.

**REUSABLE RIG-MECHANISM LESSONS (non-obvious; a future brain-swap/wedge rig needs these):**
- **A' rig = real `spt daemon brain` subprocess + real supervise_brain/run_trial** (sibling of brain_survive.rs), in-proc Broker as the stable kernel. Custom TrialEnv: `applied_state`=AppliedPending (makes it a trial), `old_gen_drained`=`!broker.any_local_controller_wedged()` (real broker truth), `ready_generation` LATCHED behind an `armed` flag (withhold the subprocess's real brain.ready from run_trial until the wedge is up — else it promotes trivially before the wedge blocks). staged_artifact_hash/ready_exe_hash=None → bytes-gate degrades to readiness-only.
- **The resume paradox:** `resume_sessions()` re-Control-attaches (by:None) EVERY session `sessions()` returns → a wedge present at boot is DISPLACED by resume. So establish conn A's wedge AFTER the candidate's boot-once resume (wait for brain.ready first) → it's the field's un-resumed wedge. Prove it via the stall-evict tally, not a pre-arm read (GREEN reaps it fast).
- **livehost MASKS the RED:** `spawn_live_host`'s reconcile loop (livehost.rs:1026) polls `query_live_session_endpoints()`→`brain.sessions()` (KIND_SESSIONS) UNCONDITIONALLY every LIVE_RECONCILE_INTERVAL_MS=5s → drives the SAME reap. A full-run_brain subprocess RED promotes at ~boot+5s regardless of the fix. Gate it: TEST-ONLY runtime env `SPT_LIVEHOST_RECONCILE_DISABLE` at the loop top (doyle Option A) → isolates the fix's core-loop drive; faithful to the field's livehost-silent trial.
- **The flood wedge SELF-HEALS at 5s:** a continuous-flood controller trips `CONTROLLER_WRITE_DEADLINE`=5s (broker.rs:833, the OUTPUT-driven channel-Full detach — a severed-drop, NO stall-evict tally), SEPARATE from the `BRAIN_WRITE_DEADLINE` stall-evict reap. false_promote.rs's flood only works because that rig finishes <5s. A slower subprocess rig self-heals → RED promotes with evicts=0. FIX: a **BURST-then-idle child** (`powershell [Console]::Out.Write('F'*1MB); Start-Sleep 600` / `sh -c "yes|head -c 1MB; sleep 600"`) — one burst sets write_blocked_since, then SILENCE means no new output → the 5s channel-detach never re-fires → the wedge PERSISTS until the reap. This is the faithful idle-brain black-hole (broker.rs BRAIN_WRITE_DEADLINE doc). Shrink maturity with `SPT_BRAIN_WRITE_DEADLINE_MS` (broker-side = the in-proc broker's process env).
- **Win UAC-740:** a test exe named `update*`/`setup*`/`install*` trips installer-detection → `os error 740` at nextest `--list`. Name it `trial_drain_drive_e2e` (see [[win-update-exe-uac740]]).
- Single `#[test]` per binary (subprocess shares SPT_HOME-derived broker_socket_name → one broker/home per test). RED run ~60s (window-elapse rollback); GREEN ~5s.