{
  "summary": "Current-tree trace complete. The decisive concurrency fact is that `BRAIN_SUBSCRIBER_STALL_EVICT` does **not** prove an OS pipe/socket write was blocked. In `controller_writer`, `write_blocked_since` is set before acquiring the per-connection `SharedSend = Arc<Mutex<SendHalf>>`; the timestamp remains set through mutex acquisition, JSON serialization, length/body `write_all`, and `flush`. It also remains set if the writer thread is descheduled after marking. Therefore the marker proves only that a controller writer made no observed progress for 15 s in the combined interval `{waiting for another writer on the same connection, executing/blocking in write_frame, scheduler-starved}`. Every output writer and broker reply for one accepted brain connection shares that mutex; one lock-holding named-pipe write can make sibling sessions on that connection independently age past 15 s and emit a multi-session cluster. The peer pump uses a distinct broker IPC connection, so it does not share this `SharedSend`; a coincident pump timeout points to a broker-process/runtime/scheduling or peer-network correlation, not direct contention on the session controller’s send mutex.\n\nThe marker is reached from two places: (1) `OutputLog::resolve_subscribe` before a new control/take subscribe, using inline stamp release; and (2) `KIND_SESSIONS` handling via `OutputLog::reap_dead_controller`, using deferred release followed by `converge_perch_stamps`. The brain-hosted live reconciler opens a fresh connection and calls `Brain::sessions()` at boot and every 5 s, so it is the periodic driver that can emit the marker even for an otherwise idle session. Output-driven controller queue saturation is a separate 5 s path (`append` → `mark_controller_gone`) and does not itself emit this marker.\n\nEviction is logical, not cancellation: the `ControllerSink` is removed and `controlled`/remote `driven_by` are converged false/None, but a writer already stuck in `send.lock()` or `write_frame` owns its own `SharedSend`, receiver, and thread stack and is not interrupted or joined. The hosted `PtySession`, child, input writer, drain, output ring/cursor, translation resources, and broker session row survive. The stale writer exits only if its write eventually returns/fails or its queue later disconnects; while blocked it can continue holding or waiting on the shared send mutex.\n\nUpdate flow in the current source is `apply_staged` → `swap_and_record(AppliedPending)` → `Brain::request_brain_restart` → broker `dispatch_brain_restart`/`BrainRestart::request` → supervisor kills and waits for old brain → increments generation and spawns `StartReason::Update` → `run_brain` calls `Brain::resume_sessions` (currently subscribes every session as a local controller), writes `brain.ready`, then starts net consumers after the first 500 ms `net_status` heartbeat. Promotion is readiness plus `ProductionTrialEnv::old_gen_drained`. Exact caveat: `old_gen_drained` negates `Broker::any_local_controller_wedged`, and that predicate becomes true only after the same 15 s threshold. An in-flight local write younger than 15 s therefore reads as “drained”; also `by=None` identifies any local controller, not a brain generation. This is a source-semantic observation, not a fix proposal.\n\nVersion warning: the factual incident document records v0.30.5 `resume_session_cursors()` cursor-only behavior and `BRAIN_RESUMED: re-established ... cursor(s)`. That symbol/text does not exist in the current tree after alternate broad searches; current `Brain::resume_sessions` re-subscribes and logs `re-attached`. Do not retroactively attribute the v0.30.5 incident to current-tree re-subscription. The marker can originate from any existing spawn/attach controller independently of update resume, as the steady-state evidence already establishes.\n\nPrioritized live evidence probes (diagnostic only; no fix proposed):\n\n1. **Pre-arm a Windows WPR/ETW circular trace across the reproduction** with CPU sampling plus CSwitch/ReadyThread and File/pipe I/O stacks; stop immediately after the marker. Filter to broker PID, old/new brain PIDs from `brain.ready`/`BRAIN_UP`, and the 15 s window. **Question:** was the marked broker writer (a) blocked in `NtWriteFile`/named-pipe `SendHalf::write` or flush, (b) sleeping on Rust/Windows mutex acquisition behind another broker writer, or (c) runnable/ready but not scheduled? A pending write stack and sleeping wait state discriminate pipe backpressure; long ReadyTime with no execution discriminates scheduler starvation. This is the strongest direct discriminator because current Rust state conflates all three.\n\n2. **Capture three closely spaced full dumps of both broker and brain around the first marker** (event-trigger from the stderr tail; e.g. immediately, +100–250 ms, +1 s; retain matching PDBs). Inspect every broker thread, not only the session named in the marker. **Question:** is there one controller writer holding the shared `SendHalf` inside `write_frame` while sibling controller writers and the connection handler wait on the same mutex, or are all candidate writers runnable/descheduled? Repeated dumps establish persistence rather than a transient sample. Correlate a blocked owner plus sibling mutex waiters with clustered session IDs; inspect the brain side for its corresponding pipe reader and whether it is asleep, CPU-bound, or blocked elsewhere.\n\n3. **Use a diagnostic build that splits the existing one timestamp into observable phases at `broker.rs::controller_writer`**: connection identity (`Arc::as_ptr(&SharedSend)` or a broker-minted conn id), sid, epoch, OS TID, `WAIT_SEND_LOCK` timestamp immediately before `send.lock`, `WRITE_FRAME` timestamp immediately after lock acquisition, and `WRITE_OK/ERR` with seq/bytes. Have the existing eviction watchdog print the sampled phase and age; do not change deadlines or ownership. **Question:** which portion of the current 15 s aggregate interval is actually aging, and do multiple evicted sessions share one connection/lock owner? This directly removes the marker’s present ambiguity between mutex wait and codec/pipe write. ETW from probe 1 remains necessary to distinguish a phase-stuck-but-descheduled thread from an executing wait.\n\n4. **Enable the already-built `SPT_PUMP_DIAG=<world-writable path>` capture and sample `identity/pump-heartbeat.json` through the same update window.** Use `LEG_ENTER/LEG_EXIT`, `NET_DIAL_DISPATCH`, `DIAL_ENTER/DIAL_EXIT`, `SYNC_REQ_SENT`, `PUMP_IPC_READER`, `PUMP_PEER_FAIL`, and `PEER_PUMP_FAIL/RESTART`. **Question:** is the nearby pump failure an independent peer/QUIC wait, a missing broker dispatch/reply on the pump’s separate IPC connection, or broad scheduling loss? `DIAL_ENTER`→`DIAL_EXIT ~10s Err` is the broker QUIC bound working; `LEG_ENTER` with no exit until 30 s and no dispatch marker is a pump IPC reply starvation; a stale pump heartbeat plus broker-wide ReadyTime supports scheduler starvation. Because the pump connection is separate, this probe is correlation evidence, not proof of controller-pipe blockage.\n\n5. **Take a high-cadence state timeline (100–250 ms around apply, slower outside it)** of `brain.ready` `{pid,generation,exe_hash}`, `spt daemon status` stall count/last time, `identity/pump-heartbeat.json`, per-endpoint `info.json` fields `controlled`, `driven_by`, `viewer_count`, and broker session `resume_seq/controller_by` via `Brain::sessions`; retain stderr generation markers. **Question:** did promotion precede the writer crossing 15 s, did logical release actually converge for every named session, did cursor progress stop before eviction, and was pump/main-brain liveness advancing concurrently? This establishes ordering and blast radius but cannot by itself distinguish mutex/pipe wait from scheduler starvation.\n\nExisting runtime state is insufficient for the central distinction: `write_blocked_since` is private and phase-conflated; `_writer.is_finished` only distinguishes exited vs still-live; `last_ok` is channel handoff time, not socket progress; `delivered_through` is a successful-write cursor without a progress timestamp; stall counters are post-event only; controller writer threads are unnamed; and no connection id/TID/lock-owner is surfaced. One useful negative inference is that marker emission itself proves some broker handler/reconcile thread was scheduled and acquired the sessions/log/blocked-state locks at that instant, though it says nothing about whether the target writer received CPU during the prior 15 s.",
  "files": [
    {
      "path": "spt-core/docs/UPDATE-WEDGE-SYMPTOMS-FOR-RCA.md",
      "description": "Factual field evidence: invariant marker, steady-state and apply-swap occurrence, v0.30.5 cursor-only resume, promotion ordering, pump failures, multi-session HFENDULEAM population. Treated as observations only."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/broker.rs",
      "description": "Core lifecycle. `SharedSend`, `ControllerSink`, `OutputLog`, `HostedSession`, `InputWriter`; `become_controller`, `controller_writer`, `append`, `controller_write_stalled`, `local_controller_wedged`, `stall_evict_controller`, `reap_dead_controller`, `clear_controller`, `detach_if`, `converge_perch_stamps`, `handle_conn`, `dispatch_spawn`, `dispatch_subscribe`, `KIND_SESSIONS` arm, `any_local_controller_wedged`, `dispatch_brain_restart`, `send_frame`. Marker at lines 1044–1051; phase-conflating mark at 1432–1435; blocking write loops at 1437–1492; per-connection shared send type at line 77."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/codec.rs",
      "description": "`write_frame`: serializes JSON, writes 4-byte length, writes body, flushes. All of this occurs while `SharedSend` is locked and while controller `write_blocked_since` remains set."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/transport.rs",
      "description": "`LocalSocketTransport`: interprocess local socket abstraction; Windows resolves to a named pipe, Unix to namespaced UDS. Establishes the OS substrate behind `SendHalf`."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/brain.rs",
      "description": "Brain-side connection and subscriber API: `BrainConn::{Whole,Split}`, `cold_start`, `cold_start_pump`, `spawn_session`, `attach`, `attach_as`, current `resume_sessions`, `sessions`, `net_status`, `request_brain_restart`, `call_deadline`, `read_event_until`, `read_frame_until`. Split pump reader uses `recv_timeout`; ordinary clients block directly."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/brainproc.rs",
      "description": "Update/restart/promotion and process ownership: `BrainRestart`, `run_brain`, `spawn_net_consumers`, `TrialEnv::old_gen_drained`, `ProductionTrialEnv`, `run_trial`, `supervise_brain`, `spawn_brain_supervisor`. Current `run_brain` resumes/subscribes before `write_ready`; net dispatcher/pump start only after first heartbeat."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/applyhost.rs",
      "description": "Apply front door: `swap_and_record` records `AppliedPending`; `apply_staged` connects on a non-session CLI connection, swaps bytes, then calls `Brain::request_brain_restart`. It does not itself subscribe to hosted PTYs."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/livehost.rs",
      "description": "Periodic eviction driver: `query_live_session_endpoints` opens a fresh Brain and sends `KIND_SESSIONS`; `spawn_live_host` invokes it at boot and every `LIVE_RECONCILE_INTERVAL_MS = 5000`, causing `reap_dead_controller`/stall checks for all sessions."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/pump/mod.rs",
      "description": "Peer pump lifecycle and deadline: `PUMP_PEER_IO_TIMEOUT = 30s`, `run_peer_pump`, per-leg diagnostics, `peer_outcome`, `connect`, `supervise_pump`, `spawn_peer_pump`. A TimedOut brain IPC read poisons the client and triggers `PEER_PUMP_FAIL` plus supervised restart."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/nethost.rs",
      "description": "Broker-side network deadline: `BROKER_QUIC_OP_TIMEOUT_MS = 10_000`, `bounded_block_on`, and `dial` diagnostics. Separates a normal 10 s peer error from the pump’s own 30 s IPC read timeout."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/pumpdiag.rs",
      "description": "Existing opt-in `SPT_PUMP_DIAG` evidence channel with epoch-ms markers to stderr and an operator-selected file; intended to arbitrate dispatch vs dial vs reply-read stalls."
    },
    {
      "path": "spt-core/crates/spt-daemon/src/endpoint.rs",
      "description": "Durable runtime surfaces: `brain_ready_path` and `pump_heartbeat_path`/`read_pump_heartbeat`, useful for generation/PID and pump-liveness correlation."
    },
    {
      "path": "spt-core/crates/spt-store/src/info.rs",
      "description": "Persistent endpoint state and setters: `InfoRecord.driven_by`, `controlled`, `viewer_count`; `set_driven_by` and related serialized mutations used by broker release/convergence."
    },
    {
      "path": "spt-core/crates/spt/src/cli.rs",
      "description": "Operator observability: daemon status queries broker `stall_evicts`, renders count and last timestamp, and surfaces pump heartbeat/staleness; useful for the high-cadence timeline probe."
    }
  ],
  "architecture": "```mermaid\nflowchart TD\n  APPLY[apply_staged CLI Brain conn\\nnot session-subscribed] --> SWAP[swap_and_record\\nAppliedPending]\n  SWAP --> REQ[request_brain_restart]\n  REQ --> BRS[Broker dispatch_brain_restart\\nBrainRestart AtomicBool]\n  BRS --> SUP[supervise_brain\\nkill+wait old child; generation++]\n  SUP --> NEW[run_brain StartReason::Update]\n  NEW --> LIST[Brain::sessions / KIND_SESSIONS]\n  LIST --> REAP[each OutputLog::reap_dead_controller\\nstall check + deferred stamp convergence]\n  NEW --> RES[Brain::resume_sessions current tree\\nsubscribe every session, by=None]\n  RES --> SUB[Broker dispatch_subscribe\\nresolve_subscribe]\n  SUB --> BC[become_controller]\n  BC --> SINK[ControllerSink + controller_writer thread]\n  SINK --> SHARED[per-connection SharedSend\\nArc Mutex SendHalf]\n  SHARED --> WF[write_frame\\nserialize + prefix/body write_all + flush]\n  NEW --> READY[write_ready]\n  READY --> TRIAL[run_trial]\n  TRIAL --> DRAINED[old_gen_drained =\\n!any_local_controller_wedged]\n  DRAINED -->|only detects by=None AND age >=15s| PROMOTE[BRAIN_PROMOTED]\n  NEW --> HB[500ms net_status heartbeat]\n  HB --> CONSUMERS[spawn dispatcher + peer pump]\n\n  PTY[PtySession output drain] --> APPEND[OutputLog::append]\n  APPEND -->|try_send nonblocking| SINK\n  APPEND -->|Full for 5s or disconnected| GONE[mark_controller_gone\\nno STALL marker]\n\n  SINK --> MARK[mark write_blocked_since\\nBEFORE send.lock]\n  MARK --> WAITLOCK[wait SharedSend mutex]\n  WAITLOCK --> WF\n  MARK -. thread may be descheduled .-> STARVE[Ready/runnable starvation]\n  WAITLOCK -. one writer can block siblings\\non same Brain conn .-> CLUSTER[multi-session 15s ages]\n\n  LIVE[brain livehost every 5s] --> LIST\n  SUB -->|before control/take| INLINE[stall_evict_controller Inline]\n  REAP -->|age >=15s| DEFER[stall_evict_controller Deferred]\n  INLINE --> LOGMARK[BRAIN_SUBSCRIBER_STALL_EVICT]\n  DEFER --> LOGMARK\n  LOGMARK --> TALLY[global count + last_ms]\n  LOGMARK --> DROP[drop logical ControllerSink]\n  DROP --> STAMP[controlled=false; remote driven_by=None]\n  DROP -. does not cancel blocked writer .-> ORPHAN[writer still owns SharedSend/rx/thread\\nuntil write returns/fails]\n  DROP -. session resources survive .-> HOSTED[PtySession child, InputWriter, Drain,\\nOutputLog/ring/cursor, Translation]\n\n  CONSUMERS --> PUMP[peer pump separate Brain conn]\n  PUMP --> PR[dedicated pump-ipc-reader]\n  PR --> RT[recv_timeout 30s]\n  PUMP --> NET[broker NetHost bound 10s op]\n  RT -->|TimedOut| PFAIL[PEER_PUMP_FAIL + restart]\n  NET -->|ordinary error first| PEERFAIL[PUMP_PEER_FAIL + redial]\n```\n\n**Ownership and isolation boundaries:** The broker process is the continuity anchor. `Broker.sessions: Arc<Mutex<HashMap<u64, HostedSession>>>` is session-wide; each `HostedSession` owns an `Arc<PtySession>`, a dedicated bounded/nonblocking-dispatch `InputWriter`, a PTY `Drain`, and an `Arc<Mutex<OutputLog>>`. Each output log owns one logical controller sink and many viewer sinks. Controller/viewer writer threads perform blocking IPC writes off the PTY drain. The important exception to per-session isolation is `SharedSend`: it is created once in `Broker::handle_conn` after splitting an accepted connection and cloned into every session role held by that brain connection, plus broker ack/error paths. Thus sessions subscribed through the same brain are coupled at the send mutex and named-pipe buffer.\n\nA connection handler tracks `my_subs`; receive EOF invokes `detach_if` for those session IDs, but only clears a role if its `SharedSend` pointer is still current. Sessions persist across the connection loss. `clear_controller` drops the sink and stamps the endpoint; it cannot forcibly cancel a thread already blocked because the thread owns cloned resources. Reattachment creates a new controller writer and advances `controller_epoch`; the epoch gate suppresses stale **initial replay** writes after they acquire the mutex, but the old live-path writer has no epoch gate and a currently blocked call cannot observe the bump until it returns.\n\nThe peer pump is process-shared but connection-isolated: `spawn_net_consumers` creates dispatcher and pump clients independently of the main brain and each other. The pump’s `BrainConn::Split` owns its own send half and a named `pump-ipc-reader`; its 30 s timeout is channel-side. On the broker, accepted connections each get their own handler thread and their own `SharedSend`. Pump and session writers can still correlate through OS scheduling, broker process CPU, the shared broker `NetHost`/Tokio runtime, and selected global locks/effect state, but they do not contend on the same controller send mutex merely by existing in the same daemon."
}