# UPDATE-WEDGE round-3 CODE-READ — reap-drive exonerated; the seam is a SharedSend self-deadlock on the brain conn (todlando, 2026-07-09) Report-before-fix per RULING 2 §5/§8. doyle asked: pin what the counter-54 reap-drive (ce053dc) blocks on each tick, especially whether an UNRESPONSIVE peer lengthens the tick >15s. Answer: **it doesn't block on anything — and the real wedge is a different seam the reap-drive only widened.** ## 1. The reap-drive does NOT block (doyle's dead-peer-tick hypothesis — FALSIFIED by code) `run_brain`'s heartbeat loop (brainproc.rs:239-278) per tick: `thread::sleep(500ms)` → `brain.net_status()` → spawn-consumers-once → `brain.sessions()` (the counter-54 reap-drive, brainproc.rs:276) → `write_ready`. It hosts NO PTY sessions, so it does not itself drain PTY output. Neither IPC call blocks on a dead peer: - **`net_status()` is a cheap field read.** `dispatch_net_status` (broker.rs:3573) returns `node_id_hex()` / `addr_json()` / `conn_count()` — no `bounded_block_on`, no peer wait. `conn_count` takes `shared.conns.lock()` only briefly; a dead-peer connect never reaches `register_conn`, so it never holds that lock. - **`sessions()` reap takes each OutputLog lock only briefly.** The `KIND_SESSIONS` handler (broker.rs:2853-2926) snapshots each session under its OWN log lock, calls `reap_dead_controller` (968), reads the cursor, releases; `converge_perch_stamps` runs OFF-lock. No peer op anywhere. - **`append` is non-blocking under the log lock.** The producer fan-out (broker.rs:790-849) is `try_send` to controller (827) and viewers (802) — never a blocking send — so it holds the log lock only momentarily. The reap's `recover_log` never queues behind it. - **The controller writer holds the SOCKET mutex, never the log lock, across its blocking write** (`controller_writer`, broker.rs:1418; docstring 1409-1411: *"a backed-up controller blocks ONLY this thread, never the session drain which used to write inline under the log lock"*). So a WEDGED writer cannot block the reap either. - **Viewers drop, never block the producer** (`try_send` + eviction, broker.rs:1086/802). Constants: `BRAIN_HEARTBEAT` = 500ms, `BRAIN_WRITE_DEADLINE` = 15s. Net: the reap-drive tick is ~ms even with dead peers present. There is NO code path by which an unresponsive peer lengthens it. **`REQ`-shaping the fix as "make the reap-drive non-blocking" targets a block that does not exist.** ## 2. The REAL seam — a SharedSend self-deadlock on the brain conn `SharedSend = Arc>` (broker.rs:78): ONE mutex guards each conn's send half. On the brain conn, that one mutex is contended by TWO writer classes: - **Subscriber writers** — `viewer_writer` (broker.rs:1333/1342) and `controller_writer` (1451) — take `send.lock()` and hold it ACROSS the blocking `write_frame` to the conn. - **The dispatch reply path** — `send_frame` (broker.rs:4221) — locks the SAME `send.lock()` to write `KIND_SESSIONS_REPLY` (2926) / `KIND_NET_STATUS_REPLY` (3592). The brain conn is BOTH a subscriber sink (resume re-subscribes N sessions onto it) AND the brain's request/reply channel. So: > when the brain conn backs up (the brain isn't draining fast enough — it only reads during > the 500ms heartbeat's `net_status`/`sessions` calls), a subscriber writer BLOCKS inside > `write_frame` while HOLDING `send.lock()` → the dispatch thread cannot acquire `send.lock()` > to send the heartbeat reply → `net_status()`/`sessions()` never return → the heartbeat loop > is stuck → the brain never drains its conn → the subscriber writer stays blocked. **A > self-reinforcing deadlock on the brain conn's send mutex.** Once the heartbeat is stuck, the brain drains nothing; every subscriber writer on the brain conn wedges; the broker-side controller writers whose sink is the brain conn cross `BRAIN_WRITE_DEADLINE` → `BRAIN_SUBSCRIBER_STALL_EVICT`. Matches the field exactly. ### Why counter-54 is the regression window (but not the root) ce053dc added a SECOND per-heartbeat reply round-trip (`brain.sessions()`) through the contended brain-conn send mutex. It did not create the deadlock — it made the heartbeat depend on TWO reply round-trips per tick surviving the mutex, widening the stall window and tightening the coupling. Regression-window-consistent; the underlying send-mutex sharing is older. ### Why the Viewer fix (v0.30.4) did NOT heal it The round-2 Viewer fix removed the resume-STEAL (no more `become_controller`), but the brain is STILL a subscriber — `viewer_writer` has the SAME `send.lock()`-across-blocking-write pattern (1333/1342) AND viewers have NO stall-evict safety valve (only controllers are reaped). So a blocked brain-viewer writer holds the send mutex indefinitely → the deadlock persists. This is why v0.30.4 field-verify wedged again. ### The dead peer is NOT causal The trigger is active-session OUTPUT backing up the brain conn faster than the 500ms heartbeat drains it — on respawn (resume subscribes N + replay burst floods the conn before the loop drains) and in steady state (an active streamer's output between heartbeats fills the socket buffer; on v0.30.3 additionally compounded by the steal). The field always had dead peers, but they are coincidental load, not the mechanism. ## 3. Fix shape (proposed — doyle to rule) — BRAIN-SIDE, seamless The daemon brain hosts NO PTY sessions (brainproc.rs:184), so it has NO consumer for the subscriptions `resume_sessions` re-establishes — it floods its own request/reply conn with output it never uses. Options, preference order: - **(A) resume_sessions does NOT re-subscribe.** Drop the undrained continuity subscription entirely — the daemon brain has no consumer for it today. The brain conn then carries only request/reply → no subscriber backpressure → no deadlock. Smallest, brain-side, seamless. Forward-seam: when genuinely daemon-driven sessions land (the live-agent adapter), THOSE attach on a drained carrier (option B). - **(B) Drain the brain conn on a DEDICATED reader thread** (`BrainConn::Split` — the exact carrier the pump already uses, brain.rs:230) so the heartbeat reply is never blocked by subscriber backpressure, and subscriptions can persist. Brain-side, seamless. - **(C) [broker-side — deployment risk, FLAG] subscriber writers use a bounded/non-blocking write** instead of holding `send.lock()` across a blocking `write_frame`. Durable root but broker-side ⇒ needs a coordinated daemon restart to land (the daemon.rs:368 constraint). Prefer **A or B** (brain-side ⇒ rides the brain-swap ⇒ seamless; the deployment risk lifts). Both touch ONLY the brain (brain.rs / brainproc.rs). C touches the broker — flagged. ## 4. Confirmer rig (RULING 2 §6, still authorized) The mechanism is a code-proven deadlock, so the rig CONFIRMS + GUARDS rather than discovers. brain+broker+PTY (`brain_decouple` template): N sessions actively producing output, the brain resume-subscribed onto its request/reply conn; RED-first — the heartbeat stalls and an active-streaming controller is stall-evicted >15s WITHOUT a dead peer needing to be present (proving dead-peer-independence). Assert BOTH severities (respawn interleave + steady-state output backup). Fix A/B must turn both green. Dead-peer arm optional (to show it is not the trigger). ## 5. Sequence Reported. doyle to rule the shape (A vs B) → mint the REQ (candidate `REQ-BRAIN-CONN-NO-SUBSCRIBER-DEADLOCK` or similar) → build the confirmer rig RED-first → fix → both severities green. No REQ / no fix until doyle rules.