# ADR-0038: Broker stream-table lifecycle truth (redispatch eligibility) - **Status:** accepted (hertz RCA + doyle seam-verify 2026-07-16, dispatched same day); **amended 2026-07-16 (REDISPATCH-STALL — see Amendment below)** - **Context:** REDISPATCH-TRUTH milestone; RCA `.claude/reports/2026-07-16-redispatch-truth/hertz-refresh-rca.txt` ## Context The broker's `NetShared.streams` table is append-only in practice: rows are inserted once (nethost.rs ~649) and `StreamLog::finish` only marks EOF — no removal or retirement path exists. Stream *classification* is derived at redispatch time by peeking ring seq 0, but the `StreamLog` data ring is bounded (4096 transport chunks) and evicts the protocol opener under traffic. Dispatcher claims are a per-process one-shot `HashSet` insert performed before worker spawn, never cleared or retried. After any brain cycle (`spt daemon refresh`, `spt update` apply — same path), the fresh `dispatch::run_dispatch_loop` therefore replays EVERY broker-held peer stream, including finished historical attach streams. A replayed historical Attach silently steals the live controller (same-identity `become_controller`, no `Displaced`) and its replayed EOF clears it — the operator sees a frozen remote PTY (field: 4/5 endpoints, twice on 2026-07-16). A high-traffic active stream whose opener evicted classifies Unknown/Failed and is permanently abandoned (claim burned pre-spawn). ## Decision Stream rows carry explicit **lifecycle truth**, and redispatch consumes it: 1. **Finished is terminal for redispatch.** A finished/terminal stream row is retired from redispatch eligibility (removed from the enumeration the dispatcher claims from, or excluded by lifecycle state). Only the post-EOF state genuinely needed by other readers (presence, log reads) survives. A fresh dispatcher never re-serves a terminal Attach. *Gate round 2 ruling: retirement is the EXCLUDED-BY-LIFECYCLE arm — eligibility-visibility only, never transport teardown. Removing the row races the worker's fire-and-forget reply send and resets the never-finished send half (requester sees torn-EOF instead of its reply); physical teardown belongs to the connection-close sweep alone.* 2. **Classification identity is restart-durable and ring-independent.** Each broker-held inbound stream pins an immutable bounded opener/ classification fact (the complete first NDJSON record or its derived family+cursor facts) OUTSIDE the evictable data ring, held until stream close. Recovery classifies from that metadata, never from ring seq 0. 3. **Claims are retryable, outcomes are classified.** The dispatcher distinguishes active/retryable vs served vs finished. A transient worker-setup failure releases/requeues the claim; terminal outcomes do not hot-loop. 4. **Live reconstruction semantics are unchanged.** The legitimate same-operator successor re-take after a brain restart (controller identity keyed on `by`) still silently re-takes; the discriminator for replay suppression is stream lifecycle, never origin identity. ## Consequences - Brain cycles (refresh/update) stop being a controller-integrity hazard: historical streams can neither take nor clear a live controller, and active streams survive opener eviction — verified by production-path regressions (the prior e2e manually re-served a stream and bypassed rediscovery entirely). - The stream table gains a retirement duty (bounded growth instead of forever-discoverable rows) and a small per-stream metadata cost (one pinned opener record per stream, ring untouched). - Rejected alternatives: clearing the whole stream table on brain restart (destroys live streams' reconstruction facts along with the stale ones); enlarging the data ring to preserve openers (unbounded memory for a correctness fact); filtering replay by origin identity (breaks the legitimate successor re-take, the exact silent-steal seam); a per-generation claimed-set handoff (patches the symptom, keeps terminal rows eligible forever). ## Amendment — REDISPATCH-STALL (2026-07-16) - **Status:** accepted (hertz v0.34 field-stall RCA + doyle seam-verify, same day, both hours apart); RCA summary `.claude/reports/2026-07-16-redispatch-stall/rca-summary.md`; hazard KNOWN-HAZARDS 7.43; build shape `docs/REDISPATCH-STALL-TRIAGE.md`. The W1 machinery this ADR shipped is field-correct on its own terms (D1/D1b hold: finished rows never serve) but two of its legs COMPOSE into a stall engine on live v0.34: Decision 3's retryable claims re-drive Decision 2's N-1 ring-peek fallback, whose guard arm is a broad `Err(_)` (`dispatch.rs:414` — intended old-broker-only, catches transport timeout/EOF/poison too), each pass installing a throwaway peek subscriber whose `StreamLog::attach` replays the whole retained ring UNDER the per-stream mutex with discarded write errors — one wedged subscriber conn = serial 15s bounded-write poison windows (33 observed, all 15,000–15,154 ms) composing into 17–62s PTY/RC stalls. Amended decisions: 1. **(amends Decision 2)** The ring-peek fallback fires ONLY on the explicit unsupported-verb/old-broker answer. Every other opener-query error (transport timeout, EOF, poison) is `Failed` → the bounded claim requeue — never a second replay subscriber. 2. **(new — subscriber write discipline)** Replay and live fan-out stop at the FIRST failed subscriber write, and the failed subscriber is REMOVED (attach replay, append, finish — all three sites). Subscriber I/O moves OFF the `StreamLog` mutex: a bounded per-subscriber writer queue, enqueue-under-lock / I/O-outside; queue overflow = detach that subscriber + resume-from-cursor (the existing gapless-resume contract), never a producer stall. 3. **(new — bounded forwarding)** Attach worker completion and forwarding are BOUNDED and cancelable: a poisoned brain subscriber cancels its paired serve/wire worker (no orphan forwarding legs). Coordinate at build with the DAEMON-LIFECYCLE C2 stream-lease work — same neighborhood, build once. 4. **(amends Decision 3)** Retries are CLASSIFIED: a pre-setup transient may retry; a deadline-poisoned replay is CIRCUIT-BROKEN (global backoff), and a replacement subscriber is never installed until the prior subscriber is fully gone. This is an upgrade of the bounded backoff, NOT a revert to the v0.33 burn-the-claim abandonment (explicitly rejected). 5. **(new — serve-side validation)** Terminal-exclusion is enforced pre-serve, and attach/detach validate ownership/generation — a stale worker can never detach a replacement controller. This also covers the UNFINISHED-stale-row steal shape (raw-close no-FIN viewports) that Decision 1's finished-row retirement cannot see. 6. **(new — dispatcher hygiene + telemetry)** The redispatch worker pool is bounded with batched cold enumeration and NO claim/sessions locks held across I/O (clone-then-drop before `detach_if`/`info.json` writes, KH 7.12 kin); dispatch telemetry is keyed per gen/stream/family/endpoint/attempt/conn with replay/poison/cancel/worker-outcome events and gauges. Rejected here: reverting retryable claims (restores permanent abandonment); widening any write deadline (hides the composition instead of removing it); tearing down all subscribers on any poison (punishes healthy streams for a wedged conn).