# REDISPATCH-STALL — wave triage (doyle, 2026-07-16)

<!-- [doc->REQ-HAZARD-REDISPATCH-STALL] -->

**Wave:** stop the v0.34 field stalls — REDISPATCH-TRUTH W1's claim retry and
the broad `Err(_)` opener fallback repeatedly drive `StreamLog::attach`'s
discard-errors replay-under-mutex into serial 15s subscriber-poison windows
that compose into 17–62s PTY/RC freezes. Counter-63 fast-follow after
MESH-RECOVERY's counter 62.

**Canonical RCA:** hertz 2026-07-16, doyle seam-verified the same day; repo
summary `.claude/reports/2026-07-16-redispatch-stall/rca-summary.md`.
**Decision record:** ADR-0038 **Amendment (2026-07-16)**.
**Hazard:** KNOWN-HAZARDS 7.43.
**Field incident:** live v0.34 boxes, recurrent 20–30s freezes, DISPATCH tails
17–62s, poison interarrival median 95.5s; 33 `CONN_WRITE_POISONED` windows all
15,000–15,154 ms (the round-4 SharedSend deadline); 4361 stream-sub-attach
records/generation; per-stream, not broker-wide (unrelated conns progressed
mid-poison — broker-wide theory falsified).

## Seam census (verified current, main @7f5ead8)

| Seam | Location | Defect |
|---|---|---|
| Broad opener fallback | `spt-daemon/src/dispatch.rs:414` (`first_line`, `Err(_) => {}` arm) | comment intends old-broker-only; catches transport timeout/EOF/poison → manufactures a throwaway peek subscriber per retry |
| Replay-under-mutex, errors discarded | `spt-daemon/src/nethost.rs:322-346` (`StreamLog::attach`) | installs subscriber, replays entire retained ring under the per-stream `Mutex`; `let _ = sub.write(&frame)` continues past failure; poisoned subscriber stays installed (same shape: `push` :288, `finish` :316; `PresenceLog::attach` :451-465) |
| Retry × fallback composition | `dispatch.rs` claim retry (500ms/1s, ×3) + the fallback above | each retry re-attaches a fresh peek subscriber → serial 15s poison windows under the lock |
| Unbounded forwarding legs | attach worker serve/wire pairing (nethost/serve_attach) | a poisoned brain subscriber leaves its paired serve/wire worker running |
| Stale-worker detach | serve/detach path (`detach_if` by ptr identity only) | no ownership/generation validation before attach/detach — a stale worker can clear a replacement controller (the UNFINISHED-stale steal shape, kin emphasys C2 raw-close no-FIN viewports) |
| Dispatcher hygiene | `dispatch.rs` per-stream `thread::spawn`, claims/sessions locks near I/O | unbounded worker fan-out on cold enumeration; lock-across-I/O kin KH 7.12 |

## Fix set (P0, accepted as filed — hertz design, doyle-ruled)

1. **Fallback narrowed** (`REQ-DISPATCH-FALLBACK-CIRCUIT`): ring-peek fallback
   ONLY on the explicit UnsupportedVerb/old-broker answer; transport
   timeout/EOF/poison = `Failed` → bounded requeue, never a second replay
   subscriber.
2. **Replay halts + failed subscriber removed**
   (`REQ-STREAMLOG-SUBSCRIBER-DISCIPLINE`): replay stops at the FIRST write
   failure and the failed subscriber is REMOVED — attach replay, append, and
   finish, all three sites.
3. **Writes OFF the StreamLog mutex** (same REQ): bounded per-subscriber
   writer queue — enqueue-under-lock, I/O outside; overflow = detach that
   subscriber + resume-from-cursor. Never a producer stall.
4. **Bounded, cancelable attach forwarding** (same REQ): a poisoned brain
   subscriber cancels its paired serve/wire worker. COORDINATE at build with
   DAEMON-LIFECYCLE C2 leases — same neighborhood, no double-build.
5. **Classified retries + circuit breaker**
   (`REQ-DISPATCH-FALLBACK-CIRCUIT`): pre-setup transient may retry;
   deadline-poisoned replay = circuit-broken global backoff; a replacement
   subscriber is never installed until the prior one is fully gone. NOT a
   revert to v0.33 abandonment.
6. **Terminal-exclusion pre-serve + ownership/generation validation**
   (`REQ-SERVE-OWNERSHIP-GENERATION`): a stale worker never detaches a
   replacement controller; covers the UNFINISHED-stale-row steal shape D1/D1b
   cannot see.
7. **Bounded worker pool + batched cold enumeration**
   (`REQ-DISPATCH-HYGIENE-TELEMETRY`): no claim locks held during I/O.
8. **Sessions-lock discipline** (same REQ): clone-then-drop the sessions lock
   before `detach_if`/`info.json` I/O (KH 7.12 kin).
9. **Keyed stage telemetry** (same REQ): per
   gen/stream/family/endpoint/attempt/conn;
   replay/poison/cancel/worker-outcome events + gauges + PTY high-water/RC
   cursor.

## Test matrix (seven legs, production-path)

1. **T1 poisoned-replay halt:** wedge a subscriber conn; `StreamLog::attach`
   replay halts at the first failed write, subscriber removed, later
   append/finish never re-feed it (unit at the StreamLog seam + poison int).
2. **T2 fallback narrowing:** opener query answering transport
   timeout/EOF/poison → `Failed`/requeue with NO peek subscriber created;
   only the explicit unsupported-verb answer falls back (unit + int).
3. **T3 writer-queue off-lock:** producer appends never block on subscriber
   I/O past the enqueue; overflow detaches + resumes-from-cursor gapless
   (unit; int: wedged subscriber, producer latency stays flat).
4. **T4 forwarding cancel:** brain-subscriber poison cancels the paired
   serve/wire worker — no orphan forwarding leg survives (int).
5. **T5 circuit breaker:** deadline-poisoned replay trips the global backoff;
   no reinstall until the prior subscriber is fully gone; the stream RECOVERS
   after the breaker window (the not-abandonment discriminator) (unit + int).
6. **T6 stale-ownership steal:** UNFINISHED-stale attach row + live current
   controller; restart the dispatcher; the stale worker neither takes nor
   clears the replacement controller (ownership/generation validated) (int).
7. **T7 mixed-image N-1:** new brain vs old broker across the fallback
   window with REAL streams carrying traffic — handshake-only N-1 is
   INSUFFICIENT (doyle rider); prove fallback fires only there and serves
   correctly (int).

**Binding on every leg:** redispatch D1/D1b stay green (the retire/steal
regressions are substrate, not victims).

**Missing-coverage census (fair, rides the wave):** D3 never poisons an
installed subscriber; `conn_blackhole` exercises conns, not StreamLog
subscribers; `daemon_refresh` doesn't cover cross-node attach continuity —
the matrix above closes these, don't re-count them as new scope.

## Gate rulings (pre-made, binding)

1. **Production-path only.** Poison/replay regressions ride the real
   `run_dispatch_loop` + `StreamLog` + serve_attach seams; hand-fed
   subscribers prove nothing about the composition.
2. **D1/D1b green every leg** — no exceptions, any red = stop.
3. **No deadline widening.** The 15s bound is correct; the fix removes the
   serial composition, never hides it behind a bigger number.
4. **Circuit breaker ≠ abandonment.** T5 must show recovery after the breaker
   window; a stream that never recovers is a regression to v0.33 burn-claim.
5. **Replay discipline is all-sites.** attach + append + finish (and
   `PresenceLog` mirrors) — a fix that patches attach only fails the gate.
6. **T7 uses real streams.** Handshake-only mixed-image is insufficient.
7. **New daemon-tree e2es enter the HEAVY list at birth** (3 identical
   strings: ci.yml Phase A + Phase B + nextest.toml).
8. **Coordinate fix 4 with DAEMON-LIFECYCLE C2** — if leases land here,
   the C2 build consumes them; no parallel second mechanism.
9. **Gate mechanics:** isolated worktree of the gate sha + fresh
   `CARGO_TARGET_DIR` + nextest + workspace clippy + `traceable-reqs check`;
   shared-seam sweep (StreamLog/dispatch touch sync/update/attach workers —
   grep + run every seam-sibling suite); both CI runners green.

## Explicitly out of scope (do not build)

- `update --restart`'s 30s rc reconnect loop — distinct and bounded, not this.
- DAEMON-LIFECYCLE C1/C3 (hybrid-online classification, poll-vs-reap race) —
  own cluster; only the C2-adjacent bounded-forwarding seam coordinates here.
- Unbounded Whole-brain read reshaping beyond what fixes 3/4 require —
  observability lands via fix 9; deeper read-path work is its own seed.

## Field discriminators (hertz, after release)

1. **Poison-window census:** the 15,000–15,154 ms `CONN_WRITE_POISONED`
   fingerprint disappears from live logs; stream-sub-attach per generation
   drops from ~4361 to O(active streams).
2. **Stall census:** no 17–62s DISPATCH tails across a daemon refresh + one
   week of live traffic.
3. **Next steal catch (if any):** capture the stolen row's `finished` +
   `retired` flags — the discriminating observable between the finished
   (D1/D1b-covered) and unfinished-stale (fix 6) shapes.

## Lane

doyle triaged (this doc) → todlando builds (branch
`build/redispatch-stall-w1`, all five REQs, flip `required_stages` at build
start per [[traceable-per-wave-activation]]) → doyle gates (rulings above) →
deployah releases (counter 63 fast-follow, bump-in-PR, counter from published
metadata) → hertz field-verify (discriminators above; they own the incident
logs).