HERTZ -> DOYLE: v0.34 PTY/RC 20–30 SECOND STALL RCA (2026-07-16)

Verdict
=======
The field symptom is a transport/redispatch regression, not evidence of a ConPTY child freeze and not a single new 20- or 30-second timer.

v0.34 W1 repeatedly enters a pre-existing blocking stream-subscriber replay/forward path. A physical broker IPC write poisons at the exact 15,000 ms deadline. The dispatcher worker can remain alive after that poison because the client uses an unbounded Whole-Brain read and Attach forwarding uses a distinct unbounded wire connection. v0.34 then retries Failed claims after 500 ms / 1 s, up to three attempts. One 15-second wedge plus cleanup/forward lag, or two deadline windows, produces the observed 20–30 seconds; longer composites produce the observed 36–62 second tail.

There is also an ownership-corruption companion: rediscovery of stale finished Attach rows can transiently steal controller ownership, then replay EOF detaches it. Whether recovery is automatic depends on which stale/current worker finishes last. This is the exact prior 4/5-frozen refresh mechanism. It has no timer by itself; the W1 retry/fallback/replay storm supplies the timing/amplification.

Observed production evidence
============================
Current cold generation, broker/brain both v0.34.0:

- 33 CONN_WRITE_POISONED incidents, all attributed to role=brain stream-subscriber.
- Every poison fires at 15,000–15,154 ms after write-start.
- Corresponding DISPATCH failures occur 15.1–62 seconds after write-start. Recent durations: 17.6, 25.4, 29.1, 30.1, 31.1, 36.6, 59.4, 61.6 seconds.
- Poison interarrival median 95.5 seconds, minimum 29.7 seconds.
- 4,361 stream-sub-attach records and 584 PUMP_PEER_FAIL 10-second incidents in this generation.
- Exact stream 3907: attach/write at wall 1784244082619; poison +15.008 s; DISPATCH failure +19.749 s; retry attach +20.354 s.
- During stream 3907's blocked write, unrelated broker conns 10180/10181/10182 completed normally; fresh conns answered about 105 ms after poison. Therefore this incident was per physical subscriber/stream, not a broker-wide IPC stop.
- Recent webbie controller attach was wall 1784245849362. Stream 4834 write began +37.354 s and poisoned at +52.366 s. This temporal match is suggestive only: current logs do not expose enough opener/endpoint identity to prove stream 4834 was webbie's active Attach.
- Brain heartbeat advanced and PID/generation remained stable during observation. Pump was alive.
- Resource snapshot: broker 116 threads/550 handles; brain 26 threads/183 handles. Therefore 4,361 attaches are cumulative churn, NOT proof that 4,361 workers were simultaneously live. Unbounded per-row spawning remains a source risk and cold-burst amplifier, but should not be presented as an observed 4k-live-thread fact.

Exact source chain
==================
1. v0.34 claim/retry regression relation
   - crates/spt-daemon/src/dispatch.rs: ClaimState retry logic, MAX_DISPATCH_ATTEMPTS=3, DISPATCH_RETRY_BASE=500 ms, 100 ms discovery poll (roughly lines 77–230, 299–365).
   - v0.33 used a process-local one-shot claim. A setup failure burned the claim. v0.34 turns Failed into retries at +500 ms and +1 s.
   - Successful Served rows retire; Failed rows do not, so poison remains retry-eligible and a new brain generation starts with an empty process-local claim map.

2. broad opener fallback creates a second replay path
   - dispatch.rs first_line (roughly 390–451) calls net_stream_opener on a fresh Brain connection, but Err(_) falls back for EVERY error, not only explicit old-broker/unsupported-verb.
   - fallback peek_first_line subscribes to the stream, reads only through the first NDJSON line, then drops.
   - Broker StreamLog::attach nevertheless synchronously replays every retained chunk. Once the throwaway client stops reading, its pipe can fill and hit the exact 15-second subscriber write poison.
   - A transport timeout can therefore be followed by a fallback subscriber deadline; this is a direct ~30-second mechanism.

3. synchronous replay under per-stream lock
   - crates/spt-daemon/src/nethost.rs StreamLog::attach/append/finish (roughly 283–351; subscribe at 1676–1693) performs sub.write synchronously while holding the per-stream StreamLog mutex.
   - write results are discarded. Replay continues iterating after the first error; cleanup/lock release waits for replay traversal to return.
   - BrokerConn production write bound is 15 seconds (broker.rs/conn.rs). CONN_WRITE_POISONED is the watchdog deadline, not proof write_frame has returned or peer EOF has propagated.

4. poison does not bound dispatcher completion
   - crates/spt-daemon/src/brain.rs ordinary BrainConn::Whole read_frame is blocking/unbounded; only pump Split has recv_timeout.
   - crates/spt-daemon/src/attach.rs serve_attach owns two independent broker IPC conns: `brain` receives network input/session output; `wire` forwards session output with fire-and-forget net_stream_send.
   - If wire/NetHost send backpressures, the single serve thread stops draining brain. Broker poisons brain after 15 s, but that cannot cancel the distinct wire conn. NetHost peer send has its own 10-second bound. Broker error reporting can itself write to a wire client that never reads, adding another up-to-15-second poison window.
   - DISPATCH logs only after serve_attach returns and updates ClaimState. Hence poison-to-DISPATCH tails of 0.1–47 seconds are expected from unbounded client read, distinct forward write, cancellation/EOF propagation, and scheduling; there is no source 20–30 timer.

5. end-to-end viewport backpressure chain
   ConPTY drain -> broker OutputLog -> target serve `brain` -> distinct Whole `wire` -> NetHost send (10 s) -> operator StreamLog -> operator RC BrokerConn (15 s) -> rc stdout write/flush (unbounded).
   Any stopped consumer can propagate backpressure in reverse. The normal blast radius is one Attach stream/viewport. Several PTYs can coincide during cold rediscovery/retry or shared scheduler/IPC pressure, but a single per-stream lock cannot freeze unrelated ConPTY children by itself.

6. stale Attach control steal companion
   - Production incident evidence already showed old/current low and high stream IDs rediscovered together.
   - Same-origin stale serve_attach may become_controller; replay EOF then detach_if clears controller. If the stale worker finishes last the freeze is indefinite until fresh rc; if the active worker finishes last after the replay/retry storm, it auto-recovers.
   - This does not supersede the W1 deadline/fallback mechanism. It explains ownership corruption; W1 replay/retry explains delayed ordering and recurrent 15–30+ second windows.

v0.33 -> v0.34 boundary
========================
v0.33 release commit: 42d06a87401a62d2a8a6a81ccff2532d43ca6ede.
v0.34 W1 runtime commits:
- ad7fde8acb97b5c839316121d341375b077092df: durable opener, retirement, retryable claims.
- 2a110eed6f97d3dab96ed301d1b6d1df9a2cc002: finished Attach terminal/unserved.
- 84253572cd5e0cef57201ba671a8c201c6dbbdd8: retirement is eligibility visibility only, not transport teardown.
- 571d6a55e9949ce1b96aef1ba00f5e007a0a9521: ADR-0038 round-2 ruling.
v0.34 release commit: eb9fd8643921930ce21cc7ef0751a27360fc14bc.

No 15-second deadline was newly added in v0.34. The regression is that retryable reconstruction, broad fallback, and stale-row rediscovery repeatedly drive that old blocking seam. The intended W1 fix prevents some permanent stale replay cases, but its failure/retry path can convert a one-shot v0.33 abandonment into recurrent transient wedges.

Ranked hypotheses after falsification
=====================================
1. HIGH: v0.34 redispatch retry + synchronous StreamLog replay/backpressure. Exact deadline/source/log match.
2. HIGH: broad ANY-Err opener fallback -> throwaway peek subscriber replay -> another deadline; likely contributor where opener fast path errors or old rows lack durable opener.
3. HIGH companion for multi-PTY/control symptoms: stale finished Attach replay steals/detaches controller; completion order determines recovery.
4. MEDIUM: separate wire/NetHost send and unbounded Whole read explain post-poison 10–47 s tail and 25/30/60 s composites.
5. MEDIUM amplifier only: unbounded dispatcher thread::spawn/connections during cold enumeration. Source defect exists, but current field snapshot does not prove thousands simultaneously live.
6. LOW as primary cause: peer pump 10-second failures. Pump remained alive; failures indicate network/load pressure but do not establish PTY causality.
7. Falsified as global primary for measured incidents: broker-wide IPC wedge. Unrelated conns progressed during exact poison windows.
8. Unproven/low: ConPTY child-local freeze. Need PTY output sequence/child counters during a visible stall; current transport evidence points above it.

Fix design, in priority order
=============================
P0 correctness/latency:
1. Narrow first_line fallback. Fall back only on an explicit UnsupportedVerb/old-broker result. Transport timeout, EOF, poison, or other I/O error must become Failed/requeue under bounded policy, never immediately open a second replay subscriber.
2. Stop StreamLog replay at the first subscriber write failure. Return/propagate io::Result, remove the failed subscriber, and do not keep a poisoned subscriber installed. Apply the same failed-subscriber removal discipline to append and finish.
3. Remove socket writes from the StreamLog mutex. Install a bounded per-subscriber writer queue, enqueue replay-before-live ordering under the lock, release the lock, perform framed I/O on the writer. On queue overflow/failure, detach and resume from cursor/ring; never block the peer/read pump.
4. Bound Attach worker completion and forwarding. Make subscription setup acknowledged or surface deterministic close; use bounded/acked wire sends or close/cancel the wire on send failure. A brain subscriber poison must cancel its paired serve/wire worker rather than waiting indefinitely on distinct IPC.
5. Preserve retries, but classify them. Transient pre-setup connect/opener race may retry. Deadline-poisoned replay must use circuit-broken/global backoff and cannot immediately reinstall until the prior handler/subscriber is fully gone. Do not revert wholesale to v0.33 permanent abandonment.

P0/P1 ownership and load:
6. Enforce terminal finished-Attach exclusion before serve_attach/subscription and prevent same-origin stale replays from becoming controller. Validate stream/session/generation ownership before attach/detach; a stale worker must never detach a replacement controller.
7. Bound redispatch concurrency with a fixed worker pool/semaphore and bounded pending queue; batch cold enumeration. Never hold claim locks during I/O.
8. Clone required session/log state, then drop global sessions lock before disconnect detach_if/info.json I/O, avoiding cleanup serialization of input/resize/attach.
9. Add keyed stage telemetry: generation, stream, family/opener, endpoint/session, attempt, conn; opener start/end/fallback reason; replay start/chunks/end/error; attach_as/controller replacement; wire send start/end; poison/cancel-issued/inflight-clear/sever/client EOF; worker outcome/claim transition; active/pending worker gauges; PTY output high-water and RC render cursor.

Required regression tests
=========================
1. Production dispatcher, Windows: five real active remote Attach/PTYs plus thousands of stale/terminal rows. Continuously send unique input and autonomous output markers. Fresh brain generation. Assert bounded maximum viewport/input gap, no control steal, no duplicate old output, bounded worker/conn counts, and independent sessions/net-status latency.
2. NetHost subscriber poison path: shorten test write deadline to 100–400 ms, black-hole a real dispatcher subscriber during retained replay. Assert replay stops on first write error; old handler is gone before retry; poison -> inflight-clear -> sever -> client EOF -> DispatchOutcome is tightly bounded; unrelated viewport progresses.
3. Opener compatibility matrix: UnsupportedVerb invokes peek fallback exactly once. TimedOut/EOF/poison/transport errors never invoke fallback.
4. Retry-after-installed-subscriber: inject failure after subscription/replay starts. Assert no repeated deadline wedges and retry stays within per-stream and global concurrency budgets.
5. Stale/current Attach ordering: finished stale Attach plus active Attach for same endpoint/session; exercise both completion orders. Stale worker cannot become/detach controller; current controller remains stable.
6. Brain refresh E2E: five remote RC streams remain attached through brain cycle with continuous markers. Child and broker PID stable, input echoes, output/render cursors advance, no reconnect banner for brain-only cycle.
7. Mixed-image E2E: v0.34 brain against v0.33 broker with real active/finished streams; assert documented degradation or require broker feature activation. Existing handshake-only N-1 test is insufficient.

Missing coverage that permitted this
====================================
- redispatch D3 injects one failure before setup; it never blocks/poisons an installed subscriber or exercises retry replay.
- conn_blackhole_lifecycle covers session controller subscribers, not NetHost StreamLog subscribers through run_dispatch_loop.
- brain_resume_conn_deadlock checks heartbeat under six PTYs, not redispatch cold-start viewport/input gaps.
- daemon_refresh_e2e proves local child/PID survival, not five cross-node Attach continuity.
- no stage telemetry maps poison stream IDs to opener family/endpoint/session, so current causality requires inference at that final identity edge.

Fast field discriminator
========================
During the next visible stall, sample every 250 ms:
- broker PTY OutputLog seq/high-water,
- affected RC render cursor/stdout progress,
- endpoint input echo,
- fresh sessions/net-status latency,
- child CPU/I/O counters,
- dispatcher stage/stream/conn IDs.

Interpretation:
- PTY seq advances, child stable, fresh probes answer, affected input/render stops -> Attach/redispatch transport cause confirmed.
- PTY seq stops but fresh probes answer -> victim ConPTY/output drain.
- fresh independent probes also delay -> broker scheduler/IPC pressure is process-wide.
- poison stream classified as affected active Attach and recovery follows retry -> direct causal proof.
- poison IDs unrelated and affected stream uninterrupted -> timing correlation incidental.

Operational caveat
==================
A full `spt update --restart` intentionally cycles the broker and rc has a visible bounded 30-second reconnect loop. That is distinct from this silent pre-sever stall. Brain-only refresh is supposed to preserve viewports continuously.

No production state, endpoints, sessions, logs, or code were mutated for this investigation. Evidence came from current daemon telemetry, source/history, and existing tests/docs.