# PUMP-TRUTH — TRIAGE & SCOPE (doyle)

**Operator-raised 2026-07-08** via deployah (peer-pump stall RCA, build-lane not release-lane). Full RCA: memory `subnet-peer-pump-stall-rca`; the 21d-old prior sighting: `peer-pump-stall-bhalf-confirmed`. This milestone makes peer discovery resilient to offline peers. Lane: **doyle scopes+REQ → todlando builds → doyle gates → deployah publishes** ([[dont-solo-across-role-lines]]).

## Symptom (fleet-wide, operator /diagnose)
HFENDULEAM + ENLYZEAM + a 0.29.x desktop: `spt subnet status` shows member counts but `nodes: []` — no node sees any peer; `peer pump STALLED (Ns since last tick)`. Correlated with enlyzeam's BS-Resources join (2→3 subnets, +~5 mostly-offline paired peers → a >30s offline-peer dial hits certain every round → pump wedged permanently).

## Root (RCA, captured via foreground `daemon run` stderr)
`PEER_PUMP_FAIL: brain IPC read deadline elapsed` + doubling `PEER_PUMP_RESTART` backoff (5→10→20→…→300s cap), **zero** `PUMP_PEER_FAIL:{peer}` fast-fail lines. Chain: pump dials paired peers via `brain.net_dial` (bounded by the brain's 30s `PUMP_PEER_IO_TIMEOUT`); most peers OFFLINE; the broker's dial to an offline-but-**resolvable** peer does not fast-fail → the brain's own 30s read-deadline fires (`TimedOut`) → poisons the pump client (REQ-HAZARD-PUMP-IPC-DEADLINE) → `run_peer_pump` returns Err → `supervise_pump` doubling-backoff restart → pump sits in backoff ~always → never advertises presence nor connects peers → `nodes:[]` fleet-wide. **Network exonerated 4 ways** (see RCA): a live `connprobe` reached enlyzeam→HFENDULEAM in 339ms over the SAME `NetEndpoint::bind` path; discovery TXT globally resolvable; relay live; clocks synced; family-gate correct.

## The load-bearing puzzle (what W1 must resolve)
The broker-side 10s QUIC-op bound **already shipped v0.8.3** — `REQ-HAZARD-BROKER-QUIC-DEADLINE` / KNOWN-HAZARDS **7.8** (NOT the b-half memory's stale "KH7.6" = that's the A-half). `NetHost::dial` wraps `endpoint.connect()` + `prove_membership` in `bounded_block_on` = `tokio::time::timeout(10s, …)` (nethost.rs:882-919, :809). **10s < 30s, so the broker should ALWAYS reply first** with an ordinary error → `peer_outcome` ordinary arm → `PUMP_PEER_FAIL` → round continues.

**Yet on 0.29.x (which HAS the v0.8.3 bound) the field shows the 30s brain deadline firing and ZERO fast-fails.** So the shipped 10s bound provably does NOT bite an offline-but-**resolvable** iroh peer. Leading hypotheses (todlando RCA-confirms FIRST, do not fix blind):
1. **Uncancellable connect** — dropping the `timeout` future on elapse may not cancel iroh's in-flight discovery/holepunch/relay connect (a resolvable peer triggers a long relay+holepunch retry that the outer `timeout` should still abort at 10s — verify it actually does under `block_on`).
2. **Pre-bound resolve span** — a discovery/address-resolve step that runs OUTSIDE the `bounded_block_on` span (or inside `connect` but not yielding to the timer).
3. **Time-driver starvation** — `bounded_block_on` uses `self.runtime.handle().block_on(...)`; under serial fan-out (many blocked dials on the broker dispatch thread) the runtime's time driver could fail to advance so the 10s timer never fires. (This couples to W2.)

**Test-level gap that let it ship:** the B-half's hermetic test uses an **unreachable** addr (fast connection-refused) + `set_quic_op_timeout(150ms)`; the A-half test simulates "broker never replies" (brain deadline). **Neither exercises a resolvable-but-offline iroh peer** — the exact field mode (discovery TXT present, endpoint dead → iroh attempts relay+holepunch for >30s). W1's int test MUST use that mode.

## Wave plan
### W1 — REQ-PUMP-DIAL-FASTFAIL (tactical, RCA-complete + fix)
The offline-but-resolvable-peer dial fast-fails within the broker bound as an ordinary per-peer error (`PUMP_PEER_FAIL`), so the round continues and the heartbeat advances — NEVER the brain's 30s `TimedOut` poison. RCA-FIRST: confirm WHY the shipped v0.8.3 10s bound doesn't bite this path (hypotheses above), then the minimal fix. **Gate:** a hermetic/real test dialing a **resolvable-offline** peer (discovery TXT present, no live endpoint) returns within the bound as `PUMP_PEER_FAIL`, the pump round completes, heartbeat advances — asserted, not the hermetic-unreachable proxy the v0.8.3 test used. Happy path unchanged (a live peer completes with zero added latency). KNOWN-HAZARDS entry extends 7.8 with the resolvable-offline mode.

### W2 — REQ-PUMP-PEER-ISOLATION (architectural, operator ruling)
"One peer must NOT block all others; peer discovery must be async / per-peer-independent." Two coupled defects in `run_peer_pump`: (1) **sequential fan-out** (`for peer in fan_targets` dials one-at-a-time, each up to the bound → peer N+1 waits behind peer N); (2) **whole-round poison** (`peer_outcome(...)?` — one TimedOut aborts the ENTIRE round → supervised restart, resetting ALL conns). Fix: per-peer concurrency + fault isolation — the pump issues non-blocking dials, the broker (already async tokio+iroh) returns connection/presence as async events (D4c presence seam); a peer TimedOut drops + reschedules ONLY that peer, NEVER aborts the round or restarts the pump. **Supervised-restart reserved for a dead BROKER conn, not a dead peer.** **Gate:** a mixed roster (1 live + N offline peers) — the live peer connects and this node advertises presence in the SAME round the offline peers fail; heartbeat advances every round; no `PEER_PUMP_RESTART` from a dead peer. (The single-thread+bounded-read A-half was defensive — it stopped the infinite wedge but coupled every peer's fate; W2 decouples.)

## Sequencing & discipline
W1 (stall-fix, ships fast) → gate → W2 (architectural) → gate → release (deployah). RCA-first on W1 (like F028-B5) — request nothing built until the timeout-not-firing root is code-confirmed.

**3-WAY AGREEMENT GATE (operator ruling 2026-07-08):** PUMP-TRUTH was first RCA'd by deployah on real boxes (foreground `daemon run` trace on enlyzeam) — deployah is the field-evidence authority. So todlando's W1 root-cause finding (WHY the shipped 10s bound doesn't fire) is CROSS-CONFIRMED with deployah before doyle blesses any fix. No fix ships until todlando + deployah + doyle all agree the root against the field evidence. The check sits at the root-confirmed gate (before build begins), so it doesn't slow the investigation — it prevents a plausible-but-wrong code hypothesis diverging from what the boxes actually showed. Instrument per RCA: `spt daemon run` FOREGROUND with `Start-Process -RedirectStandardError` (NOT a `Select-Object -First` pipe — it blocks); the daemon writes no disk trace, but `supervise_pump` eprintln's `PEER_PUMP_FAIL/PANIC/RESTART`. Watch: this is the resident-BROKER path (broker-update batch, rare) — W1's fix reaches the fleet only via a broker update, same as v0.8.3 did.

Co-authored by: doyle

## W1 ROOT-LOCK + FIX RULING (doyle, 2026-07-08 — pending round-2 empirical confirm)

**The original triage premise was WRONG and is corrected here.** Field measurement (deployah, isolated enlyzeam repro, instrumented v0.30.0) EXONERATED the dial (3 dials healthy ~119ms, conn established) and REFUTED every dial-side hypothesis (fast-fail-gap, discovery-resolve, relay-session, prove_membership, time-driver-starvation, dispatch-block — 8 refutations). **Offline peers are RED HERRINGS** (their dial fast-fails). 

**REAL ROOT (code-grounded static pin, deployah):** the pump's **sync leg reply-read** (`sync.rs:374-376`, `request_sync`) opens the sync stream + sends the Request (all bounded, all land), then BLOCKS reading the peer's response with `deadline = brain.call_deadline()` = the 30s `PUMP_PEER_IO_TIMEOUT`. A peer that **CONNECTS + accepts the sync stream but sends NO response** (a connect-then-silent / half-alive peer at fan position #1) burns the full 30s → `read_event_until` returns `TimedOut` → `request_sync` `Err(TimedOut)` → `peer_step` Err → `peer_outcome` (`pump/mod.rs:601`) sees `TimedOut` → **POISON** (Err bubbles → whole-round abort → `supervise_pump` doubling-backoff restart). = exactly the field `PEER_PUMP_FAIL: brain IPC read deadline` + always-zero `PUMP_PEER_FAIL`.

**Two defects (the reply-read is BOUNDED at 30s — not unbounded):**
1. **MIS-CLASSIFICATION (the head-of-line block):** `peer_outcome` treats ANY `TimedOut` as poison (rationale: 'a late reply could bind the wrong stream id') → a NORMAL peer-silent condition kills the WHOLE pump instead of dropping that ONE peer. Fan#1 silent ⇒ whole round dies + backoff.
2. **BUDGET COUPLING:** the 30s sync-reply budget == the 30s brain deadline, so it can NEVER surface as an ordinary per-peer fail — it always races to poison.

### Fix ruling — W1 = (a) + (b); W2 = (c) (the architectural complement)
- **(a) [core, required] — reclassify the sync peer-response timeout as an ORDINARY per-peer error** (drop THAT peer + conn, continue the round), NOT a poison. Reserve `peer_outcome`'s poison **strictly for a genuine broker-IPC-CARRIER desync `TimedOut`** (the brain lost sync with the broker — the carrier itself is broken; THAT is where the 'late-reply/wrong-stream-id' safety actually matters). Impl: `request_sync` maps its peer-response no-progress timeout to a DISTINCT non-poison error kind (source classifies its own timeout), so `peer_outcome`'s poison arm only fires on the real carrier-desync. **Verify** the conn/stream teardown on the per-peer drop prevents a late reply mis-binding a fresh stream id (the exactly-once stream discipline should already cover it — confirm at build; this is the safety the old blanket-poison was protecting over-broadly).
- **(b) [W1, budget-decouple] — shorten + DECOUPLE the sync-reply-read budget below the 30s brain deadline** (~the 10s broker-QUIC bound), so a silent peer drops PROMPTLY even in the still-sequential pre-W2 pump AND the reply-read can never race the brain deadline into poison.
- **(c) [W2, REQ-PUMP-PEER-ISOLATION] — concurrency + per-peer fault isolation.** This root VALIDATES W2 head-on: fan#1's connect-then-silent wait blocking peers 2..N IS the head-of-line block; the whole-round `peer_outcome(...)?` poison IS the whole-round abort. W2's concurrent legs + per-peer drop-and-reschedule remove BOTH structurally.

**(a) alone unblocks the round immediately; (a)+(b) = the clean W1 tactical stall-fix; +(c) W2 = the durable architectural fix.** `REQ-PUMP-DIAL-FASTFAIL` as-scoped (bound the DIAL) is fully REFUTED — the REQ RE-SCOPES to this sync-reply-read reclassification+budget-decouple. Files: `sync.rs` (`request_sync` timeout mapping + budget), `pump/mod.rs` (`peer_outcome` poison reserved for carrier-desync), tests.

**HOLD (discipline):** dispatch todlando the fix ONLY after deployah's round-2 leg-instrumentation EMPIRICALLY confirms the static pin (`LEG_ENTER i=<fan#1> → SYNC_REQ_SENT → ~30s → LEG_EXIT err[TimedOut]`). No dispatch on a static pin alone — this RCA has refuted 8 plausible-but-wrong roots, several code-grounded.

Co-authored by: doyle

## ROUND-2 EMPIRICAL LOCK — corrects the static pin: it's the UPDATE leg (doyle, 2026-07-08)

Round-2 leg-instrumented capture (deployah, enlyzeam isolated repro, 3 IDENTICAL rounds) EMPIRICALLY LOCKS the root and CORRECTS the static guess (sync → **update**):
```
DIAL_EXIT ms=96 ok · LEG i=0 registry 0 · LEG i=1 notif 8 · LEG i=2 sync 72 ok · LEG i=3 update ms=30025 err[TimedOut]  <- THE WEDGE
```
**EXACT LINE: `propagate.rs:373-375`, `request_update`** — opens the update stream + subscribes + sends `UpdRecord::Query` (`net_stream_send finish=false`, :365, all bounded, all land), then BLOCKS `read_event_until(deadline = brain.call_deadline() = 30s)` on the peer's Offer/UpToDate reply. The first fan peer (HFENDULEAM) accepts the conn + update stream but **never answers the Query** → 30s → `TimedOut` → `request_update` Err → update `peer_step` Err → `peer_outcome` POISON → whole-round abort + supervise restart. **Why sync looked innocent:** `request_sync` SKIPS the reply-read when the want-set is empty (this node had nothing to pull); `request_update` has NO such skip → ALWAYS reply-reads → always hits the connect-then-silent 30s. **`sync.rs:374-376` is the LATENT TWIN** (identical `read_event_until` pattern) — bites too the moment a sync want-set is non-empty against a silent peer.

**FIX SCOPE (corrected — covers BOTH legs):** the reclassification+budget-decouple ruling above applies to **BOTH `request_update` (propagate.rs:375, the ACTIVE wedge) AND `request_sync` (sync.rs:376, the latent twin)** — both do the identical brain-deadline reply-read. (a) both map their reply-read no-progress timeout to a DISTINCT non-poison error kind → `peer_outcome` (pump/mod.rs:601) ordinary per-peer DROP, poison RESERVED for genuine broker-IPC-CARRIER desync; (b) both budgets decoupled+shortened <30s; (c) W2 concurrency. Files: `propagate.rs` (request_update) + `sync.rs` (request_sync) + `pump/mod.rs` (peer_outcome), tests: a connect-then-silent peer at fan#0 → the update leg drops it ordinarily, round continues, heartbeat advances, no restart; + the sync-non-empty-want-set latent case. Evidence: enlyzeam `C:\spt-diag\pump-diag.log` (leg-tagged). **ROOT EMPIRICALLY LOCKED (3 identical rounds) — the hold is LIFTED, W1 fix dispatched to todlando.**