---
name: pump-truth-progress
description: "PUMP-TRUTH milestone (peer-pump resilience to offline peers) — scoped 2-wave, W1 dispatched RCA-first to todlando 2026-07-08"
metadata: 
  node_type: memory
  type: project
  originSessionId: c6203feb-91ff-4cdb-b10d-6579afc0a2f4
---

**✅ W1 SHIPPED to main 2026-07-09 (doyle gated+blessed+merged).** Fix @b4862f3 (reply-read poison reclassify + budget-decouple, REQ-PUMP-DIAL-FASTFAIL re-scoped) → PR #66 → FF-merged origin/main @b4862f3. GATE: local isolated-worktree nextest 1698 GREEN + CI both legs GREEN (kitsubito+hfenduleam). **✅ PUBLISHED v0.30.1 counter 52 2026-07-09** (deployah, PR #67, patch bump 0.30.0→0.30.1; survived a mid-build PC crash — self-hosted runner auto-resumed on reboot, release run finished clean, NO rerun/re-tag; flip verified isDraft=false/Latest=v0.30.1/update-set v52 all 3 platforms; counter cross-checked vs published 51). W1 reply-read poison fix LIVE for self-update. deployah's diag instrument stays local-uncommitted as evidence. **CI-FLAKE LESSON:** first CI run RED = ENVIRONMENTAL not the fix — Windows spurious `E0282` at pairjoin.rs:147 (untouched test, not in diff) + cascade `windows rlib` = poisoned CI incremental cache on hfenduleam (`checkout@v5` cleans git tree, NOT gitignored `target/`); Linux 2 E2E fails (`adapter_apply…`,`cross_perch_dead_owner_repin…`) = kitsubito contamination (2 leaky). Discriminator: main was ALREADY red on pure docs/triage commits (can't break compile) + reproduced pairjoin/seedproofx compile CLEAN locally on the SAME box as the CI runner. `gh run rerun --failed` self-cleared it (fresh Phase-A rebuild); no manual cache wipe needed. Local shared-checkout main lags origin by the fix (dirty diag tree blocks in-place FF) — benign, dirt-owner reconciles. **W2 (REQ-PUMP-PEER-ISOLATION) DISPATCHED to todlando 2026-07-09** — dispatch doc `docs/PUMP-W2-BUILD-GO.md @origin/main 6875542` (docs-only FF commit past b4862f3; does NOT affect deployah's release cut). Build order: (1) REBASE pump-w2-design (@02852d9, 7 behind) onto origin/main FIRST — W2 DELETES the `peer_outcome(...)?` poison W1 just reworked (same pump/mod.rs), resolve deliberately (keep W1's carrier-desync→supervise = re-pointed REQ-HAZARD-PUMP-IPC-DEADLINE, assert both dirs); (2) activate REQ [impl,unit,int]; (3) Q1 round-deadline = DERIVED CONST from `brain::PEER_REPLY_READ_BUDGET` (=`Duration::from_secs(10)`, brain.rs:1604) +~2s, SYMBOLIC not hardcoded; (4) build G1-G5 per design @3321229 + rulings @d806932; (5) fresh `.worktrees/` worktree, NOT shared main checkout. Gate = int load-bearing (1 LIVE + N OFFLINE same round, no dead-peer restart, dead-BROKER still restarts, exactly-once drain, §V4 stagger) + traceable/clippy. **BUILD STARTED (todlando, worktree .worktrees/pump-w2, rebased clean on origin/main). DELTA doc `docs/PUMP-W2-BUILD-DELTA.md @pump-w2-design` — doyle APPROVED 2026-07-09:** Δ1 = the G2 split-carrier fallback TRIGGERED for a stronger reason than backpressure = EXACTLY-ONCE correctness (a shared carrier's `read_event_until` on a pull-reply advances `presence_cursor` then `_=>continue` DROPS the presence frame → outcome lost / peer never rescheduled). Resolution = 2nd pump-mode brain `events` OWNS presence subscription+drain, primary `brain` OWNS dial-submit acks + peer_step IO (separate socket→reader-thread→channel, non-blocking `try_recv` drain, deadline bounds only the K>0 submit-wait, steady-state cadence unchanged). peer_outcome whole-round `?`-poison DELETED → per-peer `peer_leg_outcome` (Ok keep / ordinary Err drop+backoff / raw carrier TimedOut → restart). Q2 dual-permit honored (both submit_dial task + blocking dial() acquire the NetHost semaphore). **doyle GATE-ADD (load-bearing, must assert at int):** verify the broker delivers presence frames ONLY to the subscribing events-carrier, NOT broadcast to the primary brain conn — if it broadcasts, the primary STILL swallows+advances the cursor and the split does NOT fix gotcha#1. Awaiting todlando build → doyle gates.

**⚡ W2 BUILD COMPLETE 2026-07-09 (todlando, worktree .worktrees/pump-w2 off pump-w2-design rebased clean on origin/main; survived a mid-build PC crash — all Edits persisted to disk, resumed from disk state). LOCAL GATES GREEN, ready to commit + hand SHA to doyle.** Impl per design @3321229 + rulings @d806932 + delta @PUMP-W2-BUILD-DELTA.md (doyle-approved). Files: msg.rs (PRESENCE_DIAL_FAILED kind + `reason`/`remote_addr` optional fields on NetPresenceEvent + KIND_NET_DIAL_SUBMIT/SUBMITTED), nethost.rs (PresenceLog append→push+typed helpers append_connected/append_dial_failed; register_conn carries observed addr; PUMP_DIAL_CONCURRENCY=24 shared Semaphore; DialPlan struct = single-sourced connect for BOTH blocking dial() + non-blocking submit_dial; both acquire the permit — Q2), broker.rs (dispatch_net_dial_submit — derives correlation hex from addr.id via PublicKey::from_bytes, NOT the nonexistent EndpointAddr::node_id_hex; immediate KIND_NET_DIAL_SUBMITTED ack), brain.rs (submit_dial + NetDialSubmitted ack variant; net_presence_subscribe_tail = TAIL-FOLLOW from_seq=u64::MAX + cursor=None baseline-on-first — NO history replay, evicted-ring can't gap-trip a false restart; PEER_REPLY_READ_BUDGET now pub(crate)), pump/mod.rs (2nd `events` carrier owns presence drain; per-peer PeerSched{next_due,backoff} + next_peer_backoff/peer_eligible; round=submit-nonblocking→run-connected-legs→drain-until-deadline→mark_ran; peer_leg_outcome SUPERSEDES peer_outcome — ordinary/reclassified→drop+backoff, raw carrier TimedOut→Err→restart KEPT; handle_presence_event + pure presence_state_effect; resolve_submit_addr seeded-first; ROUND deadline = brain::PEER_REPLY_READ_BUDGET + 2s SYMBOLIC), peeraddrs.rs (forget/drop_seed — stale-seed self-heal on DIAL_FAILED), traceable REQ activated [impl,unit,int]. **KEY DESIGN CALLS:** (1) doyle's G2 split-carrier fallback ADOPTED — 2 pump brains (peer_step pull-reply reads on `brain` would swallow presence frames off a shared carrier = exactly-once break); PresenceLog is single-subscriber so primary brain structurally never gets presence (gate-add satisfied by construction + asserted). (2) DRAIN classification: `TimedOut`=quiet (drain done, break), any other read err (broker Disconnected/UnexpectedEof)=dead broker→Err→restart — the re-pointed IPC-deadline, no new non-blocking API needed (split carrier is reader-thread+channel; short deadline = try_recv-equivalent). (3) REQ-CONV-1 preserved through the async path: observed-addr write-back rides PRESENCE_CONNECTED (register_conn→append_connected), seeded-first resolution kept, stale-seed dropped on DIAL_FAILED; old ensure_conn/dial_seeded REMOVED, tags migrated. **LOCAL GATES: clippy --workspace --all-targets ZERO warnings; traceable check --json exit 0; 12 W2 tests GREEN** (8 unit: peer_leg_outcome, per_peer_backoff, presence_state_effect, resolve_submit_addr, msg round-trip, 3 peeraddrs; 4 int in tests/pump.rs: pump_w2_live_peer_advertised_amid_dead_peers_no_restart [load-bearing: 1 live proving-peer + 2 hermetic black-holes, live connects+advertised same rounds dead peers DIAL_FAIL, heartbeat advances, Ok-on-stop no restart], pump_w2_dead_broker_bubbles_err_for_restart [no-broker→connect()→Err = restart trigger; NOT mid-run death — deterministic variant], pump_w2_presence_reaches_only_the_subscribing_carrier [single-subscriber gate-add], + W1 regression pump_survives_a_black_holing_peer still green). **TEST GOTCHA (cost a debug cycle):** the FIRST pump round PRIMES ALL legs (`due(last=None,...)`=true regardless of cadence) → the update-pull leg runs vs a connect-then-silent live peer → blocks ~10s (PEER_REPLY_READ_BUDGET) before W1's per-peer drop → a 10s heartbeat-advance window RACES it. FIX = widen window to 30s (behavior is CORRECT — per-peer drop, no wedge/restart; parking sync/update cadence only stops REFIRE, not the round-1 prime). **EndpointAddr has NO node_id_hex (that's on NetEndpoint); derive peer hex from `addr.id.as_bytes()` via PublicKey (broker.rs:3832 pattern).** **COMMITTED @6e92be9 + pushed (force-with-lease, rebased branch) + HANDED to doyle 2026-07-09** (handoff doc docs/PUMP-W2-GATE-HANDOFF.md, child commit 49c591b; regression sweep = 712 tests GREEN 0 fail 4-leaky-benign, incl presence lifecycle + self-drive + W1 black-hole regression). **2 CALLS FLAGGED to doyle (non-blocking):** (1) dead-broker int test is the DETERMINISTIC no-broker→connect()→Err variant (Broker::stop only unblocks accept(), doesn't sever already-handled carriers → clean mid-run kill not reliably reproducible; same Err→restart code path); (2) "advertises same round" covered by live-connect-converge + fan-out ordering (registry=leg0 runs on CONNECTED before update), no explicit stream-receipt assertion — offered to add both if doyle wants. **doyle GATE = CONDITIONAL PASS 2026-07-09** (full clean gate GREEN in his warm worktree: clippy 0, traceable 0 findings[], nextest FULL workspace 1720/1720 pass 8-leaky-benign 1-env-skip; NO sibling-seam breakage across inject/resume/twohost/term — my scoped 712 held whole-suite). Code CONFORMS, drain classify pump/mod.rs:637 correct. **BOTH flagged calls RULED REQUIRED (not optional) → LANDED @ca8058e (branch pump-w2-design):** flag1 = extracted `classify_drain_read`+`DrainStep` enum from the round loop → deterministic unit `drain_read_classifies_dead_carrier_as_restart_timeout_as_quiet` (TimedOut=RoundDone, any other events-carrier read err=dead-broker-mid-drain→Restart→Err→supervise; guards the L637 non-TimedOut arm a refactor must never swallow = W1 field-bug reintro guard). REMOVED the connect-fail dead-broker int (fails BEFORE the drain, doyle's point) → pointer comment to the unit. flag2 = explicit 'advertised same round' converge (live holds A's registry-push stream via net_streams remote_id_hex==a_hex) in the load-bearing int. Local re-check post-adds: clippy 0, traceable 0, affected 7/7 green. **✅ W2 GATE = PASS + BLESSED 2026-07-09 @ca8058e** (doyle re-verified both adds: clippy 0, traceable 0, nextest FULL workspace 1720/1720 8-leaky-benign 1-env-skip, no sibling breakage; flag1 classify_drain_read unit + flag2 advertised-same-round both green). **PR #68 UP (pump-w2-design→main), doyle DRIVING the merge** — CI both legs running, doyle watching, MERGE ON GREEN (rerun-not-rework on the cache/contamination flake per [[v0301-published]] lesson). W2 branch behind main only by the v0.30.1 release (Cargo.toml+CHANGELOG; W2 touches neither = NO conflict, GitHub tests the merge result). **todlando HOLDING for doyle's merge ping** (doyle pings on merge; then deployah publishes = release lane, NOT mine; only a genuine CODE red = I fix, a flake = rerun). Build complete — clean conformance, the 2 self-flagged calls were the right calls (doyle). throwaway CARGO_TARGET_DIR=.worktrees/pump-w2/target-w2check left warm until merge. **W2 DELIVERED. Next: post-merge, deployah cuts a release (broker-update batch reaches fleet — resident-broker path); W2 is the durable cure for the peer-pump stall (W1 tactical reply-read fix shipped v0.30.1, W2 = concurrent non-blocking dials + per-peer isolation).**

PUMP-TRUTH = make peer discovery resilient to offline peers. Operator-raised 2026-07-08 via deployah (fleet-wide stall: `spt subnet status` shows member counts but `nodes:[]`; `peer pump STALLED`). Full RCA: [[subnet-peer-pump-stall-rca]]; prior 21d sighting [[peer-pump-stall-bhalf-confirmed]] (now SUPERSEDED — see below). Scope doc: `docs/PUMP-TRUTH-TRIAGE.md` @`c85e2a8` (main). Lane: doyle scopes+gates, todlando builds, deployah publishes.

**Root:** offline-but-**resolvable** peer dial doesn't fast-fail → brain 30s `PUMP_PEER_IO_TIMEOUT` read-deadline poisons the pump client → `supervise_pump` doubling-backoff (5→…→300s) → pump never advertises presence → `nodes:[]` fleet-wide. Correlated with enlyzeam's BS-Resources join (+~5 offline peers → a >30s offline dial certain every round). Network exonerated 4 ways (live connprobe reached peer in 339ms same bind path).

**KEY FINDING (reshapes the b-half memory):** the tactical broker-side 10s QUIC bound **ALREADY SHIPPED v0.8.3** = `REQ-HAZARD-BROKER-QUIC-DEADLINE`, KNOWN-HAZARDS **7.8** (NOT the b-half memory's "KH7.6" = that's the A-half). `NetHost::dial` wraps `endpoint.connect()`+`prove_membership` in `tokio::time::timeout(10s)` (`bounded_block_on`, nethost.rs:809, :882-919). 10s<30s so the broker *should* reply first. **YET 0.29.x (which HAS the bound) still stalls with the 30s brain deadline + ZERO `PUMP_PEER_FAIL` fast-fails** → the shipped bound provably does NOT bite a resolvable-offline iroh peer. The v0.8.3 hermetic test used an UNREACHABLE addr (fast refuse); the A-half test simulates broker-never-replies — neither exercises a resolvable-dead iroh peer (discovery TXT present → iroh retries relay/holepunch). That test-gap let it ship.

**W1 RCA STATUS (2026-07-08, todlando → docs/PUMP-W1-RCA-FINDINGS.md on branch rca/pump-w1-fastfail @36f5bfb):** starvation REFUTED (existing B-half test cuts a pure `pending()` on this runtime = timer fires — nethost.rs:1602). Pre-bound-resolve REFUTED (pump `dial_seeded` addr → straight into `host.dial` → the bound; no I/O outside). **LEADING root:** iroh 0.98.2 `Endpoint::connect()` to a black-holed-resolvable peer ESCAPES the outer `tokio::time::timeout(10s)` (uncancellable / non-yielding blocking-poll on the block_on driver thread) → 10s never cuts → 30s brain deadline wins. The v0.8.3 test used `pending()` not a real `connect()` = the test-gap. **ROOT DIVERGENCE to reconcile:** deployah reads the same zero-PUMP_PEER_FAIL as 'hang NOT in the wrapped dial' (unwrapped path); todlando reads 'IN the wrapped connect, escapes timeout.' Both fit the field trace. **ARBITER (doyle greenlit, RCA-only):** todlando's hermetic TEST-NET-1 (`192.0.2.1`, RFC5737 silent-drop) black-hole-dial through `NetHost::dial` — `bounded_block_on(500ms)` returns ~bound ⇒ deployah right (wrap bites, re-open W2-ward); ≫bound ⇒ todlando right (iroh connect escapes wrap → fix = dial resolved direct addr under iroh's own connect/handshake timeout or a cancel token, NOT tokio::timeout(connect)). 3-way concur AFTER repro. **DIVERGENCE COLLAPSED 2026-07-08:** deployah's field trace does NOT pin where connect stuck (only supervise_pump stderr: BRAIN_UP → 30s brain-IPC deadline; no per-dial tracing) — so deployah CANNOT arbitrate in-wrap-vs-unwrapped from the field; his 'not-in-wrapped' was an honest inference he WITHDREW (todlando's uncancellable-escape fits his trace equally). **deployah concurs, NO DISSENT: the TEST-NET-1 repro is the sole root-lock arbiter** (overrun=todlando escape → activate REQ + fix; bite=re-open). Now purely awaiting todlando's empirical repro result. NO REQ activation / fix until it lands.

**⚠ SHARED PHYSICAL CHECKOUT (doyle + deployah + TODLANDO — all three, 2026-07-08):** the three agents operate the SAME physical spt-core checkout (C:\Users\decid\Documents\projects\spt-core) — ANY merge/branch-switch/push SWITCHES everyone's working-dir HEAD (bit doyle repeatedly with 'on feat branch' surprises; bit todlando when deployah's v0.30.0 merge switched him to main mid-design and he accidentally committed the W2 design onto local main → he reconciled clean: branched to pump-w2-design + reset local main to origin, nothing pushed). During the musl-release window doyle HELD main pushes (deployah cut server-side via gh pr merge + tag, pinged before/after). **GOING-FORWARD FIX (raised to todlando, endorsed): concurrent branch work goes in per-agent `.worktrees/` worktrees (isolated HEAD, the CLAUDE.md worktree discipline), NOT the shared main checkout — W2 BUILD will run in a worktree off pump-w2-design.** Also: another-agent's-truncated-send lesson recurred — doyle's Q1-Q3 ruling send truncated (middle eaten), re-sent as a FILE (docs/PUMP-W2-DISPATCH-RULING.md addendum @ca2038b) + short pointer; long rulings ALWAYS ride files ([[lifecycle-truth-build-progress]] live-SENT-eats-heads).

**⚡ ARBITER FLIPPED 2026-07-08 (repro f366860):** the wrap BITES — `bounded_block_on` cut a black-hole dial at ~500ms in ALL 3 modes (direct relay-off 510ms, direct relay-on/production 501ms, id-only-discovery 504ms). todlando's in-wrap-escape hypothesis REFUTED; the 10s dial bound WORKS (dial + open_stream + send ALL bounded, nethost.rs:1051/1074). **So W1-as-scoped (dial doesn't fast-fail) is REFUTED — the premise was wrong.** deployah's off-wrap read was right.

**CONVERGED ROOT (3-way, 2026-07-08 — 4 refutations deep, now LEADING pending the field-lock):** doyle's W2-accumulation synthesis was REFUTED by deployah (code-grounded, brain.rs:1216): `net_dial` uses a PER-CALL `call_deadline`=now+30s, NOT a round budget — so sequential ~10s dials each fast-fail under their OWN fresh 30s and PRINT PUMP_PEER_FAIL; they can't accumulate into one 30s TimedOut. Zero-PUMP_PEER_FAIL + 30s ⇒ ONE dial exceeded 30s with the 10s bound NOT firing. doyle grep corroborated: net_dial/open_stream/stream_send all per-call-bounded (brain.rs:1216/1266/1305), pump hot path calls ONLY those three (mod.rs:683/725/726) → no non-dial unbounded op. **ROOT = the id-only-discovery-SUCCEEDS dial escapes the 10s host.dial bound.** deployah field pre-fact: enlyzeam has NO peer-addrs.json → ALL dials are id-only-discovery. todlando's NXDOMAIN repro (no TXT) fast-failed ~500ms; the direct-addr black-hole repro cut at ~500ms too (QUIC-handshake future IS cancellable). But a field peer's TXT RESOLVES → the discovery-resolve span OR the relay-wait-after-resolve escapes the outer timeout-drop → ONE dial >30s → brain TimedOut → zero PUMP_PEER_FAIL = the field. host.dial wraps `endpoint.connect()` but the id-only discovery sub-future isn't effectively cancelled (todlando's ORIGINAL escape intuition — localized to the discovery path the direct repro couldn't hit). **W1 NOT refuted — RE-SCOPED to a bound-gap on the id-only-discovery/relay-resolve path (not host.dial's QUIC handshake).** ~~W2 demoted to optional hardening~~ — **REVERSED by OPERATOR RULING 2026-07-08: W2 (concurrent/async peer pump) is MANDATORY, NOT optional.** doyle mis-demoted it by conflating 'not the stall root' with 'not required.' Even after W1 bounds the escape, a SEQUENTIAL fan-out head-of-line-blocks peer N+1 behind peer N (up to the bound each) and `peer_outcome(...)?` aborts the whole round on one failure — architecturally wrong regardless of stall root. BOTH waves are REQUIRED deliverables: W1 (bound the escape, stall-fix, gates first) + W2 (REQ-PUMP-PEER-ISOLATION: concurrent non-blocking dials + per-peer fault isolation, one peer drops/reschedules only itself, never aborts round or restarts pump; supervised-restart reserved for dead BROKER conn). W1 still RCA-locked on empirical >30s; W2 is a full design+build wave after (broker returns presence as async events, D4c seam), NOT deferred. doyle to ready a W2 dispatch ruling; scope W2 async model in parallel with the W1 root-lock wait. **TODO post-musl-tag: fix docs/PUMP-TRUTH-TRIAGE.md if it carries the 'optional' framing (currently can't push — holding main for deployah's release).**

**TRIANGULATION NARROWED 2026-07-08 (all 3 aligned):** direct-addr black-hole dial BITES (~500ms, repro f366860). todlando's Half-2 relay-wait synth (home relay usw1-1.n0-canary injected onto an absent-peer addr — DNS bypassed) BITES at 510ms (@3d95788) → relay-connect-to-absent is properly cut INSIDE the wrap → **relay-wait EXONERATED**. By elimination the >30s field escape is **Half-1 DNS-discovery-resolve** (the pkarr/DNS TXT lookup for an id-only addr — the ONLY half the synth structurally skips; enlyzeam is all-id-only-discovery, no peer-addrs.json, so every dial hits it). Mechanism: a blocking/un-cancellable resolve `tokio::timeout` can't cut, vs the cancellable QUIC connect. **REFINED FIX SITE (held pending field-lock):** SURGICAL to the resolve span — spawn_blocking the resolve / iroh's own discovery timeout / cancel-token around the resolve — NOT re-wrapping the whole `endpoint.connect()` (already cut). **⚡ HALF-1 REFUTED 2026-07-08 (refutation #5, EMPIRICAL beats by-elimination):** deployah MEASURED the actual system resolver (iroh hickory DnsDiscovery's) on a REAL published iroh TXT from hfenduleam = OK in 2.9s (public 1.1.1.1 = 1.2s), NOT >30s. So Half-1 DNS-resolve is FAST, not the escape — our by-elimination conclusion overturned. NOW all THREE isolated sub-paths fast-complete (NXDOMAIN 504ms + relay-wait 510ms + resolve 2.9s) YET a single field dial = >30s. **The >30s is a component none of us measured in isolation.** NEW SUSPECTS: (a) QUIC connect HANDSHAKE to a peer with a STALE-RELAY-SESSION (relay has a LIVE session for the now-dead peer → forwards → handshake waits for the absent peer — DIFFERS from todlando's synth where the relay had NO session → fast no-route 510ms; a live-but-stale session is UNTESTED); (b) prove_membership (in-band seed-proof, nethost.rs:901-918). BOTH inside bounded_block_on yet seemingly not cut by the 10s timeout. **NEW ARBITER (deployah running, file-log instrument 180s, survives detached respawn):** host.dial ENTER/CONNECT_ENTER/EXIT wall-times → pins WHICH segment (ENTER→CONNECT_ENTER=pre-wrap/unwrapped; CONNECT_ENTER→connect-returns=QUIC handshake; connect-returns→EXIT=prove_membership) AND whether the >30s is INSIDE the 10s wrap (=> tokio::timeout not cutting an uncancellable/blocking await) or OUTSIDE it. That segment + inside/outside = the root-lock. **DROP the resolve-span fix direction — re-opened.** W1 build STILL held till the instrument locks the segment.

**WHY 5 REFUTATIONS (todlando's synth-gap diagnosis, key insight):** every hermetic synth BIT yet the field escapes because the synths structurally SKIPPED the two field-only conditions — (1) `membership: None` → prove_membership NEVER exercised (field = `membership: Some`); (2) the synth's relay-wait peer had NO relay session → relay fast-REJECTED (knew peer absent), whereas the field's peer has a STALE-LIVE relay session → relay FORWARDS + the handshake waits for the absent peer. Those 2 gaps = the 2 untested segments = deployah's 2 suspects (prove_membership + stale-live-session handshake). Explains all false-negatives; vindicates holding for deployah's LIVE instrument over the hermetic bites. **CONFIRMATION SEQUENCE (agreed):** deployah's wall-times PIN the segment (live) → todlando synths THAT segment specifically (membership:Some prove_membership OR a stale-live-relay-session peer) to hermetically confirm + it BECOMES the int-test regression fixture → doyle greenlights the fix at the pinned site → gate → W2 build (build-ready @02852d9). Fix-site decision tree PRIMED for deployah (sent): the hung segment = fix site; if total_ms>>10s the tokio::timeout isn't cutting that segment's await (uncancellable/blocking) → fix = make THAT segment cancellable (spawn_blocking / select vs a real honored timeout), NOT re-wrap; if pre-wrap delta → unwrapped path → wrap it. W1 fix build HELD till the wall-times land + the segment locks. **SOLE GATE = deployah's host.dial wall-times.**

**ALL HERMETIC SEGMENTS EXONERATED 2026-07-08 (6 refutations total):** todlando's synths — direct dial bites; relay-no-session 510ms; prove_membership 506ms (A membership:Some connects to B membership:None+alive loopback, seed-proof READ hangs on B's silence, wrap CUTS it @1ac6c27); discovery-resolve 2.9s (deployah field measurement, refuted Half-1). **taskkill /F abrupt-kill BIT 505ms @6653bae** — pre-kill A connected to live B via relay (187ms, session confirmed live), taskkill /F B → process death → OS RST → relay TORE B's session → degraded to no-session fast path (the guard doyle flagged: verify the session persists post-kill; it did NOT). **HERMETIC-KILL EXHAUSTED — no local KILL holds a stale-live session (kill=RST→relay tears it; field needs a NETWORK-dead peer, no RST).** **CURRENT PROBE (doyle GO): SUSPEND-B synth** — freeze B (SuspendThread, not kill) → socket stays OPEN (no RST) → relay session LINGERS → frozen B won't answer the forwarded handshake = the faithful field mode, hermetic no-elevation. HANGS >30s inside the wrap ⇒ stale-live-session handshake is the escape (root LOCKED, fix=make the handshake cancellable + suspend-B is the int-test). BITES ⇒ (relay keepalive tore the frozen session) → deployah's FIELD is the only source (fallback HELD READY: world-writable DIAG C:/Windows/Temp + forced SPT_HOME=decid-home on the de-elevated child so it dials the real enlyzeam roster + writes DIAG regardless of elected-user; ~1min re-instrument+rebuild; only a REAL fleet stale-session on enlyzeam has a network-dead-peer-no-RST). **todlando CHECKPOINTING (high context) — resumes to run suspend-B.** stale-LIVE-relay-session remains the SOLE suspect (all others exonerated).

**⚡ SUSPEND-B EXONERATED stale-live-session 2026-07-08 @a7a3d41 (worktree pump-w1-stale) — HERMETIC ROUTE EXHAUSTED:** suspend-B FAITHFULLY reproduced the field mode (NtSuspendProcess froze B, socket stayed open/no-RST, a 12s LINGER-PROBE rode to 12.0155s = PROOF the relay held B's session + forwarded A2's handshake to the frozen non-answering peer — vs a no-session's ~500ms fast-reject; drop()/taskkill could NOT emulate this) — YET the dial BIT 514ms (500ms bound). **The stale-live-session handshake is CANCELLABLE at ANY bound (500ms→514ms, 12s→12.0155s)** → at a 10s field bound it would bite as PUMP_PEER_FAIL, NEVER the 30s deadline → EXONERATED. **CONCLUSION: NO single hung dial can produce the field signature (30s brain deadline + zero PUMP_PEER_FAIL) — every host.dial segment is cancellable + bites.** **LEADING ROOT (todlando, sharp): tokio TIME-DRIVER STARVATION under real 3-subnet fan-out** — under enough concurrent fan-out load the runtime's time driver can't advance, so the 10s bounded_block_on timeout NEVER FIRES, and the dial rides to the brain's 30s deadline = 30s TimedOut + zero PUMP_PEER_FAIL. REAL-FLEET-ONLY (a hermetic single-dial has no load → timer fires → bites; that's why all 7 synths bit). Secondary: sequential-fan-out accumulation (W2), though per-call-30s-deadline weakens pure accumulation — starvation is the prime. **W1 INTERIM RULING (doyle): REQ-PUMP-DIAL-FASTFAIL as-scoped ('dial doesn't fast-fail') is REFUTED — the bound WORKS. FINAL W1 ruling HELD for deployah's field-DIAG** (the arbiter, CALLED — world-writable DIAG C:/Windows/Temp + forced SPT_HOME=decid-home defeats the de-elevation; **KEY: does DIAG_DIAL_ENTER even PRINT + EXIT ride to 30s with NO 10s bite = starvation confirmed; if ENTER never prints = broker DISPATCH blocked upstream**). LIKELY RE-SCOPE: W1 → a starvation-RESISTANT bound (dedicated time-driver thread / offload the dial off the timer thread so a busy runtime can't starve the deadline) AND/OR fold into W2 (concurrency cuts the per-runtime load + sequential accumulation). This VINDICATES the RCA discipline hard: a naive W1 fix would have bounded a dial that already works. todlando RESTING (high context); deployah running the field.

**⚡ FIELD ARBITER LANDED THE ROOT 2026-07-08 (deployah, isolated repro on enlyzeam via ssh decid@enlyzeam — fresh node.key + distinct home_tag socket = ZERO disruption to the live 0.28.0 daemon; copied REAL 3-subnet roster SPT_DEV/BIGNET/BS-Resources ~8 peers incl stale SCELTOUIN + 3 BS-Res desktops + peer-addr seeds; instrumented v0.30.0, 120s foreground, SPT_PUMP_DIAG):** 3 dials in 120s ALL healthy — NET_DIAL_DISPATCH→DIAL_ENTER→DIAL_EXIT ms=119/88/119 OK, conn established (1/3/4). **DIAL EXONERATED. STARVATION + DISPATCH-BLOCK BOTH REFUTED (both doyle+todlando hypotheses killed).** Capture: enlyzeam C:\spt-diag\pump-diag.log. **REAL ROOT (deployah, field-grounded): a POST-DIAL leg — the SYNC-PULL reply read (the only leg awaiting a peer RESPONSE). A peer that CONNECTS but never ANSWERS the sync request wedges the single-threaded pump 30s → whole fan-out starves.** ⚠ **THE ORIGINAL TRIAGE PREMISE WAS WRONG:** offline peers are RED HERRINGS (their dial fast-fails 119ms) — the trigger is ANY CONNECT-then-SILENT peer at fan position #1 (a half-alive peer: connects via cached/relay path, doesn't answer the app-level sync-pull). NOT a dial/discovery/relay/membership issue at all. **ROUND 2 (doyle GO): deployah instruments the 4 post-dial legs (open_stream / send / sync-reply-read / update-read) to pin the EXACT unbounded IPC reply-read.** FIX (W1 re-scoped, todlando builds after the leg is pinned): bound the post-dial reply-read path (extend the 10s broker-QUIC deadline coverage to the reply-AWAIT, not just dial/open_stream/send) so a connect-then-silent peer fast-fails as an ordinary per-peer error instead of a 30s wedge. **VALIDATES W2 (REQ-PUMP-PEER-ISOLATION) head-on:** fan#1 connect-then-silent wedging the WHOLE single-threaded round IS the head-of-line-block + whole-round-poison W2's concurrency + per-peer isolation kills. So W1 = bound-the-reply-read (tactical), W2 = concurrency (architectural), both still needed. REQ-PUMP-DIAL-FASTFAIL as-scoped fully REFUTED (dial fast-fails fine); the W1 REQ RE-SCOPES to the reply-read bound. Awaiting round-2 leg-pin → doyle scopes the fix → todlando builds. deployah field lane; live daemon untouched throughout.

**⚡ STATIC PIN + FIX RULING 2026-07-08 (deployah pinned, doyle ruled — docs/PUMP-TRUTH-TRIAGE.md 'W1 ROOT-LOCK + FIX RULING' @f16e2e0; round-2 empirical confirm running ETA ~2min):** THE WEDGE = the pump's SYNC leg reply-read, `sync.rs:374-376` (`request_sync`): opens the sync stream + sends the Request (all bounded, all land), then BLOCKS `read_event_until(deadline = brain.call_deadline() = 30s PUMP_PEER_IO_TIMEOUT)` on the peer's response. A peer that CONNECTS + accepts the sync stream but sends NO response (connect-then-silent / half-alive at fan#1) burns the full 30s → TimedOut → request_sync Err(TimedOut) → peer_step Err → `peer_outcome` (`pump/mod.rs:601`) sees TimedOut → POISON (whole-round abort → supervise restart) = the field PEER_PUMP_FAIL brain-IPC-deadline + always-zero PUMP_PEER_FAIL. **The reply-read is BOUNDED (30s), NOT unbounded — TWO defects:** (1) MIS-CLASSIFICATION (the head-of-line block) — peer_outcome treats ANY TimedOut as poison (rationale 'late reply could bind wrong stream id') so a NORMAL peer-silent kills the WHOLE pump instead of dropping ONE peer; (2) BUDGET COUPLING — 30s reply-read == 30s brain deadline, so it can NEVER surface as an ordinary per-peer fail, always races to poison. **FIX RULING (doyle): W1 = (a)+(b); W2 = (c).** (a) [core] reclassify the sync peer-response timeout as an ORDINARY per-peer DROP (drop that peer+conn, continue round); RESERVE peer_outcome's poison strictly for a genuine broker-IPC-CARRIER desync TimedOut (where the late-reply/wrong-stream-id safety actually matters — verify the per-peer conn/stream teardown prevents mis-bind); impl = request_sync maps its read-timeout to a DISTINCT non-poison kind. (b) [W1] decouple + shorten the sync-reply budget <30s (~10s, off the brain deadline) so a silent peer drops promptly pre-W2 + can't race the deadline. (c) [W2 REQ-PUMP-PEER-ISOLATION] concurrency + per-peer isolation — this root VALIDATES W2 head-on (fan#1 silent blocking 2..N = the head-of-line block; the whole-round `?` poison = the whole-round abort). (a) alone unblocks; (a)+(b)=clean W1; +(c)=durable. **REQ-PUMP-DIAL-FASTFAIL as-scoped (bound the DIAL) FULLY REFUTED — RE-SCOPES to this sync-reply-read reclassification+budget-decouple.** Files: sync.rs (request_sync timeout+budget), pump/mod.rs (peer_outcome poison→carrier-desync-only), tests. **HOLD: dispatch todlando the fix ONLY after round-2 empirically confirms the pin (LEG_ENTER fan#1 → SYNC_REQ_SENT → ~30s → LEG_EXIT err[TimedOut]) — no dispatch on a static pin, 8 refutations deep.** todlando RESTING (loop him with the re-scoped W1 on round-2 confirm); deployah field lane, live daemon untouched.

**⚡ ROUND-2 EMPIRICALLY LOCKED THE ROOT + CORRECTED THE STATIC PIN 2026-07-08 (deployah leg-instrumented capture, 3 IDENTICAL rounds; docs/PUMP-TRUTH-TRIAGE.md 'ROUND-2 EMPIRICAL LOCK' @206884f):** it's the **UPDATE leg, NOT sync** — `LEG i=3 update ms=30025 err[TimedOut]` (dial 96ms, registry/notif/sync all fast+ok). EXACT: `propagate.rs:373-375` `request_update` ALWAYS reply-reads the peer's Query-response with a 30s budget (no empty-skip); the first fan peer (HFENDULEAM) accepts the conn+update-stream but never answers → 30s → peer_outcome POISON → whole pump dies. `sync.rs:374-376` = LATENT TWIN (looked innocent only because its want-set was empty → skipped the read). This is WHY we held for empirical over the static pin — the discipline caught the WRONG leg. **ROOT LOCKED (empirical, 3 rounds). HOLD LIFTED. W1 fix DISPATCHED to todlando** (build in a worktree): (a) reclassify BOTH request_update + request_sync reply-read timeouts → ordinary per-peer DROP (peer_outcome poison RESERVED for real broker-IPC-carrier desync) + (b) budget-decouple <30s. Files: propagate.rs + sync.rs + pump/mod.rs. **W2 (REQ-PUMP-PEER-ISOLATION) VALIDATED head-on** (fan#1 wedge = the head-of-line block concurrency kills). REQ-PUMP-DIAL-FASTFAIL re-scoped (title refuted, activation comment @traceable tail points at the reply-read fix). deployah field lane CLOSED (masterclass: isolated repro fresh-key/distinct-socket = zero disruption to live 0.28.0, 8 refutations + self-corrected the static sync→update guess). Evidence: enlyzeam C:\spt-diag\pump-diag.log. **RCA COMPLETE — grand total: dial bound (v0.8.3, refuted), discovery-resolve, relay-session, prove_membership, starvation, dispatch-block ALL refuted; the real bug was a worker-leg reply-read poison, found only by real-field leg-instrumentation. Zero fix code on any wrong root.**

**ARBITER PLAN (approved):** deployah field-instrument = PRIMARY (covers BOTH halves — DNS-resolve + relay-wait — + the starvation residual; needs a real stale TXT = fleet-only), POST-musl-tag. todlando partial hermetic synth = POST-CI cross-check (inject a real relay-url onto an absent-peer addr = forces the relay-WAIT half only, SKIPS DNS-resolve). The two TRIANGULATE WHICH half escapes (directs the fix site): todlando-synth escapes ⇒ relay-wait is (part of) it; todlando-synth BITES but field escapes ⇒ escape is in DNS-resolve. SHARED-BOX: musl CI + deployah tag have right-of-way on hfenduleam; both pump probes run AFTER ([[seedmap-test-collides-live-daemon]]). **ROOT-LOCK criterion:** field confirms ONE id-only-resolvable-dead dial >30s AND the half (resolve/relay/both) pinned → THEN activate re-scoped REQ-PUMP-DIAL-FASTFAIL + fix = bound the escaping half (iroh's own connect/discovery timeout or a cancel token around the resolve; NOT `tokio::timeout(connect)` which doesn't cancel the discovery sub-future). REQ HELD until locked. **RCA-first + the arbiter discipline caught 4 wrong roots (todlando in-wrap-escape, deployah not-in-wrap-inference, doyle W2-accumulation) before a line of fix — exactly the 3-way gate's value.**

**W2 DESIGN FORKS (doyle grounded run_peer_pump, mod.rs:428-537, for the pending W2 dispatch ruling — todlando sending a sketch+Qs):** confirmed targets — SEQUENTIAL `for (sub,peer) in fan_targets` (:510) = head-of-line block; `peer_outcome(outcome,…,&mut conns)?` (:531) `?`-bubbles a TimedOut OUT of the whole round → supervised restart = whole-round poison; single `&mut brain` (one reader-thread+channel client, one PUMP_PEER_IO_TIMEOUT per call) shared across every peer_step via PeerIo; 4 cadenced worker legs (registry→notif→sync→update) run per-peer in fan-out order. FORKS to rule: (1) concurrency model — per-peer tokio tasks vs bounded pool vs the operator-favored 'pump issues non-blocking dial REQUESTS, broker returns conn/presence as async BrokerEvents' (the D4c presence-event seam); (2) THE HARD ONE — brain-IPC concurrency: the single serial brain client can't do concurrent per-peer calls; either N brain clients (one per peer, costly) OR multiplex req/resp over one channel correlated by request-id; (3) fault isolation — replace `peer_outcome(…)?` with per-peer drop+reschedule, round never aborts, supervise_pump restart reserved for a dead BROKER conn (the channel itself), not a peer TimedOut; (4) shared state `conns`/`ops`(EpochSource)/worker per-step state safe under concurrency (per-peer conn independent; ops may need lock or per-peer partition); (5) preserve per-leg cadence + per-peer worker ordering; (6) heartbeat naturally keeps advancing (tick not blocked by a peer task). doyle rules these on todlando's sketch. **RULED 2026-07-08 → docs/PUMP-W2-DISPATCH-RULING.md @d806932 (main):** Option A endorsed (D4c-native, no new substrate). G1 EXTEND NetPresenceEvent + typed dial-outcome variants + per-peer correlation key (one cursor/exactly-once). G2 ONE carrier, submit fire-and-forget + drain non-blocking with a broker-LIVENESS deadline only (re-points REQ-HAZARD-PUMP-IPC-DEADLINE to broker-conn health; split only if backpressure entangles). G3 round closes on a DEADLINE (submit-all → drain-until-round-timeout ≈ W1 bound+margin → mark_ran, wedge-proof, preserves V4 stagger) NOT all-resolved. G4 concurrency on BROKER + bounded semaphore cap. G5 per-peer doubling backoff capped, reset on success. Paired new broker verb: non-blocking dial-submit whose outcome rides the G1 event. todlando to mint REQ-PUMP-PEER-ISOLATION + design in earnest; W1 STILL gates first (root-locked), no build while box is deployah's for musl CI. **W2 DESIGN doc up (branch pump-w2-design, docs/PUMP-W2-DESIGN.md @3321229) + doyle DESIGN-CHECK = APPROVED 2026-07-08:** all 5 rulings honored, premise grounded (D4c PresenceLog string-kind forward-compat msg.rs:904, §V4 stagger, ADR-0034 op_id/minter dedup), re-pointed IPC-deadline asserts BOTH directions (dead peer never restarts / dead broker STILL does). Design: new `PRESENCE_DIAL_FAILED` kind + optional `reason` on NetPresenceEvent, correlation by existing `remote_id_hex` (conn_id:0 sentinel on fail, no submit_id needed); new non-blocking `KIND_NET_DIAL_SUBMIT` (payload=NetDialReq); `NetHost::submit_dial` spawns under a semaphore, same bounded connect, outcome→PresenceLog; pump subscribes once, per-peer PeerSched{next_due,backoff}, round=submit-all→drain-until-deadline→mark_ran, DELETES peer_outcome(...)? poison. **doyle Q1-Q3 RULINGS:** Q1 round-deadline = DERIVED CONST = W1_final_bound + ~2s margin (NOT a config knob; pin when W1 bound lands — W1-before-W2 dep). Q2 semaphore = NetHost-SHARED (CAVEAT: the blocking dial() must ALSO acquire the same permit, else cap bypassed). Q3 peer_step mid-round fail = per-peer drop+backoff same as DIAL_FAILED, never round-wide (+ DISCONNECTED stays NO-backoff, redial-eligible immediately; only DIAL_FAILED backs off). Ready to mint+build POST-W1-gate. **BUILD in a .worktrees/ worktree (NOT the shared main checkout — see shared-checkout hazard).**

**⚡ W1 ROOT LOCKED + FIX BUILT 2026-07-08 (todlando @b4862f3, branch pump-w1-fix, worktree .worktrees/pump-w1-fix off main @5555648) — AWAITING doyle GATE.** doyle round-2 lock (deployah leg-instrument, enlyzeam, 3 identical rounds): NOT the dial (DIAL_EXIT 96ms healthy) — the UPDATE leg reply-read. `request_update` (propagate.rs:373-375) blocks `read_event_until(deadline=call_deadline()=30s)` on the peer's Offer/UpToDate reply; a connect-then-silent peer (HFENDULEAM fan#0) burns 30s → TimedOut → `peer_outcome` (pump/mod.rs:601) POISON → whole-round abort + supervise restart (= field PEER_PUMP_FAIL + always-zero PUMP_PEER_FAIL). `request_sync` (sync.rs:374-376) = latent twin (skips reply-read only when want-set empty). **FIX:** (a) `brain::read_peer_reply_until` reclassifies a reply-read no-progress TimedOut OUT of TimedOut (→Other) so peer_outcome drops ONLY that peer; TimedOut poison RESERVED for genuine broker-IPC-CARRIER desync (carrier ops net_open_stream/subscribe/send keep raw TimedOut). Abandoning the peer stream SAFE — exactly-once seq cursor stays contiguous, late reply matches no live stream id. (b) `brain::reply_read_deadline`=now+min(io_timeout,10s) decouples below the 30s carrier deadline. Both legs use them; peer_outcome doc reserves poison for carrier-desync. Files: brain.rs, propagate.rs, sync.rs, pump/mod.rs, tests/pumpdeadline.rs, traceable-reqs.toml. **REQ ACTIVATED [impl,unit,int] all satisfied.** Pre-gate GREEN: new units (deadline-decouple, reclassify, silent-peer→ordinary-drop-vs-carrier-poison) + int (pumpdeadline reply-read over REAL split-stream carrier) + regression (full-round black-holing-peer heartbeat-advances-no-restart, self-drive round, two-tier-sync + torn-pull happy paths ALL pass — budget-decouple leaves healthy streaming pulls alone, they re-arm on progress) + traceable-reqs check exit 0 + workspace clippy clean. Platform-agnostic (no path-semantics) → Linux gate leg optional (doyle's call). Sent doyle (msg2 QUEUED, offline). **NEXT: doyle gates → W2 REQ-PUMP-PEER-ISOLATION build (design APPROVED @3321229 + Q1-Q3 ruled, build-ready, in a .worktrees/ worktree off pump-w2-design; Q1 round-deadline pins to W1's 10s bound now that it's landed).**

**Waves (both queued `required_stages=[]`, activate at work-start):**
- **W1 REQ-PUMP-DIAL-FASTFAIL** — DISPATCHED to todlando **RCA-FIRST** 2026-07-08. Crack WHY the 10s bound doesn't fire (candidates: timeout-drop not cancelling iroh's in-flight relay/holepunch connect; discovery-resolve span outside `bounded_block_on`; runtime time-driver starvation under serial fan-out) → report root to doyle BEFORE building → minimal fix so a resolvable-offline dial fast-fails as ordinary `PUMP_PEER_FAIL`, round continues, heartbeat advances. Gate: test with a real resolvable-offline peer (not the unreachable proxy). KH 7.8 extended.
- **W2 REQ-PUMP-PEER-ISOLATION** — architectural (operator ruling). `run_peer_pump` two defects: SEQUENTIAL fan-out + WHOLE-ROUND poison (`peer_outcome(...)?` aborts the round). Fix: per-peer concurrency + fault isolation — non-blocking dials, broker returns presence as async events (D4c seam), one peer's TimedOut drops+reschedules ONLY it, NEVER aborts round or restarts pump; supervised-restart reserved for dead BROKER conn. Depends on W1.

**3-WAY AGREEMENT GATE (operator ruling 2026-07-08, BINDING):** deployah first RCA'd this on real boxes (foreground daemon-run trace) = field-evidence authority. todlando's W1 root finding is CROSS-CONFIRMED with deployah before doyle blesses any fix — no fix ships until todlando+deployah+doyle all agree the root against field evidence. Gate sits at root-confirmed (pre-build), doesn't slow the RCA. Both notified 2026-07-08. Committed @91460a7.

**Discipline:** RCA-first W1 (like F028-B5 — no blind fix). Resident-BROKER path → fix reaches fleet via a broker-update batch only (rare, like v0.8.3). Parallel to MUSL-TIER release (deployah), no conflict. Instrument: foreground `spt daemon run` + `Start-Process -RedirectStandardError` (NOT `Select-Object -First` — blocks); `supervise_pump` eprintln's PEER_PUMP_FAIL/PANIC/RESTART.