---
name: f023-f024-wan-idle-starvation
description: "F-023/F-024 cross-node delivery RCA'd (post-v0.19.1): WAN-arrived spool never offered to spt-hosted inject + no idle-edge drain; F-024 transport EXONERATED by receiver-confirmed (spooled) token — symptom = F-023 on remote node. Build order BUILD-F023-WANIDLE to todlando (preempts endpoint-list)."
metadata: 
  node_type: memory
  type: project
  originSessionId: 0c1a5493-59c3-4e19-aea3-c3b6de88f7b5
---

Perri findings (spt-claude-code docs/SPT-CORE-FINDINGS.md @8191587, 2026-07-01/02, both nodes spt 0.19.1 adapter 0.10.3). doyle RCA 2026-07-02, code-verified same day.

**F-023 CONFIRMED (both perri hypotheses real):** `receive_wan` (spt-daemon wan.rs:230) = access-check → replay-check → restamp → `deliver_tcp` (harness-hosted relay leg) → else spool. NO spt-hosted broker-inject leg — that leg (KIND_ENDPOINT_INPUT → translation binary, idle-gated at broker.rs ~2070–2160) exists ONLY in local `cmd_send` (REQ-SEND-SPT-HOSTED). AND no idle-edge drain exists: "ACTIVE→deferred (hook-poll)" / "IDLE+no-binary→non-deferred (relay wakes)" both strand on an spt-hosted endpoint (no relay). WAN msgs sleep until an adapter hook polls (perri timeline: idle 7min, binary pid alive, zero injection, UserPromptSubmit drained both).

**F-024 TRANSPORT EXONERATED:** `SENT(WAN):… (spooled)` is the RECEIVER-CONFIRMED reply token (wansend.rs:42 — Sent is the ONLY variant printing SENT(WAN); "(spooled)" = WanOutcome::Spooled from the peer). Perri's DIAG-7c1e probe traversed HFENDULEAM→ENLYZEAM post-restart and sits in ball-b's spool. Operator symptom "never lands" = F-023 starving remote-side. "(spooled)" not "(delivered)" also ⇒ ball-b had no live TCP listener at receive. WORKAROUND relayed: type a prompt into ball-b → UPS hook drains.

**Fix spec (BUILD-F023-WANIDLE, todlando, preempts endpoint-list):** (1) factor cmd_send's spt-hosted inject leg into shared fn; receive_wan calls it post-restamp pre-spool; delivered → wan_mark_seen_at → existing "delivered" wire token (no wire change). (2) idle-edge drain: ACTIVE→IDLE transition offers pending spool (deferred + non-deferred) through same leg; reuse hook-poll take/ack so concurrent api poll can't double-deliver. (3) from_id="" ruling: compose stamps cli@<node-label>; ingress renders empty-from as origin node label; refusal rejected. (4) residue diagnostics: recv-pull/context-sync bundles silent since 05:36Z daemon restart on HFENDULEAM (sync worker due()/wake priming — CONTEXT sync not messaging, separate defect if confirmed) + 32 net-send PENDING-no-DONE (verify every dial/write failure closes journal entry, else document as offline-peer bookkeeping). (5) REQ-WAN-SPT-HOSTED-DELIVERY + REQ-MSG-IDLE-EDGE-DRAIN + KH hazard (delivery must never depend on adapter hook cadence); red-first int = WAN-arrival to idle spt-hosted endpoint injects with NO hook poll.

**F-024 REOPENED (sub-case, 2026-07-02):** operator ran the ball-b workaround (typed prompts) — stuck messages STILL undelivered. Break on ENLYZEAM is AFTER spool insert (receiver-confirmed "(spooled)" keeps coming; doyle fresh marker DIAG-DOYLE-e024 confirmed spooled). Candidate splits: (1) SPT_HOME/owlery mismatch on ENLYZEAM — setsid/service daemon env writes one owlery, hook's spt polls another [[linux-spt-home-no-bug]] adjacent; (2) perch-resolution mismatch on spool path; (3) adapter hook drain broken that node — drains-and-DROPS or from="" envelope parse batch-kill (hall-a drained from="" fine → would be platform/plugin-tree-specific; ENLYZEAM ccs-tree reconcile = suspect). Operator running ENLYZEAM diagnostics: version, LOCAL-send control (fails too → whole drain path / works → WAN rows only), spool-db + .has-messages inspection, owlery-path comparison. Perri flagged (F024-REOPEN-FYI). Discriminator: local-send control result.

**F-024C ROOT FOUND (2026-07-02, perri ssh'd ENLYZEAM + doyle code-confirm): SESSION-PIN AUTH WEDGE.** ball-b /clear'd 06:06:30Z (old session 22ce8af0 → new 42bb5aae; operator's workaround prompts DID land in the new transcript — rc input works). But old session had transient env corruption (`hook: command not found` on every Bash call — v0.9.0 hook_cmd-cache signature despite 0.1.9 on disk) → its /clear-time `api boundary` died → perch stayed pinned to dead sid. api poll/state/boundary ALL gated()→authenticate() (proof sid vs info.json.session_id; auth.rs:88, mod.rs:587) → every hook call from the live session AUTH_REFUSED (stderr-only = silent in hooks) INCLUDING retry-boundary → ready:false, stale .idle, drain no-ops forever. ONE missed rotation = permanent strand, no self-heal. RECOVERY (relayed): full relaunch of ball-b (establish_perch dead-owner rebind re-pins; pid 55276 dead) — NOT /clear (still refused); alt: boundary with perch --token if token file exists. WAVE ADDENDUM (F024C-AUTHWEDGE-ADDENDUM): REQ-HAZARD-SESSION-PIN-WEDGE — authenticate() dead-owner fallback (sid mismatch + recorded pid dead → accept + RE-PIN loud; live-owner mismatch still refuses), precedent = establish_perch dead-owner rebind; + loud-once diagnostic on touch_has_messages create-fail (spool.rs:147 `let _` swallows — ENLYZEAM shows live same-dir stale sentinel = silent persistent create-fail in field); + 3m23s one-way traversal latency folded into pump residue diagnostics. Perri still owns: env-corruption SOURCE (what held stale SPTC_HOOK_BIN despite 0.1.9) + two-dirs enumeration for the record.

**Enumeration verdicts (2026-07-02):** two-dirs RULED OUT — exactly ONE spt-core root on ENLYZEAM (LOCALAPPDATA\spt-core), one owlery, one ball-b perch; daemon orphaned/detached from dead user-shell parent (no service-home split possible). Same-dir anomaly CONFIRMED REAL: spool.db insert 06:59:17Z (DIAG-DOYLE-e024 row 27 delivered=0) beside .has-messages mtime 2026-06-29 in ONE dir → touch_has_messages' swallowed File::create fails persistently on Windows in field (cause TBD: read-only attrib / share-lock — perri checking). Wedge timeline refined: api.token refreshed 04:49:35Z (same-sid CC respawn resumed 22ce8af0 → auth OK); wedge latched at FIRST rotation c6aec397 @05:52Z (boundary died pre-/clear); second rotation 42bb5aae @06:06:30Z also refused. Token file EXISTS → recovery (b) boundary --token viable; relaunch still preferred. Wave locked (WEDGE-LOCKED-todlando): F-023 legs first, then auth-wedge fallback (coexists with token auth untouched).

**Closing facts (2026-07-02, perri):** (1) sentinel anomaly narrowed to REAL swallowed io error in daemon context — NTFS bump works as decid (his CREATE_ALWAYS test bumped it; CAVEAT: that test CONTAMINATED ball-b's sentinel mtime — pre-test value was 2026-06-29T18:01:28Z), attrib=A, no lock; code has NO touch-skipping branch (touch unconditional post-commit) → daemon-process-specific create-fail (CFA/AV/path-form?), todlando's loud-once diagnostic names the io::Error kind on next WAN arrival. Same-version HFENDULEAM touches fine — divergence unexplained until diagnostic prints. (2) FIRST DOMINO: ENLYZEAM plugin cache holds BOTH sptc 0.1.8 AND 0.1.9; session 22ce8af0's OWN SessionStart (Jun-29T17:15:33Z, 81s post-boot) ran the 0.1.8 dispatch (unquoted printf, no suffix-strip) DESPITE 0.1.9 installedAt 16:50:22Z (24min prior) — CC plugin-resolution lag/enabled-pin, cached the bad SPTC_HOOK_BIN into per-session CLAUDE_ENV_FILE which survives every resume. Residual puzzle (stale-plugin-resolution) = perri court + hazard entry. 0.1.9/0.10.2 fixed the corruption itself; the AUTH WEDGE (no self-heal) is what turned a fixed bug's leftover into a dead node. Full addendum in spt-claude-code docs/SPT-CORE-FINDINGS.md. Operator recovery (a) relayed; awaiting relaunch result.

**F-024D PARTIAL RETRACTION + RULING (2026-07-02):** clean-room repro post-relaunch — /clear wedges in a CLEAN env (no corruption needed; negative control: relaunch-without-/clear drained all 27 rows delivered=1). TWO adapter defects (perri court, building): (a) boundary hook resolves id via whoami-keyed-by-NEW-sid → self:null (new sid unregistered = catch-22) → silent skip on `if !id.is_empty()` (SessionStart brief prefers $SPT_ENDPOINT_ID and works — session looks healthy while perch strands; broken since verb landed, v0.10.1 validated brief not rotation+delivery); (b) boundary call carries NO --session-id/--token (poll/state pass current sid fine; boundary's CC payload sid is the NEW one, old sid gone with per-session env). DOYLE RULING (F024D-BOUNDARY-RULING): Track-1 adapter now — adapter-owned state file {adapter_dir}/state/session/<endpoint_id>.sid written every SessionStart, read at clear/compact as PRIOR sid → `boundary <src> $SPT_ENDPOINT_ID --to-session-id <new> --session-id <prior>`; id via $SPT_ENDPOINT_ID first; NEVER silent-skip (loud stderr on empty-id + on refusal); NOT CLAUDE_ENV_FILE (per-session env persistence = the 0.1.8 domino mechanism, and old env is gone by design); E2E = /clear → info.json rotated + post-/clear message DELIVERS. Track-2 core (todlando doc-scope, F024D-DOCSCOPE): harness-contract boundary section documents departed-session-proof requirement + adapter persistence pattern; KH cross-ref; dead-owner re-pin explicitly does NOT cover live-pid rotation (correct squat protection — do not widen). PARKED design: pid-ancestry proof for boundary (core walks caller's real process ancestry vs info.json.pid = OS-truth self-proving rotation, eliminates adapter sid-state; needs ADR + Windows PROC_THREAD_ATTRIBUTE_PARENT_PROCESS spoof caveats).

**F-024D FORMAL TRIAGE + DOCS FIX LANDED (2026-07-02, operator directive: classify w/ evidence + durable docs).** Verdict: perri EXONERATED (implemented published api.md:99 signature verbatim — no auth flag shown); DOCS GAP proven (Rule 2 never says the matching sid = the DEPARTED one; boundary section auth-less); CORE GAP proven vs CONTEXT.md:186 (design anchors rotation on identity+parent_pid; impl gates on the departed sid via uniform gated() — undesigned, incoherent w/ deliberately-ungated purge, legacy-parity regression). Branch **boundary-rotation-credential @8110062**: ADR-0032 (3 layers: documented adapter prior-sid pattern NOW / dead-owner re-pin = F-023 wave / parent_pid-anchor ancestry proof = design-true follow-up, impl+unit stages reserved), api.md boundary rewrite (authed sig + catch-22 callout + v0.10.4 reference pattern + loud-refusal rule), integration-checklist row (proof/persistence + rotation-AND-delivery E2E), CONTEXT.md:186 rotation-credential meaning + _Avoid_, REQ-BOUNDARY-ROTATION-CREDENTIAL doc-stage ACTIVE (333/333). Handed to todlando to MERGE into wave (his KH cross-refs it; token ADR32-DOCS-8110062). Perri shipped **adapter v0.10.4** (Track-1 exact: $SPT_ENDPOINT_ID-first, .sid state file, loud-never-silent w/ AUTH_REFUSED reason capture); NEW F-015-sibling caveat ledgered: live `spt adapter update` prints DONE while install-dir swap pending endpoint quiesce (honest version-of-truth saves it). REWORDED 2026-07-02 (perri verdict, NOT closed): "verify-post-fix (F015B): mid-live adapter update on a PROFILE-composite endpoint swaps for real and reports honestly — perri runs on-node at wave-publish." She verifies against published binary: (a) install-dir version ACTUALLY swaps (version-of-truth not the DONE line), (b) KIND_APPLIED honest, (c) endpoint stays online; green → CLOSE w/ evidence. Cross-ref: perri's docs/SPT-CORE-FINDINGS.md (SaberMage/claude-spt) carries the fix pointer (8b9975d + KH 7.24 + ADR-0025 amend 2 + e2e 16eef91/10a316d). Release-ping at wave-publish MUST include this verify ask. **RETIRED 2026-07-02:** attempt #1 INVALID (stale broker image → F-025); post-bounce re-verify GREEN (@4fbca79: real swap + honest DONE + endpoint survived) → item CLOSED. MY E2E (promotes her REQ-DIST-BOUNDARY-ROTATE to int): pending operator green-light to stop/relaunch hall-a (LIVE again post-purge; holds exe lock) around adapter update; then throwaway endpoint /clear E2E: info.json rotated + post-/clear delivery + .sid rollforward. OPERATOR TRIAGE FRAME (standing): perri issue = misread docs / docs gap / core missing parity feature — classify w/ evidence, check EVERY decision vs CONTEXT.md, ALWAYS land the docs so future adapter devs don't re-meet it.

**IMPL ARCHITECTURE (todlando, 2026-07-02, wave branch f023-wan-idle-delivery @1ce9240 = 6 reqs + doyle's ADR-0032 merged @140cbf7, current w/ main).** Inject leg map: `is_spt_hosted_no_relay(target,owlery)` (cli.rs:4163, spt crate — DETECTION: controllable + spt-hosted + resolve_address==None; unit tests cli.rs:10490-10508) + `Brain::inject_endpoint(id,bytes,native)` (brain.rs:1298, spt-daemon — IPC CLIENT sending KIND_ENDPOINT_INPUT to broker) + `broker.dispatch_endpoint_input` (broker.rs:2069, SERVER — finds session, translation-binary idle-inject, v0.14.3 no-binary-spools law LIVES HERE). cmd_send (cli.rs:4204, gate at 4251/4286) + delivery.rs:258 are callers. KEY: `receive_wan` (spt-daemon wan.rs:230) CANNOT lift cli.rs code — the spt binary crate is ABOVE spt-daemon. So factor the leg INTO spt-daemon (Brain+receive_wan's crate) as a shared fn `try_spt_hosted_inject(target,body,owlery)->bool`, call from cmd_send (spt, down-dep) + receive_wan. is_spt_hosted_no_relay's helpers (resolve_address/controllable/spt-hosted read) likely in spt-store/spt-msg (shared) — verify they're reachable from spt-daemon before moving detection. INSERTION POINT exact: receive_wan between wan.rs:274 (deliver_tcp→DeliveredTcp) and :276 (spool fallback) — after replay-check(:255)+restamp(:267), before spool. delivered→wan_mark_seen_at→new DeliveredHosted outcome (reuse "delivered" wire token, WanOutcome enum ~:117-133). RED-FIRST int mirrors: broker.rs:311 endpoint_keyed_inject_without_binary_spools_not_pty (v0.14.3) + inject_control_wedge.rs (real broker PTY + .idle sentinel dispatch_endpoint_input reads). NEXT: write wan_arrival_to_idle_spt_hosted_injects_with_no_hook_poll int (reds) → factor shared fn → wire receive_wan. Tasks #5-8; endpoint-list W1 parked @025b305.

**E2E-0104 GREEN (2026-07-02, HFENDULEAM, adapter 0.10.4 + spt 0.19.1) — rotation wedge DEAD.** All 3 assertions: (1) /clear on live hall-a → info.json sid rotated (a19521de→00539523), boundary authenticated w/ prior sid; (2) state/session/hall-a.sid rolled forward in lockstep (and matched at SessionStart pre-/clear); (3) post-rotation `spt send hall-a` → SENT (true-injection). Sent to perri → REQ-DIST-BOUNDARY-ROTATE promotes to int. FIELD-REPRO'd perri's PENDING-QUIESCE caveat a 2nd time: operator's `spt adapter update` printed DONE but install dir stayed 0.10.3 (live translate held exe lock); version-of-truth caught it; working sequence = `spt endpoint stop hall-a` → update → verify via `spt adapter list --json` → `spt endpoint run --adapter claude-spt:ccs --id hall-a --start`. Backlog item (update-arc messaging: DONE should read PENDING-QUIESCE when swap deferred) carries two field repros — REWORDED to verify-post-fix (F015B), perri verifies on-node at wave-publish (see F-024D section). /clear ban LIFTED on 0.10.4 nodes; ENLYZEAM/ball-b needs same update+relaunch sequence before its /clear is safe.

**BOTH LEGS SHIPPED + F015B strengthened — WAVE COMPLETE pending doyle final gate (2026-07-02, todlando).** Branch f023-wan-idle-delivery: leg 1 (REQ-WAN-SPT-HOSTED-DELIVERY) @6811ed0, F015B fix @8b9975d, leg 2 (REQ-MSG-IDLE-EDGE-DRAIN) @d70ae73. **Shared inject leg lives in spt-daemon/src/inject.rs** `try_spt_hosted_inject(target,from,body,owlery,native)->Option<(bool,bool)>` + `is_spt_hosted_no_relay` — called by receive_wan (leg 1) AND delivery.rs drain_idle_window (leg 2) AND cmd_send. **Leg 2 take/ack = spool.rs `claim_idle_edge_at`(atomic BEGIN IMMEDIATE: SELECT undelivered + mark delivered in ONE txn → concurrent api poll can't re-take) + `release_at`(ack-failure: return uncarried row intact — id/window/deferred/ephemeral/op_id preserved).** drain_idle_window: is_spt_hosted guard → resting gate (include_deferred=!deferred_held) → claim → inject each → release first-miss + not-yet-attempted → evaporate ephemeral. v0.14.3 spool-loud honored (release, never PTY). RED-FIRST both waves proven by revert@HEAD-1: e2e(a) apply_ok=true/swap_landed=false (composite matcher skew); leg-2 int pending 1→1 (old peek-non-deferred skipped deferred). F015B e2e STRENGTHENED @10a316d: (a) composite cc:prof parent-match swap-lands, (c) foreign-adapter non-over-match. KH 7.23 (delivery-starvation, prose+table) + 7.24 (adapter-apply). traceable: IDLE-EDGE [impl,unit,int], DELIVERY-STARVATION [doc,int], ADAPTER-APPLY-SILENT-NOOP [OK]. Local: clippy --workspace 0, traceable exit 0, all units + 2 new ints green. **GOTCHA: crates/spt has NO lib → leg-2 int drives the real `spt` binary (`api state idle <id> --session-id <sid>`, auth via harness-bound sid read from info.json); compiling against the binary is what makes it clean red-first.** doyle gates on detached worktree @d70ae73 while tree stays dirty-safe. Wave-publish release-ping must carry perri's F015B verify-post-fix ask [[owl-reply-to-broken]].

**Item-4 residue verdicts (2026-07-02, todlando code-read + doyle concur):** (a) 32 PENDING-no-DONE = BY-DESIGN — broker.rs:2524/:2560 apply_once(EffectKind::NetSend) fsyncs PENDING at attempt / DONE on Ok / retains PENDING on Err for re-drive (effect.rs:256-260); Err surfaced via KIND_ERROR (broker.rs:1680); append-only file + recover()-dedup-by-key ⇒ RAW LINE-COUNT OVER-READS live pending. (b) sync-worker priming EXONERATED — due() prime-all on restart (WorkerLasts RAM-only → last=None → due first tick); RESIDUAL PARKED: recv-pull-silent-since-05:36Z field observation UNEXPLAINED (peer-availability or pump-liveness), reopenable w/ live peer-set/pump-liveness evidence — do not treat as resolved. (c) 3m23s = expected envelope (sync_pull 60s / registry 30s / notif 15s / backoff 5s→300s; messages event-driven post-F-023 legs, anti-entropy minutes-scale by design). No REQ mints; docs-only deliverable (effect.rs module doc + operator log-reading note + envelope note).

**Diagnostic lesson:** the reply-token vocabulary IS a transport diagnostic — receiver-confirmed "(spooled)" collapses "message never lands" from a transport hunt into a delivery-scheduling bug on the receiving node. Check the send token FIRST on any cross-node "not landing" report.

Related: [[counter39-field-bugs-1-3]] (predecessor milestone), [[v0143-raw-inject-removal]] (translation-binary-only law this fix must honor — no raw PTY fallback), [[v0150-w1-activity-gated-delivery]] (the activity gate the drain rides), [[spt-core-findings-backlog]].