---
name: isolated-resident-broker-field-repro
description: "Template for field-RCA'ing a resident-broker daemon on a remote box WITHOUT stopping its live daemon: fresh-key copy of its home + distinct broker socket + env-gated diag"
metadata: 
  node_type: memory
  type: reference
  originSessionId: b317273f-decf-492f-b6d7-a5558a360cdf
---

**The template (proven on the PUMP-TRUTH W1 arbiter, enlyzeam 2026-07-08 — doyle: "a keeper, the template for any future resident-broker field RCA").** How to reproduce + instrument a daemon-hosted bug on a box whose LIVE daemon you must NOT disturb (and where you can't stop it / it may auto-respawn / it runs elevated-detached).

**Two blockers this dodges at once:** (1) live-daemon disruption — stopping the resident daemon to run an instrumented one collides on identity + broker socket; (2) de-elevation capture loss — an elevated resident daemon respawns detached under a different user/SPT_HOME with null stdio, so a naive instrument writes nothing readable.

**Why it works — three isolation facts:**
1. **Broker IPC name = `spt-daemon-broker-{home_tag()}.sock`, `home_tag()=hash(spt_home())`** (`crates/spt-daemon/src/endpoint.rs:23`). A DIFFERENT `SPT_HOME` → different socket → the instrumented daemon's broker/digest/drive/tunnel/seed listeners never collide with the live one.
2. **Drop `node.key`** from the copied home → the instrumented daemon mints a FRESH iroh identity → no relay/endpoint collision with the live daemon's key (they can coexist on the network).
3. **Load-dependent / connect-then-X bugs are identity-AGNOSTIC** — the copied roster/`peer-addrs.json` make the fresh daemon dial the SAME peers; a dead/silent peer behaves identically regardless of who dials. (The bug reproduced against the real 3-subnet fan-out with a throwaway key.)

**Recipe (Windows resident box, default home `%LOCALAPPDATA%\spt-core`; adapt paths for Unix `$HOME/.spt-core`):**
- Reach: `ssh decid@<box>` (operator grants the hop). Probe: `Get-Process spt` (2 procs = broker+brain), find `identity\subnet.json` under the home.
- `setup.ps1`: `Copy-Item <home>\identity\* -> C:\spt-diag\home\identity\ -Recurse`; then `Remove-Item node.key, notifs.db, pump-ops.json` (fresh key + fresh op counters). Dump subnet.json/roster.json/peer-addrs.json to confirm the repro state.
- `run.ps1`: `$env:SPT_HOME='C:\spt-diag\home'`; `$env:SPT_PUMP_DIAG='C:\spt-diag\pump-diag.log'` (the env-gated diag sink); `Start-Process C:\spt-diag\spt.exe daemon run -RedirectStandardError ... -PassThru`; `Start-Sleep 120`; **scoped** stop `Get-CimInstance Win32_Process | Where ExecutablePath -like 'C:\spt-diag\*' | Stop-Process` (NEVER machine-wide — [[no-machinewide-killon-shared-runner]]); dump the diag log + `home\logs\daemon.stderr.log`.
- SSH-through-PowerShell triple-quoting mangles inline `-Command "..."` — **write the remote script to a `.ps1`, `scp` it, run `powershell -File`.** Don't fight nested `\"`.

**The instrument pattern (throwaway, ships-safe):** env-gated `pump_diag!` macro — inert unless `SPT_PUMP_DIAG` set; when set, `eprintln` (rides the persistent daemon stderr, LIFECYCLE-TRUTH W3 `logs/daemon.stderr.log`) AND appends the world-writable file the var names (survives detached/cross-user daemon). See `crates/spt-daemon/src/pumpdiag.rs` (uncommitted evidence on hfenduleam tree). Put ENTER/EXIT+elapsed markers around the suspect spans; **instrument ALL candidates, not just the static favorite** — round-2 leg markers corrected a static sync-guess to the update leg.

**Artifacts kept:** `C:\spt-diag\{setup,run}.ps1` on both hfenduleam (authored) + enlyzeam (deployed). Live daemon stayed up the whole time. See [[subnet-peer-pump-stall-rca]] for the RCA this cracked, [[legacy-psyche-migration-2026-07-06]] for who runs where.