--- name: emphasys-endpoint-loss-incident description: "2026-07-27 emphasys endpoint vanished (perch+spool lost, mind survived); ROOT CAUSE CLOSED: spt ring ephemeral-perch ADOPTION bug deleted the caller's real perch on timeout" metadata: node_type: memory type: project originSessionId: a7551ebf-857b-4903-b969-30c4aff3692f modified: 2026-07-28T05:34:32.831Z --- **2026-07-27 ~19:4x: emphasys (this node, HFENDULEAM) went completely missing from the endpoint registry.** Operator-reported; urgent recovery. Evidence timeline (daemon.stderr.log, epoch-anchored): - 18:09 daemon bounce → `DAEMON_RESTART_RESUME:emphasys` session 2fc845f5 under **claude-spt:ccs** pid 30516 — resumed from broker session table. - Immediately after: `DROP_INGEST_SKIP:emphasys: relative drop dir ".claude" has no endpoint cwd to resolve against` ⇒ **bind/perch record ALREADY damaged at resume**. Also `PSYCHE_SESSION_NOT_FOUND` exit 95 (psyche reseed). - 19:43:28 harness conn died (BrokenPipe, conn poisoned organic-write class); pid 30516 gone. - `owlery/emphasys` (perch records + spool.db history) ABSENT; **no PURGE/TEARDOWN/CONVERGE lines for emphasys anywhere** — deletion left no daemon-side log. `tracked/agents/emphasys` (live-role.md + live-context.md, written 19:09–19:11) SURVIVED. Recovery (done 19:49): `spt endpoint run --adapter claude-spt:ccs --id emphasys --create --start --subnet SPT_DEV` from `~/Documents/projects/omp-spt` → ONLINE, perch recreated, mind seeded session. REQ-CREATE-BIND-REST-ACTIVE (v0.45.0) made create-bind land active. Recovery note SENT to emphasys: spool lost, resend-don't-reconstruct ([[empty-inbound-is-never-material]]), open threads named. **Why:** perch loss without a daemon log line = external deletion or an unlogged destructive path — either is a defect class. **How to apply:** ROOT CAUSE OPEN — seed for [[spt-core-findings-backlog]]: (1) what deletes owlery/ without logging (purge logs? test with real SPT_HOME? manual rm?); (2) bind record was already cwd-less at the 18:09 resume ⇒ damage predates the visible loss — check whether the 18:09 bounce window or hertz's omp-spt bounce (19:1x, session 26409caa) touched it; (3) registry projection silently drops a mind-holding agent when its perch dies — absence had no alarm ([[absence-needs-sibling-probe]]). emphasys was NOT a startup_endpoint (daemon.json) — no auto-heal existed. Recovery recipe (reusable): mind at `tracked/agents/` is the identity that matters; perch is recreatable via `endpoint run --create` with correct adapter[:profile], subnet, and cwd. Find prior adapter/pid in stderr log `DAEMON_RESTART_RESUME:` lines. **Forensics round 2 (2026-07-27 ~20:3x):** - **todlando's dummy-home rigs EXONERATED on artifacts**: g6-curve.ps1 (only rig referencing real owlery) never ran (no g6-dummy-home) and only COPIES probe dirs (pattern excludes emphasys); wedge rig hermetic (per-test TestHome tempdir, todlando's own probe proved it); hosting-gate.ps1 ran 19:34+ in-window but TestHome-isolated (impl verified sound), no purge calls. - **All CLI erasers cleared by transcript sweep**: no `endpoint purge emphasys`, no `session-end --erase`, no `fork --delete-source` in ANY .claude/.ccs transcript or PSReadLine/bash history. Adapter SessionEnd hook = SOFT end (spool-preserving, source-verified). purge_endpoint_core is terminal-silent + effects-silent BY DESIGN (ADR-0043 d3) ⇒ an eraser leaves no daemon trace — itself a defect-class observation for the observability seed. - **USN journal WRAPPED** past the 19:1x-19:44 window (operator ran elevated readjournal 20:25; only hit = the output file's own creation) — build churn on this box eats the journal in <40min. Filesystem attribution DEAD; agent self-reports = last channel (todlando asked 20:3x). - **Registry-row disappearance is mechanical, not a second event**: perch death + #107 relay-death convergence + 5-min registry eviction fully explain "completely missing" — the ONLY abnormal event is the perch dir deletion. - Same-endpoint context: emphasys's row was tonight's convergence-gap specimen (inherited controllable=true exemption, todlando's pid-role build fixes it) — a "cleanup" of that stuck row is the most plausible motive; no recorded actor. - emphasys ACKed recovery, confirmed nothing on their side; their 19:11→19:43 sends/receives unrecoverable (spool). Three open threads re-stated to them from doyle's books 20:3x. **ROOT CAUSE CLOSED (2026-07-27 ~20:5x, doyle):** `spt ring` ephemeral-perch ADOPTION defect (spt-msg/src/ring.rs). - emphasys ran `spt ring hertz --timeout 60` at 19:14:33 (a liveness check ON HERTZ, from their own live session). `from_has_live_perch` (ring.rs:54) tests the READY MARKER FIRST; emphasys's marker was down (ghost state since the 18:14 re-bind) ⇒ ring decided the caller was perchless ⇒ `setup_ephemeral` ADOPTED the existing real perch (clobbered info.json with a ready_agent/"ring" record) ⇒ TIMEOUT 19:15:39 ⇒ `cleanup_ephemeral` (ring.rs:139-146) deleted ready + info.json + **spool.db** + `remove_dir_all(perch)`. Silent CLI-process fs ops = the zero-log evidence pattern. Ghost-heal pump tick noticed 19:15:52 (`ROSTER_GHOST_HEAL`); prior-doyle probe read "info.json GONE" 19:22:06. - ⭐ REQ-HAZARD-EPHEMERAL-CLEANUP guarded the LEAK direction only; the ADOPTION direction (claiming a pre-existing perch as ephemeral) was unguarded. Ready-marker absence ≠ perchless — [[absence-needs-sibling-probe]] made executable. - Exonerated along the way: todlando (evidence-backed self-report + rigs verified), hertz (new session inert), ALL CI legs (#114 test job was compiling during the window; n1_pairing fully isolated), operator console, purge/session-end/--erase (no callers anywhere). - ⚠ FLEET WARNING ISSUED (sent todlando/emphasys/hertz/perri/deployah 20:5x): NO `spt ring` from live-agent sessions until the fix ships — use `spt send`. - FIX DISPATCHED to todlando (next after #116), registry mint rides the build PR. Spec seed: setup_ephemeral create_new-refuses on pre-existing dir; loud RING_PERCH_EXISTS degrade to deliver-only; liveness probe consults perch dir+record not marker alone; unit = live-shaped perch survives timed-out ring byte-identical. **FIX MERGED — PR #117 @9d7af01 (2026-07-27 ~22:3x), same-day as discovery (find ~20:4x → root cause ~20:5x → merged ~22:3x).** ⚠ **THE FLEET WARNING STILL STANDS: merged ≠ shipped.** Every agent runs the INSTALLED spt (v0.45.0), which still has the adopting `ring`; no `spt ring` from live sessions until a release carrying #117 is published and installed ([[no-version-floor-before-it-ships]] — do not name the version until it ships). Gate record: 5 units by name PASS both OSes; the one red (inject_control_wedge g2) closed as load-family on at-sha green, ledgered as the 1st g2 sighting. **Build detail (PR #117), branch `fix/ring-perch-adoption` @52b4d70+@1b43c1a.** Mints REQ-HAZARD-RING-PERCH-ADOPTION + KNOWN-HAZARDS 3.4. Shape (doyle-ruled, my counter-shape accepted then amended by him on emphasys's objection): probe the perch DIR not the marker — record-or-spool = OCCUPIED, unreadable record / unlistable dir = OCCUPIED (ambiguity always resolves toward refusal); **empty dir is REFUSED too, self-heal REVOKED** (`endpoint run` owns an empty perch dir mid-create, so deleting it is the same race in a different hat); refusal delivers + declines to block-wait behind distinct loud reports `RING_PERCH_EXISTS` / `RING_STALE_DIR` (loudness, not repair, is what keeps a permanent refusal from being silent); leaf dir now `create_dir` so a probe/create race also refuses; only a partial setup this call made reaches cleanup. Gated green pre-push: clippy 0, 37/37 units, traceable-reqs 0. - ⭐⭐ **NEW CLASS from the red-first proof — adoption also DRAINS, and that half is a CONFIDENTIALITY defect.** With the guard disabled the survival unit returns `Replied{from:"someone", body:"mail that must survive"}`: while ring holds an adopted perch, its reply-wait `drain_one_at` consumes the victim's spool and renders it to the RINGER as the reply. **Did NOT fire in this incident** — emphasys field-bounded it: his 66s window (02:14:33.027Z→02:15:39.915Z) returned `TIMEOUT:hertz…` with zero `Replied{}`, so nothing was drained; his loss was the dir deletion alone. "Available, did not fire" is the honest phrasing — do NOT carry theft into this write-up as fact (my first draft did; emphasys corrected it). - ⭐ **The severe shape is ringer ≠ victim** (emphasys's escalation, doyle ruled it into #117's gate): his case was DEGENERATE because he rang from his own session, so a drain would have returned mail to its intended recipient — loss at worst. A THIRD party ringing a live agent drains the victim's inbox INTO THE RINGER'S OUTPUT, untraceable once the dir deletes: cross-agent mail disclosure across a multi-machine subnet, not durability. 5th unit added on that shape (victim holds a third id's mail, second id rings, mail stays put and never surfaces). - ⭐ **Self-camouflaging** (emphasys, via doyle): the damage mimics the stale-liveness condition people reach for `ring` to diagnose — he was probing a ghost-roster symptom and the probe manufactured a fresh one. Debuggers of stale liveness are disproportionately its victims and misread the wreckage as more of the symptom. - ⭐ Red-first is what surfaced it: the guard-disabled run produced a WRONG-VALUE failure (`Replied{...}`), not just a missing-refusal failure — the disproof carried information the incident write-up did not have. - Forensic method worth keeping: 30s window + "enumerate EVERY recorded command across all harness transcript stores (.claude + .ccs + .omp) inside it" found the actor after logs/USN failed. .omp sessions are NOT covered by .claude/.ccs sweeps.