---
name: e2e-leaked-daemons-shared-box
description: "nextest fail-fast cancellation leaks e2e-spawned spt daemons; on the shared box they poison SUBSEQUENT bringup-ONLINE e2e runs for EVERYONE — sweep by exe Path under the target dir used, delete throwaway targets promptly"
metadata: 
  node_type: memory
  type: project
  originSessionId: 5a63f400-f291-4ea5-ba2e-6b0f1cb46429
  modified: 2026-09-07T06:27:24.945Z
---

**Failure class (counter-39 gate, 2026-07-01):** bringup-ONLINE e2e tests (attach_wedge, dummy_harness) started failing box-wide, cross-version (previously-green committed shas flipped red mid-session), even rested + `--test-threads=1`. RCA: nextest **fail-fast cancellation leaks the detached spt daemons** in-flight e2e tests had spawned. Four leaked `spt.exe` from `Temp\c39-gate-target\debug\` (doyle's gate runs) + disk at 97% with ~60G of orphaned throwaway CARGO_TARGET_DIRs. todlando's kill sweeps were scoped to `workspace\target` — correct for HIS artifacts, blind to another agent's temp targets. `NET_FAMILY_GATE: binding IPv4-only` in brain stderr = informational red herring.

**Why:** identical committed code flipping pass→fail across a session = box state, not code (todlando's differential logic — correct). Multiple agents share HFENDULEAM; each agent's kill scope must cover every build dir, or one agent's leak becomes the other's phantom regression.

**ESCALATION (v0.25.0 release, 2026-07-04): leaked gate daemons starved a CI RELEASE JOB, not just a gate.** PR #51's Windows `test` leg went RED on `seedmap::tests::stop_op_acks_then_serve_returns` TIMEOUT (240s TERMINATING, not an assertion) — while the FIRST CI run on byte-identical content passed. Root: 8 leaked `spt-w6gate3-target\debug\spt.exe` (brain+run daemons from my own W6 gate subagents that never reaped) had accumulated into hfenduleam's steady state between run-1 and run-2, starving the CI seedmap unit (CI's full `test` filter does NOT exclude seedmap the way my gates do). Swept all 8 scoped-by-path → rerun-failed green. TWO TAKEAWAYS: (1) my gate SUBAGENTS must reap their target-dir daemons before returning — a leaked daemon outlives the gate and poisons CI for the release; verify `Get-Process spt` is clean after each gate agent, don't trust its self-report. (2) SEED: CI self-hosted `test` job should exclude-or-serialize seedmap (`not test(/seedmap/)`) like the gates, else it's starvation-flaky on the shared runner. "identical bytes pass then fail across runs" = box-state/starvation, the [[gate-cross-platform-leg]] sibling on the time axis.

## ⭐ 2026-07-22 (v0.40.0 cut, PR #56) — THE SWEEP-BY-PATH RECIPE BELOW IS BLIND TO CI-RUNNER LEAKS (session 0)

**Two distinct leak populations exist on this box, and the recipe in this file only reaches one of them.**

*Population A — session 1, agent/gate-spawned.* Sweepable exactly as documented. This cut: **12 `spt.exe` running from `.worktrees/gate-d289787/.gate-target/debug/`**, leaked from a gate run 4 hours earlier and still holding handles. Killed 8 by enumerated pid with path re-verified at kill time (4 more exited as children). Box went **48 → 32 procs, 6864 → 4187 handles (−39%)**.

*Population B — **session 0**, spawned by the `actions.runner.*` service.* **Invisible to every technique in this file.** Symptoms that look like corrupted/zombie processes but are not: `ExecutablePath` blank, `CommandLine` unreadable, `GetOwner` unresolved, `Stop-Process -Force` AND `taskkill /F /T` both **Access is denied** — while `Responding=True` and the process is perfectly healthy. **Cause: the runner service runs as `.\decid`, the same account name you are logged in as, but service-spawned children live in session 0 with a service token. Same name, different security context.** A session-1 unelevated shell — agent OR human — cannot read their paths or kill them. Do not diagnose these as wedged/zombie; they are healthy and simply out of reach.

**⭐ ORPHAN-CONHOST SOLVED (was 7 unexplained sightings).** `taskkill /F /T` on the leaked daemons revealed hidden children: `conhost.exe`, session 0, one per leaked `spt.exe`. The recurring "orphan conhost" is nothing exotic — it is the console host of a leaked session-0 CI test daemon that nothing ever reaped. This is also the prime suspect for the `Device or resource busy` pin on `crates/spt-daemon` in stale worktree skeletons (a session-0 process holding a cwd there), which is why those pins **survive killing every session-1 daemon** and typically need a reboot.

**⭐ THE LEAK IS PER-RUN, AND THAT IS THE MECHANISM.** The session-0 leaks were born inside the failing CI run's own execution window; they did not exist for the two runs before it (those contended with population A instead). So **every CI run leaves session-0 daemons behind that contend with the next run, while also leaking its own mid-run.** Observable signature across 3 runs of byte-identical code: a *different single test* dies each run, **with no panic and no assertion output** — just `running 1 test` → `(test failed with exit code 1)`, i.e. a process-level death — while sibling tests in the same family pass alongside, and disk sits healthy. ⚠ A "load-flake family" of N observed-once tests may therefore be ONE environmental cause selecting a random casualty per run; hardening members individually chases the victim and never closes the family. Check the population before filing member N+1.

**⭐ OPERATOR RULING 2026-07-22: moving the CI runner off HFENDULEAM (family-wave leg 3 / option d) is a NON-OPTION for now — stop proposing it. Cause-side mitigation = legs 1+2 (shipped PR #58: in-job reap+census + Windows Phase-A bound).**

> ✅ **CHALLENGED AND UPHELD 2026-08-04 — THE RULING ABOVE STANDS, DO NOT RE-PROPOSE LEG 3.** doyle
> carried a conflicting status ("OPERATOR DECISION … REQUESTED 2026-07-22, still open") and planned
> to re-surface leg 3 at operator contact with fresh evidence. I flagged the conflict rather than
> guessing; he checked and ruled THIS file current and his own line stale, then corrected it by
> replacement and dropped the re-surface. **Mechanism, and it is the reusable part:** his line was
> written in a WAVE-SHAPE/PLAN doc when the request went OUT and was never updated when the answer
> came BACK — a plan document records intentions and silently drifts against the ruling record that
> answers them ([[stale-carried-forward-sentence]]). A leg marked "requested" in a plan is not
> evidence the request is still open. **Standing consequence:** evidence may keep accumulating in
> this file if the operator ever reopens the "for now" hedge — accumulating it is MEASUREMENT,
> re-asking is DIRECTION, and the ruling forbade the second
> ([[report-measurement-never-issue-direction]]).

**⭐ 2026-08-04 — THE FAMILY IS AT STAGE 4, AND THE PER-RUN-RANDOM-VICTIM RULE ABOVE PREDICTED IT.**
Golden 30873007187 on 4b37512 (#145), same sha, two attempts, Windows/hfenduleam only:
attempt 1 killed `spt-daemon daemon::tests::a_tree_teardown_reaches_a_grandchild_the_service_spawned`
(daemon.rs:2396, "grandchild outlived the tree teardown"); attempt 2 killed a DIFFERENT test,
`spt-daemon::registry_lifecycle multichunk_feed_applies_with_exactly_one_snapshot_write`
(registry_lifecycle.rs:610, `left: 0, right: 1` — zero snapshot writes observed, an OBSERVATION
COUNT rather than a wrong value). Linux/kitsubito ran BOTH tests and passed BOTH at that sha
(verified by name in the log, not inferred from the job-level green; the grandchild test is a plain
`#[test]` with no cfg gate, so its Linux green is a real arm). twohost-a/-b were red on attempt 1
and GREEN on attempt 2. **Two attempts, two different victims, one sha = the signature in the
2026-07-22 entry above on a smaller n.** doyle ruled the family CONFIRMED and re-rooted here, and
placed us at stage 4 (cause-side legs 1+2 landed and the family still fires). The multichunk red is
the THIRD sighting of a specific 2026-07-22 seed (snapshot-write-count race) — its
loop-to-characterize trigger fired; hertz holds it, as it holds the grandchild investigation.
**Transferable, and it is the diagnostic that did the work:** a code regression is DETERMINISTIC —
it fails the SAME test on the same sha. A different victim per attempt on ONE sha is an environment
signature, and that single question separates "my lane broke it" from this family faster than any
amount of log reading. Kin [[cascade-red-from-a-skipped-driver-is-zero-information]] (the other
non-evidence red from that same run), [[is-this-red-mine-delta-test]],
[[daemon-lib-tests-deadlock-on-live-host]].

> ### ⭐ SAME RUN, ATTEMPTS 3 AND 4 — THE SIGNATURE REACHED n=4 ON ONE SHA IN ONE NIGHT
>
> Same golden 30873007187, same `4b37512`, Windows/hfenduleam only, Linux green throughout:
> **attempt 3** killed a THIRD distinct victim (the `brain_split` respawn window); **attempt 4**
> killed a FOURTH — `spt::resume_no_control_steal_e2e`
> `brain_respawn_keeps_every_session_controller_and_still_promotes`
> (resume_no_control_steal_e2e.rs:488, Phase A, FAIL 5.619s). **Four attempts, four different
> victims, ONE sha** — the per-run-random-victim rule above at the strongest n we have ever
> measured, and the twohost legs went green on the same attempt (margin +59.59s), so the box was
> not merely "slow". That same row PASSED a1 6.957s / a2 11.379s / a3 5.914s at this identical
> tree: intermittent in BOTH directions, which is the family's shape and not a regression's.
>
> **⚠ THE GATER'S OWN RIG IS A POPULATION-A SOURCE.** Pre-attempt-4, deployah's census caught
> **four leaked daemons from doyle's own gate rig**, and he reaped them. The agent gating this
> family is one of the rigs feeding it — consistent with the standing note that his last four reds
> were his own rig's. Nobody is outside population A; run the sweep on YOUR rig before attributing
> a red to someone else's.
>
> **⭐ THE a4 PANIC NAMES A MECHANISM ITS OWN DATA CONTRADICTS.** The assertion text is
> PRE-AUTHORED — it says "Failure A" (a control-steal at resume) whichever way the gate trips, so
> the message is not evidence of mechanism. deployah's log forensics broke the steal story on
> three measurements: (1) one session frozen while TWO siblings kept their controllers through the
> SAME `resume_sessions` call, which a displacing steal cannot do; (2) session 0 was spawned FIRST
> (`CONN_LIFECYCLE` conn=1 attach mono_ms=0, conn=2 at 37, conn=3 at 69) yet held 4 ticks against
> 17/16 at the post-resume snapshot — **already 4x behind BEFORE the measured window opened**, and
> a steal AT resume cannot produce a PRE-window deficit; (3) the rig's claimed immunity to a dead
> producer is a COMMENT (resume_no_control_steal_e2e.rs:246-247, "the child keeps ticking either
> way") with nothing measuring child liveness at t1 — a starved or dead ticker child yields
> byte-identical evidence. doyle WITHDREW his "candidate product defect" framing on point 2 and
> reclassified victim 4 to candidate family member; the standing order is that nobody writes
> "control-steal recurrence" anywhere until a discriminator speaks. See
> [[pre-authored-panic-names-an-unmeasured-mechanism]].
>
> **Discriminator dispatched (2026-08-04):** rig-side child-pid liveness + producer-side counter at
> t1 (hertz), product-side per-session attach-intent logging out of `resume_sessions` (todlando —
> `RESUME_ATTACH_INTENT`, brain.rs, epoch-stamped per call, joins the broker's `SUBSCRIBE_DECISION`
> on `session`). Splits real-steal from starved-producer before any seam reading.

> **⭐ FREE INSTRUMENT — nextest's `leaky` COUNT IS A PER-RUN PROCESS-LEAKAGE MEASUREMENT, AND IT IS
> ALREADY IN EVERY LOG WE HOLD.** The summary line reads `2676 tests run: 2675 passed (13 leaky), 1
> failed, 1 skipped` — **leaky = tests that returned while leaving processes still running**, i.e. a
> direct count of this file's entire subject, emitted by the harness on every run at zero cost, green
> runs included. Golden 30873007187: **a1=12, a3=10, a4=13** (doyle's read; no a2 value in hand).
> Note what this does NOT yet show: the counts do not visibly track which test died, so this is a
> COVARIATE to carry into the family analysis, not a finding — do not promote it to cause. Its value
> is that it is retrospective (every archived log has one), continuous (a number, not a
> present/absent), and independent of whoever swept the box. Kin
> [[measure-what-costs-one-command]].
>
> Also from the a4 log: exactly ONE summary line, which is how you tell **Phase B never ran** — a
> Phase-A death takes the rest of the battery with it, so "the other phase passed" is never inferable
> from a log with a single summary.

**How to clear population B:** you cannot, unelevated — `Restart-Service` on the runner is denied from session 1 too. It needs an **elevated** shell (operator): restart `actions.runner.*` **while the runner is idle** (`Runner.Listener.exe` present, NO `Runner.Worker.exe` = no job executing) to reap the whole session-0 tree; or kill by pid elevated; or reboot (also clears the pinned worktree skeletons). **Fix direction for the rig: a post-job reap leg** — the runner must kill its own spawned test daemons by target-path at battery end, before worktree removal. Leaks that outlive the job are the actual defect; sweeping afterwards is the shim.

**⭐ 2026-09-07 (W2 attachment rig, todlando's reds 5+6) — A FIXED RIG PORT IS ANSWERED BY THE PREVIOUS
RUN'S LEAKED DAEMON, WHICH HAS A DIFFERENT SPT_HOME.** Signature: fetch 404 while the rig daemon's own
`serve list --json` HOLDS the entry, correct in every field — registry RIGHT, surface WRONG. That is not a
registration bug, it is ANOTHER PROCESS on the port: netstat named pid 54684, the daemon his previous run
autostarted (DAEMON_AUTOSTART) with no teardown, holding 5493 with its own registry. The DEFAULT port hides
the same thing worse: 5474 hands you the FLEET daemon (pid 14444, up since Sep 4). Under the ephemeral flag
a leak cannot answer the next run, so hertz's flag sweep is safe for rigs that never mint a URL; a rig that
FETCHES a minted URL needs BOTH halves: port chosen PER RUN (bind :0, read the number, feed the daemon AND
every CLI call — the flag must be OFF or it eats the fed port, IR-78) and a Drop-guard reaper armed before
the first CLI call that stops the daemon on every exit path. Gate read: `DOCS_SERVER_UP` port == URL port
in the raw, reaper breadcrumb present, no spt.exe from the rig's SPT_HOME after the leg. Sibling of
[[ephemeral-ports-flag-silently-beats-spt-docs-port]]; product fix = releases#282 (URL from the bound port).

**How to apply:**
- After ANY failed/cancelled e2e run: sweep `Get-Process spt | Where Path -like '<the CARGO_TARGET_DIR you used>*'` and kill by pid. NEVER machine-wide; NEVER owl.exe or `spt-core\bin\spt.exe` (live infra, [[no-machinewide-killon-shared-runner]]).
- ⭐ **KILL DAEMONS BEFORE BRAINS — a live `daemon run` respawns its `daemon brain` child.** Leaks come in PAIRS (`daemon run --detached` + `daemon brain --generation 0 --start-reason cold`, brain parented by the daemon). Kill the brains first and the daemons re-mint them; kill the daemons first and the brains die WITH their parent. Measured 2026-08-19 (er-fix rig target reap, disk-floor dispatch): 3 daemon+brain pairs, 3 kills covered all 6, zero survivors. Enumerate pairs before killing and order the list parents-first.
- ⭐ **PROVE THE TWO POPULATIONS DISJOINT POSITIVELY, don't just avoid the install path.** "I killed by path, not by name" is a negative guard; it does not tell you whether a live endpoint is bound to a pid you are about to kill. The positive check is one read: `%LOCALAPPDATA%\spt-core\owlery\<id>\info.json` carries `pid` / `parent_pid` / `pid_started_at` / `pid_role` (`binder` or `relay`) per endpoint. Assert the rig pid-set and the live endpoints' pid-set are disjoint, THEN kill. Same 2026-08-19 reap: fleet ran 15 procs from `AppData\Local\spt-core\bin\spt.exe` while the rig set was 6 from the lane target; doyle 24948, deployah 33140, hertz 45676, todlando 11208 all installed-bin, none in the rig set. Pair it with re-verifying `ExecutablePath` at kill time as a [[pid-reuse-across-reboot]] guard.
- **A rig target's `POOL-OWNER.json` lives INSIDE the pool**, so reaping the subtree drops the claim by construction — but say it that way. It is not the `pool-release` verb having run, and the distinction matters when the verb is BLOCKED: `cargo run -p xtask` from the main tree refuses with `SPT_POOL_FOREIGN` whenever another lane holds main's own pool ([[pool-claim-holder-death-is-not-lane-state]]).
- A CI `test` leg that TIMES OUT on `seedmap::stop_op_acks_then_serve_returns` (vs asserts) on a previously-green tree = starvation, not regression: `Get-Process spt` WITH Path, sweep leaked target-dir daemons, rerun-failed. Don't park it as a real fail.
- Delete throwaway CARGO_TARGET_DIRs as soon as their gate/bisect purpose is done — they're 20G+ each and stack up fast.
- When e2e reds appear mid-session on previously-green code: `Get-Process spt` WITH `Path` column FIRST, before diagnosing code. Kin [[seedmap-test-collides-live-daemon]], [[gate-clean-target-not-incremental]], [[hfenduleam-disk-full-ci]].