--- name: e2e-leaked-daemons-shared-box description: "nextest fail-fast cancellation leaks e2e-spawned spt daemons; on the shared box they poison SUBSEQUENT bringup-ONLINE e2e runs for EVERYONE — sweep by exe Path under the target dir used, delete throwaway targets promptly" metadata: node_type: memory type: project originSessionId: 5a63f400-f291-4ea5-ba2e-6b0f1cb46429 modified: 2026-07-22T09:31:51.031Z --- **Failure class (counter-39 gate, 2026-07-01):** bringup-ONLINE e2e tests (attach_wedge, dummy_harness) started failing box-wide, cross-version (previously-green committed shas flipped red mid-session), even rested + `--test-threads=1`. RCA: nextest **fail-fast cancellation leaks the detached spt daemons** in-flight e2e tests had spawned. Four leaked `spt.exe` from `Temp\c39-gate-target\debug\` (doyle's gate runs) + disk at 97% with ~60G of orphaned throwaway CARGO_TARGET_DIRs. todlando's kill sweeps were scoped to `workspace\target` — correct for HIS artifacts, blind to another agent's temp targets. `NET_FAMILY_GATE: binding IPv4-only` in brain stderr = informational red herring. **Why:** identical committed code flipping pass→fail across a session = box state, not code (todlando's differential logic — correct). Multiple agents share HFENDULEAM; each agent's kill scope must cover every build dir, or one agent's leak becomes the other's phantom regression. **ESCALATION (v0.25.0 release, 2026-07-04): leaked gate daemons starved a CI RELEASE JOB, not just a gate.** PR #51's Windows `test` leg went RED on `seedmap::tests::stop_op_acks_then_serve_returns` TIMEOUT (240s TERMINATING, not an assertion) — while the FIRST CI run on byte-identical content passed. Root: 8 leaked `spt-w6gate3-target\debug\spt.exe` (brain+run daemons from my own W6 gate subagents that never reaped) had accumulated into hfenduleam's steady state between run-1 and run-2, starving the CI seedmap unit (CI's full `test` filter does NOT exclude seedmap the way my gates do). Swept all 8 scoped-by-path → rerun-failed green. TWO TAKEAWAYS: (1) my gate SUBAGENTS must reap their target-dir daemons before returning — a leaked daemon outlives the gate and poisons CI for the release; verify `Get-Process spt` is clean after each gate agent, don't trust its self-report. (2) SEED: CI self-hosted `test` job should exclude-or-serialize seedmap (`not test(/seedmap/)`) like the gates, else it's starvation-flaky on the shared runner. "identical bytes pass then fail across runs" = box-state/starvation, the [[gate-cross-platform-leg]] sibling on the time axis. ## ⭐ 2026-07-22 (v0.40.0 cut, PR #56) — THE SWEEP-BY-PATH RECIPE BELOW IS BLIND TO CI-RUNNER LEAKS (session 0) **Two distinct leak populations exist on this box, and the recipe in this file only reaches one of them.** *Population A — session 1, agent/gate-spawned.* Sweepable exactly as documented. This cut: **12 `spt.exe` running from `.worktrees/gate-d289787/.gate-target/debug/`**, leaked from a gate run 4 hours earlier and still holding handles. Killed 8 by enumerated pid with path re-verified at kill time (4 more exited as children). Box went **48 → 32 procs, 6864 → 4187 handles (−39%)**. *Population B — **session 0**, spawned by the `actions.runner.*` service.* **Invisible to every technique in this file.** Symptoms that look like corrupted/zombie processes but are not: `ExecutablePath` blank, `CommandLine` unreadable, `GetOwner` unresolved, `Stop-Process -Force` AND `taskkill /F /T` both **Access is denied** — while `Responding=True` and the process is perfectly healthy. **Cause: the runner service runs as `.\decid`, the same account name you are logged in as, but service-spawned children live in session 0 with a service token. Same name, different security context.** A session-1 unelevated shell — agent OR human — cannot read their paths or kill them. Do not diagnose these as wedged/zombie; they are healthy and simply out of reach. **⭐ ORPHAN-CONHOST SOLVED (was 7 unexplained sightings).** `taskkill /F /T` on the leaked daemons revealed hidden children: `conhost.exe`, session 0, one per leaked `spt.exe`. The recurring "orphan conhost" is nothing exotic — it is the console host of a leaked session-0 CI test daemon that nothing ever reaped. This is also the prime suspect for the `Device or resource busy` pin on `crates/spt-daemon` in stale worktree skeletons (a session-0 process holding a cwd there), which is why those pins **survive killing every session-1 daemon** and typically need a reboot. **⭐ THE LEAK IS PER-RUN, AND THAT IS THE MECHANISM.** The session-0 leaks were born inside the failing CI run's own execution window; they did not exist for the two runs before it (those contended with population A instead). So **every CI run leaves session-0 daemons behind that contend with the next run, while also leaking its own mid-run.** Observable signature across 3 runs of byte-identical code: a *different single test* dies each run, **with no panic and no assertion output** — just `running 1 test` → `(test failed with exit code 1)`, i.e. a process-level death — while sibling tests in the same family pass alongside, and disk sits healthy. ⚠ A "load-flake family" of N observed-once tests may therefore be ONE environmental cause selecting a random casualty per run; hardening members individually chases the victim and never closes the family. Check the population before filing member N+1. **⭐ OPERATOR RULING 2026-07-22: moving the CI runner off HFENDULEAM (family-wave leg 3 / option d) is a NON-OPTION for now — stop proposing it. Cause-side mitigation = legs 1+2 (shipped PR #58: in-job reap+census + Windows Phase-A bound).** **How to clear population B:** you cannot, unelevated — `Restart-Service` on the runner is denied from session 1 too. It needs an **elevated** shell (operator): restart `actions.runner.*` **while the runner is idle** (`Runner.Listener.exe` present, NO `Runner.Worker.exe` = no job executing) to reap the whole session-0 tree; or kill by pid elevated; or reboot (also clears the pinned worktree skeletons). **Fix direction for the rig: a post-job reap leg** — the runner must kill its own spawned test daemons by target-path at battery end, before worktree removal. Leaks that outlive the job are the actual defect; sweeping afterwards is the shim. **How to apply:** - After ANY failed/cancelled e2e run: sweep `Get-Process spt | Where Path -like '*'` and kill by pid. NEVER machine-wide; NEVER owl.exe or `spt-core\bin\spt.exe` (live infra, [[no-machinewide-killon-shared-runner]]). - A CI `test` leg that TIMES OUT on `seedmap::stop_op_acks_then_serve_returns` (vs asserts) on a previously-green tree = starvation, not regression: `Get-Process spt` WITH Path, sweep leaked target-dir daemons, rerun-failed. Don't park it as a real fail. - Delete throwaway CARGO_TARGET_DIRs as soon as their gate/bisect purpose is done — they're 20G+ each and stack up fast. - When e2e reds appear mid-session on previously-green code: `Get-Process spt` WITH `Path` column FIRST, before diagnosing code. Kin [[seedmap-test-collides-live-daemon]], [[gate-clean-target-not-incremental]], [[hfenduleam-disk-full-ci]].