# RCA — fleet daemon pid 14444 died 08:03:14.87Z on hfenduleam, 2026-09-07

Analyst: hertz. No box claimed; every reading below is read-only (log reads, WMI census, gh logs).
Cause NOT named. What follows is a timeline, a list of things RULED OUT with the evidence that
rules them out, and the two hypotheses still standing.

## Timeline, measured

| time (UTC) | event | source |
|---|---|---|
| 07:52:54 | CI job `unit (self-hosted, Windows, hfenduleam)` STARTS, PR #199 run 34097634781 | gh job timing |
| 07:57:00.9 | step runs `cargo nextest run --workspace -E 'kind(lib) + kind(bin)'` | CI log :545 |
| 08:03:08.0 | `Finished \`test\` profile ... in 6m 04s` — BUILD ends | CI log |
| **08:03:08.0 → 08:03:15.8** | **7.8 s gap: nextest's LIST phase — 226 test binaries executed (`26` + `200` skipped)** | CI log |
| **08:03:14.87** | **daemon pid 14444 dies. Last log line `wall_ms=1788768194871`, mid-stream, no shutdown line** | rotated daemon log |
| 08:03:15.80 | nextest prints run ID and the FIRST test result | CI log |
| 08:03:20.4 | new generation-0 daemon pid 48232 `daemon run --detached`, parent 59384 (a transient CLI, gone) | WMI + new log |
| 08:03:25.4 | 11 `deadline-<session>.json` rewritten — the DAEMON_RESTART_RESUME fan-out | spt home mtimes |
| 08:07:13 | CI job ends, SUCCESS | gh job timing |

**The death falls INSIDE the 7.8-second list phase and 0.93 s BEFORE the first test cell ran.**

## RULED OUT, each with the measurement that rules it out

- **Any action of mine.** No kill / Stop-Process / taskkill / `spt node stop|restart` / `spt update`
  / reap-by-path in this session; I was idle between turns at 08:03Z. (Caveat I raised rather than
  hid: my prior reply carried a shortform dispatch the adapter delivers FOR me at a time I do not
  control — that is the one spt.exe invocation in my name I cannot timestamp from here.)
- **A test cell.** The tests had not started — first result 0.93 s AFTER the death.
- **Life-before-main in a listed binary.** `#[ctor]`/`#[used]`/CRT-section/static_init: **zero**
  matches workspace-wide. Under `--list`, libtest's harness main runs, not `spt`'s `main()`.
- **A machine-wide kill from product code.** All three `taskkill` sites are `/PID <pid>`, never
  `/IM`: daemon.rs:3472 (comment: "Reap scoped (never machine-wide — shared runner)"), daemon.rs:3525
  (a test killing its own `ping` child), shellhost.rs:639.
- **pid 0 blast radius.** `process_descendants(0)` returns empty at :390-392 and `kill_pid_tree(0)`
  returns early at :431-433. Both guarded.
- **Disk exhaustion.** C: had **103.2 GB** free.
- **A Windows fault.** No Application-log error, no System-log entry 00:58–01:06 local, and
  **no WER ReportArchive/ReportQueue entry** after 00:50. The process left no OS-level trace —
  consistent with TerminateProcess or a silent exit, not a crash.
- **A panic/abort inside the daemon.** The last 8 MB of its stderr contain no `panic`/`fatal`/
  `abort`/OOM line; the only match was one benign `PAIR_MEET_SERVE_ERR:ws272-rig` handshake error.

## STILL STANDING

1. **pid reuse under mass process churn.** 226 binaries executed in 7.8 s is the textbook condition,
   and the reuse band is LIVE on this box, measured: processes created at 08:03:21 took pids
   **3904, 4176, 9416, 11812**, and **44** currently-live processes sit within 2000 of **14444**.
   So 14444 is squarely inside the range Windows was handing out during the window. Any component
   holding a stale 14444 for something of its own — liveness-check then kill — would have struck the
   daemon. **What is missing is a killer: nobody has shown one.** Plausible, not demonstrated.
2. **Silent self-exit.** No shutdown line, but also no evidence of one.

## SEPARATE DEFECT, worth filing regardless of the cause

The dead daemon's stderr was **303,128,498 bytes (303 MB)** over ~72 h uptime (`mono_ms` 259.19M),
~100 MB/day. In the LAST 2 MB alone: **3238** `CONN_LIFECYCLE ... event=write-start` + **3086**
`event=transport-close poisoned=false timed_out=false`, both `role=brain`, plus **1163**
`NET_FAMILY_GATE: binding IPv4-only`. Conn ids had reached **720837**, and `CONN_WRITE_RETIRED`
says conns are "poisoned + retired, never reused". A brain conn opened and closed thousands of times
per 2 MB of log, under `[REQ-CONN-POISON-ATTRIBUTION]`, is its own finding.

## NEXT MEASUREMENTS (cheap, none need a box held)

1. **Instrument the killer, do not hunt it retroactively.** The window is reproducible: it is every
   `cargo nextest run` list phase on this box. Watch the fleet daemon's pid across one and record
   whether it dies again.
2. **Ask whether any component kills by a REMEMBERED pid without re-verifying identity at kill time**
   (image path or start time), which is the only way reuse becomes a kill. My own standing rule from
   the orphan reap is exactly this: re-verify the path AT kill time.
3. **Bound the log volume** independently of the cause.

## releases#286 — stderr follow-up (2026-09-09)

<!-- [doc->REQ-CONN-HEALTHY-LIFECYCLE-BOUNDED] -->

The historical 303 MB incident above remains the reported baseline, not a
newly reproduced event. A read-only sample of the currently installed
`%LOCALAPPDATA%/spt-core/logs/daemon.stderr.log` ended at byte **4,163,267**:
the last **1,999,958 complete-line bytes** spanned wall stamps
**1788997126331–1788998151874** (1025.543 s). It contained **5127** first writes
(656,158 bytes), **5092** unpoisoned/non-timeout closes (850,622 bytes),
**3** organic retirements, **0** deadline poisons, **107** family-gate lines
and **108** rendezvous-up lines. Healthy start/close records alone consumed
**75.3%** of that sample. A subsequent live append observation over
**307.203 s** added **79,741 bytes** (**259.57 B/s**), with **104** starts,
**104** healthy closes and **30** family gates paired with **30** rendezvous
binds. These are different observation windows, not a claimed constant rate.
No daemon was restarted or reconfigured for either measurement.

**Cause discrimination.** `role=brain` identifies *every broker IPC client*,
not the supervised coordinator. `run_brain` already keeps its heartbeat
`Brain` alive. In contrast, the pre-fix
`livehost::query_live_session_endpoints` constructed `Brain::cold_start`,
called `sessions`, and dropped the carrier on every invocation from the
five-second reconcile loop. This is a source-confirmed avoidable reopen
cause, consistent with the recurring five-second bare-brain start/close
pairs in the live log, **not an attribution of all 5127 connections to that
caller**. Other short-lived callers include dispatch idle queries and
per-operation dispatch/CLI clients; their lifetimes are not changed here.

The livehost census now owns one carrier across polls. Its `BRAIN_CONN_OPEN`
breadcrumb names `caller=livehost-session-query` and distinguishes
`reason=initial-query` from `reason=previous-query-error`. Every query error
discards the carrier, returns an unavailable census (never an empty
authoritative census), and permits a fresh connection only on the next
scheduled query. The one-shot boot orphan census remains one-shot.
No read/write deadline, poison semantics, or retry cadence changed.

**Family gate exonerated.** The sole diagnostic site in
`NetEndpoint::bind_scoped` is already once per real bind attempt, not per IPC
connection. `pairhost::spawn_meet_rotation` binds a new derived identity for
each attached subnet at every 30-second TOTP boundary; the observed
`BIGNET`, `SPT_DEV`, and `SPT_MANTLE` windows account for the three lines per
boundary. These listeners cannot share a stable identity without changing
the pairing protocol. Bind and diagnostic policies are intentionally
unchanged; a process-global "log once" would hide later real binds.

**Explicit contract amendment.** Issue #286 supersedes only the
`REQ-CONN-POISON-ATTRIBUTION` leg-(d) requirement for unconditional
*healthy start/close lines*. Across the broker process these become
`CONN_LIFECYCLE_SUMMARY` totals, at most once per 60-second monotonic interval,
flushed by the first subsequent lifecycle counter event. Starts count the
first committed write attempt; closes count unpoisoned physical drops.
There is no timer thread and no shutdown flush, so a quiet tail remains
pending until another counter event (and can be lost on process exit).
Poison, organic failure, poisoned close, and all role-transition records
remain unconditional per connection. They retain conn id, role facts,
wall/monotonic stamps **and the original first-write wall/monotonic time**;
a connection that never wrote says `first_write=none`.

The deterministic regression cells exercise real framed socket exchanges
over both retained and fresh carriers with an isolated fixed log clock/sink,
assert one interval summary for N healthy connections, preserve exceptional
records through the same sink, and drive livehost census reuse plus recovery
from a malformed reply. Focused commands (not executed in the preparation
lane; integration owner runs them after landing):

```text
cargo test -p spt-daemon --lib conn::tests::
cargo test -p spt-daemon --lib livehost::tests::session_poll_reuses_healthy_carrier_and_reopens_after_query_error
cargo test -p spt-daemon --test conn_blackhole_lifecycle
traceable-reqs check --json
```