--- name: hfenduleam-disk-full-ci description: "hfenduleam Windows CI failures with LNK1318/os-error-112 = disk full, not code; cargo clean the dev target" metadata: node_type: memory type: project originSessionId: 6b7355ca-c59b-4b57-a4e8-b248ce4673c6 modified: 2026-07-22T05:45:13.901Z --- HFENDULEAM (Windows self-hosted CI runner) Windows `test` job failures showing `LINK : fatal error LNK1318: Unexpected PDB error; LIMIT (12)`, `rustc-LLVM ERROR: IO failure on output stream: no space on device`, or `os error 112 (not enough space)` are **disk-full, NOT a code regression**. Re-run the CI job after reclaiming space — do not chase a phantom test bug. **Why:** 2026-06-13 PR #8 (m9 Wave 2 G2 gate) Windows test went RED at 1m41s during *build/link*; all logic tests + Linux + traceability passed. Root cause: C: had only **2 GB free**. The dev target dir `C:\Users\decid\Documents\projects\spt-core\target` had bloated to **396 GB** (330k files — branch/profile accumulation). It shares the disk with the CI runner workspace (`C:\actions-runner\_work\spt-core\spt-core\target`, ~10 GB). **How to apply:** `cd` to the dev checkout, `cargo clean` (reclaimed 396 GB → 374 GB free), then `gh run rerun --failed`. The dev target is regenerable; the live listener is owl.exe (plugin), never target/ — see [[spt-daemon-is-live-infra]]. Watch the dev target — it regrows fast; clean periodically before it strands CI again. **UPDATE 2026-07-01 (v0.19.0 PR#41 cut) — the hogs are BROADER than one dev target.** hfenduleam C: filled to <1 GB mid-release; the CI runner target `C:\actions-runner\_work\spt-core\spt-core\target` was only 21 GB. The real mass: (a) STALE SIBLING WORKTREE TARGETS — every `git worktree` under `C:\Users\decid\Documents\projects\spt-core-*/target` carries its own full ~15-31 GB target (bug2 17G, bug6 20G, translate-proof 31G, ufx 29G, wedgeval 18G, delivery 4G = ~119 GB for SHIPPED/merged branches); (b) TEMP GATE SCRATCH — doyle's throwaway `CARGO_TARGET_DIR`s at `C:\Users\decid\AppData\Local\Temp\.tmp*/cv-gate7*` accumulate to ~177 GB (dozens of dirs, each a full target). Reclaim recipe when C: tightens: `rm -rf` the sibling `spt-core-*/target` dirs (rebuildable; work is merged/shipped) + the TEMP `.tmp*/cv-gate7*` gate targets — freed ~130 GB this cut. CAVEATS: a busy worktree subdir errors `Device or resource busy` (another agent's live checkout — skip it, don't force); scope to `target/` and throwaway scratch only — the 1.8 TB baseline is the user's non-project data, never touch. Also note: a Windows `test` fail that is `1 timed out` (not a link error) on a busy hfenduleam is usually the [[seedmap-test-collides-live-daemon]] starvation flake, NOT disk — check `df` first, then just `--failed` rerun. **UPDATE 2026-07-05 (remote-truth bless matrix) — LOCAL gates die on this too, tell = `tail: write error: No space left on device`.** C: hit literally 0 bytes free mid-matrix; shared checkout target was 193.7 GB again (regrew from the 07-01 clean in 4 days of remote-truth gating). A dying local gate leg with pipe write errors = ENOSPC, not code — the leg before it can even be green. Recipe: warn the co-agent on the shared tree (HOLD builds), `cargo clean` the shared checkout (took ~3 min, freed 186 GB), ALL-CLEAR, rerun only the dead legs. Gate worktree targets are small (~5 GB) — the shared incremental target is always the hog. **UPDATE 2026-07-19 (RC-RENDER-TRUTH v0.38.1 W4 gate @622eb84) — NEW tells: `LNK1180` AND a build/nextest KILLED MID-COMPILE are disk-full too.** During doyle's whole-branch battery the nextest build went RED with `LNK1180` while C: sat at 100% (2.5 GB free). Root: doyle's two concurrent gate builds + a STALE 29 GB orphaned prior-gate target `gate-2a110ee-target` left in the claude temp tree (`%LOCALAPPDATA%\Temp\claude\...`). Reclaiming the orphan → 32 GB free, re-run went clean. Separately my OWN full Phase-A nextest re-run got `killed mid-compile` (harness stopped it, no error) — first blamed on CI contention, but doyle's df showed it was almost certainly the SAME disk pressure. So: a build that dies during compile/link with LNK1180 (or is silently killed mid-compile) on this box ⇒ check `df -h /c` FIRST. Box runs hot on disk generally (1.9 TB volume at 99% even post-reclaim). Sweep target: orphaned `gate--target` dirs in the claude temp tree (doyle's throwaway CARGO_TARGET_DIRs), same class as the `.tmp*/cv-gate7*` hogs above. NOTE: rustc INCREMENTAL ICE (`STATUS_STACK_BUFFER_OVERRUN 0xc0000409`) is a SEPARATE known Windows gotcha (clear incremental dir + `CARGO_INCREMENTAL=0`) — doyle hit BOTH this cut; don't conflate them, but `clippy --all-targets` passing proves it's not a real compile error either way. **UPDATE 2026-07-16 (v0.34.0 counter-61 cut, REDISPATCH-TRUTH W1) — NEW tell: the CI job fails with NO code log at all.** hfenduleam Windows `test` job went RED at 2m26s; `gh run view --job --log-failed` and `--log` both returned `log not found: ` (the runner couldn't WRITE its own log). The diagnosis lived ONLY in the **check-run annotation**: `System.IO.IOException: There is not enough space on the disk. : 'C:\actions-runner\_diag\pages\...log'`. So when a self-hosted job fails fast with unfetchable logs, `gh api repos///check-runs//annotations` is the grounding move — it surfaced the disk IOException directly. `df -h /c` confirmed 100% used / 16 GB free. Hog THIS time was the dev checkout target `C:\Users\decid\Documents\projects\spt-core\target` = **234.7 GB** (141,678 files); `cargo clean` freed all of it → 240 GB free, then `gh run rerun --failed` went green (Windows test 14m11s). Only that one job failed (disk); Linux test + both n1-gates + traceability were green — a lopsided one-runner fail with unfetchable log ⇒ check `df` + annotations FIRST, never chase a phantom regression. Also swept my own leftover `%TEMP%\spt-release-v0.3*` publish dirs (tiny, ~0.11 GB each) as tidy-up. ## RESOLVED 2026-07-19 — OPERATOR-APPROVED `deps` CLEAN, 10.60 GB -> 243.33 GB free **Three disk-full incidents in ONE DAY, two of which broke CI runs** (one on a PR that had already passed a full local bar AND doyle's gate). Symptoms presented as COMPILE ERRORS, never as "disk full": `rustc-LLVM ERROR: IO failure on output stream: no space on device` / `could not compile spt-daemon` / `os error 112`. Every `incremental` clean bought HOURS, not days; a single cold gate cost ~30 GB while a warm build cost ~1.5 GB — **a COLD GATE is the expensive act on this box, not a build.** **The fix:** `\target\debug\deps` = **233.65 GB**, deleted in a DECLARED WINDOW with both agents parked. `incremental` (0.36 GB) and `build` (0.43 GB) left alone; nothing outside the repo's target dir touched; live daemons run from `AppData\Local\spt-core\bin` and were never in scope. **PROCEDURE THAT MADE IT SAFE (reuse this):** (1) get OPERATOR approval — a cold rebuild imposed on every lane is THEIR call, not an agent's; (2) DECLARE the window and tell every agent on the box to stop; (3) VERIFY rather than assume — check for live `cargo|rustc|nextest|link` processes AND for anything executing out of the target dir; (4) delete; (5) confirm DELETED not PARTIAL; (6) announce the window closed. **THE NEAR-MISS THAT JUSTIFIES STEPS 2-3:** todlando had a clippy + 10x loop running and killed it on the announcement. **A partial delete with rustc mid-write leaves a half-populated deps dir, and the next build fails with something that LOOKS LIKE A CODE ERROR** — the exact failure shape being cleaned up, manufactured by the cleanup itself. **LESSON: this was the first cleanup all day that happened BEFORE the failure rather than after.** doyle had been re-filing it as a standing hazard repeatedly instead of putting it to the operator as a DECISION. **Filing a hazard is not escalating it.** When a hazard has fired more than once, the next step is a decision request with numbers attached, not another note. **UPDATE 2026-07-22 (v0.40.0 cut, PR #56) — ⭐ NEW SYMPTOM CLASS: disk pressure presenting as TEST-TIMING failures, not build/link errors.** Every tell in this file so far is a compile/link/IO error (LNK1318/1180, os-error-112, ENOSPC, unfetchable logs). This time C: at **3.2% free (58.8 GB)** with **CPU idle at 18%** produced *green builds* and *failing asserts*: `brain_decouple::suspended_brain_controller_is_stall_evicted_take_completes_viewer_ticks` red twice consecutively on byte-identical code (`CONN_LIFECYCLE ... event=writer-exit reason=write-failed kind=TimedOut`), plus an unrelated `adapter_translate_proof_gates_on_commit` red appearing only on the second run (1 fail → 2 fails). **A stall-eviction/deadline test going red on a starved disk looks exactly like a real timing regression.** Check `df` before believing any timing-assert failure on this box — and note CPU-idle does NOT exclude disk (this box was idle and still starved). Compounding trap that made it worse: the change under suspicion (the v0.40.0 observability rider) had added a synchronous per-subscribe log write **under the broker lock**, i.e. fs I/O on the exact path the failing test times. Rider-alone / disk-alone / rider-amplified-by-disk all fit the same evidence, so neither could be dismissed from the CI signal — the honest move was to fix the box first and re-run the same code as the discriminator, before spending a 60-run A/B on a starved box (an A/B interleave decorrelates box *drift*, never a box that is out of space in *both* legs). **Hog location has MOVED — look under `.worktrees/`, not sibling `spt-core-*` dirs.** Since the operator worktree ruling, every lane lives at `/.worktrees//target`. Census: obs-rider **34.16 GB**, release-lane 12.35, r4-stale-guard 0.92, docs-reorg 0.71, rest ~0. **One merged build lane's target was 3/4 of the entire reclaim.** Sweep rule that worked: `target/` dirs die unconditionally (zero risk, that is where the GB are); the checkout itself goes only if clean AND fully pushed; gated-but-alive lanes (spec-pilot/TLA+) stay untouched. Result 59 → 105.2 GB (3.2% → 5.65%), ~46 GB. **⭐ OPERATOR RULED 2026-07-21 (post-escalation, direct):** recycle bin emptied (+11.6 GB, emptiable at will); **>100 GB free = "plenty for CI"** — no volume migration, no games-footprint reclaim. The operating posture is run-near-full WITH standing sweep discipline: doyle gate-rig leg 0 df+stale-target sweep, deployah lane-close target sweep, df-before-believing any timing-assert red here. ~100 GB ≈ 3 cold workspace targets — the envelope holds only while the discipline does. ⚠ **AND THE SWEEP DID NOT FIX THE BOX — this is the escalation datum.** Profile-wide: projects 368.6 GB, AppData\Local 53.5, Roaming 29.6, Temp 3.1, .cargo 2.1, .ccs 1.6 ≈ **458 GB of 1803 GB used. ~1345 GB lives OUTSIDE the user profile entirely** and outside any agent's authority. Worktree/target sweeps can only ever buy tens of GB on a 1.9 TB volume that is structurally full; they are a shim, not a fix. Per the 2026-07-19 lesson at the top of this file — *filing a hazard is not escalating it* — the correct next move is an operator DECISION REQUEST with these numbers attached, not another sweep. A full workspace target measures ~34 GB here, so 105 GB is ~3 cold targets of headroom: enough to proceed, not enough to call the box healthy. Residual hygiene: 3 dead gate worktrees + an obs-rider skeleton refuse deletion with `Device or resource busy` **all pinned at the same subpath (`crates/spt-daemon`)** — orphan-conhost, seventh sighting. ~7 MB total, zero space impact; defer to a reboot window rather than pid-hunting mid-release. ## ⭐ OPERATOR RULING 2026-07-22 — RUN-NEAR-FULL IS THE ACCEPTED POSTURE (escalation CLOSED) The decision request above was answered: **recycle bin emptied (+11.6 GB, and emptiable at will going forward); >100 GB free is the accepted CI envelope; NO volume migration and NO reclaim outside our own footprint.** So the 664 GB games/emulation and the ~520 GB personal data are permanently off the table, and "move CI+dev to a dedicated volume" is REFUSED. Do not re-propose either — this is settled, not pending. **The posture that replaces it, three standing disciplines (all binding):** 1. **Lane-close target sweeps (deployah)** — a merged/closed lane's `target/` dies at close, not eventually. One merged build lane was 34 GB of the 46 GB reclaimed this cut; that is the whole ballgame. 2. **Gate-rig leg-0 `df` + stale-target sweep (doyle)** — already the mechanism at the bottom of this file; now doctrine rather than a good idea. 3. **`df` BEFORE believing any timing red on this box** — the new symptom class above (timing asserts, not link errors) means a red test is not evidence of a code defect until the disk figure is in hand. **Framing that matters for the next incident:** near-full is now the DESIGNED operating point, so "the box is at 95%" is no longer itself a finding. The finding is always the delta — what changed, what grew, and whether free fell below the ~100 GB envelope. Report the envelope breach, not the percentage. ⚠ **AND THE RULING COVERS DISK ONLY.** It does not touch the separate contention finding from the same incident: 43 live spt-family processes / 6486 handles on the box while CI runs 1881 tests in full parallel, producing a process-level test death (no panic, exit 1) with a DIFFERENT random victim per run while disk sat healthy at ~106 GB. That is [[e2e-leaked-daemons-shared-box]] / [[seedmap-test-collides-live-daemon]] territory, it is still open, and disk headroom will not fix it — see the release lane's own record for the reframe (a "load-flake family" of N flaky tests may be ONE environmental cause selecting a random casualty). **UPDATE 2026-07-21 (shell mini-wave gate @5a38c7d) — the hog was FOUR STALE GATE TARGETS in the claude temp tree, and disk-full poisons BUILD-SCRIPT OUT_DIRS too.** C: hit literally 0.0 GB mid-battery (LNK1180 on leg 4 after legs 0-3 green). Census one PowerShell loop over `%LOCALAPPDATA%\Temp\claude\C--Users-decid-Documents-projects-spt-core\*/scratchpad`: gate-target 66.4G + gate-w3-target 33.1G + tgt-919ed95 15.7G + tgt-rig 7.8G = **123 GB of throwaway targets from CLOSED gates whose sessions died before their own cleanup**. Verified no process ran from any (exe-path scan), deleted all four → 160.8 GB free. THEN the rerun failed differently: `couldn't read .../build/libsqlite3-sys-*/out/bindgen.rs (os error 2)` — the disk-full build had left build-script OUT_DIRS half-written, same poisoned-target class as the LNK1285 PDB truncation but in `debug/build/`. Purge the named `build/-*` dirs, not just deps/PDBs. **MECHANISM (add to every gate rig): leg 0 checks `df` and reclaims stale `gate-*-target`/`tgt-*` dirs in the claude temp tree BEFORE building — every finished gate that cleans its own target only helps if the session survives to do it; the box needs the sweep at the START of the next gate, not the end of the last one.**