---
name: instrument-soundness-guards
description: "Running list of guards on test instruments — a gate whose failure signature matches the fault it guards is broken; instrument tests are never requirement evidence; per-attempt provenance stamps; mutation harnesses need a proven-green baseline, a commit to restore to, and a DISTRIBUTION not one sample; piped gates read the pipe's exit code; CI's runner may be green where the hand-run lane is not."
metadata: 
  node_type: memory
  type: feedback
  originSessionId: b5e592de-a93b-44ea-9e05-99ea04229fcb
  modified: 2026-08-03T06:19:30.531Z
---

Two guards doyle ratified 2026-07-26 (the endpoint_autostart_e2e torn-read fix, PR #80). Apply to any gate, not just this one:

1. **A gate whose failure signature is indistinguishable from the regression it guards is a broken instrument** — fix it before trusting any red or green from it. The autostart gate polled a log the daemon was concurrently writing; a torn read failed the poll to its deadline and panicked as "must bring the saved endpoint back up" — the exact message a genuinely-never-launched endpoint produces. CI called it a load flake and reran green, which closed the *environment trigger* only. Rerun-green is the [[premature-closure-guards]] heal case wearing CI clothes.
2. **Instrument tests are not requirement evidence** — a test that exercises the harness's own read/setup seam would pass on a build where the feature is entirely broken, so tagging it `[int->REQ-*]` is coverage inflation under traceability rule 1. The requirement's evidence stays the test that actually drives the product. Put the reasoning in the commit message; that's its durable home.

**Why:** both make a suite *look* more trustworthy while making it less so — one hides a fault behind a plausible message, the other hides thin coverage behind a tag count.

**How to apply:** when a poll reads a file another process is writing, the satisfy-predicate and the judge-predicate must be **one predicate over the whole expected shape**, and any unterminated trailing fragment gets dropped before it can be matched. Splitting them across the loop boundary lets a truncated line satisfy the loop and then fail the post-condition on its own truncation. Watch for a reader that *terminates* a fragment (e.g. pushing `'\n'` per file) — that promotes it to a complete-looking line and is the harder half of the bug.

4. **The checker's own transport can forge the fault it checks for** (added 2026-08-01, v0.50.0 release-body parity — the purest instance yet). The runbook's body-vs-CHANGELOG parity step is a *mojibake* check; my checker transcoded the text on the way to disk, so it reported em-dashes as replacement characters and a 10-byte delta — a textbook mojibake publish that would have justified a body repair on a clean release. `gh ... --jq .body > body.txt` mangled one side; piping `git show` through a text pipeline mangled the *other*. **The only thing that saved it was two runs disagreeing about WHICH side was broken** — a single run would have shipped a false report. Guard: for any byte-parity check, pull both sides as **raw bytes in-process** (API response body; `git show blob > file` in binary), decode explicitly as UTF-8, compare programmatically, and **never print the text to a console or redirect it through a shell**. Corollary beyond encodings: when a checker and its subject share a transport, the transport's failure mode is inside the checker's answer space.

3. **Outcome-correlated provenance stamps fabricate experimental arms** (added 2026-07-30, lia's Gate-4 retraction). A provenance stamp keyed on the *entity* and overwritten by the *latest attempt* — while the result file keeps the best/longest attempt — makes "unstamped" mean "succeeded on attempt 1", so splitting outcomes by the stamp measures the outcome itself (lia's p=0.0166 "prompt effect" was exactly this; honest per-attempt numbers were p=0.393). Guard: a stamp must travel WITH the result it describes, one per attempt, never keyed on the entity; before trusting any split-by-provenance analysis, ask how the stamp is *written*, not what it looks like it records.

4. **Reachability is part of the finding, not a detail after it** (added 2026-07-31, hertz's proof-poll retraction). A writer with a provable torn/truncate-on-open window is NOT a defect until the reader is shown to run *concurrently* with it — three "racy poll" sites were retracted because the reader only ran after a synchronous-to-completion call (`pulse_tick` via `run_bounded_stdin`) had already reaped the writer, making the window unobservable; the one real case (advisory) differed exactly in having a long-running process writing concurrently with the poll. Guard: before filing a torn-window finding, falsify it — inject a guaranteed window and show the unfixed shape actually fails. Hertz's injected 2s window passing the old shape is the model.

5. **A mutation harness must prove a green baseline before trusting any red, and sha-verify its restore** (added 2026-07-31, todlando's W4 T5). His mutation script timed out mid-run and left a mutation applied; he checked for leftovers by grepping the markers he *remembered* inserting, missed it, and a later run used the mutated file as baseline — two "reds" were partly the stale bug, not the mutation under test. Grepping remembered markers is not a restore check: it only finds what you remember, and a timeout can leave an edit with no marker at all. Guard: before any mutation run, assert the suite is green on the claimed-clean tree; after restore, sha-compare against the pristine file. A red is evidence only relative to a proven-green baseline.

6. **A gate read through a pipe reads the PIPE's exit code** (added 2026-08-01, two independent instances in one day). doyle's golden pre-gate `cargo clippy ... | tail -15` reported exit 0 on a compile RED (tail's exit; caught only because the visible text contradicted it); perri's gate suite piped through `tail` let an `&&` chain commit-and-push past a FAILING mirror leak guard. Guard: capture the instrument's own exit (`PIPESTATUS[0]`, or run unpiped and post-process the file); any green that plumbing alone can produce is not a green. Same family as guard 1 — the instrument's signature must be producible only by the thing it measures. Deployah's sharpening: this is not a flaky check, it is a check that was NEVER CONNECTED — piped through tail, the gate structurally cannot fail, in any run.

7. **A fail-toward test must assert it reaches the arm it names** (added 2026-08-01, todlando's HANDRAIL W2). The natural fixture for "unreadable file" — a plain FILE standing where a directory belongs, then reading a path through it — returns `Ok(false)` on Windows, i.e. the ABSENT arm, not the error arm. A test written that way passes while proving nothing about unreadability, and it looks correct in review. Guard: assert the instrument first (`assert!(marker.try_exists().is_err(), "must reach the UNREADABLE arm")`) so the fixture's platform behavior is itself under test. Portable Err source for a path probe: an **interior NUL** in the path — Unix cannot build the C string, Windows cannot build the wide string, both surface `Err`. Same family as guard 1: absent and unreadable are two arms that a careless fixture collapses into one.

8. **Commit before mutating, always** (added 2026-08-01, same session, a mistake I made). Mid-mutation restore via `git checkout <file>` resets to HEAD — which, while the work is still UNCOMMITTED, is the genuine pre-state, silently wiping the entire build being proven. The next mutation then fails to find its anchor, which is the only reason it got caught. Guard: the mutation cycle is commit → mutate → run → `git checkout` → verify `git diff --quiet HEAD`. Restoring to a *commit* is what makes `git checkout` a safe restore; without one, it is a delete. Extends guard 5 — a sha-verified restore is worthless if the sha it restores to was never recorded anywhere durable.

9. **A one-sample baseline cannot attribute causation on a flaky suite** (added 2026-08-01, releases#58 re-audit — my own error, caught within the session). `cargo test -p spt-daemon --lib` in the PRIMARY tree red 2 tests; I established those as pre-existing correctly. Then in a clean worktree one baseline run came back 760/760 green, my branch red 2 — and I told doyle the reds were mine. Three more clean runs: 2, 2, 1 failures. The clean tree flakes in exactly the same band; the single green was the lucky sample. Guard: on any suite with known shared-process contention, a baseline is a DISTRIBUTION (≥3 runs), not a run — and compare failure-name POOLS, not counts. Two discriminators that settled it here: single-threaded moved the failure to a different pool member instead of clearing it (⇒ scheduling, not logic), and no test in the module I changed ever appeared in the pool. Same shape as the mtime slip perri caught: a derived signal read ONCE cannot separate two causes that both produce it. Extends guards 5 and 8 — "prove a green baseline" means prove it is *reliably* green.

10. **The lane a human reaches for by hand may be broken while CI is green** (added 2026-08-01, same session). `cargo nextest run -p spt-daemon --lib` = 760/760 (process-per-test, CI's unit lane since releases#47), while plain `cargo test -p spt-daemon --lib` is reliably 1–3 red on hfenduleam from a pool of four names (broker `the_engine_room_refuses_every_viewport_and_every_off_node_attach`, dispatch `the_receiver_decides_origin_and_flood_not_the_sender`, config `upsert_startup_endpoint_appends_then_replaces_preserving_other_knobs`, seedmap `request_stop_barrier_holds_until_no_listener`) — all pass in isolation. CI structurally cannot see this, so anyone verifying a fix by hand reads someone else's flake as their own regression. Guard: when a suite's CI runner differs from its ergonomic invocation, the gap belongs in the crate docs or AGENTS.md, not in tribal memory — and always confirm a fix under the CI form before believing a hand-run red.

11. **A MINIMUM is not a coverage guard when truncation lands on a legal value** (added 2026-08-01, DOORBELL W2 remedy pin). To prove a text-extractor wasn't stopping short I asserted a depth floor (`path.len() >= 2`). It catches truncation on a 2-segment command — and misses it on the 3-segment one, which truncates to a legal 2 and passes the floor AND the downstream grammar walk. The defect hides inside its own truncation, and the arm was green until a mutation surfaced it. Guard: pin the extractor to the FULL SPAN of what it was supposed to consume, re-derived by a different route than the extractor takes (here: split the source on its backticks, compare to the walked path), never to a threshold — a threshold only excludes the degenerate case, and real truncation is rarely degenerate. Corollary: when writing a guard on an instrument, ask which WRONG outputs still satisfy it, not whether the right one does.

12. **A list cannot testify against itself — derive the expectation from the OTHER source** (added 2026-08-01, DOORBELL W3 census, doyle's catch on my own fix). I replaced a stale hand count (`ALL.len() == 14`) with a wildcard-free `census_index` match plus a bijection check. It closes the add direction at COMPILE time and catches interior drops — but both `claimed` and `expected` derive from `ALL`, so a variant added to the enum, given a census arm claiming the TAIL index, and omitted from `ALL` leaves the sets equal and everything GREEN. Mutation-confirmed: full crate 766/766 green with a family that exists, dispatches in production, and appears in no family walk. The interior case only reds because removing an entry SHIFTS its neighbours into disagreement; at the tail nothing shifts. Two carry-forwards: (a) the author reaches that state by doing exactly what the compiler asks — rustc names three match sites, they fill three, and the one unforced edit is the data list, because **data cannot be non-exhaustive**; (b) real closure needs enumeration from the type (`strum::EnumIter` behind `#[cfg_attr(test, derive(...))]`, then `for v in T::iter() { assert!(ALL.contains(&v)) }`), which also supplies the inverse that lets the failure NAME the absentee. Guard: when a test compares a set to itself under a transformation, ask which source each side came from; if it is one source, the check is an internal-consistency check wearing a coverage costume.

13. **A TOOL's negative needs the same positive control a TEST's negative does** (added 2026-08-01, DOORBELL W5e; doyle put it in his gate notes). Verifying my new e2e was in the nextest heavy class, `cargo nextest list --message-format json` showed it with no `test-group` — which reads as "the filter did not match" and was one keystroke from being reported as a config bug. It reads identically for `multi_subnet_bringup_e2e`, a *known-heavy* binary: that JSON carries no group field **at all**. The instrument was blind, not the config. `cargo nextest show-config test-groups -p <pkg> --test <bin>` is the one that answers, and it prints the binary inside the matching override. Guard: before believing a tool's silence/absence/empty-result, run it against a case you KNOW is positive — if the known-positive looks the same, you learned nothing. Third instance in one day of an instrument's silence nearly becoming a claim (the others: guard 9's lucky-green baseline, and a mutation that "passed" because the mutation missed, not because the guard was blind). The unifying rule: **absence of a signal is evidence only from an instrument proven able to emit it.**

14. **A forced predicate must be DECLARED beside the evidence, naming a CHECKABLE owner** (added 2026-08-01, doyle's W5e ruling). My e2e clears three inherited agent env markers so the engine-room ceremony will run — legitimate hermeticity (the markers state nothing true of a test child; the ancestry half of the guard still runs against an empty roster), but invisible to a reader, and an undeclared forced predicate is indistinguishable from a bypass. Doyle required it declared in the test body. The upgrade that mattered: my first version named the REQ that owns the refusal row, which is a trust-me; the accepted version names the UNIT TESTS (`engine_room_ceremony_decision_table`, `agent_ground_describes_itself_exactly_as_the_ceremony_always_did`) — a claim a reader can open. Verify the names EXIST before citing them: a dangling test name inside such a declaration reads as coverage that cannot be opened, which is worse than no declaration. Corollary: adding the declaration MOVES THE SHA — re-report it even when told not to, because under ff-only main a gate against a vanished commit breaks tested-sha-equals-merged-sha.

15. **A parse MISS must not be gradeable as a verdict — and the positive control is what catches it** (added 2026-08-02, BAROMETER W4 mutation sweep). My driver graded each mutation by regexing nextest's summary for `N tests run`. nextest prints **`1 test run`** — singular — for a single row, so every one-row mutation parsed as no-summary and fell into a verdict bucket meaning "not killed". First run: 1/9 KILLED, on a build where all nine rows had visibly failed with exactly the right panic. Two things made it recoverable: the run's raw FAIL lines were in the log, and — the part that actually did the work — an inert **positive control** mutation was in the table, so a "harness broken" state had a *shape*: M0 came back `NO-SUMMARY` when it had plainly PASSED, and a control that cannot report SURVIVED invalidates every KILLED beside it. Guards: (a) a parser that fails to match must produce a DISTINCT terminal verdict (`NO-SUMMARY`, hard error), never fall through into a substantive one — mine did keep them distinct, which is why it was visible at all; (b) every mutation table carries an inert control whose required verdict is SURVIVED, and the summary line states the control's verdict FIRST, because it gates the meaning of the rest; (c) when the driver is wrong, **fix the driver and re-run — do not hand-grade the old logs**, even when the raw output is unambiguous; a table that says KILLED must have been produced by an instrument, not by me reading. Extends guards 5/8/13: the same "absence of a signal is evidence only from an instrument proven able to emit it" rule, applied to the grading layer rather than the test layer.

16. **A hardened reaper can fail safe in the WRONG direction, and the arm that proves it looks identical** (added 2026-08-03, deployah's catch on my IR-18 lane before it was built). Replacing a bare `taskkill /PID /F /T` with the suite's `authenticated_kill` is strictly correct — unless the `expected_exe` is wrong per pid, in which case EVERY kill returns `REFUSED-foreign-image`, both supervised services leak, and the test still passes because they exit on their own; the log prints exactly the authenticated reap line a reviewer wants to see. Bare kill leaks nothing and endangers strangers; refuse-everything endangers nobody and leaks everything — strictly worse than what it replaced, invisible in the same log. The concrete trap here: `boot_pid`/`rel_service_pid` run the staged **svcmock** image, only `brain.ready`'s pid is `spt.exe`, so passing `spt_bin` for all three refuses all three. Guard: **an arm that must kill must assert it killed** — never infer success from zero survivors, since "nothing survived" is satisfied equally by a kill and by a refusal followed by a natural exit. General form: when a remedy's failure mode is *inaction*, absence-of-damage is not evidence the remedy ran. **The spelling matters and my first one was wrong** (corrected same night by hertz): a blanket "zero `verdict=REFUSED-` lines" gate REDS THE HAPPY PATH — `reap.rs`'s own contract says a refusal is the correct outcome when the test already stopped its daemon (`Refused("gone")` is leak insurance, not primary teardown). Someone then loosens the gate to go green, and the loosened version is exactly the one blind to `foreign-image`: the detector dies by maintenance, wearing a green — the shape the gate existed to catch. Correct spelling: split the reasons and assert only the TRAP class absent — `foreign-image`, `unreadable-image`, `unproven-identity` (something WAS there and authentication declined) — while `gone`, `reused`, `breadcrumb-moved`, `self`, `ancestor`, `no-breadcrumb` are SOUND and expected. Assert structurally on the returned `Verdict`, not by scraping stderr a harness may not capture, and say in the comment that `Refused("gone")` is expected or the next reader "fixes" it. Same family as guards 1 and 13 — otherwise the entry's remedy mints the next entry, wearing a green.

17. **Run a proposed instrument against the conformance case that motivated it — BEFORE it is specced** (added 2026-08-03, my own falsified spelling, IR-18). A per-pid teardown gate is blind to a process whose pid was never captured (`[a, b].into_iter().flatten()` drops `None`; the feeding `mock_pid` collapses absent / denied / IO-error / garbage into one `None`), so I proposed a POPULATION assertion — "zero surviving svcmock/spt.exe **descending from the test process**" — citing a real 12-process leak as its conformance case. hertz falsified it with that same case: teardown kills the daemon FIRST, the leaked service is the daemon's child, and a live BFS cannot traverse a dead middle hop — so `descendants(test_pid)` is empty precisely in the leak state and would have caught **0 of 12**. The INTENT (an observable independent of the broken enumeration) was right; the spelling failed on its own evidence. What survived: svcmock by **image-path-under-unique-home** (zero bookkeeping), spt.exe by **live-seeded descent ∩ image** (seeds captured while alive, machine-wide image selection still forbidden). Then the same question turned on the respelling found the residual: both halves select on image PATH, so a process whose path is unreadable is invisible — a zero that cannot see, [[husk-reads-as-default-defeats-durability]] and IR-8's open entry, third instance in one night — hence an unreadable-path bucket that makes the zero REFUSE when nonzero. Guard: before writing a proposed gate into a spec, execute it in your head against the specimen you are citing to justify it and state what it would have caught; and when someone respells your instrument, re-run the question on THEIR spelling — an instrument is hardened by its author refusing it, not by its author defending it.

Related: [[premature-closure-guards]], [[stale-snapshot-equality-proxy]], [[alarm-every-test-run]], [[husk-reads-as-default-defeats-durability]], [[unit-lane-inprocess-isolation]], [[discriminator-question]], [[zero-match-filter-reads-as-absent]], [[rig-inherits-the-defect-it-studies]].

8. **A rebuilt instrument is validated against its SEALED OUTPUT, and the seal loses to source** (added 2026-08-29, todlando's v0.65.0 post-cut emission census). The precut census script did not survive a context clear — only its three hashed output files did. Rebuilding it from a prose description and running it at the new sha would have been a *plausible* number with nothing behind it, so it was first pointed at the PRECUT sha and required to reproduce the sealed artifacts: 487/487 `file:line`+token rows, 383/383 tokens, 26/26 consumer rows, exact. **Two things that only this ordering shows.** (a) The PRIMARY KEY reproduced while a DERIVED column did not — 7 of 487 macro cells disagreed, and reading the source at the sha said the *seal* was wrong (it attributed the macro from an earlier line in the lookback window when the token's own line carried it, always a `println!("{s}")` in a neighbouring match arm). A sealed artifact is a REFERENCE that validates the reproduction; where they disagree, SOURCE adjudicates, and the seal is corrected in TEXT, never by editing a file both parties hashed. (b) A predicate could not be recovered from the prose at all — "mentions a token" gave 72 consumer files against the sealed 26 — and was recovered instead by asking which rule admits exactly the two tokens the seal scored in one file and rejects the six it did not (answer: the token must be followed by a COLON; bare matching pulls in prose like "the IDLE case"). **Recover a lost predicate from the sealed OUTPUT, not from your own description of it.**

**Why:** a rebuilt instrument agreeing with your expectations is the failure mode — it is exactly the state in which nothing was measured. Only a hashed prior reading can refuse it.

**How to apply:** never point a rebuilt generator at the new sha first. Run it at the OLD sha, diff against the seal, and treat every disagreement as an open question for the source — some will be the seal's defect and those are findings worth reporting. Also measured the same day: the sealed tokens file **had no final newline**, so `wc -l` read 382 for 383 records — the undercount the "assert the logical count in the generator" requirement was written against, still sitting inside the artifact and invisible to anyone who counts it with a shell.

3. **A LAUNCHED PROCESS THAT CANNOT WRITE IS INDISTINGUISHABLE FROM A QUIET ONE** (hertz,
   2026-09-07, arming the fleet-daemon watcher during the hfenduleam incident). I launched the
   sampler with `Start-Process powershell -File watch.ps1`. The process existed and its log stayed
   empty. Cause: `Get-ExecutionPolicy -List` shows **LocalMachine = AllSigned**, so `-File` refused
   to load an unsigned script — and the refusal went to a hidden window's stderr that nobody reads.
   A process census said ARMED; the instrument had never executed a line. Fixed with
   `-EncodedCommand`, to which execution policy does not apply.
   **How to apply:** an instrument is armed when its FIRST LINE IS ON DISK, never when its process
   exists — check for the line, not the pid. Make the first act of every watcher a write (an ARM
   line, and a pidfile), so "no output" can only mean "never started". And carry a NEGATIVE CONTROL
   in the instrument itself: mine probes a nonexistent pid at startup and logs
   `CONTROL probe of a nonexistent pid returned: NULL (detector can report DEAD)` — chert's
   make-it-red-on-purpose rule, paid once at arm time so every later ALIVE means something.
   Corollary for long watches: log EVERY Nth poll, so a stale last timestamp is itself proof the
   WATCHER died rather than proof the subject survived.