# Golden CI operating protocol Builder pull requests run `.github/workflows/ci.yml`: lint, unit tests, and requirement traceability only. The gater assembles every milestone PR into one `golden/` merge chain. `.github/workflows/golden.yml` runs the full cross-platform suite on that branch. Main advances by fast-forward only to the exact green golden SHA. An urgent fix is a golden batch of one. ## Dispatch and concurrency Pushing the assembled `golden/**` branch starts the full suite, including both coordinated two-host jobs. `workflow_dispatch` is the explicit rerun/manual avenue and must target the same candidate ref; its `twohost` input defaults on and is an explicit opt-out only when isolating other golden evidence. The two-host jobs wait for the suite as a rendezvous barrier but run after any non-cancelled suite result, so an unrelated red cannot turn seam evidence into a dependency skip. Verify two-host evidence from the job list and conclusions—an absent or skipped job is silence, not a pass. Golden runs never cancel in progress. Thin runs may cancel superseded work on the same non-main ref; main evidence is exempt from cancellation. ## Red protocol 1. Record the exact candidate SHA, platform, job, and failing test id. Do not rebuild or amend the candidate. 2. Diagnose the named failure against that batch first. Fix a deterministic product or test defect on its owning builder branch, rebuild the golden chain, and run the new SHA. 3. A failure may use the flake avenue only when `.github/ci/flake-registry.json` has an active entry matching the exact test id and platform. Every entry must contain a prior green same-SHA rerun, its run URL/id, and a concrete retirement condition. `.github/ci/flake-registry.py` validates these fields mechanically. 4. A registry match permits exactly one `workflow_dispatch` rerun of the unchanged SHA. Green clears that named failure for the batch. Red again blocks the batch and immediately invalidates the flake assumption; diagnose or fix it. 5. An unregistered failure gets no automatic rerun. If diagnosis cannot name the failing PR or shared interaction, bisect the assembled merge chain as the last resort. 6. Preserve both run URLs in the gate record. Never treat a green run of rebuilt bytes as evidence that the original red was a flake. Registry entries are test-infrastructure code. Review additions and removals like workflow changes. Retire an entry as soon as its condition is met; a registry is not a permanent ignore list. ## Step timings leave as an artifact Every wall-clock-bearing step in `golden.yml` is wrapped by `.github/bench/wrap.sh` (bash legs) or `.github/bench/wrap.ps1` (Windows legs), which append one JSONL row per step and propagate the wrapped command's exit code unchanged. Each job uploads its rows as a `bench-ledger-*` artifact; nothing is committed, and nothing gates on a duration. Read them with `cargo run -p xtask -- bench-report` at milestone close-out, where a slow step or a step that ran twice on one input key becomes a filing. Sub-second guards (free-space preflight, census/reap, temp sandbox, selftests) are deliberately unwrapped — a report that averages them beside a thirteen-minute phase says less, not more. ## Shared-runner quiet-window predicate A shared box is quiet only when **both** conditions hold: 1. zero non-terminal CI runs; and 2. no local `cargo`, `rustc`, or `cargo-nextest` process alive on the box. A GitHub-only drain check reads clean while another agent is mid-local-build. That is the contention class that muddied the g6 evidence twice. These axes are proxies for contention, not a definition of quiet; a proxy returning “quiet” is not itself a measurement of quiet. ### Queue status is not occupancy `gh run list` status is not an occupancy probe. A run can report `queued` while its job is already executing on the box; this was observed on #121/#122 with `Runner.Worker` and `cargo-nextest` live and a suite mid-flight. Therefore `queued` never means “not started,” and an idle-looking queue is not evidence of a deaf runner. The authoritative occupancy signal is the process table. `Runner.Worker` means a job is running on this box. `cargo-nextest` means a suite is running, which is the seedmap-starvation collision. ### Attribute process load before counting it The two predicate axes are not independent: a CI job manifests in the local process table as `cargo` and `rustc`. Their presence does not imply a second occupant. Walk each PID's parent chain before counting it as new load: - a chain rooted at `Runner.Worker.exe`, `Runner.Listener.exe`, or `RunnerService.exe` is the CI axis already counted; - a chain rooted at a user shell is a genuine second occupant, so hold. Owner queries alone are insufficient. The runner service account can return empty owner and command-line fields; use the parent chain. Report attributed load, not bare process presence: “live local cargo” and “box free of a second occupant” can both be true. ### The link is a third axis, and it is measured rather than assumed Both axes above are process-shaped. A box can be quiet by both and still be a bad place to run a two-host rendezvous, because neither axis can see the link. Run 30771155390 is the specimen: zero non-terminal runs, no local build, and a link so degraded that a 1-second checkout took 321 seconds with bidirectional 10-second QUIC dial timeouts. Its `PUMP_PEER_FAIL` counts moved 0→3→0 on one host and 8→22→8 on the other across green/red/rerun, and the red was triaged as if it were a product failure because nothing in the run recorded what the network was doing. kitsubito's only link is wifi, indefinitely (operator-confirmed 2026-08-02). The link cannot be hardened, so the predicate has to see it. Both two-host jobs therefore probe the peer immediately before rendezvous, via `.github/bench/link-probe.sh` (Linux) and `.github/bench/link-probe.ps1` (Windows), and each writes its result into the run's bench ledger. Read the probe's line like this: ``` LINK peer=kitsubito ip=100.98.197.12 path=direct samples=5/5 min=10ms med=15ms max=83ms rtts=10,12,15,81,83 ``` - **`path=relay`** means tailscale fell back to DERP. That is a different network than the one the rig's timings were tuned on, and it is the first thing to check when a rendezvous is slow rather than broken. - **`samples=n/5` below full** is loss. A rendezvous over a lossy link produces timeouts that look exactly like a product wedge. - **`max` far above `med`** is jitter, and jitter is the failure shape here — the motivating run's link mostly worked. A median alone would have read healthy through it, which is why both numbers are carried: the ledger gets `link-rtt-med-` and `link-rtt-max-` rows, so the spread is a trend across runs and not one number in one log. - **`probe=NO-REPLY` or `probe=UNAVAILABLE`** is a labelled hole, not a silence. The `ok:false` row says a measurement was attempted and did not land, which is a different claim from a run that never probed. - **`ip=` must match that job's `SPT_TWO_HOST_PEER_IP`.** The probe names its peer and the rig addresses one; a disagreement means they are not talking about the same box. The probe is an instrument, not a gate: it always exits 0, and no threshold refuses a rendezvous today. Whether a number should is a ruling nobody has made, and an instrument that reds a run converts "the link was slow" into "the suite failed" — the exact confusion it exists to remove. ### Post-restart daemon churn is a known blind spot The fleet daemon's own resume storm—reconnects, pulses, and disk I/O for minutes after restart—is real load that neither predicate axis sees. This is an existing 2026-07-27 g6 finding with an existing mitigation, not a new discovery: a timed-window run adjacent to a daemon restart records the fleet-`spt` CPU-seconds delta sampled before and after, so contention is measured rather than inferred. Recording a rediscovery as a discovery loses the mitigation someone already paid for. The preserved instruments are `.github/ci/g6-postbounce.ps1`, for the immediate post-restart sample, and `.github/ci/g6-curve.ps1`, for the private-daemon age/CPU dose-response curve. Both require caller-supplied scratch and fixture-worktree paths; neither embeds an expired agent-session scratchpad. The curve's throwaway `SPT_HOME`, home-hashed IPC socket, `daemon run`, and PID-scoped bounce are deliberate safety boundaries: `daemon start` or `daemon stop` may reach the fleet service and are prohibited in this rig. This is a second reason never to restart a wedged daemon merely to tidy the box before a run: that trades a known abnormality for an unmeasured one. ## Windows ad-hoc rigs that outlive the tool call Use `Start-Process` with **`-WindowStyle Hidden -PassThru` and no `-RedirectStandardOutput` / `-RedirectStandardError`** when a Windows rig must outlive its invoking PowerShell tool call. Give the driver durable output paths that **the driver opens itself**; its lifetime must not depend on the caller's output handles. This is launch discipline, not a new helper or a change to CI, service hosting, or product daemons. Qualify the boundary with a harmless, bounded child in the **actual tool host** before starting expensive work: 1. Record the child PID and process birth, and a child-written start event. 2. Let the launching tool call return. In a later call, prove that the **same child** is alive and writes another durable event after that boundary. 3. Finish the child, retain its native exit result and output, and close owned handles. A returned PID, hidden window, or start-only receipt is not completion. A missing finish/refusal record is incomplete execution evidence, not a pass. **Origin and limits (IR-137, YPLFG3CG).** Doyle reported the #304 S3 Windows attempt-2 launcher at2026-09-15T07:17Z under the **Claude Code PowerShell tool host**: redirected launch PID47560 left only a start line and no refusal file; `rustc` died with `0xc000013a` (`STATUS_CONTROL_C_EXIT`) seconds after tool-call return. The same driver, hidden with no redirects (PID37536), ran25minutes and exited0. No tracked redirected launcher exists; `.github/ci/g6-curve.ps1` already uses hidden/no redirects and is intentionally unchanged. The bounded2026-09-15 discriminator under **OMP `functions.bash` → pwsh** observed **both** redirected and hidden/no-redirect children survive a normal caller return, write `AFTER_CALLER_RETURN`, and exit0 after a finish marker. Native handles were pinned before return and process births verified; all children finished and handles closed. This does **not** reproduce or discharge the Claude Code failure, nor establish that redirects are safe in another host. An initial15s observer-handshake timeout is excluded from the valid two-arm result and preserved as such. Receipt and reproduction scripts: `.spt/preserved/hertz-ir137-caller-return/receipt.json`, SHA256 `e1c9b207544e6313eac5421ebb692786ec18b3a165756b451bff3a704899b178`. The receipt links per-arm native results, durable events, script hashes and cleanup evidence. It is not a cargo/test-suite run. ## One-shot builds set `CARGO_INCREMENTAL=0` A build whose `target/` is thrown away afterwards never reads the incremental state it writes. Gate rigs, fresh `.worktrees/` lanes, bundle deploys, and any `rm -rf`-then-clone cache are that shape, and there the setting is free money: **-29.8% wall and -5.65 GB of target, n=3, with the slowest OFF run beating the fastest ON run** (releases#103 leg 3). Prefix the rig's build and test commands with it: ``` CARGO_INCREMENTAL=0 cargo nextest run --workspace … # bash $env:CARGO_INCREMENTAL = "0" # pwsh, before the lane's builds ``` The disk half is not a footnote on this box: `.github/workflows/golden.yml` enforces a 32 GiB free-space floor and an exhausted disk surfaces as `LNK1318`, not as a disk message. **Scope it to one-shot, and say why when you don't.** Incremental exists to make the *next* build of the same tree cheap, so the setting is a straight loss anywhere a target dir survives to be rebuilt. The CI runners' `_work` target is exactly that — persistent and warm across runs — so the golden legs deliberately keep incremental on. The one CI exception is the n1-gate's pinned old-broker cache, which is deleted and rebuilt when the pin moves and consumed as a binary in between; it carries the flag and a comment saying so. Do not apply it to a warm tree because it helped a cold one; the measurement was taken on cold builds and does not transfer. ## Lock-touching lanes A lane that changes `Cargo.lock` moves one crate on purpose and proves it moved only that. Four steps, in order. **1. Update by name.** `cargo update -p ` states the intent in the command. A bare `cargo update` re-resolves the whole graph, so a one-crate intent arrives as a graph move nobody reviewed; when a wider move really is wanted, name it as the lane's subject rather than acquiring it as a side effect. **2. Read the diff twice — blocks, then edges.** Count the changed `[[package]]` blocks, then diff each surviving block's dependency lines. The second read is the one that pays. At `acaaa4f` the package set was identical — no block added, none removed, not one `name` or `version` line in the diff — while eight blocks had their `windows-sys` edge moved from 0.61.2 to 0.59.0 (consumer counts 18→10 and 5→13, with both version blocks still present, so a version-presence check also reads clean). That commit's own message says it "changed one package and left 132 dependencies untouched": true of the set, false of the edges. A reviewer who counts blocks is told nothing happened. **3. Two resolutions that disagree are evidence about toolchains, not dependencies.** If re-resolving the same manifests locally produces a different lock than the one on the branch, stop and compare toolchains against CI — golden prints `cargo`, `rustc`, and `clippy` versions on both OS legs for exactly this comparison. Ship a lock after the versions are reconciled, not by picking the diff that looks tidier. **4. A hand-edited lock is fine when `cargo check --workspace --locked` passes.** Editing the file directly is a legitimate way to land a minimal delta; `--locked` is what turns the check into evidence, because without it cargo repairs the lock it was asked to verify and reports success about a file that no longer matches what you are shipping. Then run `cargo run -p xtask -- check` on the lane. It asserts every load-bearing `[patch.crates-io]` pin is still in force in both ways such a pin lapses — the stanza deleted, or the lock re-resolved back to the registry — and prints, on refusal, why that patch is load-bearing and the condition under which it may be dropped. ## Rig disposal Disposal is a rig step and runs **last**: sweep survivors, remove the owned target, then report free bytes. Never perform a multi-gigabyte deletion under a live CI job. Worktree and target bloat are a CI failure mode, not untidiness; `.github/workflows/golden.yml` enforces a 32 GiB free-space floor. A survivor sweep is scoped by executable path under the rig's own throwaway target, never machine-wide by process name. Concretely, require `Path.StartsWith($TARGET)` before reaping. Installed fleet binaries under `AppData\Local\spt-core\bin` are then unreachable by construction, so another agent's live repro survives the battery and disposal without relying on memory. Before classifying an unregistered file as junk, read it and identify its producer and purpose. Stray files are where unowned diagnostic instruments accumulate; path shape alone is not evidence of disposability. Preserve or promote paid-for instruments before removing the containing worktree.