# Golden CI: milestone-batch integration with fast-forward-only main Status: accepted (operator-ruled 2026-07-27, grill session with doyle) Per-PR full-suite CI on two single-box runners produced eight-hour merge sweeps dominated by queue wait and environmental flakes, and the merge-then-verify gap let a PR merge on a stale green (#106, 2026-07-27 freeze). We decided: PRs run a **thin lane** only (lint, unit, traceability); the full suite runs once per milestone on a **golden branch** — a merge chain of every milestone PR, assembled by a builder and **ruled by the gater** — and on green, main advances by **fast-forward to the exact tested sha**. Nothing else ever moves main; an urgent fix is a golden batch of one. “Gater-assembled” would have the gater checking his own work, which is the failure this project has already paid for. Milestone A ran the corrected shape in practice: doyle assigned assembly to todlando and retained ruling. A gate loses its independence the moment its author rules it. Drift between documented and practised procedure is a finding to fix, never leave standing. ## Considered options - **Bisection-first red handling ("inverse merge pyramid")** — the original proposal: split the batch in halves until the failure isolates. Rejected as the *primary* mechanism because this repo's measured flake rate (silent-peer: 4/30 on a quiet box before its root fix) makes half-verdicts unreliable, and each level costs a serialized full suite on single-box runners. Kept as the *last resort* behind the red protocol: names first, diagnose-and-fix against the batch, registry-matched flakes get one same-sha rerun. - **Green-then-merge-individually** — golden as a gate but PRs merge normally after. Rejected: recreated merge commits mean the final main sha was never tested, which is the exact stale-green class the strategy exists to kill. - **GitHub merge queue** — requires branch protection this repo deliberately lacks (agent-watcher enforcement), does rolling batches not milestone batches, cannot express the red protocol. - **Eject-and-proceed on conflict** — shipping the batch minus a conflicting PR. Rejected: a milestone ships whole; conflicts bounce to their author and the golden run waits. ## Consequences - Retired once the pilot proves the mechanics: sacred-baseline runs, merge freezes, the main-run concurrency-cancellation hazard, and per-PR merge choreography. - The flake registry (checked-in, test-infra-owned, entries reviewed like code) becomes load-bearing: the red protocol reads it mechanically. - A milestone's first full-suite execution of a diff happens at golden time; builders keep local gate rigs for pre-batch confidence. - Rollout: pilot on the milestone after FIELD-TRUTH W1, delivered as one CI lane bundle (absorbs the seeded concurrency-exemption + workflow_dispatch fix); old procedures retire only after the pilot's first green fast-forward. ## The silent-skip class Milestone A's golden pilot went green while the milestone's defining seam had never executed. The two-host cross-machine legs came back `skipped`, and a skipped job is **silence about its seam, never evidence**. Six distinct mechanisms produce evidence-shaped silence, and all six were present in the milestone's release path: - **Face 1 — the marker is absent.** The leg is opt-in on a token (`[twohost]` in the head commit message). Measured: it had executed **zero times across the last 40 runs**, including a release run and every main merge. From inside a run history, *standing design* and *standing oversight are indistinguishable* — the survey that shows a leg never running is the strongest available defence for concluding it was never meant to, which is precisely how the gap survived. - **Face 2 — a trigger-impossible expression sits in a GATE position.** `contains(github.event.pull_request.title, …)` in `golden.yml@6bf2bd9`, whose triggers are `push: golden/**` + `workflow_dispatch`: `github.event.pull_request` is always null there, so the clause is unreachable text that *reads* as a live route and invites a probe designed against it. **The defect is the gate position, not the reference.** The identical expression is defensive and correct elsewhere in the same file — `ref: ${{ github.event.pull_request.head.sha || github.sha }}` at `golden.yml@ce4386e:822` resolves correctly on push via its fallback, and `PR_NUMBER: ${{ github.event.pull_request.number }}` at `golden.yml@ce4386e:833` is empty on push and guarded by `[ -n "$PR_NUMBER" ]` in the consuming script. Those stay. What cannot stand is the gate: **a gate cannot express its own impossibility** — it silently returns false and is indistinguishable from a gate that simply did not match. A fallback announces its own absence; a gate does not. `workflow_dispatch` with no declared inputs is the same trap: `head_commit` and `pull_request` are both null on that event, so every `contains()` gate is false and dispatch is not the escape hatch it appears to be. - **Face 3 — the dependency skips.** A `needs:` upstream that fails, combined with a gate carrying no `always()` / `!cancelled()`. A single *bystander* unit failure on one box would have skipped both two-host legs — manufacturing the exact instrument failure the red protocol classifies as blocking, in the run that was supposed to prove the seam. **Live proof:** on golden run `30501468421`, Windows `test` failed while both `twohost-a` and `twohost-b` still existed as queued job records; `!cancelled()` preserved the seam legs through the exact upstream-red shape this ruling was designed for. Queued existence proves the gate, not the seam—the seam remains unproven until both jobs conclude non-skipped. - **Face 4 — the attribution parse finds nothing.** `ci-notify.sh` extracts a line-anchored `Co-authored by:` trailer. A commit body containing literal backslash-`n` characters leaves the apparent trailer mid-line, so the extractor matches nothing and the co-author silently receives no notification. This is the same class: silence that reads as “there was none.” A missing parseable trailer must emit a workflow warning and explicitly report the doyle-only fallback. Notification delivery remains best-effort and cannot change the gate verdict. - **Face 5 — the red exits without evidence.** Golden run `30501468421`, Windows job `90742055246`, reported `worker_lifecycle_e2e` as only `running 1 test` followed by exit code 1: every child outcome, assertion, and observability point sat *after* a `taskkill /PID /F /T` reaper. A hard-killed harness cannot unwind, so it cannot print a panic or assertion. Emit complete child status/stdout/stderr before any destructive boundary. A breadcrumb pid is not identity: bind the observed daemon to its creation time, then require the live pid to retain that start time and resolve to the exact expected executable; refuse self, ancestors, unreadable identity, or mismatch. A red carrying no reason is closer to silence than evidence. - **Face 6 — a competent run measures the wrong tree.** The first reassembled tip `aa487b9` sat on a sibling line from `ce4386e`, omitting the six-commit repair batch—including the `SPT_HOME` isolation repair and the two-host independence workflow that produced the seam evidence. A golden run there would likely have gone green while testing none of the repairs it purported to validate. Every other check verified *what* the run measured; none verified **which tree** it measured. The discrepancy surfaced only because falsifiable source facts disagreed: the requirement seed promised five production `spt_home()` resolutions in `daemon_inhibit`, while the candidate carried the old three-call shape. This is provenance of the subject, not competence of the instrument. ### Rulings 1. **A load-bearing seam is never opt-in on the golden branch.** Golden legs are **default-on**, with an explicit `workflow_dispatch` opt-*out* reserved for intentionally isolating other evidence. Omission must not be able to skip a seam; only a deliberate, recorded act may. The mechanism in force: `if: ${{ !cancelled() && (github.event_name != 'workflow_dispatch' || inputs.twohost) }}` — on a `golden/**` push this reduces to `!cancelled()`, so the legs run unconditionally; on dispatch a typed `twohost: boolean, default: true` governs. This makes face 1 *structurally impossible* rather than merely unlikely. 2. **A `needs:` used as a rendezvous barrier must say so with `!cancelled()`.** Where a dependency exists only to release coordinated roles together (each self-hosted box has one slot, so an unguarded `needs:` lets one role starve behind the other machine's leg), the gate must make the seam evidence independent of the suite's result. An unrelated red must not be able to erase it. 3. **A dead clause inside a live gate is a booby trap — delete it, do not leave it.** It documents a route that does not exist and will be designed against. 4. **Verdicts come from the job list and its conclusions, never from a notification.** An absent or `skipped` job is silence, not a pass. The run-level `success` is compatible with every face above. Missing attribution must likewise be loud even though notification remains best-effort: best-effort delivery is a promise about the channel, never a licence for the producer to fail quietly. 5. **A milestone's defining seam must have an *executing* leg in the tree that ships.** Prefer probing the tree that ships over a marker-commit probe of an earlier tree: a probe of the pre-repair tree can be skipped by the very defect under repair. 6. **After anything touches a workflow — including your own fix — re-read every verification procedure written against it.** The repair that closed face 1 retired the marker token as a gate, which silently turned “confirm the token is present” into a check measuring a mechanism that no longer existed. Both the gate's author and its verifier had written procedures against the old instrument. Quote conditions as `@`, and read both the workflow and requirement registry at the sha under test, never in a working checkout. A stale local ref is a shared instrument, so its staleness is a fleet-wide defect rather than a personal slip; comparing against a branch name hides that defect. Prefer an explicit sha, or verify the ref before using it. Treat a workflow moved between files as a *condition migration* — re-verify triggers, not just the `if:`. **A grep proves a string is present; a parse proves the file the runner will read has the structure you think it has.** 7. **Where a gate has several ways to silently not fire, two independent checkers verify it and a DISAGREEMENT between them is itself the finding.** A check with a single owner is not a mechanism. 8. **Verify the subject before trusting its evidence.** Before pushing a golden candidate, assert the ruled repair tip is an ancestor, then independently assert file-level facts that characterize the intended tree; ancestry alone cannot catch a bad cherry-pick. Use explicit blob hashes for load-bearing files and workflow conditions read at the candidate SHA. Literal source counts must name a fixed-string predicate and its tool—for this incident, `grep -F -c 'spt_home()'`. A regex dialect where `()` is a group would silently match bare `spt_home` and return 10 instead of the intended 5; POSIX BRE treats the parentheses literally and returns 5, so a reported count without its tool and dialect is not reproducible evidence. A falsifiable count or hash can trip; a narrative cannot. A future workflow lint should derive each workflow file's declared trigger set and reject unreachable `github.event.` references in `if:` positions while allowing the same reference behind explicit fallbacks and presence guards. It must inspect the whole workflow after any gate edit, because repairing one instance does not repair sibling instances of the mechanism. The lint remains an inactive requirement seed until its delivery milestone. ## Retired-host side-effect class This is distinct from the six silent-evidence faces: **a mechanism was removed and a responsibility it discharged only as a side effect was not re-homed.** The dedicated release PR used to advance `main` when merged. Golden CI correctly retired that PR vehicle, but the original procedure added no explicit replacement because “advance main” had never been written as an obligation. The resulting release sequence could tag and publish the correct signed candidate while `origin/main` silently lagged it, causing the next milestone to branch from stale code. The omission became visible because the release driver published his exact commands for review before running them: his first step fast-forwarded only a local branch and no later step pushed `main`. **Retired-host side-effect rule: when you retire a mechanism, enumerate its side effects and re-home each one—the work and the sentence that explains the work.** This retirement lost both in opposite directions: a responsibility vanished with its host (the release-PR merge stopped advancing `main`, and no step named the work), while a by-product outlived the sentence explaining it (the replacement fast-forward push still fires `push:[main]` CI, but the text saying what that run is for disappeared). One stopped happening unnoticed; one kept happening without a known purpose, so the same run could be read as noise or scripture. For releases, the ruled tip must be fast-forwarded and pushed to `main` before the tag is created. That push's non-cancelling policy preserves the thin CI record but does not grant it authority: golden remains the milestone's full-suite authority. Wait for both before local signing because both occupy the runner, not because the thin run blesses the SHA. A thin-run red at a golden-green SHA neither auto-blocks publication nor gets waved through; the driver stops for one gater ruling. This status is earned: at `af65ac0`, `the_spawn_environment_carries_the_cli_capability` passed in golden's full-suite Linux job and failed in main's thin Linux job, exposing a real latent process-global bystander defect. A tag points at a commit already on `main`, never the reverse. A third paid precedent completes the class in the other direction: the name `docs-publish.yml` outlived the workflow itself after its docs step moved into `release.yml`, so a release driver followed the stale runbook name and waited for a workflow that could never appear (the documentation drift repaired for PR #113). The three failures are distinct consequences of one missing retirement inventory: responsibility vanished with its host, a surviving by-product lost the sentence that gave it meaning, and a name survived the thing it named. ## Release shape under golden CI The release runbook's dedicated release-PR rule and this ADR's fast-forward-only rule conflict under golden CI. A release PR stacked after the ruled tip would put the tag on a commit no golden run tested: the runbook's own feared unvalidated release commit, reached through the other door. **Tested SHA equals shipped SHA outranks bump-in-PR.** The version bump, lockfile, generated docs, and changelog ride the milestone batch and are tested with it. Main advances only to a SHA whose golden run is **green**. “Tested” means the suite passed; an explained red is not golden evidence, regardless of whether its mechanism predates the batch. The update-set is signed, and every node's `spt update` verifies that signature: shipping on an explained red would make the signature attest to provenance grounded in a narrative rather than the gate. A signature must never attest to something weaker than the gate claims. **v0.45.0 having shipped carrying the same hazard is not precedent for repeating it — it is how the hazard became known.** Release-shape independence remains substantive but moves out of band: at the ruled tip, the release driver records an audit that decodes the counter from the last published signed metadata, audits the changelog against the actual commit range rather than the milestone narrative, proves update-set compatibility constants unchanged at that SHA, and justifies the bump level from observable behavior changes. There are **no post-golden edits**—not a changelog date or rider line. After the run, the tree is frozen. Any delta requires a new batch commit and rerun; otherwise ship exactly what was tested. ### Amendment, 2026-08-02 (operator-ruled): the absolute narrows to a provable-delta exception **Struck:** the sentence above read as an absolute — *any* delta requires a rerun. **Replaced by:** post-golden edits are refused **by default**, and the refusal narrows only for a delta whose defect provably sits outside what golden validated. **Tested SHA equals shipped SHA** is unchanged as the governing principle for everything golden actually validated. What was over-broad was treating every byte of the tree as validated. Golden's verdict rests on what its gates inspect; a delta no gate reads cannot invalidate a verdict that never rested on it, and re-running the suite against a byte-identical compiled tree re-purchases evidence already held. The rule as written therefore charged a full window — both self-hosted test legs, both n1-gates, both two-host legs — to confirm a result it already had. The amended form: **tested SHA equals shipped SHA, or a provable-code-free delta from a tested SHA.** **The provability bar, which is the whole of the safety.** The delta must be *provable by diff, not plausible by argument*: 1. A name-only diff confined to non-code — `git diff --name-only .. | grep -c '\.rs$'` returns 0 — inspected, not summarised. 2. **Plus** the compiled-in exceptions checked by measurement. Version material is the standing one: its diff looks like prose, but it is compiled into the binary, and version-sensitive gates exist (the embedded-anchor release verify, the update-set counter and rollback gates, the broker's compiled-image report). Reasoning that these "should not move" is exactly the argument this bar refuses. **A delta that needs an argument rather than a diff is not eligible.** It buys the full window. Worked example, v0.51.0 (counter 86, `6312b01`): the tested head `65d2c98` carried an all-nine-jobs-green run; the candidate added a doc-prose fix, a one-line REQ doc-stage activation, the version bump and the changelog section — five files, zero `.rs`. The compiled-in exception was measured rather than argued: 92 version/anchor/update-set/manifest tests run green on the *bumped* tree, including `release_verify_e2e published_release_verifies_against_embedded_anchor`. Thirteen seconds of targeted tests in place of a fifty-minute suite, with the signature attesting to a tree whose code was gate-proven and whose delta was diff-proven. **Why this is not the hazard the original rule guarded.** That rule exists so a signature never attests to something weaker than the gate claims. It still does not: the code the signature covers carries a green golden run, and the part the run did not cover is proven inert by inspection rather than narrative. The failure the rule prevents is a tag pointing at a commit whose *behavior* nobody tested. A provable-code-free delta produces a commit whose behavior is byte-identical to one that was. **Retirement inventory (this ADR's own rule, applied to this amendment).** The absolute was also doing a second job: it discouraged post-hoc tinkering of any kind, including edits that look harmless but are not. That job is re-homed onto the provability bar rather than dropped — the bar refuses precisely the "surely this is harmless" class, because it demands a diff and a measurement instead of a judgement. Sibling instance re-homed with it: `docs/RELEASE-RUNBOOK.md` step 1 carried the same absolute and now carries the same amendment.