# Release runbook
> M6-D6/D7 (ADR-0015). How a release ships. CI builds; the maintainer signs
> locally — release keys never enter CI.
## One-time setup (done 2026-06-05)
- Key ceremony: `cargo run -p xtask -- release-keygen rel-primary-2026` (and
`rel-recovery-2026`); both seeds in the password manager (+ separate paper
backup); both PUBLIC keys embedded in `BUILTIN_RELEASE_KEYS`
(`crates/spt-daemon/src/release.rs`). The recovery key is never used until
the primary is lost/compromised — then it signs the next release, which
rotates in a fresh primary.
- `RELEASES_TOKEN` repo secret on spt-bs-core: fine-grained PAT, Contents
read/write on `BigscreenVR/spt-bs-releases` only.
## Per release
### Preflight — derive the seam set, never recall it
Before handing a candidate sha to the gater, the release driver preflights it.
The suite is the usual one (workspace clippy, the touched crate's lib tests,
the seam integration binaries, `traceable-reqs check`) — but **how the seam set
is chosen is the part that goes wrong**:
- **Derive the seam binaries from the changed symbols, mechanically.** For every
function the change adds, wraps, or alters, grep for its callers and for the
tests that drive it:
```sh
git diff --name-only ..HEAD -- '*.rs' # what actually changed
grep -rl '' crates/*/src crates/*/tests # who exercises it
```
Run **every** test binary that grep returns, plus the binaries for the seam
those callers sit on.
- **Never hand-list the seam set from memory.** A remembered list is a snapshot
of the siblings that existed when it was memorised; it goes stale silently as
tests are added, and it fails in exactly the case that matters — a sibling
that drives the changed seam through a different fixture.
- **Why this is here (paid twice):** the v0.40.0 observability rider wrapped the
broker's subscribe-decision function, adding a synchronous log write on the
decision path. A hand-listed preflight set (broker, controller lease,
redispatch, resize, attach-truth) came back fully green; the gate then went red
on `brain_decouple` — a sibling driving the same subscribe/take seam, absent
from the list only because the list was recalled rather than derived. A grep
for the wrapped function's name would have returned it immediately. The same
shape had already been paid once before, in v0.8.2.
- **A failure inside a seam the change touched does not get filed as a load
flake on one observation.** Repeat-proof it (looped repro, A/B against the
pre-change sha, under comparable load) before either dismissing it or fixing
it. "Rerun went green" closes nothing here.
### Golden-head intake
The gater hands the assembled head to the release driver, who verifies it
**before** golden runs — the check gates the run, not just the merge.
- **Greenlit-form parity.** The head must contain and fulfil exactly the
requests the operator greenlit. Take the baseline **as a snapshot at
greenlight**, not reconstructed at hand-off: record the milestone's live
sub-issue set the day it is greenlit and check drops and adds against that
list. v0.46.0's milestone shipped omitting four operator-greenlit requests
unexplained, which is what reconstructing-from-memory buys. A drop needs a
reason comment on the milestone issue *and* relocation-or-back-to-eval before
the run; a conversation or a commit message does not discharge it.
- **The head must already be release-shaped — and this is a CHECKED leg, not a
preference.** Bump, regenerated docs and changelog section ride *in* the
candidate (step 1). A code-complete head is not a release candidate. Render
`spt --version` from the head's own binary and believe the render: a head
reporting the previous version is the disproof, not a detail to explain away.
Run it as two commands against the candidate sha, and record the answers in
the intake comment beside the other legs:
```sh
git show :Cargo.toml | grep -m1 '^version' # must be the version being cut
git show :CHANGELOG.md | grep -m1 '^## \[' # must be `## []`
```
**Neither answer is inferable from the other legs.** Ancestry, patch-id, blob
fidelity, greenlit form, `traceable-reqs`, docs-drift and clippy can all come
back clean on a head that is a version behind — every one of them did at the
v0.57.0 intake, which is how an unshaped head reached a GO with a pinned tag
sha. Tagging such a head fails `release.yml`'s changelog extraction and ships
binaries rendering the previous version.
**Expect the head NOT to be shaped; the documented default is the minority
practice.** Measured over the four cuts to v0.57.0, the bump rode inside the
golden candidate exactly once:
| Cut | Tagged commit | Shape rode in the candidate? |
|---|---|---|
| v0.54.0 | `86f0d84` "bump, and the changelog the tested head never carried" | no — added after golden |
| v0.55.0 | `0a25b77` "bump workspace to 0.55.0 … onto the tested tree" | no — added after golden |
| v0.56.0 | `60d74ea` "v0.56.0 shape onto the respin head" | **yes** — folded into the respin head |
| v0.57.0 | `a0f9ecd` "release: v0.57.0 — version material only" | no — added after golden |
**Measured continuation (through v0.65.0):** v0.58.0 rode the provability-bar
route again (golden tested a head carrying 0.57.0); then the norm flipped and
held for **seven consecutive cuts, v0.59.0 through v0.65.0** — each tag IS a
golden push-run tip, verified per cut at the exact tagged sha (v0.59.0
`c62904e7`/run 32482048369, v0.60.0 `517c9f6f`/32631346858, v0.61.0
`618a35dc`/32715787327, v0.62.0 `12ab4a7a`/32796133406, and v0.63.0–v0.65.0
the same property; v0.57.0/v0.58.0 are NOT any golden tip, consistent with
their provability-bar record). v0.59.0 was shaped **before** the run by
construction — ruled sha == main tip == golden ref == tag, one object with
four names. Two shapes of the norm since, and the difference is
load-bearing: v0.62.0/v0.63.0 authored the shape ONCE and let fixes land on
top of it (shape-on-top, surviving one and three respins); v0.65.0 authored
the shape **fresh on every respin head** — four separate shapes, none
carried forward, each intake running an explicit not-an-ancestor check
against every previously discarded head. **A respin RE-SHAPES rather than
inherits**: a respin quietly built on a discarded shape carries the bump
with it and passes every leg *including the shape leg*, since it genuinely
reads the right version — the not-an-ancestor check is what catches that,
and it is part of this leg, not an optimization.
**Make it construction, not discipline (IR-54):** the gater's assembly
checklist asks the release-shaping question **at intake** — "is this head
release-shaped for the version being cut?" — and when the answer is no, the
release driver authors the version material on top of the assembled head
**before pushing the golden ref**, never after the run. The provability-bar
route in step 1 remains the fallback for a shortfall discovered post-golden;
it is no longer the ordinary outcome, and treating it as ordinary is how five
of six cuts each argued a post-golden delta inert. What is never acceptable
is discovering it after the tag.
- **Name the never-executed cells before the run.** Ask of the head: which
test cells in it have never executed in CI at this depth? A lane that
deferred its int evidence to the gate ships cells whose CI denominator is
ZERO, and a red on one is structural-until-shown-otherwise wearing a
flake's face — no rate argument, no same-sha rerun, mechanism first.
Pre-declaring the list turns that triage from an RCA into a lookup: both
reds of v0.56.0's first golden (run 32205195794) were first-ever-executed
cells, ruled rig defects only after each got a from-scratch mechanism read
(#181 comments 5336602572/5336611544/5336764652), and the KNOCK-169 ladder
reds the same week were first-executions too. The gater compiles the list
at milestone intake; the release driver re-checks it at hand-off.
**The list is a hand-off ARTIFACT, not an understanding (IR-54).** The
hand-off message itself carries it, beside the greenlit-form delta: which new
cells' first CI execution is this golden, and where each HAS executed so far
(lane gate / assembled head). A hand-off arriving without the list is a
hand-off defect the driver bounces back, not a gap the driver fills by
asking — PORTER's hand-off omitted it and the driver had to ask, which
worked only because the driver knew to.
**Name the changed seam's CONSUMERS beside it (#272 r1, 2026-09-08).** The
never-executed list enumerates cells the batch ADDED; it cannot see a
pre-existing cell whose denominator went stale because a composer under it
changed. The first golden of #272 (run 34239258523) went red on three `spt`
e2e cells that had last passed at W1: W2 changed what every message envelope
carries (its own commit message predicted "cells that assert a raw spool
will red, and each one gets repinned"), nobody re-ran the `spt` e2e crate
after it, and the thin-CI unit job runs libs only. The second hand-off
artifact is therefore: for every `pub fn` the batch's diff touched in a shared
composer/renderer/parser, which test crates drive it, and the sha of each
crate's last full execution (from the preserved raws, the same source the
never-executed list reads). A crate whose last full run predates the change
is run in full on both OSes BEFORE the head is handed off — the gater's
targeted legs are a claim that the untargeted cells cannot see the change,
and a commit that predicts reds has refuted that claim in writing.
**The push run IS the golden run.** Pushing the head to `golden/**` runs the
full suite; dispatching afterwards runs an identical second one, and per-ref
concurrency (`cancel-in-progress: false`) makes them serial, so the duplicate
costs a whole window. The mechanism is `golden.yml`'s two-host condition —
`!cancelled() && (github.event_name != 'workflow_dispatch' || inputs.twohost)`
on the `twohost-a` and `twohost-b` jobs — where a `push` satisfies the left
disjunct and runs both two-host legs unconditionally. The input only ever
*subtracts*, and only on dispatch. **A workflow edit that breaks that property
breaks this rule**, so grep `golden.yml` for that condition TEXT on both twohost
jobs and re-read it before trusting the property. Cite it by text or job name,
never by line number: an earlier revision of this paragraph said "lines 598 and
677", the riders landed on main pushed the condition to 845/958, and a reader
following the stale numbers lands mid-way through an unrelated job and concludes
whatever is there (deployah, golden-head intake for #145, 2026-08-04 — the
property held; the citation had already rotted). Pin the push run's id at push time
(`gh run list --commit --event push`) and gate on that id; dispatch
survives only for a same-sha rerun with no new push, and for deliberate
`twohost=false` evidence isolation.
**Pass that `` in FULL — all 40 characters.** `--commit` matches only the
complete sha: given a short one it exits 0 and prints NOTHING, which is the same
output as "the workflow never fired for this commit". Measured by controlled
variation at one sha, seconds apart, on a run that was already `in_progress`
(deployah, v0.67.0 golden, 2026-08-30): `--commit da71b785` returned empty while
`--commit da71b78521b425a23dfa55fb264445c256fb2996` returned
`33296634901 push golden`. Only the sha LENGTH varied. Every other git command
in this runbook takes a short sha happily, so the habit is built in and the false
read is expensive: "golden never triggered" invites a re-push or a dispatch, and
`cancel-in-progress: false` makes the duplicate run SERIAL behind the real one —
a whole window. Use `git rev-parse [` to produce the id, and when any
filtered run-list comes back empty, list UNFILTERED and look for the branch and
sha before concluding the workflow did not fire. An empty filtered list is a
statement about the FILTER until you have varied it.
#### The respin test — a defect found on an assembled head
Two questions. Answer them **separately** and in this order. Q2 never feeds Q1.
**Q1 — RESPIN (cost): did the golden verdict REST on the defective artifact?**
Classify by **where the defect lives**, never by which files the fix touches.
- **The refusal is the default.** If the defect sits in something golden
validated — code, build config, tests, CI config, gated docs — the head's
evidence is invalidated. Respin: the head is rebuilt and the fix rides the new
sha. `tested SHA equals shipped SHA` demands it, and the window is owed
because the evidence for the corrected tree does not exist yet.
- **The shortcut is the exception, and it must be earned.** If the defect sits
outside golden's evidence, the head ships; the fix re-homes and buys its own
evidence on the head it lands with. This is **never** "merge untested" —
everything still lands via a future golden head, ff-only. Re-homing changes
*which* head carries the evidence, never *whether* evidence is bought.
**The provability bar.** The delta must be **provable by diff, not plausible by
argument**: a name-only diff confined to non-code (`git diff --name-only
.. | grep -c '\.rs$'` returns 0), **plus** the compiled-in
exceptions checked by measurement rather than reasoned about. Version material
is the standing exception — it is compiled into the binary even though its diff
looks like prose. Worked example, v0.51.0: delta of five files and zero `.rs`,
with 92 version/anchor/update-set/manifest tests run green on the *bumped* tree
— thirteen seconds against a fifty-minute suite. **Cite that count with the
anchor row armed.** `release_verify_e2e
published_release_verifies_against_embedded_anchor` sits inside it and
contributes nothing to it unless `SPT_RELEASE_E2E` is set, so a bare count
establishes that the anchor was exercised only when the armed form was the one
run. **A delta that needs an argument rather than a diff is a Q1-YES and buys
the full window.**
**Arm the anchor row, and read its duration.** The row that carries this bar's
weight is `release_verify_e2e published_release_verifies_against_embedded_anchor`
— it proves a published release still verifies against the embedded two-key trust
anchor. It is env-gated, so run it armed and cite the armed run:
```sh
SPT_RELEASE_E2E=1 SPT_RELEASE_E2E_TAG= \
cargo test -p spt --test release_verify_e2e
```
**Read the elapsed time as the discriminator.** A `~1.6s`-class duration means the
published `*.release.json` assets were fetched over the gh carrier and verified.
A `0.00s` duration means the gate was unset and the row returned before doing any
of that. Both forms print the identical `test result: ok. 1 passed; 0 failed;
0 ignored; 0 measured; 0 filtered out` summary — counts, colour and filter state
are the same in each — so the duration is the only in-band signal that separates
a verification from a skip. The skip does emit
`release_verify_e2e: SPT_RELEASE_E2E unset - skipping`, but libtest **captures a
passing test's output**, so that line does not reach the terminal on a pass — it
surfaces only under `--nocapture` or when the test fails. On an ordinary run the
duration is the only signal that survives capture.
This step is written armed because the unarmed form was cited as bar-clearing
evidence at the **v0.54.0 cut** and caught on duration rather than on procedure:
`0.00s` looked wrong for a network-and-crypto e2e, and re-running armed against
the published v0.53.0 assets took `1.60s` and did the real work. The bar is
cleared by the armed run; an unarmed green measures nothing, and "nothing" has no
failure mode.
**Q2 — SEVERITY (quality, independent of Q1): does shipping this mislead a user
or misstate shipped behavior?** If yes, it gets a board home before the arc
proceeds, sized explicitly as user-facing correctness rather than tidying. If
no, ordinary backlog. Severity decides urgency and sizing; it never decides
which sha carries the fix. Collapsing the two is the original defect: v0.51.0's
stale `knocking.md` passage mattered (Q2-yes) and was code-inert (Q1-no), and
reading the first answer as the second nearly bought a second full window for a
delta no gate reads.
#### A superseding ruling must land where the reader looks
A dated ruling is evidence of what was true **then**. When it is overtaken, the
superseding ruling must land where the reader will look — at the latest, before
any actor carries the old sentence into a plan.
Worked example from the v0.51.0 arc: `releases#71` comment `5153399070` read
"this gates PUBLISH", the operator subsequently cleared the acceptance to run
*post*-publish, and the record went un-updated for an interval during which the
stale sentence was carried into the release plan twice. The instrument that
caught it was checking `created == updated` on the comment and confirming no
later comment superseded it; the repair was asking the record's owner to correct
it (comment `5155301974`) rather than editing around it. Before acting on a
dated ruling, read its condition and check it still holds.
1. **Under golden CI, the release shape rides the milestone batch.** Put the
version bump (`[workspace.package] version` in root `Cargo.toml` and the
matching first-party `spt-*` lines in `Cargo.lock`), regenerated docs
(`cargo run -p xtask -- gen`), and changelog section (step 2) into the
assembled golden candidate. The golden run validates the exact commit that
gets tagged. **Tested SHA equals shipped SHA outranks bump-in-PR.**
**Main advances only to a SHA whose golden run is green.** An explained red
is still red: a diagnosis or a pre-existing mechanism cannot substitute for
the suite passing. The update-set is signed, so shipping on an explained red
would put the project's signature over an artifact whose provenance rests on
a narrative rather than the gate. A signature must never attest to something
weaker than the gate claims. **v0.45.0 having shipped carrying the same hazard
is not precedent for repeating it — it is how the hazard became known.**
Do not stack a thin release PR on the ruled golden tip. Its tag would point
at a commit no golden run tested—the exact unvalidated-release-commit failure
the dedicated-PR rule intended to prevent. The dedicated release PR form is
therefore retired under golden CI; its release-shape gate is preserved as a
**recorded out-of-band audit at the ruled tip**, performed by the release
driver:
- decode the next counter from the last published signed metadata;
- audit the changelog against the actual commit range, not the milestone
narrative — **mechanically (IR-28)**: list the range's user-facing
commits (`git log vPREV.. --oneline`, keep the fix/feat
class), match each against a changelog mention, and account for every
unmatched row before the run. A wave-gap rider is board-visible and
narrative-invisible: v0.53.0's notes omitted `184f2ac` (the releases#125
fix) while the release verb promoted #125 publicly as shipped — two
published surfaces contradicting until a docs-only repair. A notes pass
written from the milestone narrative never enumerates the range; the
repair path (append to the published body verbatim + `gh release edit`,
no retag) is precedent twice over, but the audit is what makes it
unnecessary;
- prove the update-set compatibility constants unchanged at the candidate
SHA; and
- justify the bump level from observable behavior changes.
This retires only the vehicle, never the substance or independent release
review. Counter freshness and release shape are established at the explicit
SHA before the golden run.
**Post-golden edits are refused by default.** Not a changelog date, rider
line, lockfile adjustment, or regenerated byte. After the run the tree is
frozen: ship exactly what was tested, or add the delta to a new batch commit
and rerun golden CI.
**Amended 2026-08-02 (operator-ruled).** The refusal above stands as the
default and the shortcut is the exception, not the other way round. A delta
whose defect sits *outside* golden's evidence does not invalidate the run
that already passed, and re-running the suite against a byte-identical
compiled tree re-purchases evidence already held. Such a delta may ride the
tested head **only** when it clears the provability bar in *Golden-head
intake → the respin test*: a name-only diff confined to non-code, plus the
compiled-in exceptions (version material) checked by measurement. A delta
that needs an argument rather than a diff buys the full window. This narrows
which edits are refused; it does not weaken *tested SHA equals shipped SHA*,
which still governs everything golden actually validated.
The older dedicated-release-PR rule applied before milestone-batch golden CI,
when that PR's own full CI could validate the tagged commit. It cannot be
carried forward into an ff-only lane whose golden run is the full-suite
authority.
**The lockfile-refresh rule is a PROPERTY, and the command is only a
vehicle (IR-54):** the refresh must touch workspace members only and leave
every third-party pin alone, **proven by its DIFF, never by counting version
strings** — `git diff Cargo.lock` must show exactly one line pair per
first-party crate and nothing else. `cargo metadata --offline` is one
vehicle that has that property; `cargo update --workspace --offline` is
another (v0.59.0 used it and proved the property by diff — 14 first-party
line pairs, 90 third-party pins untouched — ruled equivalent). Any vehicle
is acceptable exactly when its diff proves the property; no vehicle is
acceptable on its name alone.
Third-party crates share the workspace's version-number space, and the
collision has been **paid twice, both times the same way** — a third-party
crate already sitting at the version being bumped *to*, inflating the count
so a clean bump looks over-applied:
- `quick-xml` was already at `0.39.4` during the v0.39.4 cut. The count came
back 12 against 11 first-party crates.
- `aws-lc-sys` was already at `0.41.0` during the v0.41.0 cut. Same shape,
same false reading: 12 against 11, while the bump was in fact clean.
The converse direction is real but has never been paid, because the tooling
prevented it: at the v0.40.0 cut the *same* `quick-xml` pin at `0.39.4` was
now the **old** version being bumped away from, where a blind `sed` of the
old version string would have rewritten a third-party pin and corrupted it.
That did not happen only because the bump went through
`cargo metadata --offline` rather than a text substitution.
So: the count is unreliable in both directions — it reads high when a
third-party crate sits at the new version, and it cannot see a third-party
pin wrongly rewritten to the new one. The diff is unambiguous in both.
A member-scoped cargo verb is what makes the diff trustworthy: it touches
workspace members only and leaves third-party pins alone — which is the
property named above, and the diff is how each cut proves its vehicle
actually had it.
2. **Write the user-facing changelog** — add a `## [0.X.Y] - ` section at
the top of `CHANGELOG.md` (under the intro) with **Added / Changed / Fixed**
subsections. This section becomes the GitHub Release **body** verbatim
(`release.yml` extracts it via `--notes-file`), so it is the changelog every
spt user reads. Rules:
- **spt-user-facing UX only.** What a person running the `spt` CLI notices or
does differently. Name the actual commands/flags they type.
- **No internal lingo** — no requirement ids, crate names, commit hashes,
milestone/hazard codes, or under-the-hood mechanics. A reader who has never
seen the source must understand every line. This bans implementation nouns
that leak from the fix's own vocabulary — `replay`, `subscriber`, `serve
lease`, `stream lock`, `circuit breaker`, `write-deadline`, telemetry token
names (`DISPATCH_EV`), worker-pool/mutex internals. State the **observable
effect**, not the mechanism: not "replay halts at the first failed write and
the poisoned subscriber is dropped" but "a stuck viewer can no longer freeze
other sessions." An internal-diagnostics change collapses to one plain line
(e.g. "Improved logging granularity for attached endpoints") under
**Internal** — never sell the mechanism as an **Added** feature.
- **Impersonal voice — do not address the reader.** Avoid "you"/"your";
describe the software's behavior, not the person's. Say "users" only when a
subject is unavoidable. Pattern that reads well: *"Improved the stability of
X. Previously, Y."* Example: not "You keep control of a session across a
restart" but "Improved the stability of controlled sessions across a daemon
restart. Previously, session controllers could mix up and drop their
attached sessions."
- **Use the product's user-facing nouns, precisely.** Sessions reached over
`spt rc` are **attached sessions**, not "remote sessions" (they need not be
remote). Match the vocabulary a user sees in the CLI (controller, viewer,
attach, daemon), not the code's.
- **Introduce a feature or concept ONCE, by name; afterwards call it by that
name and stop explaining it (operator-ruled 2026-08-30).** The release that
first ships a feature or concept OWES THE READER AN INTRODUCTION, written in
step: **state its name, then say what it does or adds.** Name first, then
function — a description with the name withheld makes the reader guess what
they are reading about.
From that point on — in every later bullet of the SAME release, and in every
FUTURE release — assume the reader knows it. **Call it by its name.** Do not
re-explain what it does, and do not refer to it obliquely by its behaviour
instead of its name. Re-explaining pads the notes; describing-instead-of-
naming reads as vague and leaves the reader unsure whether a familiar
feature or a new one is meant.
Introducing (first release):
> **Engine room.** A shared view every agent on a node can post to and read,
> so an operator sees one briefing instead of asking each agent in turn.
Later bullets, and every release after (right):
> The engine room now keeps its briefing across a daemon restart.
Wrong, re-explaining a known concept:
> The engine room, the shared view agents post to so an operator sees one
> briefing, now keeps its briefing across a daemon restart.
Wrong, describing instead of naming:
> The shared place agents post briefings to now survives a restart.
The name used must be the **user-facing** one, per the nouns rule above and
the no-internal-lingo rule: introduce `engine room`, never an internal
codename, requirement id or crate name. A feature with no user-facing name
yet does not get an internal one borrowed for the occasion — name it in the
product first, then in the notes.
- **Flag breaking changes** prominently under Changed.
- The release **fails loudly** if the tagged version has no `## [0.X.Y]`
section — the changelog is not optional.
- **After inserting the new section, assert the version ladder is
contiguous**: `grep '^## \[' CHANGELOG.md` and check the versions descend
with no gap. Inserting the new heading with an editor whose anchor spans the
previous heading can silently *consume* that heading (v0.34.0 did this to
`## [0.33.0]`), folding the prior release's body into the new section — and
since the body is published verbatim, the release notes then carry two
releases. The grep is a two-second self-check; a candidate `xtask`/CI lint.
3. **Advance `main` to the ruled tip, fast-forward only, before tagging.**
The retired dedicated release PR used to advance `main` as a side effect of
its merge. Removing that vehicle without re-homing this responsibility would
produce a correct signed release while `origin/main` silently lagged, so the
next milestone would branch from stale code. The gap became visible only
because the release driver published the exact commands for review before
running them: his local fast-forward never pushed.
```sh
git fetch origin main
git switch main
git merge --ff-only
git push origin main
git merge-base --is-ancestor origin/main
```
The merge and push must succeed, and the final ancestry check must exit 0.
`--ff-only` refuses divergence rather than manufacturing an untested merge
commit. **A release tag points at a commit already on `main`, never the
reverse.**
Pushing `main` fires `ci.yml` at the exact golden SHA. Under a golden
milestone this run is **not a separate baseline authority**: it is thin by
design—traceability, changes, lint, unit—a strict subset of the golden run
already green at this SHA, so its green adds no coverage and must not be
waited on **as evidence**. Wait for it **as occupancy**: it holds the box,
and step 5 is local Cargo. Measured on 2026-07-29: lint 55s, unit Linux
2m53s, unit Windows 8m30s—roughly ten minutes, not a cycle.
If this thin run reds at a SHA whose golden run is green, that is a
contradiction between two runs of overlapping scope, which is information:
it does not auto-block the publish and it is not waved through. Stop and
refer it to the gater for a ruling. Precedent 2026-07-29:
`the_spawn_environment_carries_the_cli_capability` passed in golden's
full-suite Linux job and failed in main's thin Linux job at the same SHA
`af65ac0`—a real latent defect the full suite hid, not a flake. Main's
`cancel-in-progress: false` preserves this run's record **only once the run
has STARTED**: a run still queued in the concurrency group is superseded and
cancelled by the next push regardless, and leaves no record at all (zero
jobs). Measured 2026-08-05: run 30973909279 at `327f1f8` sat queued behind
`0a25b77`'s running thin CI and was cancelled at 04:03:49Z—one second after
`e8805f7`'s run entered the group (04:03:48Z), with `0a25b77`'s run still in
flight until 04:07:59Z. So under a rapid push sequence the record you get is
the NEWEST sha's run plus any run that had already begun; an intermediate
sha's thin-CI verdict may simply not exist, and its absence is supersession,
not evidence of anything about that sha. A started run's record IS preserved,
and it is still not the same claim as the record being authoritative.
**Preservation is not deference.** (Filed as IR-41 in
`docs/INFRA-REGISTER.md`, where the mechanism and its evidence live.)
4. **Tag**: `git tag v0.X.Y && git push origin v0.X.Y` — **always the
RULED SHA explicitly, never bare HEAD.** A shared checkout's HEAD can carry
post-golden commits (docs, register work) even while the merge queue is held;
bare `git tag` would pin the tag to an untested tip.
**Expect that gap — it is the normal state at tag time, not a slip to guard
against.** Filing register and docs work on `main` after a golden run is
routine practice, and holding the merge queue does not prevent it: the
commits that open the gap land *before* the hold, and they are precisely the
ones documenting the release just tested. So treat the ruled sha and `main`'s
tip as different objects by default, and **print both before pushing** — that
makes the tag a measurement instead of an assumption, and it costs one
command:
```sh
git rev-list -n 1 v0.X.Y # what the tag actually points at
git rev-parse origin/main # what a bare `git tag` would have taken instead
```
Two field instances in three releases, which is why this reads as a standing
condition rather than an anecdote:
- **v0.53.0** — tagged with local `main` seven docs commits past the golden
sha.
- **v0.55.0** — tagged at `0a25b77` while `origin/main` was already
`e8805f7`: two register commits ahead, neither golden-tested, both
*documenting* the release rather than being part of it. A bare `git tag`
would have published that register prose as the release.
This triggers exactly
one workflow — `release.yml`. Both runners build; the assemble job extracts
this version's `CHANGELOG.md` section into the release body (a missing
section fails the job — the changelog is not optional), builds the docs
bundle, and creates a **draft** release on spt-releases with
`spt-x86_64-linux`, `spt-x86_64-linux-musl`, `spt-x86_64-windows.exe`,
`SHA256SUMS`, `manifest.schema.json`, `mock-adapter.zip` and
`spt-docs.tar.gz`.
**There is no second workflow.** Docs ship as the `spt-docs.tar.gz` asset
built by that same assemble job (ADR-0036 §4, `xtask docs-bundle`) — the
`docs-publish.yml` this step used to name was **retired** when its docs step
moved into `release.yml`, and a release driver waiting for it to appear in
the Actions list is waiting for a workflow that no longer exists. The
bundle is picked up by step 5 along with the binaries and flipped public
with them; it is not in `SHA256SUMS` (that file covers the `spt-x86_64-*`
binaries the install scripts verify) — its integrity rides the signed
update-set entry step 5 builds. A draft missing the bundle publishes a
docs-less set loudly rather than blocking the binaries.
5. **Sign + publish** (local, one command):
Before invoking local Cargo, enter a quiet window: the golden run, the thin
main `ci.yml` run fired by step 3, and the tag's `release.yml` run must all
be terminal. This wait is about **box occupancy**, not the thin run blessing
the SHA: golden remains the full-suite authority. Required golden evidence
must be green, the complete seven-asset draft must be present, and any thin-
run red must have its gater ruling. Attribute any remaining Cargo/rustc
process by parent chain: a process rooted at
`Runner.Worker`/`RunnerService` belongs to an already-counted CI axis; one
rooted at a user shell is real contention and blocks signing.
```sh
SPT_RELEASE_SEED_CMD='' \
cargo run -p xtask -- release-publish \
--tag v0.X.Y --key-id rel-primary-2026 --version
```
It downloads the draft's binaries, **verifies them against SHA256SUMS
before signing**, signs each (`*.release.json` in the exact
`SignedRelease` shape `spt update` verifies), uploads the metadata
assets, and flips the draft public. `--version ` is the **monotonic
release counter** (u64; strictly increasing across releases — it is not
the semver).
**Seed handling — `_CMD` only, on any shared machine.** `SPT_RELEASE_SEED`
(raw hex) exists as a by-hand fallback for pasting from the manager, and it
is **forbidden on any machine that hosts a CI runner or has more than one
user.** Use the `_CMD` form there, so the seed is read per-invocation and
never persisted.
The failure mode is not theoretical, and it is not about the value leaking
to a person — it is about what already executes on the box. A CI runner
compiles and runs arbitrary third-party code: every dependency's
`build.rs`, every proc macro, and every test binary runs with read access
to the environment it inherits. A seed placed in a persistent environment
variable on such a machine is readable by all of it, survives reboots, and
outlives the release that needed it. Prefer a form that leaves nothing
behind.
**Recorded operator ruling — `HFENDULEAM`, 2026-07-26.** On that one host the
primary seed is stored in the **machine-scope** `SPT_RELEASE_SEED` variable,
and the operator has ruled that placement deliberate and permanent. Publish
there with the raw-env form — run `release-publish` **without**
`SPT_RELEASE_SEED_CMD` set, so `xtask` falls back to reading
`SPT_RELEASE_SEED` — and **do not remove the variable.** This is a recorded
exception for one named host, **not a softening of the rule above**: the
rule and its rationale stand unchanged on every other machine, including
every other CI-runner or multi-user box.
- **The accepted risk, named:** `HFENDULEAM` hosts a self-hosted CI runner,
so every job step it executes — each dependency's `build.rs`, every proc
macro, every test binary — inherits the machine environment and can read
the seed, and the seed survives reboots. That is precisely the exposure
the paragraph above describes. The operator owns it knowingly; it is not
an oversight and it is not news.
- **Do not re-escalate it per release.** The finding was raised at the
v0.43.0 cut (release driver stopped before signing) and ruled then.
Re-raising it at each cut changes nothing and costs a release window.
Revisiting the ruling is an operator decision, never a release-driver
block.
- **Windows practicality.** The `_CMD` path runs the command under
`powershell` and hex-decodes its stdout, so feeding it a seed that is
already hex double-decodes and panics (`seed is 32 bytes:
TryFromSliceError`) — inside `signing_identity()`, before any sign or
upload, so such a run is side-effect-free and safe to re-run.
**Probing whether a secret is set — never with `${VAR:-...}`.** That arm
expands to the *variable's value* when the variable is set, so the "check"
prints the secret:
```sh
[ -n "${VAR:+x}" ] && echo set || echo unset # correct — cannot emit the value
```
Use only the `:+` form above. The `:-` form reads like a default and
behaves like an echo.
6. **Associate the shipped requests — on the board, not on the release body.**
**This step has TWO actions and they sit on OPPOSITE SIDES of publish: the
ACCEPTANCE cascade before it, the `release` verb after it.** Publishing does
not discharge this step — it lands in the middle of it. Doing only the first
half leaves every shipped request CLOSED-but-ACCEPTANCE and the hub card
empty, and nothing downstream complains: the release is public, Latest is
flipped, every verification in step 5 passes, and the board is silently
unfinished. Measured v0.67.0, 2026-08-30 (deployah): cascade driven
pre-publish, publish verified at source, step 6 read as discharged — `#23`
and all 8 members sat CLOSED-but-ACCEPTANCE for ~50 minutes and the card read
empty until the operator reported it and the gater drove the verb. The verb
was in this runbook the whole time; it was the SHAPE that hid it, a
post-publish action filed as the last sub-bullet of a step whose title and
bulk are both about pre-publish timing. Tick the halves separately.
**The release body carries no milestone or request links** (operator-ruled
2026-07-31). It is the changelog section and nothing else. A links section
was tried on the v0.48.0 cut and stripped the same day; do not re-add one,
and do not read its absence as an oversight. The question *which requests
did this release deliver* is answered by the board, which is where the
states live anyway.
The association is one verb, with an ordering constraint on either side:
- **Before publish — the ACCEPTANCE close-cascade must already have
landed, and under golden CI it must be DRIVEN, not swept.** The alchemy
sweep reconciles per-request merge-closes, and golden CI produces none —
one ff of an assembled head closes no individual request — so the sweep
finds NOTHING to key on and a sweep-only driver publishes past open
requests (deployah, v0.56.0 cut). Drive the cascade explicitly:
`spt shell cmd alchemy-0 state acceptance` BEFORE
publish. The `release` verb gates on each request having been closed
*before* the release was published (`closedAt < publishedAt`). A request
closed afterwards is not picked up, so closing the board out is a
pre-publish step, not a victory lap.
**Two consumers read that timing, and only one of them forgives a late
close (measured v0.66.0, 2026-08-30).** The `release` verb has a GRACE
window: it took `#242` at a close 7 minutes past publish and said so —
`(late close, within grace)`. The Hub Daemon that builds the Discord
release CARD does not: it buckets strictly on `closedAt` in
`(prev.publishedAt, this.publishedAt]`. So a late close can be DONE on the
board and still land on the WRONG card, or on none. Two releases were
mis-carded this way before anyone noticed — v0.65.0 announced `0 shipped`
while its four CONDUIT members closed 56-66 SECONDS after its publish, and
those five then surfaced on v0.66.0’s card. The GitHub record was correct
throughout; only the card was wrong, which is why the repair is never a
state edit (cards recompute each cycle, so a daemon fix heals them
retroactively with zero issue edits).
Consequence for the driver, until the daemon shares the verb’s grace:
the MILESTONE’s own close must land pre-publish too, not just its
members’. Driving the cascade late leaves the milestone outside the
strict window and it will surface on the NEXT release’s card. `#242`
closed 01:50:10Z against a 01:42:43Z publish and is queued to do exactly
that on v0.67.0 — a known, filed off-by-one rather than a surprise.
- **After publish — run the verb:**
```sh
spt shell cmd alchemy-0 release v0.X.Y
```
It promotes the shipped requests to DONE and posts the Shipped Requests
roundup to Discord. The v0.47.0 roundup is the exemplar to match.
- **If the ordering slipped**, re-date the release past the closes by
toggling it through draft and back, then re-run the verb:
```sh
gh release edit v0.X.Y --draft=true
gh release edit v0.X.Y --draft=false
```
The tag, the assets and the body all survive this (verified on v0.48.0,
2026-07-31). Two consequences to weigh before reaching for it, both
observed rather than theoretical:
- **It rewrites `publishedAt`.** v0.48.0 moved from `10:16:13Z` to
`12:35:31Z` this way, so the release page's date is the *repair*
moment, not the moment the artifacts were signed and published. Cite
the signing time from your own record when the two disagree — the
release page is no longer authoritative for it.
- **While it is a draft the release is not the latest.** GitHub excludes
drafts from `releases/latest`, so a client asking for the latest
release in that window is told the *previous* version. Keep the two
commands adjacent, and never toggle while a rollout or a field-verify
leg is in flight.
**Publish is not the last board action.** The `release` verb above still
owes the DONE promotion and the roundup; a release whose requests never
promote looks finished from every angle except the board and the card.
A request that rode in **reduced scope** still records where the remainder
went — but on the milestone issue, as the golden-head intake step already
stipulates, not here. The board carries the association; the artifact
carries the user-facing notes. Neither borrows the other's job.
7. **Teardown: preserve FIRST, release SECOND — as two messages, in that
order (ruled doyle + deployah, convergently, 2026-08-29).** When any hold
protecting an artifact is about to lift — an instrumented tree, an
unlanded diff, a specimen the arc named worth keeping:
- **Preservation is its own step with a NAMED OWNER**, completed and
confirmed on the record by path and hash *before* the release call goes
out. The pools-released / reap call is then a **second message, gated on
that confirmation**, and it carries either "preservation confirmed by
" or "nothing named worth keeping".
- **Why two messages:** a broadcast has no ordering, so a precondition
written as a clause inside the release message is not a precondition —
it is a starting gun handed to everyone able to destroy the artifact.
Paid for at the v0.65.0 close: the release call itself named the
rca-236 instrument as the thing worth keeping, and the 830-line test
instrument survived its own reap by about a minute, saved only because
the lane owner preserved on his own initiative before touching anything.
- **Restore cost is part of the preservation.** State whether the
preserved artifact applies at the current head: the surviving v0.65.0
patch does NOT apply straight at head (its test file had moved; the src
files apply under `--3way`), and that stated limit is worth more than
the patch — a preserved artifact everyone assumes is restorable is worse
than one whose restore cost is written down.
## Notes
- The wire-protocol version (REQ-ARCH-3) and the release counter and the
workspace semver are three independent numbers. Never conflate.
- The installer scripts trust HTTPS + `SHA256SUMS` on first fetch; the
`*.release.json` metadata is what `spt update` verifies thereafter against
the embedded two-key anchor.
- Revoking a leaked key without a rebuild: add its id to `"revoked"` in
`identity/release-keys.json` on each node (the file overlays the builtin
set); ship the next release signed by the other key.
]