# Release runbook

> M6-D6/D7 (ADR-0015). How a release ships. CI builds; the maintainer signs
> locally — release keys never enter CI.

<!-- [doc->REQ-REL-2] -->

## One-time setup (done 2026-06-05)

- Key ceremony: `cargo run -p xtask -- release-keygen rel-primary-2026` (and
  `rel-recovery-2026`); both seeds in the password manager (+ separate paper
  backup); both PUBLIC keys embedded in `BUILTIN_RELEASE_KEYS`
  (`crates/spt-daemon/src/release.rs`). The recovery key is never used until
  the primary is lost/compromised — then it signs the next release, which
  rotates in a fresh primary.
- `RELEASES_TOKEN` repo secret on spt-bs-core: fine-grained PAT, Contents
  read/write on `BigscreenVR/spt-bs-releases` only.

## Per release

### Preflight — derive the seam set, never recall it

Before handing a candidate sha to the gater, the release driver preflights it.
The suite is the usual one (workspace clippy, the touched crate's lib tests,
the seam integration binaries, `traceable-reqs check`) — but **how the seam set
is chosen is the part that goes wrong**:

- **Derive the seam binaries from the changed symbols, mechanically.** For every
  function the change adds, wraps, or alters, grep for its callers and for the
  tests that drive it:

  ```sh
  git diff --name-only <base>..HEAD -- '*.rs'          # what actually changed
  grep -rl '<changed_fn>' crates/*/src crates/*/tests   # who exercises it
  ```

  Run **every** test binary that grep returns, plus the binaries for the seam
  those callers sit on.
- **Never hand-list the seam set from memory.** A remembered list is a snapshot
  of the siblings that existed when it was memorised; it goes stale silently as
  tests are added, and it fails in exactly the case that matters — a sibling
  that drives the changed seam through a different fixture.
- **Why this is here (paid twice):** the v0.40.0 observability rider wrapped the
  broker's subscribe-decision function, adding a synchronous log write on the
  decision path. A hand-listed preflight set (broker, controller lease,
  redispatch, resize, attach-truth) came back fully green; the gate then went red
  on `brain_decouple` — a sibling driving the same subscribe/take seam, absent
  from the list only because the list was recalled rather than derived. A grep
  for the wrapped function's name would have returned it immediately. The same
  shape had already been paid once before, in v0.8.2.
- **A failure inside a seam the change touched does not get filed as a load
  flake on one observation.** Repeat-proof it (looped repro, A/B against the
  pre-change sha, under comparable load) before either dismissing it or fixing
  it. "Rerun went green" closes nothing here.

<!-- [doc->REQ-GOLDEN-CI-LANE] -->

1. **Under golden CI, the release shape rides the milestone batch.** Put the
   version bump (`[workspace.package] version` in root `Cargo.toml` and the
   matching first-party `spt-*` lines in `Cargo.lock`), regenerated docs
   (`cargo run -p xtask -- gen`), and changelog section (step 2) into the
   assembled golden candidate. The golden run validates the exact commit that
   gets tagged. **Tested SHA equals shipped SHA outranks bump-in-PR.**

   **Main advances only to a SHA whose golden run is green.** An explained red
   is still red: a diagnosis or a pre-existing mechanism cannot substitute for
   the suite passing. The update-set is signed, so shipping on an explained red
   would put the project's signature over an artifact whose provenance rests on
   a narrative rather than the gate. A signature must never attest to something
   weaker than the gate claims. **v0.45.0 having shipped carrying the same hazard
   is not precedent for repeating it — it is how the hazard became known.**

   Do not stack a thin release PR on the ruled golden tip. Its tag would point
   at a commit no golden run tested—the exact unvalidated-release-commit failure
   the dedicated-PR rule intended to prevent. The dedicated release PR form is
   therefore retired under golden CI; its release-shape gate is preserved as a
   **recorded out-of-band audit at the ruled tip**, performed by the release
   driver:
   - decode the next counter from the last published signed metadata;
   - audit the changelog against the actual commit range, not the milestone
     narrative;
   - prove the update-set compatibility constants unchanged at the candidate
     SHA; and
   - justify the bump level from observable behavior changes.

   This retires only the vehicle, never the substance or independent release
   review. Counter freshness and release shape are established at the explicit
   SHA before the golden run.

   **There are no post-golden edits.** Not a changelog date, rider line,
   lockfile adjustment, or regenerated byte. After the run the tree is frozen:
   either ship exactly what was tested, or add the delta to a new batch commit
   and rerun golden CI.

   The older dedicated-release-PR rule applied before milestone-batch golden CI,
   when that PR's own full CI could validate the tagged commit. It cannot be
   carried forward into an ff-only lane whose golden run is the full-suite
   authority.
   **Verify the lockfile bump by its DIFF, never by counting version
   strings.** After `cargo metadata --offline`, the check is
   `git diff Cargo.lock`: it must show exactly one line pair per first-party
   crate and nothing else.

   Third-party crates share the workspace's version-number space, and the
   collision has been **paid twice, both times the same way** — a third-party
   crate already sitting at the version being bumped *to*, inflating the count
   so a clean bump looks over-applied:
   - `quick-xml` was already at `0.39.4` during the v0.39.4 cut. The count came
     back 12 against 11 first-party crates.
   - `aws-lc-sys` was already at `0.41.0` during the v0.41.0 cut. Same shape,
     same false reading: 12 against 11, while the bump was in fact clean.

   The converse direction is real but has never been paid, because the tooling
   prevented it: at the v0.40.0 cut the *same* `quick-xml` pin at `0.39.4` was
   now the **old** version being bumped away from, where a blind `sed` of the
   old version string would have rewritten a third-party pin and corrupted it.
   That did not happen only because the bump went through
   `cargo metadata --offline` rather than a text substitution.

   So: the count is unreliable in both directions — it reads high when a
   third-party crate sits at the new version, and it cannot see a third-party
   pin wrongly rewritten to the new one. The diff is unambiguous in both.
   `cargo metadata --offline` is what makes the diff trustworthy: it touches
   workspace members only and leaves third-party pins alone.

2. **Write the user-facing changelog** — add a `## [0.X.Y] - <date>` section at
   the top of `CHANGELOG.md` (under the intro) with **Added / Changed / Fixed**
   subsections. This section becomes the GitHub Release **body** verbatim
   (`release.yml` extracts it via `--notes-file`), so it is the changelog every
   spt user reads. Rules:
   - **spt-user-facing UX only.** What a person running the `spt` CLI notices or
     does differently. Name the actual commands/flags they type.
   - **No internal lingo** — no requirement ids, crate names, commit hashes,
     milestone/hazard codes, or under-the-hood mechanics. A reader who has never
     seen the source must understand every line. This bans implementation nouns
     that leak from the fix's own vocabulary — `replay`, `subscriber`, `serve
     lease`, `stream lock`, `circuit breaker`, `write-deadline`, telemetry token
     names (`DISPATCH_EV`), worker-pool/mutex internals. State the **observable
     effect**, not the mechanism: not "replay halts at the first failed write and
     the poisoned subscriber is dropped" but "a stuck viewer can no longer freeze
     other sessions." An internal-diagnostics change collapses to one plain line
     (e.g. "Improved logging granularity for attached endpoints") under
     **Internal** — never sell the mechanism as an **Added** feature.
   - **Impersonal voice — do not address the reader.** Avoid "you"/"your";
     describe the software's behavior, not the person's. Say "users" only when a
     subject is unavoidable. Pattern that reads well: *"Improved the stability of
     X. Previously, Y."* Example: not "You keep control of a session across a
     restart" but "Improved the stability of controlled sessions across a daemon
     restart. Previously, session controllers could mix up and drop their
     attached sessions."
   - **Use the product's user-facing nouns, precisely.** Sessions reached over
     `spt rc` are **attached sessions**, not "remote sessions" (they need not be
     remote). Match the vocabulary a user sees in the CLI (controller, viewer,
     attach, daemon), not the code's.
   - **Flag breaking changes** prominently under Changed.
   - The release **fails loudly** if the tagged version has no `## [0.X.Y]`
     section — the changelog is not optional.
   - **After inserting the new section, assert the version ladder is
     contiguous**: `grep '^## \[' CHANGELOG.md` and check the versions descend
     with no gap. Inserting the new heading with an editor whose anchor spans the
     previous heading can silently *consume* that heading (v0.34.0 did this to
     `## [0.33.0]`), folding the prior release's body into the new section — and
     since the body is published verbatim, the release notes then carry two
     releases. The grep is a two-second self-check; a candidate `xtask`/CI lint.
3. **Advance `main` to the ruled tip, fast-forward only, before tagging.**
   The retired dedicated release PR used to advance `main` as a side effect of
   its merge. Removing that vehicle without re-homing this responsibility would
   produce a correct signed release while `origin/main` silently lagged, so the
   next milestone would branch from stale code. The gap became visible only
   because the release driver published the exact commands for review before
   running them: his local fast-forward never pushed.

   ```sh
   git fetch origin main
   git switch main
   git merge --ff-only <ruled-tip>
   git push origin main
   git merge-base --is-ancestor <ruled-tip> origin/main
   ```

   The merge and push must succeed, and the final ancestry check must exit 0.
   `--ff-only` refuses divergence rather than manufacturing an untested merge
   commit. **A release tag points at a commit already on `main`, never the
   reverse.**

   Pushing `main` fires `ci.yml` at the exact golden SHA. Under a golden
   milestone this run is **not a separate baseline authority**: it is thin by
   design—traceability, changes, lint, unit—a strict subset of the golden run
   already green at this SHA, so its green adds no coverage and must not be
   waited on **as evidence**. Wait for it **as occupancy**: it holds the box,
   and step 5 is local Cargo. Measured on 2026-07-29: lint 55s, unit Linux
   2m53s, unit Windows 8m30s—roughly ten minutes, not a cycle.

   If this thin run reds at a SHA whose golden run is green, that is a
   contradiction between two runs of overlapping scope, which is information:
   it does not auto-block the publish and it is not waved through. Stop and
   refer it to the gater for a ruling. Precedent 2026-07-29:
   `the_spawn_environment_carries_the_cli_capability` passed in golden's
   full-suite Linux job and failed in main's thin Linux job at the same SHA
   `af65ac0`—a real latent defect the full suite hid, not a flake. Main's
   concurrency policy never cancels this run; that preserves the record, which
   is not the same claim as the record being authoritative. **Preservation is
   not deference.**

4. **Tag**: `git tag v0.X.Y && git push origin v0.X.Y`. This triggers exactly
   one workflow — `release.yml`. Both runners build; the assemble job extracts
   this version's `CHANGELOG.md` section into the release body (a missing
   section fails the job — the changelog is not optional), builds the docs
   bundle, and creates a **draft** release on spt-releases with
   `spt-x86_64-linux`, `spt-x86_64-linux-musl`, `spt-x86_64-windows.exe`,
   `SHA256SUMS`, `manifest.schema.json`, `mock-adapter.zip` and
   `spt-docs.tar.gz`.

   **There is no second workflow.** Docs ship as the `spt-docs.tar.gz` asset
   built by that same assemble job (ADR-0036 §4, `xtask docs-bundle`) — the
   `docs-publish.yml` this step used to name was **retired** when its docs step
   moved into `release.yml`, and a release driver waiting for it to appear in
   the Actions list is waiting for a workflow that no longer exists. The
   bundle is picked up by step 5 along with the binaries and flipped public
   with them; it is not in `SHA256SUMS` (that file covers the `spt-x86_64-*`
   binaries the install scripts verify) — its integrity rides the signed
   update-set entry step 5 builds. A draft missing the bundle publishes a
   docs-less set loudly rather than blocking the binaries.
5. **Sign + publish** (local, one command):

   Before invoking local Cargo, enter a quiet window: the golden run, the thin
   main `ci.yml` run fired by step 3, and the tag's `release.yml` run must all
   be terminal. This wait is about **box occupancy**, not the thin run blessing
   the SHA: golden remains the full-suite authority. Required golden evidence
   must be green, the complete seven-asset draft must be present, and any thin-
   run red must have its gater ruling. Attribute any remaining Cargo/rustc
   process by parent chain: a process rooted at
   `Runner.Worker`/`RunnerService` belongs to an already-counted CI axis; one
   rooted at a user shell is real contention and blocks signing.

   ```sh
   SPT_RELEASE_SEED_CMD='<your password-manager CLI read command>' \
     cargo run -p xtask -- release-publish \
       --tag v0.X.Y --key-id rel-primary-2026 --version <N>
   ```

   It downloads the draft's binaries, **verifies them against SHA256SUMS
   before signing**, signs each (`*.release.json` in the exact
   `SignedRelease` shape `spt update` verifies), uploads the metadata
   assets, and flips the draft public. `--version <N>` is the **monotonic
   release counter** (u64; strictly increasing across releases — it is not
   the semver).

   **Seed handling — `_CMD` only, on any shared machine.** `SPT_RELEASE_SEED`
   (raw hex) exists as a by-hand fallback for pasting from the manager, and it
   is **forbidden on any machine that hosts a CI runner or has more than one
   user.** Use the `_CMD` form there, so the seed is read per-invocation and
   never persisted.

   The failure mode is not theoretical, and it is not about the value leaking
   to a person — it is about what already executes on the box. A CI runner
   compiles and runs arbitrary third-party code: every dependency's
   `build.rs`, every proc macro, and every test binary runs with read access
   to the environment it inherits. A seed placed in a persistent environment
   variable on such a machine is readable by all of it, survives reboots, and
   outlives the release that needed it. Prefer a form that leaves nothing
   behind.

   **Recorded operator ruling — `HFENDULEAM`, 2026-07-26.** On that one host the
   primary seed is stored in the **machine-scope** `SPT_RELEASE_SEED` variable,
   and the operator has ruled that placement deliberate and permanent. Publish
   there with the raw-env form — run `release-publish` **without**
   `SPT_RELEASE_SEED_CMD` set, so `xtask` falls back to reading
   `SPT_RELEASE_SEED` — and **do not remove the variable.** This is a recorded
   exception for one named host, **not a softening of the rule above**: the
   rule and its rationale stand unchanged on every other machine, including
   every other CI-runner or multi-user box.
   - **The accepted risk, named:** `HFENDULEAM` hosts a self-hosted CI runner,
     so every job step it executes — each dependency's `build.rs`, every proc
     macro, every test binary — inherits the machine environment and can read
     the seed, and the seed survives reboots. That is precisely the exposure
     the paragraph above describes. The operator owns it knowingly; it is not
     an oversight and it is not news.
   - **Do not re-escalate it per release.** The finding was raised at the
     v0.43.0 cut (release driver stopped before signing) and ruled then.
     Re-raising it at each cut changes nothing and costs a release window.
     Revisiting the ruling is an operator decision, never a release-driver
     block.
   - **Windows practicality.** The `_CMD` path runs the command under
     `powershell` and hex-decodes its stdout, so feeding it a seed that is
     already hex double-decodes and panics (`seed is 32 bytes:
     TryFromSliceError`) — inside `signing_identity()`, before any sign or
     upload, so such a run is side-effect-free and safe to re-run.

   **Probing whether a secret is set — never with `${VAR:-...}`.** That arm
   expands to the *variable's value* when the variable is set, so the "check"
   prints the secret:

   ```sh
   [ -n "${VAR:+x}" ] && echo set || echo unset   # correct — cannot emit the value
   ```

   Use only the `:+` form above. The `:-` form reads like a default and
   behaves like an echo.

6. **Associate the shipped requests — on the board, not on the release body.**

   **The release body carries no milestone or request links** (operator-ruled
   2026-07-31). It is the changelog section and nothing else. A links section
   was tried on the v0.48.0 cut and stripped the same day; do not re-add one,
   and do not read its absence as an oversight. The question *which requests
   did this release deliver* is answered by the board, which is where the
   states live anyway.

   The association is one verb, with an ordering constraint on either side:

   - **Before publish — the ACCEPTANCE close-cascade must already have
     landed.** The `release` verb gates on each request having been closed
     *before* the release was published (`closedAt < publishedAt`). A request
     closed afterwards is not picked up, so closing the board out is a
     pre-publish step, not a victory lap.
   - **After publish — run the verb:**

     ```sh
     spt shell cmd alchemy-0 release v0.X.Y
     ```

     It promotes the shipped requests to DONE and posts the Shipped Requests
     roundup to Discord. The v0.47.0 roundup is the exemplar to match.
   - **If the ordering slipped**, re-date the release past the closes by
     toggling it through draft and back, then re-run the verb:

     ```sh
     gh release edit v0.X.Y --draft=true
     gh release edit v0.X.Y --draft=false
     ```

     The tag, the assets and the body all survive this (verified on v0.48.0,
     2026-07-31). Two consequences to weigh before reaching for it, both
     observed rather than theoretical:

     - **It rewrites `publishedAt`.** v0.48.0 moved from `10:16:13Z` to
       `12:35:31Z` this way, so the release page's date is the *repair*
       moment, not the moment the artifacts were signed and published. Cite
       the signing time from your own record when the two disagree — the
       release page is no longer authoritative for it.
     - **While it is a draft the release is not the latest.** GitHub excludes
       drafts from `releases/latest`, so a client asking for the latest
       release in that window is told the *previous* version. Keep the two
       commands adjacent, and never toggle while a rollout or a field-verify
       leg is in flight.

   A request that rode in **reduced scope** still records where the remainder
   went — but on the milestone issue, as the golden-head intake step already
   stipulates, not here. The board carries the association; the artifact
   carries the user-facing notes. Neither borrows the other's job.

## Notes

- The wire-protocol version (REQ-ARCH-3) and the release counter and the
  workspace semver are three independent numbers. Never conflate.
- The installer scripts trust HTTPS + `SHA256SUMS` on first fetch; the
  `*.release.json` metadata is what `spt update` verifies thereafter against
  the embedded two-key anchor.
- Revoking a leaked key without a rebuild: add its id to `"revoked"` in
  `identity/release-keys.json` on each node (the file overlays the builtin
  set); ship the next release signed by the other key.