# Release runbook

> M6-D6/D7 (ADR-0015). How a release ships. CI builds; the maintainer signs
> locally — release keys never enter CI.

<!-- [doc->REQ-REL-2] -->

## One-time setup (done 2026-06-05)

- Key ceremony: `cargo run -p xtask -- release-keygen rel-primary-2026` (and
  `rel-recovery-2026`); both seeds in the password manager (+ separate paper
  backup); both PUBLIC keys embedded in `BUILTIN_RELEASE_KEYS`
  (`crates/spt-daemon/src/release.rs`). The recovery key is never used until
  the primary is lost/compromised — then it signs the next release, which
  rotates in a fresh primary.
- `RELEASES_TOKEN` repo secret on spt-bs-core: fine-grained PAT, Contents
  read/write on `BigscreenVR/spt-bs-releases` only.

## Per release

### Preflight — derive the seam set, never recall it

Before handing a candidate sha to the gater, the release driver preflights it.
The suite is the usual one (workspace clippy, the touched crate's lib tests,
the seam integration binaries, `traceable-reqs check`) — but **how the seam set
is chosen is the part that goes wrong**:

- **Derive the seam binaries from the changed symbols, mechanically.** For every
  function the change adds, wraps, or alters, grep for its callers and for the
  tests that drive it:

  ```sh
  git diff --name-only <base>..HEAD -- '*.rs'          # what actually changed
  grep -rl '<changed_fn>' crates/*/src crates/*/tests   # who exercises it
  ```

  Run **every** test binary that grep returns, plus the binaries for the seam
  those callers sit on.
- **Never hand-list the seam set from memory.** A remembered list is a snapshot
  of the siblings that existed when it was memorised; it goes stale silently as
  tests are added, and it fails in exactly the case that matters — a sibling
  that drives the changed seam through a different fixture.
- **Why this is here (paid twice):** the v0.40.0 observability rider wrapped the
  broker's subscribe-decision function, adding a synchronous log write on the
  decision path. A hand-listed preflight set (broker, controller lease,
  redispatch, resize, attach-truth) came back fully green; the gate then went red
  on `brain_decouple` — a sibling driving the same subscribe/take seam, absent
  from the list only because the list was recalled rather than derived. A grep
  for the wrapped function's name would have returned it immediately. The same
  shape had already been paid once before, in v0.8.2.
- **A failure inside a seam the change touched does not get filed as a load
  flake on one observation.** Repeat-proof it (looped repro, A/B against the
  pre-change sha, under comparable load) before either dismissing it or fixing
  it. "Rerun went green" closes nothing here.

1. **Bump inside a dedicated release PR cut off `main` — never on a feature or
   wave branch, and never as a separate push to `main`.** Put the version bump
   (`[workspace.package] version` in root `Cargo.toml` **and** the matching
   first-party `spt-*` lines in `Cargo.lock` — leave third-party lines alone),
   the regenerated docs (`cargo run -p xtask -- gen`), and the changelog section
   (step 2) **in that one PR**. Its CI validates the **exact commit that gets
   tagged**, so on merge you tag straight away — the green PR run is the gate.

   Why the release PR must stand alone, even when only one feature PR is
   shipping:
   - **Gate separation.** A feature PR is gated on its *content*, by that work's
     gater. A release PR is gated on *release shape* — counter, changelog, bump,
     regen — by the release driver. Stacking the bump onto the feature branch
     fuses the two, and the commit that gets tagged is then never validated **as
     a release commit** by anyone.
   - **Counter freshness.** The release counter is read from published
     releases-repo metadata *at release-PR authoring time*. Frozen into a wave
     branch, it goes stale while other work merges ahead of it — the counter
     drift this rule exists to prevent.
   - **No latency is reintroduced.** The 2026-06-23 goal is preserved intact:
     the release PR's own CI validates the tagged commit, so there is still no
     serial second `push:[main]` wait before tagging. Only the *vehicle*
     narrowed, never the speed.

   > **Supersedes:** as operator-directed 2026-06-23 (bump-in-PR, v0.14.1
   > onward) this step also permitted the bump to ride *the feature PR's final
   > commit*. Wave-branch and counter-drift rulings, and practice across
   > v0.38.0–v0.39.2, narrowed it to the dedicated form; the text is updated to
   > match 2026-07-21. The OR-form is retired — it is not bounded to a
   > remaining case.

   - **Do not** author the bump as a direct push to `main` and then wait for a
     second full `push:[main]` CI pass before tagging. The code is byte-identical
     to what the feature PR already proved green (only the version string,
     changelog, and lockfile lines differ); that second pass is ~one CI cycle of
     pure latency.
   - Merging the release PR still fires `push:[main]` CI automatically — let it
     run as a **non-blocking backstop**; it is **not** a tag gate. Tag the merged
     `main` HEAD as soon as the PR is merged.
   **Verify the lockfile bump by its DIFF, never by counting version
   strings.** After `cargo metadata --offline`, the check is
   `git diff Cargo.lock`: it must show exactly one line pair per first-party
   crate and nothing else.

   Third-party crates share the workspace's version-number space, and the
   collision has been **paid twice, both times the same way** — a third-party
   crate already sitting at the version being bumped *to*, inflating the count
   so a clean bump looks over-applied:
   - `quick-xml` was already at `0.39.4` during the v0.39.4 cut. The count came
     back 12 against 11 first-party crates.
   - `aws-lc-sys` was already at `0.41.0` during the v0.41.0 cut. Same shape,
     same false reading: 12 against 11, while the bump was in fact clean.

   The converse direction is real but has never been paid, because the tooling
   prevented it: at the v0.40.0 cut the *same* `quick-xml` pin at `0.39.4` was
   now the **old** version being bumped away from, where a blind `sed` of the
   old version string would have rewritten a third-party pin and corrupted it.
   That did not happen only because the bump went through
   `cargo metadata --offline` rather than a text substitution.

   So: the count is unreliable in both directions — it reads high when a
   third-party crate sits at the new version, and it cannot see a third-party
   pin wrongly rewritten to the new one. The diff is unambiguous in both.
   `cargo metadata --offline` is what makes the diff trustworthy: it touches
   workspace members only and leaves third-party pins alone.

2. **Write the user-facing changelog** — add a `## [0.X.Y] - <date>` section at
   the top of `CHANGELOG.md` (under the intro) with **Added / Changed / Fixed**
   subsections. This section becomes the GitHub Release **body** verbatim
   (`release.yml` extracts it via `--notes-file`), so it is the changelog every
   spt user reads. Rules:
   - **spt-user-facing UX only.** What a person running the `spt` CLI notices or
     does differently. Name the actual commands/flags they type.
   - **No internal lingo** — no requirement ids, crate names, commit hashes,
     milestone/hazard codes, or under-the-hood mechanics. A reader who has never
     seen the source must understand every line. This bans implementation nouns
     that leak from the fix's own vocabulary — `replay`, `subscriber`, `serve
     lease`, `stream lock`, `circuit breaker`, `write-deadline`, telemetry token
     names (`DISPATCH_EV`), worker-pool/mutex internals. State the **observable
     effect**, not the mechanism: not "replay halts at the first failed write and
     the poisoned subscriber is dropped" but "a stuck viewer can no longer freeze
     other sessions." An internal-diagnostics change collapses to one plain line
     (e.g. "Improved logging granularity for attached endpoints") under
     **Internal** — never sell the mechanism as an **Added** feature.
   - **Impersonal voice — do not address the reader.** Avoid "you"/"your";
     describe the software's behavior, not the person's. Say "users" only when a
     subject is unavoidable. Pattern that reads well: *"Improved the stability of
     X. Previously, Y."* Example: not "You keep control of a session across a
     restart" but "Improved the stability of controlled sessions across a daemon
     restart. Previously, session controllers could mix up and drop their
     attached sessions."
   - **Use the product's user-facing nouns, precisely.** Sessions reached over
     `spt rc` are **attached sessions**, not "remote sessions" (they need not be
     remote). Match the vocabulary a user sees in the CLI (controller, viewer,
     attach, daemon), not the code's.
   - **Flag breaking changes** prominently under Changed.
   - The release **fails loudly** if the tagged version has no `## [0.X.Y]`
     section — the changelog is not optional.
   - **After inserting the new section, assert the version ladder is
     contiguous**: `grep '^## \[' CHANGELOG.md` and check the versions descend
     with no gap. Inserting the new heading with an editor whose anchor spans the
     previous heading can silently *consume* that heading (v0.34.0 did this to
     `## [0.33.0]`), folding the prior release's body into the new section — and
     since the body is published verbatim, the release notes then carry two
     releases. The grep is a two-second self-check; a candidate `xtask`/CI lint.
3. **Tag**: `git tag v0.X.Y && git push origin v0.X.Y`. This triggers exactly
   one workflow — `release.yml`. Both runners build; the assemble job extracts
   this version's `CHANGELOG.md` section into the release body (a missing
   section fails the job — the changelog is not optional), builds the docs
   bundle, and creates a **draft** release on spt-releases with
   `spt-x86_64-linux`, `spt-x86_64-linux-musl`, `spt-x86_64-windows.exe`,
   `SHA256SUMS`, `manifest.schema.json`, `mock-adapter.zip` and
   `spt-docs.tar.gz`.

   **There is no second workflow.** Docs ship as the `spt-docs.tar.gz` asset
   built by that same assemble job (ADR-0036 §4, `xtask docs-bundle`) — the
   `docs-publish.yml` this step used to name was **retired** when its docs step
   moved into `release.yml`, and a release driver waiting for it to appear in
   the Actions list is waiting for a workflow that no longer exists. The
   bundle is picked up by step 4 along with the binaries and flipped public
   with them; it is not in `SHA256SUMS` (that file covers the `spt-x86_64-*`
   binaries the install scripts verify) — its integrity rides the signed
   update-set entry step 4 builds. A draft missing the bundle publishes a
   docs-less set loudly rather than blocking the binaries.
4. **Sign + publish** (local, one command):

   ```sh
   SPT_RELEASE_SEED_CMD='<your password-manager CLI read command>' \
     cargo run -p xtask -- release-publish \
       --tag v0.X.Y --key-id rel-primary-2026 --version <N>
   ```

   It downloads the draft's binaries, **verifies them against SHA256SUMS
   before signing**, signs each (`*.release.json` in the exact
   `SignedRelease` shape `spt update` verifies), uploads the metadata
   assets, and flips the draft public. `--version <N>` is the **monotonic
   release counter** (u64; strictly increasing across releases — it is not
   the semver).

   **Seed handling — `_CMD` only, on any shared machine.** `SPT_RELEASE_SEED`
   (raw hex) exists as a by-hand fallback for pasting from the manager, and it
   is **forbidden on any machine that hosts a CI runner or has more than one
   user.** Use the `_CMD` form there, so the seed is read per-invocation and
   never persisted.

   The failure mode is not theoretical, and it is not about the value leaking
   to a person — it is about what already executes on the box. A CI runner
   compiles and runs arbitrary third-party code: every dependency's
   `build.rs`, every proc macro, and every test binary runs with read access
   to the environment it inherits. A seed placed in a persistent environment
   variable on such a machine is readable by all of it, survives reboots, and
   outlives the release that needed it. Prefer a form that leaves nothing
   behind.

   **Recorded operator ruling — `HFENDULEAM`, 2026-07-26.** On that one host the
   primary seed is stored in the **machine-scope** `SPT_RELEASE_SEED` variable,
   and the operator has ruled that placement deliberate and permanent. Publish
   there with the raw-env form — run `release-publish` **without**
   `SPT_RELEASE_SEED_CMD` set, so `xtask` falls back to reading
   `SPT_RELEASE_SEED` — and **do not remove the variable.** This is a recorded
   exception for one named host, **not a softening of the rule above**: the
   rule and its rationale stand unchanged on every other machine, including
   every other CI-runner or multi-user box.
   - **The accepted risk, named:** `HFENDULEAM` hosts a self-hosted CI runner,
     so every job step it executes — each dependency's `build.rs`, every proc
     macro, every test binary — inherits the machine environment and can read
     the seed, and the seed survives reboots. That is precisely the exposure
     the paragraph above describes. The operator owns it knowingly; it is not
     an oversight and it is not news.
   - **Do not re-escalate it per release.** The finding was raised at the
     v0.43.0 cut (release driver stopped before signing) and ruled then.
     Re-raising it at each cut changes nothing and costs a release window.
     Revisiting the ruling is an operator decision, never a release-driver
     block.
   - **Windows practicality.** The `_CMD` path runs the command under
     `powershell` and hex-decodes its stdout, so feeding it a seed that is
     already hex double-decodes and panics (`seed is 32 bytes:
     TryFromSliceError`) — inside `signing_identity()`, before any sign or
     upload, so such a run is side-effect-free and safe to re-run.

   **Probing whether a secret is set — never with `${VAR:-...}`.** That arm
   expands to the *variable's value* when the variable is set, so the "check"
   prints the secret:

   ```sh
   [ -n "${VAR:+x}" ] && echo set || echo unset   # correct — cannot emit the value
   ```

   Use only the `:+` form above. The `:-` form reads like a default and
   behaves like an echo.

## Notes

- The wire-protocol version (REQ-ARCH-3) and the release counter and the
  workspace semver are three independent numbers. Never conflate.
- The installer scripts trust HTTPS + `SHA256SUMS` on first fetch; the
  `*.release.json` metadata is what `spt update` verifies thereafter against
  the embedded two-key anchor.
- Revoking a leaked key without a rebuild: add its id to `"revoked"` in
  `identity/release-keys.json` on each node (the file overlays the builtin
  set); ship the next release signed by the other key.