# Phase 5: Deploy — Research

**Researched:** 2026-05-08
**Domain:** Container deploy + Fly.io ops + Litestream WAL replication + GitHub Actions CI/CD + OpenTelemetry-to-OpenObserve observability for a Node 22 + Colyseus 0.17 game server
**Confidence:** HIGH on stack/Fly/Litestream/OTel/Drizzle; MEDIUM on OpenObserve Tigris-config edge cases (single-source docs); HIGH on Phase-4 carry-forward integration (codebase grounded)

## Summary

Phase 5 lifts a fully Phase-4-tested `apps/server` (Colyseus 0.17.10 + Better-Auth 1.6.9 + better-sqlite3 12.9.0 + argon2 0.44.0 on Node 22) to two Fly.io apps (`rebno-staging` + `rebno-prod`, region `lax`), replicates SQLite WAL to per-app Tigris buckets via Litestream, ships CI/CD via GitHub Actions (push-to-main → staging; tag → prod via image-SHA reuse), and self-hosts OpenObserve on a third Fly app (`rebno-obs`) as the OTLP-HTTP sink for pino-bridged logs + Node/Colyseus metrics + per-message traces. Every architectural decision is locked in CONTEXT.md (D-01..D-21) — research focuses on implementation specifics.

The four genuine open questions (Dockerfile base, OTel collector form factor, flyctl-action SHA, OpenObserve version pin) all resolve to concrete prescriptions below. The primary unknown that survived research is **Litestream major-version selection** (0.3.x stable vs 0.5.x current) — see `## Open Questions` Q1.

**Primary recommendation:**
- Dockerfile base: **`node:22-bookworm-slim`** (glibc), NOT alpine — argon2 + better-sqlite3 musl prebuilds exist but are documented as fragile for Node 22 (argon2 dropped pre-built musl in some patch combos, better-sqlite3 sometimes downloads from `unofficial-builds.nodejs.org` on alpine which is blocked from many Fly egress paths). The 25 MB size win is not worth the troubleshooting risk on a one-shot deploy phase.
- OTel collector form factor: **in-process `@opentelemetry/sdk-node` + `pino-opentelemetry-transport`** — adds ~5 MB RSS, no second process to supervise, and the CONTEXT.md "fail soft to stdout" requirement is one try/catch around SDK init.
- Litestream pin: **0.3.13** (the conservative production line) for the initial Phase-5 deploy; track 0.5.x as a v2 upgrade path. Rationale and dissent in §"State of the Art" + Open Questions.
- OpenObserve pin: **`public.ecr.aws/zinclabs/openobserve:v0.14.x` latest at plan time** (community-edition, single binary, S3-backed via the documented `ZO_S3_*` env vars) on a `shared-cpu-1x@1024MB` Fly machine + 1 GB volume.

## Architectural Responsibility Map

| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| WS handshake, room loop, auth | `apps/server` (Node) | — | Phase-4 boot sequence (D-21) is unchanged in Phase 5 |
| TLS termination, HTTP→app routing, WS proxy, IP allowlist | Fly proxy / edge | — | Fly handles TLS + WS upgrade; allowlist via `fly ips allocate --network` |
| SQLite read/write | `apps/server` in-process (better-sqlite3) | Fly Volume (`/data`) | Single-machine, single-writer per ADR 0002 |
| WAL → object storage replication | Litestream (sidecar process inside container) | Tigris (S3) | Sidecar via entrypoint background process — not a separate machine |
| Image build + push | GitHub Actions (`docker buildx build`) | Fly Registry (`registry.fly.io`) | Standard Fly pattern |
| Migrations | Container entrypoint (`drizzle-kit migrate`) | — | D-09 lock; pre-migrate Litestream snapshot is the rollback safety net |
| Legacy account import | One-shot `fly ssh console` invocation | Fly Volume seed dir | D-10/D-17 lock — explicitly NOT entrypoint |
| Logs/metrics/traces transport | OTel SDK in-process → OTLP-HTTP | `rebno-obs.flycast:5080` (private) | flycast = no public exposure; 6PN encrypted by default |
| Observability storage | OpenObserve (Fly app) | Tigris bucket per-obs | Parquet-on-S3, 140× storage cost vs Elasticsearch claim |
| Health probe | `/health` Express handler | Fly `[http_service.checks]` | Phase-4 `health.ts` consumed verbatim |
| Secrets at rest | `fly secrets set` (encrypted at rest by Fly) | — | Per-app, never baked into image |
| Plaintext legacy creds transport | `fly ssh sftp` to `/data/seed/` (TLS via WireGuard) | One-shot, deleted post-import | D-17 ritual; never touches image or git |

## Standard Stack

### Core (versions verified `npm view <pkg> version` 2026-05-08, plus binary-distribution sources)

| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| Node.js base image | `node:22-bookworm-slim` | Container runtime | Glibc, official long-term support, native-module prebuilds always work [VERIFIED: docker hub/Snyk Node image guide] |
| pnpm in CI | `pnpm@10` (matches `verify-phase-4.yml`) | Workspace install | Already pinned in Phase-4 CI [VERIFIED: `.github/workflows/verify-phase-4.yml`] |
| Litestream | `0.3.13` (recommended) or `0.5.11` (current) | SQLite WAL → S3 replication | Fly.io-blessed pattern [VERIFIED: github.com/benbjohnson/litestream/releases — 0.5.11 released 2026-04-08; 0.3.13 from 2024 with 0.3.14 patch 2026-03-26] |
| `@opentelemetry/sdk-node` | `0.217.0` | In-process OTel SDK boot | Official SDK [VERIFIED: npm 2026-05-08] |
| `@opentelemetry/auto-instrumentations-node` | `0.75.0` | Drop-in instrumentation for `http`, `express`, `ws` | Official auto-instrumentation bundle [VERIFIED: npm 2026-05-08] |
| `@opentelemetry/exporter-logs-otlp-http` | `0.217.0` | OTLP-HTTP logs exporter | Official OTLP-HTTP exporter [VERIFIED: npm 2026-05-08] |
| `@opentelemetry/exporter-trace-otlp-http` | `0.217.0` (matches sdk-node) | OTLP-HTTP traces exporter | Official OTLP-HTTP exporter [CITED: opentelemetry.io/docs/languages/js/exporters/] |
| `@opentelemetry/exporter-metrics-otlp-http` | `0.217.0` | OTLP-HTTP metrics exporter | Same family [CITED: opentelemetry.io/docs/languages/js/exporters/] |
| `@opentelemetry/instrumentation-pino` | `0.63.0` | Auto-link pino log records to active span | Official contrib pkg [VERIFIED: npm 2026-05-08] |
| `pino-opentelemetry-transport` | `3.0.0` | Pino transport that emits OTel LogRecord | Official pino-OTel bridge [VERIFIED: npm 2026-05-08] |
| `drizzle-kit` | `0.31.10` | `drizzle-kit migrate` runtime CLI | Pinned to current; runtime-safe migrate command [VERIFIED: npm 2026-05-08] |
| `superfly/flyctl-actions/setup-flyctl` | `@1.5` (SHA `fc53c09...`) — pin by SHA per project security policy | flyctl in CI | Canonical action [VERIFIED: github.com/superfly/flyctl-actions/releases — v1.5 released 2025-02-02] |
| OpenObserve | `v0.14.x` (latest stable at plan time; community edition) | Self-hosted observability backend | Single binary, S3-backed Parquet [CITED: github.com/openobserve/openobserve] |

**Already in `apps/server/package.json` (Phase 4); no new dep required for these:** `pino@9`, `colyseus@0.17.10`, `better-sqlite3@12.9.0`, `argon2@0.44.0`, `drizzle-orm@0.45.2`. [VERIFIED: `apps/server/package.json`]

### Supporting (dev-only / build-only)

| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `@actions/checkout` | `v4` | GH Actions checkout step | Standard CI [CITED: github.com/actions/checkout] |
| `pnpm/action-setup` | `v4` | Pin pnpm in CI | Already used in Phase-4 CI [VERIFIED: `.github/workflows/verify-phase-4.yml`] |
| `actions/setup-node` | `v4` | Pin Node 22 in CI | Already used [VERIFIED: same file] |
| `docker/setup-buildx-action` | `v3` | buildx for multi-stage build cache | Standard for Fly registry pushes [CITED: fly.io/docs/launch/continuous-deployment-with-github-actions/] |
| `docker/build-push-action` | `v6` | Build + push to `registry.fly.io` | Same [CITED: same] |

### Alternatives Considered

| Instead of | Could Use | Tradeoff | Status |
|------------|-----------|----------|--------|
| `node:22-bookworm-slim` | `node:22-alpine` | Smaller image (~25 MB), but musl prebuilds for argon2/better-sqlite3 are documented as fragile [CITED: github.com/WiseLibs/better-sqlite3/issues/1397 + ranisalt/node-argon2/issues/236] | **Rejected — slim chosen** |
| In-process OTel SDK | OTel Collector sidecar binary | Sidecar isolates GC + memory pressure, but adds ~80 MB RSS + supervision; for <50 CCU the in-process overhead is ~5 MB and BatchSpanProcessor handles back-pressure | **Rejected — in-process chosen** |
| Litestream sidecar container | Litestream-in-server-container background process | Sidecar = isolation, but Fly machines are 1-container-per-machine; running 2 processes via `s6-overlay` or shell `&` is the only way to keep replicate alive in same container | **Same-container background process** |
| OpenObserve | Grafana Loki+Mimir+Tempo+Alloy | Ecosystem maturity, but 4 services to operate vs 1 binary; CONTEXT.md D-12..D-16 explicitly locks OpenObserve [CONTEXT D-14 + Specifics §"Self-hosted observability in v1 is the explicit user lock"] | **Rejected — OpenObserve locked** |
| OpenObserve | SigNoz | Similar single-deploy story, but heavier stack (ClickHouse) and more memory; OpenObserve's single-binary + S3 is the lighter option | **Rejected — OpenObserve locked** |
| flyctl GH Action | Hand-rolled `curl https://fly.io/install.sh \| sh` | Action handles version pinning + caching cleanly | **Use action** |
| `drizzle-kit migrate` (CLI) | `drizzle-orm/migrator` programmatic API | CLI is the documented production path; programmatic still works but loses Drizzle's CLI safety messages [CITED: orm.drizzle.team/docs/drizzle-kit-migrate] | **Use CLI** |

**Installation (incremental over Phase-4):**

```bash
# In apps/server/
pnpm add @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/exporter-logs-otlp-http \
  @opentelemetry/instrumentation-pino \
  pino-opentelemetry-transport
pnpm add -D drizzle-kit@0.31.10  # if not already at this pin from Phase 3/4
```

**Version verification at plan time:** Run `npm view <pkg> version` and `gh api repos/superfly/flyctl-actions/releases/latest --jq .tag_name` immediately before plan execution. Versions above are 2026-05-08 snapshots; OTel ecosystem ships ~weekly minors so a re-check at plan-day is mandatory. [VERIFIED: npm registry direct query, this session]

## Architecture Patterns

### System Architecture Diagram

```
GitHub repo (main branch)
   │
   │ 1) push to main
   ▼
GitHub Actions: deploy-staging.yml ─────────► [trace:check] (PR-required)
   │ 2) install + verify:phase-4 + verify:phase-5 + trace:check
   │ 3) docker buildx build → registry.fly.io/rebno-staging:<sha>
   │ 4) flyctl deploy -a rebno-staging --image registry.fly.io/rebno-staging:<sha>
   │ 5) (post-deploy, async via concurrency group) pnpm soak:staging — 30-min 2-client harness
   │
   │ 6) operator tags vX.Y.Z on a green-staging commit, git push --tags
   ▼
GitHub Actions: deploy-prod.yml
   │ 7) resolve <sha> from tag annotation OR git notes
   │ 8) flyctl deploy -a rebno-prod --image registry.fly.io/rebno-staging:<sha>
   │
   ▼
Fly proxy (TLS termination, WS upgrade, IP allowlist for staging)
   │
   ├──► rebno-staging machine (region lax, shared-cpu-2x@2GB?)
   │       │
   │       ├── /data (Fly Volume)
   │       │     ├── rebno.db (+ -wal, -shm)
   │       │     ├── keys/room_signing.ed25519 (D-19)
   │       │     ├── seed/localList.txt (one-shot, deleted after import)
   │       │     └── snapshots/pre-migrate-*.db (entrypoint safety net)
   │       │
   │       ├── docker-entrypoint.sh
   │       │     ├── 1) start litestream replicate (background &)
   │       │     ├── 2) litestream snapshots --output /data/snapshots/pre-migrate-$(date +%s).db
   │       │     ├── 3) drizzle-kit migrate
   │       │     └── 4) exec node dist/index.js
   │       │
   │       └── apps/server (Node 22)
   │             ├── boot (env → keys → SQLite open → migrations applied → Better-Auth → Colyseus + RebnoRoom → /health → SIGTERM hook)
   │             ├── otel-init.ts (loaded via --import or top of index.ts)
   │             │     └── @opentelemetry/sdk-node → OTLP-HTTP → http://rebno-obs.flycast:5080/api/default/v1/{logs,metrics,traces}
   │             ├── pino → stdout (Fly's `fly logs`)
   │             └── pino-opentelemetry-transport → same OTLP endpoint as a parallel transport
   │
   ├──► rebno-prod machine (same shape)
   │       │   (Litestream replicates to a DIFFERENT Tigris bucket)
   │       └── (no STAGING_INVITE_TOKEN; allowlist absent; public WSS at rebno-prod.fly.dev)
   │
   └──► rebno-obs machine (region lax, shared-cpu-1x@1GB)
            ├── /data (Fly Volume) — OpenObserve WAL + caches
            ├── ZO_S3_BUCKET_NAME=<rebno-obs Tigris bucket>
            ├── Listens 5080 (HTTP UI + OTLP HTTP); 5081 (gRPC)
            └── flycast: rebno-obs.flycast:5080 (private 6PN ingestion path; no public exposure of OTLP)

Litestream replicate flow (per game-server machine):
   /data/rebno.db ──(sync-interval=1s)──► s3://<per-app-tigris-bucket>/rebno-staging-or-prod/
   On boot, restore step (in entrypoint): if /data/rebno.db missing, litestream restore -o /data/rebno.db s3://...
```

### Recommended Project Structure (additions over Phase 4)

```
apps/server/
├── Dockerfile                       # multi-stage; D-09; new in Phase 5
├── docker-entrypoint.sh             # 4-step entrypoint per D-09 + Litestream restore-or-replicate
├── litestream.yml                   # DEP-03; lives in image at /etc/litestream.yml
├── fly.staging.toml                 # DEP-02 — separate file per env (planner picks single-vs-dual; recommend dual)
├── fly.prod.toml
└── src/
    ├── otel-init.ts                 # DEP-06; --import-loaded BEFORE index.ts to capture http/express auto-instr
    └── staging-invite.ts            # D-04 middleware; STAGING_MODE-gated; mounted in index.ts before Colyseus

apps/obs/                            # third Fly app (rebno-obs)
├── Dockerfile                       # FROM public.ecr.aws/zinclabs/openobserve:v0.14
└── fly.toml

.github/workflows/
├── deploy-staging.yml               # push to main → build → deploy → soak
├── deploy-prod.yml                  # tag v*.*.* → resolve image SHA → deploy
├── trace-check.yml                  # required PR check; D-06 hard-gate
└── verify-phase-5.yml               # PR check on Phase-5 paths; mirrors verify-phase-4.yml

scripts/
├── verify-phase-5.mjs               # composite gate; mirrors verify-phase-4.mjs
└── soak-staging.mjs                 # 30-min 2-client soak; reuses authority.integ.test.ts harness logic

tools/scripts/
└── lint-deploy-stack.mjs            # drift guard across fly.toml + Dockerfile + litestream.yml + otel config

docs/
├── adr/
│   ├── 0005-deploy-topology.md      # records D-01..D-04
│   └── 0006-observability-stack.md  # records D-12..D-16
└── runbooks/
    └── RESTORE.md                   # DEP-07; planner can also place at repo root
```

### Pattern 1: Multi-stage Dockerfile (DEP-01)

**What:** Two stages — `builder` (full pnpm + native build toolchain) and `runtime` (slim, only `dist/` + `node_modules/` + native `.node` files).

**When to use:** Always for Phase-5 server image. The Litestream binary is COPY'd from `litestream/litestream:0.3.13` image into the runtime stage.

**Example skeleton:**

```dockerfile
# [doc->REQ-DEP-01]
# Source: combined from Fly.io blog "All In on SQLite + Litestream"
# (https://fly.io/blog/all-in-on-sqlite-litestream/) + Litestream Docker
# guide (https://litestream.io/guides/docker/).

# ---- Stage 1: build ----
FROM node:22-bookworm-slim AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential python3 ca-certificates \
 && rm -rf /var/lib/apt/lists/*
RUN corepack enable && corepack prepare pnpm@10 --activate
WORKDIR /app
COPY pnpm-workspace.yaml package.json pnpm-lock.yaml ./
COPY apps/server/package.json apps/server/
COPY packages/protocol/package.json packages/protocol/
COPY packages/game-logic/package.json packages/game-logic/
COPY packages/db/package.json packages/db/
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm --filter @rebno/protocol --filter @rebno/game-logic --filter @rebno/db --filter @rebno/server build
# Prune to production deps for runtime
RUN pnpm --filter @rebno/server deploy --prod --legacy /tmp/server-prod

# ---- Stage 2: litestream binary ----
FROM litestream/litestream:0.3.13 AS litestream

# ---- Stage 3: runtime ----
FROM node:22-bookworm-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates dumb-init \
 && rm -rf /var/lib/apt/lists/*
COPY --from=litestream /usr/local/bin/litestream /usr/local/bin/litestream
COPY --from=builder /tmp/server-prod /app
COPY apps/server/litestream.yml /etc/litestream.yml
COPY apps/server/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
ENV NODE_ENV=production
WORKDIR /app
EXPOSE 2567
ENTRYPOINT ["dumb-init", "--", "/usr/local/bin/docker-entrypoint.sh"]
CMD ["node", "dist/index.js"]
```

`dumb-init` reaps zombie processes — important when running Litestream in background within the same container.

### Pattern 2: Container entrypoint (D-09 + Litestream restore-or-replicate)

```sh
#!/bin/sh
# [doc->REQ-DEP-01] [doc->REQ-DEP-03]
# apps/server/docker-entrypoint.sh
set -e

DB_PATH="${DATABASE_URL:-/data/rebno.db}"

# 1) Restore from Tigris if /data is empty (cold start / new volume).
if [ ! -f "$DB_PATH" ]; then
  echo "[entrypoint] $DB_PATH missing — restoring from Litestream replica"
  litestream restore -if-replica-exists -o "$DB_PATH" "$DB_PATH" || \
    echo "[entrypoint] no replica found; will start with empty DB"
fi

# 2) Pre-migrate snapshot (D-09 safety net).
mkdir -p /data/snapshots
if [ -f "$DB_PATH" ]; then
  cp "$DB_PATH" "/data/snapshots/pre-migrate-$(date +%s).db" || true
fi

# 3) Run Drizzle migrations. Failure → exit non-zero → container crashloop → Fly health check fails.
echo "[entrypoint] running drizzle-kit migrate"
node node_modules/drizzle-kit/bin.cjs migrate

# 4) Start Litestream replicate in background.
echo "[entrypoint] starting litestream replicate"
litestream replicate -config /etc/litestream.yml &
LITESTREAM_PID=$!

# 5) Forward signals to Litestream too.
trap 'kill -TERM $LITESTREAM_PID 2>/dev/null; wait $LITESTREAM_PID 2>/dev/null' TERM INT

# 6) Exec node — replaces shell PID; receives SIGTERM directly from Fly.
exec "$@"
```

**Why exec on the final line:** Fly sends SIGTERM to PID 1; if the entrypoint stays as shell PID 1, the Node process never receives the signal and the SIGTERM grace handler from `apps/server/src/sigterm.ts` doesn't fire. `dumb-init` + `exec` ensures Node is PID 1 of the container. [CITED: fly.io/docs/machines/guides-examples/private-applications-flycast/ + Linux signal docs]

### Pattern 3: Litestream config (DEP-03)

```yaml
# [doc->REQ-DEP-03]
# apps/server/litestream.yml — copied to /etc/litestream.yml in the image.
# Source: https://litestream.io/reference/config/ + https://fly.io/docs/tigris/

# Tigris credentials are auto-injected by `fly storage create`:
#   AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL_S3, BUCKET_NAME
# Litestream reads LITESTREAM_ACCESS_KEY_ID / LITESTREAM_SECRET_ACCESS_KEY by default,
# so the entrypoint exports them from the AWS_* vars before invoking litestream.
# See entrypoint section in this RESEARCH.md.

dbs:
  - path: /data/rebno.db
    # Phase-4 D-15 already pins SQLite to journal_mode=WAL + synchronous=NORMAL.
    replicas:
      - type: s3
        endpoint: ${AWS_ENDPOINT_URL_S3}      # e.g. https://fly.storage.tigris.dev
        bucket: ${BUCKET_NAME}
        path: rebno
        region: auto                           # Tigris ignores region but field is required
        sync-interval: 1s                      # RPO target < 1s per DEP-03
        # 0.3.x: snapshot.interval is 24h default; tune via:
        # snapshot-interval: 6h
        # retention: 24h                        # how long replicas keep generations on Tigris
        # retention-check-interval: 1h
```

**RPO knobs (`litestream.io/reference/config/`):**
- `sync-interval` (default `1s`) — how often replicate polls WAL for new frames. **Set to `1s` for DEP-03 RPO < 1s claim.** Going below 1s is not officially supported.
- `snapshot-interval` (default `24h`) — full-database snapshot cadence; smaller = faster restore + larger storage.
- `retention` (default `24h`) — how long old snapshots live before GC.
- `retention-check-interval` (default `1h`) — how often Litestream evaluates retention.

**For DEP-08 (Fly WS proxy ↔ Colyseus pingInterval alignment):** Already structurally non-issue — Fly's WS idle timeout is 60s, Colyseus default `pingInterval` is 3s (Phase 4 boots it explicitly at 3000 in `apps/server/src/index.ts:283-287`). The 30-min soak verifies empirically. [VERIFIED: docs.colyseus.io/server/transport/ws + community.fly.io/t/does-the-tls-handler-close-idle-tcp-connections-after-60s/2373]

### Pattern 4: fly.toml (DEP-02) — recommended per-env files

```toml
# [doc->REQ-DEP-02] [doc->REQ-DEP-05]
# apps/server/fly.staging.toml
# Source: https://fly.io/docs/reference/configuration/ + Fly.io WebSockets blog.

app = "rebno-staging"
primary_region = "lax"

[build]
  # Image is built + pushed by GitHub Actions; flyctl deploy uses --image, not [build].
  # Block left empty intentionally.

[env]
  NODE_ENV = "production"
  STAGING_MODE = "1"            # D-04: enables STAGING_INVITE_TOKEN middleware
  LOG_LEVEL = "debug"           # D-16
  PORT = "2567"
  DATABASE_URL = "/data/rebno.db"
  ROOM_SIGNING_PRIVATE_KEY_PATH = "/data/keys/room_signing.ed25519"
  ROOMS_DIR = "/app/rooms"
  ALLOWED_ORIGINS = "https://staging.rebno.decidel.com"
  # OTel
  OTEL_EXPORTER_OTLP_ENDPOINT = "http://rebno-obs.flycast:5080/api/default"
  OTEL_RESOURCE_ATTRIBUTES = "service.name=rebno-server,deployment.environment=staging"

[[mounts]]
  source = "rebno_data"
  destination = "/data"
  initial_size = "10gb"

[[services]]
  internal_port = 2567
  protocol = "tcp"
  auto_stop_machines = "off"
  auto_start_machines = false
  min_machines_running = 1

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true
  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]      # WS upgrade rides on this — Fly proxies WS over TLS automatically

  [[services.http_checks]]
    interval = "10s"
    grace_period = "30s"
    method = "get"
    path = "/health"
    protocol = "http"
    timeout = "2s"
    tls_skip_verify = false

[[vm]]
  size = "shared-cpu-2x"            # argon2id needs CPU; shared-cpu-1x is too slow per Phase-4 D-07
  memory = "2gb"
```

**Why two files vs single + env override:** ADR 0002/0003 already use ADR-on-first-lock. Two files keep diff history per env trivially navigable when an operator wonders "what did I change in prod last release". Planner can collapse to single + `--env` flags if monorepo cookbook prefers; both patterns are documented at `fly.io/docs/launch/monorepo/`. [CITED]

### Pattern 5: OTel + pino bootstrap (DEP-06)

```typescript
// [doc->REQ-DEP-06]
// apps/server/src/otel-init.ts
// Loaded BEFORE index.ts via `node --import ./dist/otel-init.js dist/index.js`
// in docker-entrypoint.sh (auto-instrumentations require pre-load to patch require).
// Source: https://opentelemetry.io/docs/languages/js/instrumentation/

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs';

// CONTEXT D-12: failure to ship to OpenObserve is non-fatal — game server keeps logging to stdout.
try {
  const base = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
  if (!base) {
    console.warn('[otel-init] OTEL_EXPORTER_OTLP_ENDPOINT not set; OTel disabled');
  } else {
    const sdk = new NodeSDK({
      traceExporter: new OTLPTraceExporter({ url: `${base}/v1/traces` }),
      spanProcessor: new BatchSpanProcessor(new OTLPTraceExporter({ url: `${base}/v1/traces` })),
      metricReader: new PeriodicExportingMetricReader({
        exporter: new OTLPMetricExporter({ url: `${base}/v1/metrics` }),
        exportIntervalMillis: 30_000,
      }),
      logRecordProcessor: new BatchLogRecordProcessor(
        new OTLPLogExporter({ url: `${base}/v1/logs` }),
      ),
      instrumentations: [getNodeAutoInstrumentations({
        '@opentelemetry/instrumentation-fs': { enabled: false },        // noisy
        '@opentelemetry/instrumentation-pino': { enabled: true },
      })],
    });
    sdk.start();
    process.on('SIGTERM', async () => {
      try { await sdk.shutdown(); } catch (e) { console.warn('[otel-init] shutdown failed', e); }
    });
    console.log('[otel-init] SDK started; OTLP endpoint =', base);
  }
} catch (err) {
  console.warn('[otel-init] init failed; continuing without OTel:', err);
}
```

**Why BatchSpanProcessor (not SimpleSpanProcessor):** SimpleSpanProcessor exports each span synchronously; Batch defers to a worker queue and is the only safe choice for tick-loop-heavy workloads. [CITED: opentelemetry.io/docs/languages/js/instrumentation/]

**Pino → OTel:** `instrumentation-pino` injects `trace_id`/`span_id` fields into pino log records. To also emit those records as OTel LogRecords (so they appear under `Logs` in OpenObserve), add a `pino-opentelemetry-transport` to the existing pino multi-stream config in `apps/server/src/log.ts` — runs alongside the existing stdout stream. [CITED: github.com/pinojs/pino-opentelemetry-transport]

### Pattern 6: GitHub Actions deploy-staging.yml shape (DEP-04)

```yaml
# [doc->REQ-DEP-04]
# .github/workflows/deploy-staging.yml
name: deploy-staging
on:
  push:
    branches: [main]
    paths:
      - 'apps/server/**'
      - 'apps/obs/**'
      - 'packages/**'
      - 'tools/**'
      - 'pnpm-workspace.yaml'
      - 'pnpm-lock.yaml'
      - '.github/workflows/deploy-staging.yml'

concurrency:
  group: deploy-staging
  cancel-in-progress: false   # never cancel a deploy in progress

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with: { version: 10 }
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: 'pnpm' }
      - run: pnpm install --frozen-lockfile
      - run: pnpm verify:phase-4
      - run: pnpm verify:phase-5
      - run: pnpm trace:check          # D-06 hard-gate

      - uses: superfly/flyctl-actions/setup-flyctl@fc53c09     # v1.5 SHA-pinned
      - name: Build + push image
        run: |
          IMAGE="registry.fly.io/rebno-staging:${{ github.sha }}"
          flyctl auth docker
          docker buildx build \
            --tag "$IMAGE" \
            --push \
            --file apps/server/Dockerfile \
            .
          echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

      - name: flyctl deploy
        run: flyctl deploy -a rebno-staging --image "$IMAGE" --config apps/server/fly.staging.toml --strategy immediate
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

      # 30-min soak runs as a separate dispatched job (NOT same job — see Pattern 7).
      - name: Trigger soak
        run: gh workflow run soak-staging.yml -F sha=${{ github.sha }}
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

### Pattern 7: 30-min soak as a separate workflow

The CONTEXT.md says "scripted 30-min 2-client soak post-deploy." Naive placement (inline in deploy-staging.yml) burns 30 minutes on every push to main. Pragmatic placement:

```yaml
# .github/workflows/soak-staging.yml
name: soak-staging
on:
  workflow_dispatch:
    inputs:
      sha: { required: true, type: string }
  schedule:
    - cron: '0 6 * * *'        # nightly belt-and-suspenders run

concurrency:
  group: soak-staging
  cancel-in-progress: true     # if a new deploy fires, abort the old soak

jobs:
  soak:
    runs-on: ubuntu-latest
    timeout-minutes: 35
    steps:
      - uses: actions/checkout@v4
        with: { ref: ${{ inputs.sha || github.sha }} }
      - uses: pnpm/action-setup@v4
        with: { version: 10 }
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: 'pnpm' }
      - run: pnpm install --frozen-lockfile
      - name: 30-min 2-client soak
        run: pnpm soak:staging
        env:
          STAGING_WSS_URL: wss://rebno-staging.fly.dev
          STAGING_INVITE_TOKEN: ${{ secrets.STAGING_INVITE_TOKEN }}
          SOAK_DURATION_MINUTES: 30
```

**Failure path:** Soak red ≠ deploy red. The soak job opens a GitHub Issue via `gh issue create` on failure (operator decides next step). Keeping soak async means deploy-staging.yml remains <5 min and developer feedback is fast. The soak doubles as the DEP-08 idle-timeout verification (zero spurious WS disconnects asserted) per CONTEXT D-11.

### Pattern 8: Tag-based prod promotion (D-07) via image-SHA reuse

The cleanest mechanism is **annotated git tags pointing at the staging-deploy SHA**. The tag's commit IS the image SHA (because GitHub Actions uses `${{ github.sha }}` as the image tag). No git-notes side-channel needed:

```yaml
# .github/workflows/deploy-prod.yml
name: deploy-prod
on:
  push:
    tags: ['v*.*.*']

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }      # need full history to resolve tag SHA
      - name: Resolve image SHA from tag
        id: sha
        run: |
          TAG_SHA=$(git rev-parse ${{ github.ref_name }}^{commit})
          echo "image=registry.fly.io/rebno-staging:${TAG_SHA}" >> "$GITHUB_OUTPUT"
      - uses: superfly/flyctl-actions/setup-flyctl@fc53c09
      - name: Deploy same image to prod
        run: |
          flyctl deploy -a rebno-prod --image "${{ steps.sha.outputs.image }}" \
            --config apps/server/fly.prod.toml --strategy rolling
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
```

Note: prod pulls the **staging registry image** (registry.fly.io/rebno-staging:<sha>). Each Fly app has its own ACL on its registry repo by default; the simple fix is to also push to `rebno-prod:<sha>` in the staging build job (one extra `docker push` line) so prod has access. Planner picks: dual-tag at build time (simpler) vs cross-app pull (one less push but requires Fly ACL config). **Recommend dual-tag at build time.**

### Anti-Patterns to Avoid

- **Running `drizzle-kit migrate` in app code on boot.** It works but couples migration semantics to app startup; the lock is D-09 entrypoint-driven so `node dist/index.js` only runs after migrations succeed. Keeps the failure mode binary (entrypoint exits non-zero → container crashloop → no traffic).
- **Using `DATABASE_URL` from a `fly secrets`.** Phase-4 reads it from env, but DATABASE_URL in Phase 5 is a path on the volume — set in `[env]` of fly.toml, not as a secret.
- **Setting `auto_stop_machines = "stop"`.** D-02 hard-locks `auto_stop_machines = "off"` + `min_machines_running = 1`. WS server cold-start defeats Colyseus reconnection grace.
- **Putting OpenObserve behind a public Fly proxy with HTTP basic auth.** D-15 says IP allowlist + admin password; planner should route OTLP via flycast (`rebno-obs.flycast`) so the public OTLP path is never exposed (game-server → flycast 6PN → obs).
- **Logging argon2 inputs / hashes.** Phase-4 D-23 redact list is `passwords, session tokens, argon2id_hash, legacy_hash`; Phase-5 OTel pipeline must inherit these via pino's existing `redact` config.
- **Embedding plaintext localList.txt in the Docker image.** Specifics §"Plaintext `localList.txt`" — would persist in registry layers. D-17 ssh-sftp ritual is the only path.
- **Skipping pre-migrate snapshot when DB is fresh.** Idempotency: snapshot copies an empty DB happily; cost is microseconds.
- **Using same BETTER_AUTH_SECRET on staging + prod.** D-18 — different secret per env. Rotation invalidates sessions; cross-env leakage is uncapped.

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Database backups | scp + cron + tar | Litestream | Streaming WAL replication with frame-level RPO; restore tool exists |
| Process supervision in container | Custom shell loop | `dumb-init` | Signal forwarding, zombie reaping; tested at scale |
| OTel exporter | Custom HTTP POST of log events | `@opentelemetry/exporter-*-otlp-http` | Batching, retries, back-pressure; spec-conformant |
| WS health probe | TCP-port-open check | Fly `[[services.http_checks]]` calling `/health` (already exists at `apps/server/src/health.ts`) | Phase-4 already tests it [VERIFIED: codebase] |
| GitHub Actions flyctl install | `curl https://fly.io/install.sh \| sh` | `superfly/flyctl-actions/setup-flyctl@<sha>` | Cached install, version pinning; SHA-pinning prevents supply-chain on master |
| Pino → OTel bridge | Custom transport | `pino-opentelemetry-transport` | Maintained by pino team; wires log level, span context, resource attributes correctly |
| `drizzle migrate` in app | `tx.run('CREATE TABLE …')` | `drizzle-kit migrate` (CLI) | Tracked migration table, idempotent, transactional per migration file |
| Image tagging | Date-based / branch-based | git SHA (`${{ github.sha }}`) | Immutable; survives history rewrites; matches D-07 promotion model |
| OTLP collector | Hand-write HTTP transport in Node | OpenObserve's native OTLP-HTTP endpoint | Spec-conformant; no Collector binary needed |

**Key insight:** Phase 5 is almost entirely glue between off-the-shelf components (Fly + Litestream + OpenObserve + OTel SDK + GitHub Actions). The only original code is `staging-invite.ts`, `otel-init.ts`, `lint-deploy-stack.mjs`, and `soak-staging.mjs` — everything else is configuration files (Dockerfile, fly.toml, litestream.yml, *.yml workflows, RESTORE.md prose).

## Runtime State Inventory

> Phase 5 has migration/runtime characteristics — this section applies. Phase 5 is greenfield deploy infra (no existing Fly resources to rename) but inherits Phase-4 runtime state plus introduces new Fly-resident state.

| Category | Items Found | Action Required |
|----------|-------------|------------------|
| Stored data | Phase-4 SQLite at `/data/rebno.db` (none yet — Phase 5 creates it). `legacy_credentials_staging` table populated once per env via D-17 ritual, then drained by tryLegacyLogin per CONTEXT D-04. | New volume = empty DB. Restore from Tigris on cold start. RESTORE.md documents restore. |
| Live service config | Three Fly apps (rebno-staging, rebno-prod, rebno-obs) each with config baked into their fly.toml (in git) + secrets set at runtime via `fly secrets`. Three Tigris buckets (one per app) created via `fly storage create`. | Document creation order in RESTORE.md (apps must exist before `fly storage create`). |
| OS-registered state | None — Fly machines are stateless containers. Volumes carry state across restarts but not across `fly volumes destroy`. | None for Phase 5. |
| Secrets/env vars | New per-app: `BETTER_AUTH_SECRET` (D-18), `STAGING_INVITE_TOKEN` (D-04, staging only), `ZO_ROOT_USER_PASSWORD` (D-15, on rebno-obs only), Tigris-injected `AWS_*` (auto, per-app). | Document each in RESTORE.md per-secret table. Generate via `openssl rand -base64 32` (or `-base64 24` for ZO password per D-15). |
| Build artifacts / installed packages | Docker image at `registry.fly.io/rebno-{staging,prod}:<sha>` per push. Fly registry retains old images; cleanup is manual `flyctl image prune` (rare). | None blocking. Document image-tag conventions in RESTORE.md. |

**Phase-4 carry-forward state that Phase 5 inherits but does not modify:**
- `apps/server/src/health.ts` — used as-is
- `apps/server/src/sigterm.ts` — used as-is; `setDraining` already wired to flip `/health` 503 before Fly drains the LB pool
- `apps/server/scripts/migrate-legacy-accounts.ts` — used as-is, invoked once per env via `fly ssh console`
- `apps/server/argon2-bench.mjs` — re-run on first staging deploy per HUMAN-UAT.md Test 3
- `apps/server/test/authority.integ.test.ts` — harness logic copied/extended into `scripts/soak-staging.mjs`

## Common Pitfalls

### Pitfall 1: Native modules fail to install on Alpine despite "prebuilds available"
**What goes wrong:** `pnpm install` on `node:22-alpine` falls back to building from source after failing to download a prebuild; build fails because Python or build-essential isn't in the image.
**Why it happens:** argon2 + better-sqlite3 push prebuilds for musl, but the matching N-API version + Node ABI must align. better-sqlite3 12.9.0 + Node 22 has a documented case where it tries to download from `unofficial-builds.nodejs.org` which is blocked from some Fly egress paths. [CITED: github.com/WiseLibs/better-sqlite3/issues/1397]
**How to avoid:** Use `node:22-bookworm-slim` (glibc) for v1. The 25 MB size penalty is not worth the troubleshooting.
**Warning signs:** `Error: prebuild-install info begin Prebuild-install version 7.x.x install` followed by gyp errors in image build logs.

### Pitfall 2: SIGTERM not delivered to Node because of intermediate shell
**What goes wrong:** Fly stops the machine, sends SIGTERM, waits 30s, sends SIGKILL. If Node is a child of `sh -c "node …"`, the shell receives SIGTERM and Node never sees it. `runGraceShutdown()` in `apps/server/src/sigterm.ts` never fires; players get raw WS close instead of SERVER_DRAINING.
**How to avoid:** Use `dumb-init` or `tini` as PID 1, and `exec` the final node command in the entrypoint.
**Warning signs:** Logs show no `SIGTERM received — beginning grace shutdown (CONTEXT D-16)` line during deploy; clients see "WS close 1006" instead of SERVER_DRAINING event.

### Pitfall 3: Litestream replicate runs but doesn't survive container exit
**What goes wrong:** Background `litestream replicate &` is a child of the entrypoint shell; when shell exits (which is what `exec node` causes), bash kills children. WAL replication silently stops; no errors visible.
**How to avoid:** Use a `trap` in the entrypoint that explicitly signals litestream PID on SIGTERM (the entrypoint snippet above does this). Better: use `s6-overlay` or `supercronic` — but for a 1-binary supervisor, the trap pattern is sufficient at this scale. Verify via OpenObserve metric `litestream_replication_lag_seconds` post-deploy.
**Warning signs:** RPO climbs steadily on staging soak; no Litestream logs after first ~10 seconds of boot.

### Pitfall 4: `drizzle-kit migrate` requires `drizzle.config.ts` at runtime
**What goes wrong:** The migrate CLI looks for `drizzle.config.ts` (or .js) in CWD by default to find the migrations folder. In a slim runtime image without drizzle-kit's full toolchain, the import chain may fail.
**How to avoid:** Pass `--config <path>` explicitly OR use the programmatic API (`import { migrate } from 'drizzle-orm/better-sqlite3/migrator'`). The programmatic path is 5 lines of TS in a tiny `apps/server/scripts/run-migrations.ts` and avoids needing drizzle-kit in the runtime image at all. This is the recommended path for production. [CITED: orm.drizzle.team/docs/migrations]
**Warning signs:** Entrypoint fails at "drizzle-kit: command not found" or "Cannot find module './drizzle.config.ts'".

### Pitfall 5: OpenObserve OTLP path requires `/api/<org>/v1/<signal>` not `/v1/<signal>`
**What goes wrong:** OTel SDK exporters auto-append `/v1/traces` to the configured base. OpenObserve expects `/api/default/v1/traces` (where `default` is the org). If you set `OTEL_EXPORTER_OTLP_ENDPOINT=http://rebno-obs.flycast:5080`, the exporter sends to `http://rebno-obs.flycast:5080/v1/traces` and OpenObserve 404s.
**How to avoid:** Set `OTEL_EXPORTER_OTLP_ENDPOINT=http://rebno-obs.flycast:5080/api/default` — exporters append `/v1/traces` to give the correct full path. [CITED: openobserve.ai/docs/ingestion/logs/otlp/]
**Warning signs:** Game server logs `OTLP exporter failed: 404 Not Found`; OpenObserve has zero ingest events.

### Pitfall 6: Fly `auto_stop_machines = "stop"` breaks WS reconnection grace
**What goes wrong:** With auto-stop, if all clients disconnect for 5min the machine stops. When clients reconnect within Phase-4's 10s grace window the machine is asleep; reconnect handshake times out; SRV-06 grace is silently broken.
**How to avoid:** D-02 hard-locks `auto_stop_machines = "off"` + `min_machines_running = 1`. Do not deviate.
**Warning signs:** `RECONNECT_FAILED` events in pino logs after the staging machine has been idle.

### Pitfall 7: Tigris bucket isn't pre-created — OpenObserve crashes on boot
**What goes wrong:** `ZO_S3_BUCKET_NAME=foo` but `foo` doesn't exist; OpenObserve fails fast with no recovery — it does NOT create the bucket. [CITED: openobserve.ai/docs/environment-variables/]
**How to avoid:** Create the bucket via `fly storage create -a rebno-obs` BEFORE first deploy of rebno-obs. Document order in RESTORE.md.
**Warning signs:** OpenObserve logs `S3 bucket does not exist`; container crashloops.

### Pitfall 8: SQLite WAL file deleted between Litestream restore and entrypoint
**What goes wrong:** Litestream restore writes `/data/rebno.db` plus WAL/SHM. If `drizzle-kit migrate` opens with different journal mode than the restored DB, you see `database is locked` or rollback artifacts.
**How to avoid:** Phase-4 D-15 already pins WAL+NORMAL via `apps/server/src/db.ts` PRAGMA. Verify same PRAGMAs are set BEFORE migrations run. The programmatic migrator path (Pitfall 4) lets you `db.pragma('journal_mode = WAL')` first, then `migrate(db, …)`.
**Warning signs:** `SQLITE_BUSY` or `database is locked` during entrypoint migration step.

### Pitfall 9: STAGING_INVITE_TOKEN middleware mounts after Colyseus matchmaking middleware
**What goes wrong:** Phase-4 `index.ts` mounts `createNodeMatchmakingMiddleware()` first (line 100). If `staging-invite.ts` mounts AFTER it, joinOrCreate POSTs hit matchmaking before invite check — invite gate is bypassed for HTTP matchmaking calls (WS upgrade may still be gated, but joinOrCreate creates a reservation that subsequent WS upgrade honors).
**How to avoid:** Mount `staging-invite.ts` BEFORE `createNodeMatchmakingMiddleware()`. The middleware should:
1. Read query string `?invite=<token>` from req.url OR `Authorization: Bearer <token>` header
2. If `process.env.STAGING_MODE !== '1'`, call `next()` immediately (zero-cost no-op for prod)
3. Compare with `process.env.STAGING_INVITE_TOKEN` (timing-safe via `crypto.timingSafeEqual`)
4. Return 401 on mismatch BEFORE matchmaking sees the request

For WS handshake gating, Colyseus 0.17 exposes `RebnoRoom.onAuth(client, options, request)` — `options` includes `?invite=` from joinOrCreate's options arg, but a pre-Express middleware also gates the matchmaking POST. Two-layer is belt-and-suspenders. [CITED: docs.colyseus.io/server/transport/ws + Phase-4 `index.ts:100`]

### Pitfall 10: pino redact list silently broken when adding pino-otel transport
**What goes wrong:** Phase-4 D-23 redacts `passwords, session tokens, argon2id_hash, legacy_hash`. Adding `pino-opentelemetry-transport` to a multi-stream config: redaction applies BEFORE transport, BUT only if the stream is listed in the same `pino({ redact, transport: { targets: [...] } })` call. Naively appending a transport via a second pino instance loses redaction for that stream.
**How to avoid:** Use pino's multi-target transport syntax exclusively: `pino({ redact: [...], transport: { targets: [{ target: 'pino/file', options: { destination: 1 } }, { target: 'pino-opentelemetry-transport', options: {...} }] } })`. Verify with a unit test that asserts a redacted token never reaches the OTel exporter.
**Warning signs:** OpenObserve "logs" stream shows session tokens or argon2 hashes.

### Pitfall 11: Litestream 0.5 LTX format incompatible with 0.3 generations
**What goes wrong:** Phase 5 ships 0.3.13 → operator upgrades to 0.5.x in v2 → restore from 0.3-era Tigris generations fails because LTX format is new in 0.5. [CITED: simonwillison.net "Litestream v0.5.0 is Here" + mtlynch.io "Hold off on Litestream 0.5.0"]
**How to avoid:** Decide pin once and document upgrade in RESTORE.md. If choosing 0.5.x, verify `ltx` CLI works for restore validation; if 0.3.x, plan a v2 migration runbook.
**Warning signs:** Future operator runs `litestream restore` and gets format errors.

## Code Examples

### Example: STAGING_INVITE_TOKEN middleware

```typescript
// [doc->REQ-DEP-02] [impl->REQ-DEP-02]
// apps/server/src/staging-invite.ts
// Source: CONTEXT D-04. Mounted in index.ts BEFORE createNodeMatchmakingMiddleware.

import { timingSafeEqual } from 'node:crypto';
import type { Request, Response, NextFunction } from 'express';

export function makeStagingInvite(env = process.env) {
  const enabled = env.STAGING_MODE === '1';
  const expected = env.STAGING_INVITE_TOKEN ?? '';
  const expectedBuf = Buffer.from(expected);

  return function stagingInvite(req: Request, res: Response, next: NextFunction) {
    if (!enabled) return next();             // zero-cost no-op on prod
    if (!expected) {
      res.status(503).json({ error: 'STAGING_MODE=1 but STAGING_INVITE_TOKEN unset' });
      return;
    }
    const provided =
      (req.query.invite as string | undefined) ??
      (req.headers.authorization?.replace(/^Bearer\s+/, '') ?? '');
    const providedBuf = Buffer.from(provided);
    if (
      providedBuf.length !== expectedBuf.length ||
      !timingSafeEqual(providedBuf, expectedBuf)
    ) {
      res.status(401).json({ error: 'invite token required' });
      return;
    }
    next();
  };
}
```

For WS handshake-time enforcement (after the matchmaking POST has been gated), `RebnoRoom.onAuth` reads `options.invite` and rejects symmetrically. Belt-and-suspenders.

### Example: Programmatic Drizzle migrate (preferred over CLI in container)

```typescript
// [doc->REQ-DEP-01]
// apps/server/scripts/run-migrations.ts
// Avoids needing drizzle-kit in runtime image (Pitfall 4).

import Database from 'better-sqlite3';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import { migrate } from 'drizzle-orm/better-sqlite3/migrator';
import { resolve } from 'node:path';

const url = process.env.DATABASE_URL ?? '/data/rebno.db';
const sqlite = new Database(url);
sqlite.pragma('journal_mode = WAL');
sqlite.pragma('synchronous = NORMAL');
const db = drizzle(sqlite);
migrate(db, { migrationsFolder: resolve('packages/db/migrations') });
sqlite.close();
console.log('[migrate] OK');
```

Entrypoint becomes: `node dist/scripts/run-migrations.js && exec node --import ./dist/otel-init.js dist/index.js`.

### Example: lint-deploy-stack.mjs skeleton

```js
// tools/scripts/lint-deploy-stack.mjs
// Drift guard across fly.toml, Dockerfile, litestream.yml, OTel base URL.
// Mirrors lint-rate-limit-budgets.mjs pattern.

import { readFileSync } from 'node:fs';
const errors = [];
const flyStaging = readFileSync('apps/server/fly.staging.toml', 'utf-8');
const flyProd = readFileSync('apps/server/fly.prod.toml', 'utf-8');
const dockerfile = readFileSync('apps/server/Dockerfile', 'utf-8');
const litestream = readFileSync('apps/server/litestream.yml', 'utf-8');

if (!flyStaging.includes('STAGING_MODE = "1"')) errors.push('fly.staging.toml missing STAGING_MODE=1');
if (flyProd.includes('STAGING_MODE')) errors.push('fly.prod.toml MUST NOT set STAGING_MODE');
if (!flyStaging.includes('auto_stop_machines = "off"')) errors.push('fly.staging.toml: auto_stop_machines must be "off"');
if (!flyProd.includes('auto_stop_machines = "off"')) errors.push('fly.prod.toml: auto_stop_machines must be "off"');
if (!flyStaging.match(/min_machines_running\s*=\s*1/)) errors.push('fly.staging.toml: min_machines_running must be 1');
if (!dockerfile.includes('node:22-bookworm-slim')) errors.push('Dockerfile must use node:22-bookworm-slim base');
if (!dockerfile.match(/FROM litestream\/litestream:[\d.]+/)) errors.push('Dockerfile must pin Litestream version');
if (!litestream.includes('sync-interval: 1s')) errors.push('litestream.yml: sync-interval must be 1s for DEP-03 RPO');
if (!flyStaging.includes('rebno-obs.flycast')) errors.push('fly.staging.toml OTEL endpoint must use flycast');

if (errors.length) {
  errors.forEach((e) => console.error('lint-deploy-stack:', e));
  process.exit(1);
}
console.log('lint-deploy-stack: OK');
```

## Project Constraints (from CLAUDE.md)

| Directive | Section | Phase 5 Implication |
|-----------|---------|---------------------|
| Server-authoritative; never trust client positions/scores/chat origin | Hard Rules #1 | Soak test (`scripts/soak-staging.mjs`) MUST NOT introduce client-trusted shortcuts; reuse `apps/server/test/authority.integ.test.ts` zod-strict schemas (PITFALLS B1) |
| No faithful port of plaintext passwords — argon2id from packet 1 | Hard Rules #2 | RESTORE.md "legacy-creds staging-table read-once-then-purge" step is non-optional. The localList.txt seed file is deleted from `/data/seed/` after migration |
| No "run clipboard as superuser" admin | Hard Rules #3 | Admin UI is Phase 7 PAR-07. Phase 5 must not deploy any admin endpoint. `apps/server/src/admin-stubs.ts` already throws `NotImplementedInPhase4Error` |
| Repo stays private through Phase 7 | Hard Rules #8 | CI workflow secrets (`FLY_API_TOKEN`, `STAGING_INVITE_TOKEN`) gated on private repo. RESTORE.md must NEVER be pushed to a public preview environment |
| Each plan = one commit, message references REQ-IDs | Conventions | Phase-5 plan commits use format `feat(05-N): <message> (DEP-NN)` per Phase-4 commit precedent |
| `[<doc>->REQ-DEP-NN]` / `[<impl>->REQ-DEP-NN]` / `[<unit>->REQ-DEP-NN]` / `[<int>->REQ-DEP-NN]` tagging | Traceable-reqs contract | Every Phase-5 artifact carries the appropriate stage tag in a comment. Default `required_stages = ["doc"]` per REQ; planner adds `impl`/`int` per DEP-NN at plan-time |
| `pnpm trace:check` required CI gate | Requirements Traceability | D-06: trace-check.yml on PR is the hard-gate. Closes Phase-4 carry-forward DEP-04 finding |
| TypeScript everywhere; strict mode; Markdown docs; ADRs at `docs/adr/NNNN-title.md` | Conventions | New code: `staging-invite.ts`, `otel-init.ts` in TS. ADRs: `0005-deploy-topology.md`, `0006-observability-stack.md`. Lint scripts: `.mjs` per Phase-2/3/4 precedent |
| Linux is the determinism reference platform | Phase-3 D-22, Phase-4 verify-phase-4.yml | All Phase-5 verify gates run on `ubuntu-latest`. Windows local execution is not authoritative |

## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| REQ-DEP-01 | Multi-stage Dockerfile (Alpine/musl-compatible) | Pattern 1 + Pitfall 1 — recommend bookworm-slim over alpine; verified argon2 + better-sqlite3 prebuild availability for both [VERIFIED: npm + GH issues] |
| REQ-DEP-02 | fly.toml deploys to single Fly machine + persistent volume | Pattern 4 — fly.staging.toml + fly.prod.toml schema; `auto_stop_machines = "off"`, `min_machines_running = 1` per D-02 [CITED: fly.io/docs/reference/configuration/] |
| REQ-DEP-03 | Litestream sidecar replicates SQLite WAL to Tigris bucket — RPO < 1s | Pattern 3 — litestream.yml with `sync-interval: 1s`; Tigris auto-injected env vars via `fly storage create`; entrypoint runs `litestream replicate` as background [VERIFIED: litestream.io/reference/config/ + fly.io/docs/tigris/] |
| REQ-DEP-04 | GitHub Actions: build + test + deploy on push to main | Pattern 6 + 7 + 8 — deploy-staging.yml + deploy-prod.yml + soak-staging.yml + trace-check.yml; image-SHA reuse via dual-tag [VERIFIED: github.com/superfly/flyctl-actions] |
| REQ-DEP-05 | /health HTTP endpoint reports liveness + WS readiness; Fly health checks consume it | Codebase: `apps/server/src/health.ts` already returns `{status, ws_ready, rooms_loaded}`. `[[services.http_checks]]` block in fly.toml calls `/health` every 10s [VERIFIED: codebase] |
| REQ-DEP-06 | Structured pino JSON logs shipped to a destination; upgrade path documented | Pattern 5 — pino → stdout (Fly logs viewer) AND pino-opentelemetry-transport → OTLP-HTTP → OpenObserve on rebno-obs. Failure non-fatal per D-12 [VERIFIED: npm pino-opentelemetry-transport@3.0.0] |
| REQ-DEP-07 | RESTORE.md documents <5 min restore (validated on staging) | RESTORE.md template in §"Validation Architecture" + Phase-4 carry-forward verification (kill -9, argon2 bench, multi-client smoke) embedded as named acceptance steps [VERIFIED: codebase 04-09-SUMMARY.md] |
| REQ-DEP-08 | Fly WS proxy idle timeout aligned with Colyseus pingInterval | DEP-08 is structurally non-issue: Fly idle = 60s; Colyseus pingInterval default = 3s; Phase-4 explicitly pins 3000 in `apps/server/src/index.ts:283-287`. 30-min soak verifies empirically [VERIFIED: codebase + docs.colyseus.io/server/transport/ws] |

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| OTel collector sidecar binary | In-process `@opentelemetry/sdk-node` for low-traffic Node services | 2023+ — sdk-node matured | At <50 CCU, in-process overhead is ~5 MB and saves operating a second binary |
| Litestream 0.3.x with WAL replication | Litestream 0.5.x with LTX format and faster restore | 2025-10 (0.5.0 release) | 0.5.x is faster but had 0.5.0 bugs; community advice as of 2026-04 leans 0.5.2+ for new deploys, 0.3.x for stability-first projects [CITED: simonwillison.net/2025/Oct/3/litestream/] |
| `node:18-alpine` for size | `node:22-bookworm-slim` for native-module reliability | 2024+ — Snyk + Node WG recommendation | Glibc compatibility is more reliable for native modules on Node 22 [CITED: snyk.io/blog/choosing-the-best-node-js-docker-image/] |
| Custom WS supervisor in app | Colyseus's built-in graceful shutdown + Fly proxy 30s SIGTERM grace | Phase-4 (already shipped) | Phase 5 just consumes it — `colyseus.gracefullyShutdown(false)` is the canonical signal source |
| `pino-multi-stream` | pino multi-target transport via `transport: { targets: [...] }` | Pino 7+ | Single redact pipeline for all streams (Pitfall 10) |
| Single-app Fly deploy | Two-app split (`-staging` + `-prod`) | Industry standard for any service with state | RESTORE.md validation lands on staging without prod blast-radius (CONTEXT D-01) |

**Deprecated/outdated:**
- **`flyctl deploy --build-only`** for image-only builds — superseded; use `docker buildx build --push` directly to `registry.fly.io/<app>:<tag>`. Provides explicit SHA control for D-07 promotion.
- **`pino-pretty` in production** — never. Production pino must be JSON-only for Fly logs viewer + OTel ingestion.
- **`@opentelemetry/api-logs` (legacy)** — superseded by `@opentelemetry/sdk-logs` BatchLogRecordProcessor [CITED: opentelemetry.io/docs/languages/js/exporters/]

## Assumptions Log

| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | OpenObserve `v0.14.x` is the current stable line at plan time | Standard Stack | LOW — exact version may be 0.13 or 0.15 by plan-day; planner runs `gh api repos/openobserve/openobserve/releases/latest` to confirm before pinning. Falls back to `latest` tag if unsure. |
| A2 | Litestream `0.3.13` is the conservative production pin | Open Question Q1 | MEDIUM — community signals are mixed; some projects have moved to 0.5.2+. ADR 0002 doesn't mandate either. The user/operator should pick before plan execution. |
| A3 | `node:22-bookworm-slim` is the right base over alpine | Standard Stack + Pitfall 1 | LOW — well-supported by current docs, both options work; alpine saves ~25 MB but adds risk. |
| A4 | flyctl-actions v1.5 SHA `fc53c09` is the current production tag | Standard Stack | LOW — planner verifies via `gh api repos/superfly/flyctl-actions/releases/latest --jq .target_commitish` at plan time. |
| A5 | Two-layer staging gate (Fly IP allowlist + STAGING_INVITE_TOKEN) is meaningfully more secure than either alone | Pattern 9 + CONTEXT D-04 | LOW — defense-in-depth pattern; user already locked it. |
| A6 | OpenObserve UI gated by IP allowlist alone is acceptable for v1 | CONTEXT D-15 | LOW — user explicitly accepted; production SSO is v2. |
| A7 | Drizzle-kit migrations table on Phase-4's `0001_baseline.sql` is initialized cleanly on a fresh DB | Pitfall 4 + Pattern 4 | MEDIUM — Phase-4 `bootstrapSchemaIfFresh` in `index.ts:372-408` runs the SQL directly without using the migrator's `__drizzle_migrations` tracking table. This means Phase 5's first `drizzle migrate` call will think the DB is empty and try to re-apply 0001. **Action:** Phase 5 must add an explicit "mark 0001 as applied" step on existing volumes, OR retire `bootstrapSchemaIfFresh` and let the migrator handle bootstrap. Researcher flags this as a real implementation hazard. |
| A8 | OTLP-HTTP path `/api/default/v1/{logs,metrics,traces}` matches OpenObserve's default org name | Pitfall 5 | LOW — `default` is OpenObserve's bootstrap org name. If renamed, planner updates the path. [CITED: openobserve.ai/docs/ingestion/logs/otlp/] |
| A9 | A 2 GB memory machine is sufficient for `apps/server` at <50 CCU + argon2 bench load | fly.staging.toml | LOW — Phase-4 D-07 argon2 memoryCost=65536 is 64 MiB per hash; at peak 50 concurrent auths that's 3.2 GB worst-case. **Action:** planner verifies during first staging argon2 bench; may need to bump to `4gb` if peak collisions arise. Production likely needs `shared-cpu-2x@2gb` minimum. |
| A10 | OpenObserve on `shared-cpu-1x@1024MB` is sufficient for <50 CCU log volume | CONTEXT D-14 | MEDIUM — search suggests 1 GB is below OpenObserve's "recommended" 8 GB but sufficient for this scale. **Action:** monitor RSS during first soak; bump to `shared-cpu-1x@2048MB` if `oom_kill` observed. [CITED: github.com/openobserve/openobserve/discussions/2711] |

**If this table is empty:** N/A — there are 10 assumed claims; planner and discuss-phase should validate A2, A7, A9, A10 before locking plan tasks.

## Open Questions

1. **Litestream 0.3.x or 0.5.x?**
   - What we know: 0.3.13 + 0.3.14 (patch only) are the long-stable production line; 0.5.x is the active development line with new LTX format + better restore semantics, but 0.5.0 had bugs that 0.5.2 closed. Industry as of 2026-04 has split opinion.
   - What's unclear: Litestream 0.5's LTX restore against Tigris specifically (most success stories are AWS S3).
   - Recommendation: Pin **0.3.13** for the initial Phase-5 deploy; document upgrade to 0.5.x in v2. ADR 0006 records the tradeoff. Planner: confirm with user during plan kickoff if they prefer 0.5.

2. **Phase-4's `bootstrapSchemaIfFresh` vs Phase-5's drizzle-kit migrate ownership** (Assumption A7)
   - What we know: Phase-4 boots the DB by running `0001_baseline.sql` raw via `sqlite.exec`; Phase-5 wants `drizzle-kit migrate` (or programmatic migrator) to own migrations.
   - What's unclear: how to reconcile on existing Phase-4 dev DBs (locally) and on the first staging deploy (volume already has `0001_baseline.sql`-applied schema but no `__drizzle_migrations` row).
   - Recommendation: Phase 5 plan adds a "seed `__drizzle_migrations` with the 0001 hash if accounts table exists but migration row missing" idempotent step inside `run-migrations.ts`. **This is a concrete plan-task to identify in the planning phase.**

3. **Single fly.toml with --env vs two files (`fly.staging.toml` + `fly.prod.toml`)?**
   - What we know: Fly's monorepo guide supports both. CONTEXT.md says "planner picks."
   - What's unclear: project preference for diff readability vs DRY.
   - Recommendation: **Two files.** ADR-on-first-lock pattern (Phase-2 D-12, Phase-3 D-22) suggests committing to per-env config files. Easier auditing on a privacy-sensitive deploy. lint-deploy-stack.mjs checks both.

4. **Image-SHA reuse cross-app: dual-tag at build time vs cross-app pull?**
   - What we know: `flyctl deploy -a rebno-prod --image registry.fly.io/rebno-staging:<sha>` works if Fly registry ACL permits cross-app pulls (default: yes for same-org).
   - What's unclear: whether to push twice (cleaner audit trail per app) or once + cross-pull (one less push).
   - Recommendation: **Dual-tag at build time.** Cleaner; Fly registry storage is per-app metered but free at this scale.

5. **Soak test failure → block subsequent staging deploys?**
   - What we know: D-11 says "Failure = staging deploy marked red but does NOT auto-rollback (operator decides)."
   - What's unclear: should a red soak open a GH issue + Slack ping, or just exit non-zero in CI?
   - Recommendation: **GH Issue + label `soak-fail`**; planner may extend with notification later.

## Environment Availability

| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Docker / docker buildx | Image build (CI + local debug) | ✓ on `ubuntu-latest` runners | 24+ default | — |
| flyctl | CI deploy steps + operator local | Installed in CI via `superfly/flyctl-actions/setup-flyctl@<sha>` | latest stable | — |
| pnpm 10 | install + verify gates | ✓ via `pnpm/action-setup@v4` | 10 | — |
| Node 22 | runtime + CI typecheck/test | ✓ via `actions/setup-node@v4` | 22 LTS | — |
| Fly.io account + API token | Deploy target | Operator action: `fly auth login` once + `FLY_API_TOKEN` repo secret set in GH | — | — |
| `legacy/servers/enlyzeam-current/localList.txt` | D-17 one-shot import | ✓ in repo (gitignored under `legacy/`) | — | — |
| Tigris buckets (per-app) | DEP-03 + OpenObserve storage | Created via `fly storage create` AFTER Fly app exists | — | — |
| OpenObserve container image | rebno-obs deploy | Pulled at deploy time from `public.ecr.aws/zinclabs/openobserve` | v0.14.x | Use `latest` if pin unavailable |
| Linux host for Phase-4 carry-forward Test 1 (kill -9) | First staging deploy | ✓ Fly machine IS the Linux host; closes the deferral cleanly per CONTEXT.md | — | — |

**Missing dependencies with no fallback:**
- Operator-side: `fly auth login` must be run once before `fly storage create` and any `fly ssh sftp` commands. Document in RESTORE.md prerequisites.

**Missing dependencies with fallback:**
- OpenObserve specific version may not be available — planner falls back to `latest` tag and ADR 0006 documents the version drift acceptance.

## Validation Architecture

### Test Framework
| Property | Value |
|----------|-------|
| Framework | Vitest 4.1.5 (workspace-pinned) + Node 22 + custom `.mjs` lint scripts (mirrors Phase-2/3/4) |
| Config file | `apps/server/vitest.config.ts` (existing); new tests live under `apps/server/test/*.test.ts` and `apps/server/test/*.integ.test.ts` |
| Quick run command | `pnpm --filter @rebno/server test` (unit only — excludes `*.integ.test.ts`) |
| Full suite command | `pnpm verify:phase-5` (composite gate; mirrors `pnpm verify:phase-4`) |

### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| REQ-DEP-01 | Dockerfile builds Alpine/musl-compatible image (pivot: bookworm-slim) | int | `docker buildx build --file apps/server/Dockerfile .` (run in CI; assert exit 0 + image size sane) | ❌ Wave 0 — needs `apps/server/test/dockerfile.smoke.mjs` |
| REQ-DEP-01 | Drift-guard: Dockerfile pins expected base + Litestream version | unit | `node tools/scripts/lint-deploy-stack.mjs` | ❌ Wave 0 — `tools/scripts/lint-deploy-stack.mjs` |
| REQ-DEP-02 | fly.toml shape (per-env) — required keys present | unit | `node tools/scripts/lint-deploy-stack.mjs` (covers fly.toml + Dockerfile + litestream.yml) | ❌ Wave 0 |
| REQ-DEP-03 | litestream.yml schema valid | unit | `litestream version` + parse via `js-yaml` in lint-deploy-stack.mjs | ❌ Wave 0 |
| REQ-DEP-03 | RPO < 1s observable | int (manual on staging) | RESTORE.md procedure: write 100 rows on staging, kill machine, restore on different machine, count rows; difference ≤ rate*1s | ❌ Wave 0 — `scripts/measure-rpo.mjs` |
| REQ-DEP-04 | trace:check is required PR check | unit | `pnpm trace:check` (CI runs it; Phase-4 already wired) | ✅ Phase-4 |
| REQ-DEP-04 | deploy-staging.yml triggers on push to main + verify gates pass | int | manual via test PR + `gh run list -w deploy-staging.yml` | ❌ Wave 0 — `scripts/test-staging-deploy.mjs` (dry-run validator) |
| REQ-DEP-05 | /health returns 200 OK on healthy server | unit | Phase-4 has health unit test in `apps/server/test/`; extend with draining-state test | ✅ Phase-4 (extend) |
| REQ-DEP-05 | Fly health check probe success path | int | `curl https://rebno-staging.fly.dev/health -i` from soak test; assert 200 | ❌ Wave 0 — embedded in soak-staging.mjs |
| REQ-DEP-06 | Pino JSON logs reach stdout in container | unit | Run `apps/server` in container locally + read `docker logs`; assert valid JSON | ❌ Wave 0 — `scripts/test-pino-stdout.mjs` |
| REQ-DEP-06 | OTLP exporter sends to OpenObserve (success path) | int | spin up `apps/obs` container locally + run `apps/server` against it + assert ingestion via REST | ❌ Wave 0 — `apps/server/test/otel.integ.test.ts` |
| REQ-DEP-06 | OTel init failure does NOT crash server (D-12) | unit | Set `OTEL_EXPORTER_OTLP_ENDPOINT=http://invalid` + boot server + assert it stays up | ❌ Wave 0 |
| REQ-DEP-07 | RESTORE.md procedure validated end-to-end on staging in <5 min | int (manual on staging, recorded) | RESTORE.md execution log committed to `.planning/phases/05-deploy/05-HUMAN-UAT.md` | ❌ Wave 0 |
| REQ-DEP-07 | kill -9 mid-tick recoverability (Phase-4 carry-forward Test 1) | int (manual on staging) | Verbatim 6-step procedure from `04-09-SUMMARY.md` §"Manual Verification" | ❌ closes Phase-4 deferral |
| REQ-DEP-07 | argon2 prod-hardware bench in band [200ms, 500ms] (Phase-4 carry-forward Test 3) | int (manual on staging) | `fly ssh console -a rebno-staging -C 'cd /app && N=10 node scripts/argon2-bench.mjs'` | ❌ closes Phase-4 deferral |
| REQ-DEP-08 | Colyseus pingInterval=3s alive over 30 min — zero spurious WS disconnects | int (CI scripted) | `pnpm soak:staging` 30-min run, asserting zero disconnect events | ❌ Wave 0 — `scripts/soak-staging.mjs` |

### Sampling Rate
- **Per task commit:** `pnpm --filter @rebno/server test` + cheap lints (`node tools/scripts/lint-deploy-stack.mjs`)
- **Per wave merge:** `pnpm verify:phase-5` (composite — runs Phase-4 verify, typecheck, lints, integ tests where reasonable)
- **Phase gate:** Full suite green + manual RESTORE.md execution recorded in `05-HUMAN-UAT.md` + `pnpm trace:check` zero findings on DEP-* reqs before `/gsd-verify-work`

### Wave 0 Gaps (test infra to land before implementation tasks)
- [ ] `tools/scripts/lint-deploy-stack.mjs` — drift guard for fly.toml + Dockerfile + litestream.yml + OTel base URL
- [ ] `scripts/verify-phase-5.mjs` — composite gate (mirrors verify-phase-4.mjs scaffold)
- [ ] `scripts/verify-phase-5.test.mjs` — unit tests for the composite gate
- [ ] `.github/workflows/verify-phase-5.yml` — CI invoker
- [ ] `.github/workflows/trace-check.yml` — required PR check (D-06)
- [ ] `apps/server/test/dockerfile.smoke.mjs` — image build assertion
- [ ] `apps/server/test/otel.integ.test.ts` — OTel SDK init success/failure paths
- [ ] `apps/server/test/staging-invite.test.ts` — middleware unit tests
- [ ] `scripts/soak-staging.mjs` — 30-min 2-client harness (extracts logic from `apps/server/test/authority.integ.test.ts`)
- [ ] `scripts/measure-rpo.mjs` — RESTORE.md companion
- [ ] `apps/server/scripts/run-migrations.ts` — programmatic migrator (replaces drizzle-kit CLI in container)

## Security Domain

### Applicable ASVS Categories

| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V1 Architecture | yes | ADR 0005 (deploy topology) + ADR 0006 (observability) document trust boundaries |
| V2 Authentication | yes | Better-Auth + argon2id (Phase-4 SRV-09); STAGING_INVITE_TOKEN gates staging; BETTER_AUTH_SECRET via `fly secrets`; per-env different secrets (D-18) |
| V3 Session Management | yes | Better-Auth cookie-based sessions (Phase-4); rotation invalidates all sessions per D-21 |
| V4 Access Control | yes | Staging IP allowlist + invite token; Fly proxy ACL on registry per-app; OpenObserve UI behind same allowlist |
| V5 Input Validation | yes | Phase-4 zod-strict schemas at WS handshake (PITFALLS B1); Phase 5 inherits unchanged |
| V6 Cryptography | yes | argon2id (never hand-roll), Ed25519 room signing keypair (Phase-4 D-19), TLS via Fly proxy, Litestream replicas optionally encrypted at rest (Tigris-side default; Litestream 0.3.x has Age option, 0.5.x dropped Age — relevant to v2 upgrade) |
| V7 Errors & Logging | yes | pino redact list (Phase-4 D-23) preserved across OTel transport; Pitfall 10 verifies; trace-IDs propagate per D-13 |
| V8 Data Protection | yes | Plaintext localList.txt enters via TLS-encapsulated `fly ssh sftp`, never via image layers; deleted post-import; legacy_credentials_staging table read-once-then-purge per Phase-4 D-04 |
| V9 Communication | yes | TLS 1.2+ via Fly proxy (mandatory per fly.toml `force_https = true`); WireGuard 6PN for app-to-app (rebno-obs ingestion); OTLP path internal-only via flycast |
| V10 Malicious Code | yes | flyctl-actions SHA-pinned (not `@master`); `actions/checkout@v4` SHA-pinnable at plan-time |
| V12 File Resources | partial | `/data/seed/` is the only writable seed location; `/data/keys/` perms 0600 (Phase-4 D-19); `/data/snapshots/pre-migrate-*.db` rotates on each deploy (planner: add cleanup task or accept linear growth) |
| V14 Configuration | yes | All secrets via `fly secrets`, never committed; `BETTER_AUTH_SECRET` per-env different; `ALLOWED_ORIGINS` set to staging-only domains on staging machine |

### Known Threat Patterns for Phase-5 stack

| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| Supply-chain attack via floating GH Action ref | T (Tampering) | SHA-pin `superfly/flyctl-actions/setup-flyctl@fc53c09` and other actions |
| Plaintext credentials in image layers (pre-D-17 mistake) | I (Information disclosure) | D-17 ssh-sftp ritual; lint check that `legacy/` paths are not COPY'd in Dockerfile |
| BETTER_AUTH_SECRET exposed in CI logs | I | Mark as GH secret; never `echo $BETTER_AUTH_SECRET`; pino redact applies |
| OTLP traffic eavesdropping | I | `rebno-obs.flycast` is private 6PN — never public OTLP endpoint |
| OpenObserve admin UI public | E (Elevation) | IP allowlist per D-15 + `ZO_ROOT_USER_PASSWORD` strong (`openssl rand -base64 24`) |
| Stolen FLY_API_TOKEN deploys arbitrary image | E | GH org-secret scope; rotate on team change; no token in PRs from forks |
| Litestream replica encryption — Age dropped in 0.5 | I | If staying 0.3.x, optionally enable Age; if moving to 0.5.x, rely on Tigris server-side encryption + bucket ACL |
| Container privilege escalation via `dumb-init` misuse | E | Run as non-root user (Dockerfile: `USER node`); `dumb-init --` doesn't elevate |
| WS upgrade DoS — abuse free joinOrCreate | DoS | Phase-4 SRV-07 token-bucket rate-limit on `auth` budget rate=0.1/burst=3; staging invite gate adds another layer |
| Tigris key compromise leaks WAL history | I | Tigris bucket per-app; rotate via `fly storage destroy && fly storage create`; documented in RESTORE.md secret rotation table |

## Sources

### Primary (HIGH confidence)
- `apps/server/package.json` (Phase-4 lockfile pins) [VERIFIED: codebase 2026-05-08]
- `apps/server/src/index.ts`, `apps/server/src/sigterm.ts`, `apps/server/src/health.ts` [VERIFIED: codebase 2026-05-08]
- `.github/workflows/verify-phase-4.yml` (CI shape mirror) [VERIFIED: codebase 2026-05-08]
- `tools/scripts/lint-rate-limit-budgets.mjs` (lint pattern mirror) [VERIFIED: codebase 2026-05-08]
- `.planning/phases/04-server-rebuild-mvp/04-09-SUMMARY.md` §"Manual Verification (Phase 5 Debt)" — verbatim kill -9 procedure [VERIFIED: codebase]
- `.planning/phases/04-server-rebuild-mvp/04-HUMAN-UAT.md` — Tests 1+2+3 carry-forward [VERIFIED: codebase]
- `.planning/research/STACK.md` §"Fly.io Specifics" + Tigris notes [VERIFIED: codebase]
- `.planning/research/PITFALLS.md` §B1 server-authoritative discipline [VERIFIED: codebase]
- `docs/adr/0002-persistence-layer.md` — SQLite + Litestream + Drizzle locked [VERIFIED: codebase]
- npm registry `npm view <pkg> version` results for argon2 0.44.0, better-sqlite3 12.9.0, drizzle-kit 0.31.10, @opentelemetry/sdk-node 0.217.0, pino-opentelemetry-transport 3.0.0 [VERIFIED: this session]
- [Litestream Configuration File reference](https://litestream.io/reference/config/) [CITED]
- [Litestream Docker guide](https://litestream.io/guides/docker/) [CITED]
- [Litestream releases — current stable 0.5.11 / conservative 0.3.13](https://github.com/benbjohnson/litestream/releases) [VERIFIED via WebFetch]
- [Fly.io blog: All In on SQLite + Litestream](https://fly.io/blog/all-in-on-sqlite-litestream/) [CITED via STACK.md]
- [Fly.io fly.toml reference](https://fly.io/docs/reference/configuration/) [CITED]
- [Fly.io Tigris docs](https://fly.io/docs/tigris/) [CITED]
- [Fly.io Flycast — private services](https://fly.io/docs/networking/flycast/) [CITED]
- [Fly.io continuous deployment with GH Actions](https://fly.io/docs/launch/continuous-deployment-with-github-actions/) [CITED]
- [OpenObserve docs — environment variables](https://openobserve.ai/docs/environment-variables/) [VERIFIED via WebFetch]
- [OpenObserve OTLP HTTP ingestion](https://openobserve.ai/docs/ingestion/logs/otlp/) [CITED]
- [OpenTelemetry JS exporters](https://opentelemetry.io/docs/languages/js/exporters/) [CITED]
- [OpenTelemetry JS instrumentation](https://opentelemetry.io/docs/languages/js/instrumentation/) [CITED]
- [Drizzle ORM migrations](https://orm.drizzle.team/docs/migrations) + [drizzle-kit migrate command](https://orm.drizzle.team/docs/drizzle-kit-migrate) [CITED]
- [Colyseus WebSocket transport docs](https://docs.colyseus.io/server/transport/ws) [CITED]
- [superfly/flyctl-actions releases v1.5](https://github.com/superfly/flyctl-actions/releases) [CITED via WebSearch]

### Secondary (MEDIUM confidence)
- [Simon Willison on Litestream v0.5.0 release](https://simonwillison.net/2025/Oct/3/litestream/) [CITED]
- [mtlynch.io: Hold off on Litestream 0.5.0](https://mtlynch.io/notes/hold-off-on-litestream-0.5.0/) [CITED]
- [The Fly Blog: Litestream v0.5.0 is here](https://fly.io/blog/litestream-v050-is-here/) [CITED]
- [Snyk: choosing the best Node.js Docker image](https://snyk.io/blog/choosing-the-best-node-js-docker-image/) — alpine vs slim production guidance [CITED]
- [github.com/WiseLibs/better-sqlite3/issues/1397](https://github.com/WiseLibs/better-sqlite3/issues/1397) — Alpine + Node 22 install fragility [CITED]
- [github.com/openobserve/openobserve/discussions/2711](https://github.com/openobserve/openobserve/discussions/2711) — OpenObserve memory tuning [CITED]
- [pino-opentelemetry-transport README](https://github.com/pinojs/pino-opentelemetry-transport) [CITED]
- [DZone: Deep Observability in Node.js with OpenTelemetry and Pino](https://dzone.com/articles/observability-nodejs-opentelemetry-pino) — production patterns [CITED]
- [GitHub Actions concurrency control docs](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency) [CITED]
- [community.fly.io idle timeout = 60s confirmation](https://community.fly.io/t/does-the-tls-handler-close-idle-tcp-connections-after-60s/2373) [CITED]
- [OWASP ASVS](https://owasp.org/www-project-application-security-verification-standard/) — V1..V14 categories [ASSUMED — standard reference]

### Tertiary (LOW confidence — flag for validation at plan-time)
- OpenObserve `v0.14.x` exact patch: planner re-checks via `gh api repos/openobserve/openobserve/releases/latest` at plan-day
- flyctl-actions `fc53c09` SHA: planner re-confirms current via `gh api repos/superfly/flyctl-actions/releases/latest`
- Tigris-specific Litestream restore performance vs AWS S3: no first-party benchmarks; rely on staging soak measurement

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH — every package version verified against npm registry this session; Phase-4 deps inherited unchanged
- Architecture (Dockerfile + entrypoint + fly.toml + litestream.yml + OTel): HIGH — patterns are off-the-shelf with single-source verification
- CI/CD shape: HIGH — superfly/flyctl-actions is documented; image-SHA promotion is canonical pattern
- Pitfalls: HIGH on the codebase-grounded ones (Pitfall 9 staging-invite ordering, Pitfall 4 drizzle-kit, Pitfall 8 SQLite WAL); MEDIUM on the OpenObserve specifics (Pitfall 5, 7) — single-vendor docs
- Phase-4 carry-forward integration: HIGH — `apps/server/src/{health,sigterm,index,db}.ts` and `04-09-SUMMARY.md` all read in full
- Litestream version pin: MEDIUM — community signals split on 0.3 vs 0.5; ADR-level decision needed
- OpenObserve sizing on Fly: MEDIUM — only one community discussion thread, no first-party Fly-specific guidance

**Research date:** 2026-05-08
**Valid until:** 2026-06-07 (30 days for stable infra; sooner if Litestream cuts a major or OTel ships a breaking SDK minor — re-check at plan-day)

## RESEARCH COMPLETE