# Bigscreen Server Fleet Launch Checklist **Scope:** A human-followable checklist for shipping a complete server fleet update to production. Assumes the new fleet has already been built by Jenkins pipelines, client tokens are updated, and client builds are ready. **Before you start, fill these in:** | Variable | Value | |--------------------|--------------------------------------------| | NEW fleet | `main-_______` (e.g., `main-ocean`) | | OLD fleet | `main-_______` (currently live, keep warm) | | Launch window | _______ UTC | | Launch lead | _______ | | Rollback on-call | _______ | > URL substitution used throughout: `$NETWORK = main`, `$FLEET = the new fleet name`. > Wherever you see `$NETWORK-$FLEET-...`, expand it for your fleet. --- ## Phase 0 — Prerequisites Confirm the following is true before starting anything else. If any item fails, **stop** and resolve it. - [ ] Jenkins is reachable at and you can log in. - [ ] You have access to the `Server_Keys` folder in Dropbox and `$BIGSCREEN_DROPBOX_PATH` is set locally (see `SERVERS.md`). - [ ] You know the NEW fleet name and it exists on Jenkins. - [ ] You know the OLD (currently live) fleet name and it is still running — **do not decommission it until post-launch verification is complete**. - [ ] Jenkins build history shows a green run for each of these pipelines against the new fleet: - [ ] `001.init` (fleet infra + DNS records) - [ ] `002.init_redis_cluster` - [ ] `010.setup_cloud_database` - [ ] `003.build_all` (or `003.apis` + `003.admin_apis`) - [ ] `007.webapps` (and/or `009.arda_admin_api` for `main-kiwi`) - [ ] `004.deploy_media_servers_digitalocean` - [ ] `005.build_media_servers_aws` - [ ] Cloudflare dashboard is reachable and you have permission to edit the `bigscreencloud.com` zone. - [ ] You have access to the AWS console (`us-west-2`) and the S3 buckets: `main-melon-bigscreen-website`, `main-bigscreen-website-preview`, `main-bigscreen-beyond-scans`. - [ ] You have credentials for Steamworks and the Meta Quest developer dashboard. - [ ] The `#releases` (or equivalent) Discord channel is monitored. --- ## Phase 1 — Server APIs are accessible There are no dedicated `/health` or `/ping` endpoints — verify by hitting the actual service URLs and confirming a 200/expected payload. Pull the live DNS for each instance first by running the info pipeline: - [ ] Trigger Jenkins `099.info` for the new fleet (e.g., `aws_network_info main_` under the **Info** tab). This confirms the instances exist and prints their public DNS, plus the fleet's config files. Then hit each public endpoint. Substitute `$NETWORK-$FLEET` (e.g., `main-ocean`): - [ ] `curl -I https://$NETWORK-$FLEET-api.bigscreencloud.com/` → returns a TLS-valid response (not `SSL_ERROR`, not `NXDOMAIN`). - [ ] `curl -I https://$NETWORK-$FLEET-cloud-api.bigscreencloud.com/` → same. - [ ] `curl -I https://$NETWORK-$FLEET-admin-api.bigscreencloud.com/` → expect the ALB to reject from public internet (by design — Admin API is not public-facing). Confirm via a browser on the team VPN / team-access IP instead. - [ ] WebSocket probe: `wscat -c wss://$NETWORK-$FLEET-cloud.bigscreencloud.com` (or equivalent) → handshake succeeds. - [ ] For each service, SSH via the Jenkins build server (see `SERVERS.md` §1–9) and confirm: - [ ] `pm2 list` shows all expected processes `online` (no `errored`, no repeated restarts). Admin API on `main` should show 15 PM2 instances on ports 3001–3015. - [ ] `sudo systemctl status nginx` is `active (running)`. - [ ] Latest log has no stacktraces: `pm2 logs --lines 200 --nostream` or tail the relevant `out.log` / `error.log` under `/home/ubuntu/networks/$NETWORK/$FLEET/`. - [ ] ALB target groups in AWS console show all instances `healthy` (EC2 → Target Groups → filter by fleet name). - [ ] Redis reachability from an API instance: `redis-cli -h -p 6379 ping` → `PONG`. - [ ] Postgres reachability: - [ ] API instance: `psql` connection to the primary DB succeeds (use creds from the fleet `.env`). - [ ] Cloud API instance: `psql` connection to the cloud DB succeeds. --- ## Phase 2 — Arda is live and accessible Arda is team-only; access is restricted via the "General Team Access" security group. - [ ] Fleet-specific Arda loads and you can log in: - [ ] Canonical Arda (`WEBAPP_ARDA_HOST` in the fleet config) loads: — note this currently points at whichever fleet owns the canonical CNAME; it will become the new fleet once Phase 5 is complete. - [ ] On the webapps server (SSH via Jenkins), `pm2 list` shows the `arda` processes `online`. - [ ] Test a sample admin action in the new fleet's Arda (e.g., load a user, load a content listing) — confirm the page renders without errors. - [ ] Browser devtools network tab shows Arda's API calls going to the NEW fleet's admin-api URL, not the old fleet. --- ## Phase 3 — New RDC build is visible in Arda but not activated The RDC ("Release Candidate") config lives at `/home/ubuntu/networks/$NETWORK/$FLEET/config/${NETWORK}_${FLEET}_rdc.config.ts` on the Jenkins server. `099.info` prints it in the console output. - [ ] Rerun `099.info` and sanity-check the RDC config section — it should list the intended RDC version, not a stale one. - [ ] In Arda, navigate to the builds / release candidates section (or wherever the new build should appear). - [ ] New RDC build is **listed** and **visible** to admins. - [ ] New RDC build is **NOT yet marked active / live** — end users should not receive it until the switchover (Phase 9). - [ ] The "current live" RDC visible to users still points at the OLD fleet's active build. - [ ] If the RDC lists a Unity client version / compatibility window, the versions listed match what will be shipped in Phase 4. --- ## Phase 4 — Unity builds visible on stores **Note:** Client build publishing is handled outside this repo. Confirm with whoever owns the Unity client pipeline that these steps are complete. - [ ] Steam (Windows/Oculus PC): - [ ] New build uploaded to Steamworks (via `steamcmd` / Steamworks partner tools, run by the client team). - [ ] Build is set on the intended branch (e.g., `default`, `beta`). - [ ] Client team has confirmed the build launches cleanly via the Steam beta branch. - [ ] Meta Quest store: - [ ] New `.apk` uploaded via `ovr-platform-util` / the Meta developer dashboard. - [ ] Build is visible on the correct release channel (RC / LIVE / Alpha / Beta) but not yet promoted to LIVE. - [ ] App ID / entitlement matches the new server's `allowed_apps` config. - [ ] (If applicable) Meta Quest app lab / PCVR / any other distribution target: new build uploaded and visible. - [ ] Record the build numbers / changeset IDs in the launch notes so rollback can target the previous build. - [ ] Spot-check: a test device / account can download the new build from the pre-release channel and, pointed at the NEW fleet, plays through a basic session (launch → sign in → join a room). --- ## Phase 5 — Server DNS points at the new fleet DNS is Cloudflare (`bigscreencloud.com` zone). Records are CNAMEs created by `assign_dns()` in `Jenkins/server_builder_v8.sh` with **TTL 120s** (fast failover). The switchover is the actual "go live" moment. **Before you flip:** - [ ] Record the OLD fleet's current ALB DNS names for each service (copy from AWS console or from a prior `099.info` run). You need these for rollback. | Service | OLD ALB DNS | |------------------|-----------------------------------| | api | _______ | | admin-api | _______ | | cloud-api | _______ | | cloud (ws) | _______ | | arda (canonical) | _______ | - [ ] Record the NEW fleet's ALB DNS names (from `terraform output` or AWS console): | Service | NEW ALB DNS | |------------------|-----------------------------------| | api | _______ | | admin-api | _______ | | cloud-api | _______ | | cloud (ws) | _______ | | arda (canonical) | _______ | **Flip the records (Cloudflare):** the canonical client-facing hostnames (whatever the shipped Unity clients resolve — for example `api.bigscreencloud.com`, `cloud-api.bigscreencloud.com`, `cloud.bigscreencloud.com`, `admin-api.bigscreencloud.com`, `arda.bigscreencloud.com`) need to be updated to point at the NEW fleet's ALB DNS. The fleet-specific `$NETWORK-$FLEET-*` records are already created by `001.init`; the switchover is the canonical CNAMEs. - [ ] Update each canonical CNAME to point at the corresponding NEW ALB DNS. - [ ] Confirm TTL stays at 120s so a rollback propagates in ~2 minutes. - [ ] Wait for propagation and verify from at least two networks: - [ ] `dig api.bigscreencloud.com` resolves to the NEW ALB DNS. - [ ] `dig cloud-api.bigscreencloud.com` resolves to the NEW ALB DNS. - [ ] `dig cloud.bigscreencloud.com` resolves to the NEW ALB DNS. - [ ] `dig arda.bigscreencloud.com` resolves to the NEW ALB DNS. - [ ] In a private browser window, load `https://arda.bigscreencloud.com` and confirm devtools show requests going to the NEW admin-api. --- ## Phase 6 — Website points at the new fleet The website is a React static site built from the `website` repo, deployed to S3 via `Jenkins/pipelines/006.build_website.jenkinsfile`. The target bucket depends on the `TARGET_BUCKET` argument: - `TARGET_BUCKET=live` → `s3://main-melon-bigscreen-website` (live, 60s cache) - Anything else → `s3://main-bigscreen-website-preview` (no-cache) The API URL the website talks to is baked in **at build time** from the fleet's `main_${FLEET}_website.env` file, so the website must be **re-built against the new fleet** and then promoted to the live bucket. > Heads-up: `006.build_website.jenkinsfile` currently hardcodes `FLEET="ocean"` > and `TARGET_BUCKET="preview"`. If you're launching a fleet other than > `ocean`, update the pipeline (or use a parameterized copy) before running. - [ ] Trigger `006.build_website` against the NEW fleet, `TARGET_BUCKET=preview` first. - [ ] Verify the preview URL loads and devtools show the website calling the NEW fleet's API URL: (or the Cloudflare-fronted preview URL). - [ ] Trigger `006.build_website` again with `TARGET_BUCKET=live` to push to `s3://main-melon-bigscreen-website`. - [ ] `aws s3 ls s3://main-melon-bigscreen-website/` shows fresh timestamps on `index.html` and static assets. - [ ] Load (or whichever public host fronts the live bucket), hard-refresh (Ctrl+Shift+R), and confirm devtools network tab shows requests hitting the NEW API URL. - [ ] 60s cache headers visible on `index.html`; hashed assets are `max-age=60`. --- ## Phase 7 — Face scanning endpoints work Face scanning writes to `s3://main-bigscreen-beyond-scans` (bucket policy at `AWS/S3/beyond-scans.json`). The Lambda(s) that read/write this bucket are **not defined in this repo** — verify via AWS console and an end-to-end test. - [ ] In AWS console, the `main-bigscreen-beyond-scans` bucket exists and its policy (should match `AWS/S3/beyond-scans.json`) is intact. - [ ] Lambda(s) handling face scans are deployed, enabled, and their execution role still has the IAM grants required by the bucket policy. - [ ] If face scanning endpoints are fronted via API Gateway or a direct URL referenced by the Unity client, do a test scan from a dev device pointed at the NEW fleet and confirm: - [ ] Upload succeeds (no 403 / 5xx). - [ ] Scan result is returned to the client. - [ ] Corresponding object appears in `s3://main-bigscreen-beyond-scans`. - [ ] CloudWatch logs for the scanning Lambda(s) show no new errors during the test. --- ## Phase 8 — Anything else Things easy to miss. - [ ] **Redis clusters (v1 and v2) healthy**: AWS ElastiCache shows both `$NETWORK-$FLEET-redis` and `$NETWORK-$FLEET-redis-v2` as `available`. (v2 is legacy post–Cloud Worker removal, but still in place — see `SERVER_TOPOLOGY_SPEC.md` §5.2.) - [ ] **Cloud PostgreSQL is encrypted at rest** (differs from primary — see §4.2). Confirm in RDS console for the new fleet's DB. - [ ] **TLS certificates**: ACM certificate used by ALBs is valid and not expiring soon (`openssl s_client -connect host:443 2>/dev/null | openssl x509 -noout -dates`). - [ ] **Firebase service account keys** are present on the Jenkins build server at the expected fleet path and are not expired. - [ ] **JWT keypair** for the new fleet is distinct from the old fleet's and matches what was baked into the client builds (Phase 4). - [ ] **Brightcove token, allowed-apps config**: both show the new fleet's values in `099.info` output. - [ ] **Media servers (AWS + DigitalOcean)**: at least one of each is reachable on UDP 10000–65535 from a test client (or run a short session end-to-end). - [ ] **Admin API egress hardening**: confirm the Admin API security group still has egress restricted to `172.31.0.0/16` (`SERVER_TOPOLOGY_SPEC.md` §7.3). A misconfiguration here is easy to miss and security-relevant. - [ ] **Load test passed** on the new fleet before launch: `098.begin_load_test` (default 15 instances × ~75 users = ~1,125 simulated users). Run `098.end_load_test` to tear it down. - [ ] **Discord launch notifications** (`Jenkins/deps/discord.sh`) fired to the release channel for each pipeline — scan the channel to confirm no `BUILD FAILED` messages were missed. - [ ] **Metabase, Jenkins, TURN server** are unaffected by the switch — those hosts are not part of a fleet and should be unchanged (see `SERVERS.md`). - [ ] **Monitoring**: `AWS_TERRAFORM_AUDIT.md` flags the lack of CloudWatch alarms / health-check monitoring as a known gap. Plan to manually tail logs for 30–60 min post-cutover. --- ## Phase 9 — Server transition (the actual cutover) This is the sequence of go-live steps. Each checkpoint must pass before moving to the next. 1. **T–30 min — final sanity check** - [ ] Phases 1–4, 7, 8 all green. - [ ] Announce in `#releases` that cutover is starting in 30 min. - [ ] Old fleet's `pm2 logs` window open in one terminal; new fleet's in another. 2. **T–0 — flip DNS (Phase 5 action)** - [ ] Update Cloudflare canonical CNAMEs to NEW ALB DNS. - [ ] Watch `dig` from two networks until resolution flips everywhere. 3. **T+2 min — verify traffic on new fleet** - [ ] NEW fleet `pm2 logs` shows real client traffic ramping up (not just health pings). - [ ] NEW API: request rate climbing in AWS ALB metrics. - [ ] OLD API: request rate decaying. - [ ] No spike in 5xx on the NEW ALB target groups. 4. **T+5 min — promote website and activate RDC** - [ ] Phase 6 complete: website live bucket points at new fleet. - [ ] In Arda, **activate the new RDC build** (the one staged in Phase 3). - [ ] Confirm the "current live" RDC is now the new build. 5. **T+10 min — promote Unity builds to LIVE channels** - [ ] Steam: promote from beta branch to `default` (client team runs this). - [ ] Meta Quest: promote from RC to LIVE. - [ ] Spot-check: a real device without a dev token can launch into the new build and complete a session. 6. **T+15 min — soak period** - [ ] Tail NEW fleet `pm2` logs for 15 min — no spikes in errors. - [ ] Quick face-scan E2E from a device (Phase 7). - [ ] DAU / concurrent users metric (Metabase) not cratering. - [ ] Support channel / Discord not seeing user reports. 7. **T+60 min — keep old fleet warm** - [ ] Do **not** decommission the old fleet. Leave PM2 running and instances up for at least 24 hours in case of delayed rollback. - [ ] Record the time the new fleet went live; the earliest acceptable decommission time for the old fleet is T+24h (longer if any unresolved issue). 8. **Close out** - [ ] Post launch summary in `#releases` with: - NEW fleet name - RDC build number - Steam / Meta build numbers - Start / finish timestamps - Any incidents during the window - [ ] File a retro note for anything that went wrong so next launch is smoother. --- ## Rollback If something is broken and you can't fix forward within the soak window, roll back. Rollback is fast because TTL is 120s. 1. **DNS rollback (fastest, ~2 min):** - [ ] In Cloudflare, change canonical CNAMEs back to the OLD ALB DNS values recorded in Phase 5. - [ ] Verify with `dig` from two networks. - [ ] Confirm `pm2 logs` on the OLD fleet shows traffic returning. 2. **Website rollback:** - [ ] Re-run `006.build_website` with the OLD fleet name and `TARGET_BUCKET=live` to rebuild the live bucket pointing at the old fleet's API URLs. 3. **Arda / RDC rollback:** - [ ] In Arda, re-activate the previous RDC build. The new build can stay listed but should not be the active one. 4. **Client build rollback:** - [ ] Steam: revert the `default` branch to the previous build (Steamworks build history → set live). - [ ] Meta Quest: revert LIVE channel to the previous build. 5. **Single-service rollback (if you only need to revert one API):** See `README.md` "Handling Urgent Rollbacks". Each build lives in a timestamped folder `${NETWORK}_${FLEET}_${SERVICE}_${BUILD_NUMBER}` on the instance — SSH in, `cd` to the previous folder, `pm2 start .js`. 6. **After rollback:** - [ ] Post in `#releases` that rollback is complete. - [ ] Leave the NEW fleet running for post-mortem inspection. Do not destroy it until root cause is understood. --- ## Appendix — Quick reference **Canonical fleet URLs (substitute `$NETWORK-$FLEET`):** | Service | URL | |------------------|--------------------------------------------------------------| | API | `https://$NETWORK-$FLEET-api.bigscreencloud.com` | | Admin API | `https://$NETWORK-$FLEET-admin-api.bigscreencloud.com` | | Cloud API | `https://$NETWORK-$FLEET-cloud-api.bigscreencloud.com` | | Cloud WebSocket | `wss://$NETWORK-$FLEET-cloud.bigscreencloud.com` | | Arda (fleet) | `https://$NETWORK-$FLEET-arda.bigscreencloud.com` | | Arda (canonical) | `https://arda.bigscreencloud.com` | | Website (per-net)| `https://$NETWORK-website.bigscreencloud.com` | **Jenkins pipelines touched during a launch:** | Pipeline | When | |---------------------------------|---------------------------------------------| | `099.info` | Phase 0, 1, 3, 8 (check live state) | | `006.build_website` | Phase 6 (preview + live) | | `098.begin_load_test` / `end` | Phase 0 (pre-launch validation) | | `097.restart_media_servers` | Only if media issues (rolling restart) | **Files worth re-reading before launch:** - `SERVER_TOPOLOGY_SPEC.md` — what the fleet looks like. - `SERVERS.md` — how to SSH into anything. - `README.md` (rollback section) — single-service rollback recipe. - `Jenkins/server_builder_v8.sh` — the actual `assign_dns` and `build_website_s3` implementations. **Known gaps / watch-outs** (things this repo doesn't give you): - No `/health` endpoints — rely on real traffic + ALB target-group health. - No CloudWatch alarms (flagged in `terraform/v6/AWS_TERRAFORM_AUDIT.md`). - Unity client publish pipeline is not in this repo — coordinate with the client team. - Face-scan Lambdas are not defined in this repo — verify via AWS console. - `006.build_website.jenkinsfile` has a hardcoded `FLEET="ocean"`; edit before running for any other fleet.