# Bigscreen Devops

Infrastructure-as-code, CI/CD scripts, and operational tooling for the Bigscreen
server platform. This repo is what Jenkins uses to provision, build, and deploy
every fleet in every network (`main`, `dev`, `test`).

## What's in here

| Path                          | Purpose                                                                                   |
|-------------------------------|-------------------------------------------------------------------------------------------|
| `Jenkins/`                    | Jenkins pipeline definitions and the shell library (`server_builder_v8.sh`) they call.   |
| `Jenkins/pipelines/`          | Numbered `.jenkinsfile`s, one per pipeline (fleet init, API builds, media deploys, etc.).|
| `Jenkins/server_builder_v8.sh`| Shared bash library. Every pipeline sources it and invokes one or more of its ~35 functions (fleet provisioning, app builds, DNS, Discord notifications, load tests). |
| `terraform/v6/`               | Current Terraform modules: `fleet/`, `databases/`, `cloud-database/`, `redis-cluster-v2/`, `media-server-aws/`, `media-server-digitalocean/`, `user-sim-load-test/`. |
| `SecretsInstaller/`           | TypeScript CLI that uploads secrets and config to AWS Secrets Manager. Replaces the legacy `ServerEnvironmentBuilder`. |
| `ServerEnvironmentBuilder/`   | Legacy Node.js tool that generates `.env` files on the Jenkins build server. Being phased out in favour of `SecretsInstaller`. |
| `AndroidBigMediaPlayer/`      | Build scripts for the native ExoPlayer/ffmpeg plugin used by the Unity Android client.    |
| `AWS/S3/`                     | AWS S3 helper scripts.                                                                     |
| `DRM/`                        | Brightcove DRM key-provisioning scripts.                                                   |
| `.archive/`                   | Prior-version source retained for historical reference — `server_builder_v2.sh` through `v7`, `terraform/v4` and `v5`, old `UnityBuildServer`, and the previous `README.md`. Not used by any live pipeline. |

## Authoritative references

Before changing anything in this repo, read the doc that applies:

- [`SERVER_TOPOLOGY_SPEC.md`](SERVER_TOPOLOGY_SPEC.md) — the specification of the
  live server topology: networks, fleets, service roles, data tier, security
  model, port reference. The "what" of the infrastructure.
- [`SERVERS.md`](SERVERS.md) — operational guide. How to find a server's current
  DNS (via the Jenkins "Info" tab), how to SSH to it through the Jenkins build
  server, Dropbox key layout, financial-report runbook, Metabase, TURN server.
- [`LAUNCH_CHECKLIST.md`](LAUNCH_CHECKLIST.md) — human-followable runbook for
  shipping a new fleet to production. Lists every pipeline, every Cloudflare
  change, and every verification step in order.
- [`CODE_SIGNING_WORKSHEET.md`](CODE_SIGNING_WORKSHEET.md) — AWS CloudHSM
  code-signing setup notes.
- [`terraform/v6/AWS_TERRAFORM_AUDIT.md`](terraform/v6/AWS_TERRAFORM_AUDIT.md)
  and
  [`terraform/v6/SECRETS_MANAGER_MIGRATION_PLAN.md`](terraform/v6/SECRETS_MANAGER_MIGRATION_PLAN.md)
  — current Terraform audit and the Secrets Manager migration plan.

## How the pieces fit together

A typical fleet deploy (e.g. `main-ocean`) runs, in order:

1. **`001.init`** — `init_server_fleet_config_files` + `terraform_deploy_fleet`.
   Creates the fleet's EC2 instances, ALBs, security groups, IAM users, DNS.
2. **`002.init_redis_cluster`** — `terraform_deploy_redis_cluster`.
3. **`010.setup_cloud_database`** — `terraform_deploy_cloud_databases`.
4. **`003.build_all`** (or the narrower `003.apis` / `003.admin_apis`) —
   `build_api`, `build_admin_api`, `build_cloud_api_next`, `build_cloud_websocket`,
   `build_webapp_arda`, `run_cloud_setup_next`. Each function rsyncs the app
   from the `BigscreenVR/cloud` repo to the fleet, writes config, and starts
   it under PM2.
5. **`007.webapps`** and **`009.arda_admin_api`** — webapp builds.
6. **`004.deploy_media_servers_digitalocean`** and
   **`005.build_media_servers_aws`** — media infrastructure.
7. **`006.build_website`** — marketing site to S3.

Each pipeline `source`s `Jenkins/server_builder_v8.sh` and then calls one or
more of its functions. The library is the single source of truth for build
logic; the pipelines are thin orchestration.

## Security rules

- **Never commit an unencrypted key file, `.env`, or any file containing a
  secret.** The `.gitignore` blocks `*.pem`, `*.pub`, `*.key`, `*.env`, and
  `*.template` — do not bypass it.
- All runtime secrets are either pulled from AWS Secrets Manager (current) or
  generated at build time by Jenkins from Dropbox sources (legacy). This repo
  contains neither.
- Avoid committing public DNS or IP addresses. Use the Jenkins `info` pipelines
  in `Jenkins/pipelines/099.info.jenkinsfile.sh` to look these up at runtime.

## Common one-off commands

### Generate a JWT key pair

```bash
# Private key
ssh-keygen -t rsa -b 2048 -f KEY_NAME.key

# Public key
openssl rsa -in KEY_NAME.key -pubout -outform PEM -out KEY_NAME.key.pub
```

### Generate an API key

```bash
uuidgen | sha256sum
```

### Roll back a build

Each app is deployed to a versioned folder on its host, e.g.
`main_ocean_admin_api_34/`. To roll back, SSH to the host (via the Jenkins
build server — see [`SERVERS.md`](SERVERS.md)), `cd` into the older folder, and
restart under PM2:

```bash
cd ~/main_ocean_admin_api_34
pm2 restart ecosystem.config.js   # or the app's pm2 entry
```

Don't decommission the currently-live fleet until post-launch verification is
green — see the rollback section of [`LAUNCH_CHECKLIST.md`](LAUNCH_CHECKLIST.md).