# Bigscreen Server Topology Specification

**Version:** 1.0
**Last Updated:** 2026-04-10
**Status:** Living Document
**Scope:** Complete server infrastructure topology for the Bigscreen platform

---

> This document describes **what** the Bigscreen server topology is, not how it is
> provisioned. It is tool-agnostic by design, though implementation examples
> reference Terraform where helpful. For provisioning details see the
> `devops/Jenkins/` pipeline definitions and `devops/terraform/v6/` modules.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Network Architecture](#2-network-architecture)
   - 2.1. [Networks and Fleets](#21-networks-and-fleets)
   - 2.2. [VPC and Subnet Layout](#22-vpc-and-subnet-layout)
   - 2.3. [Naming Conventions](#23-naming-conventions)
3. [Service Roles](#3-service-roles)
   - 3.1. [API Server](#31-api-server)
   - 3.2. [Admin API Server](#32-admin-api-server)
   - 3.3. [Cloud API Server](#33-cloud-api-server)
   - 3.4. [Cloud WebSocket Server](#34-cloud-websocket-server)
   - 3.5. [WebApps Server](#35-webapps-server)
   - 3.6. [Website](#36-website)
   - 3.7. [Service Summary Matrix](#37-service-summary-matrix)
4. [Database Tier](#4-database-tier)
   - 4.1. [Primary PostgreSQL](#41-primary-postgresql)
   - 4.2. [Cloud PostgreSQL](#42-cloud-postgresql)
   - 4.3. [Replication Topology](#43-replication-topology)
5. [Caching Tier](#5-caching-tier)
   - 5.1. [Redis v1 Cluster](#51-redis-v1-cluster)
   - 5.2. [Redis v2 Cluster](#52-redis-v2-cluster)
   - 5.3. [Access Control Matrix](#53-access-control-matrix)
6. [Media Infrastructure](#6-media-infrastructure)
   - 6.1. [AWS Media Servers](#61-aws-media-servers)
   - 6.2. [DigitalOcean Media Servers](#62-digitalocean-media-servers)
   - 6.3. [Media Groups](#63-media-groups)
7. [Security Model](#7-security-model)
   - 7.1. [Security Group Architecture](#71-security-group-architecture)
   - 7.2. [TLS Termination](#72-tls-termination)
   - 7.3. [Egress Policies](#73-egress-policies)
   - 7.4. [SSH Access Model](#74-ssh-access-model)
   - 7.5. [Authentication and Secrets](#75-authentication-and-secrets)
8. [Environment Sizing](#8-environment-sizing)
   - 8.1. [Compute Sizing by Environment](#81-compute-sizing-by-environment)
   - 8.2. [Database Sizing by Environment](#82-database-sizing-by-environment)
   - 8.3. [Storage Allocation](#83-storage-allocation)
9. [Deployment Model](#9-deployment-model)
   - 9.1. [Pipeline Architecture](#91-pipeline-architecture)
   - 9.2. [Fleet Lifecycle](#92-fleet-lifecycle)
   - 9.3. [Build and Deploy Flow](#93-build-and-deploy-flow)
   - 9.4. [Load Testing](#94-load-testing)
10. [Appendix](#10-appendix)
    - A. [Port Reference](#a-port-reference)
    - B. [Current Security Group IDs](#b-current-security-group-ids)
    - C. [Current Subnet IDs](#c-current-subnet-ids)
    - D. [Current AMI Reference](#d-current-ami-reference)

---

## 1. Overview

Bigscreen is a virtual reality platform that enables shared experiences across VR headsets, web, and mobile clients. The server infrastructure supports real-time media streaming, REST APIs, WebSocket communication, content management, and internal tooling.

The architecture is a **fleet-based, multi-environment** system deployed primarily on **AWS** (us-west-2) with **DigitalOcean** as a secondary provider for media workloads. Each deployment unit ("fleet") contains a complete, independent set of application services that can be provisioned, scaled, and torn down independently.

```mermaid
graph TB
    subgraph "Internet"
        clients["Clients<br/>(VR Headsets, Web, Mobile)"]
    end

    subgraph "AWS us-west-2"
        subgraph "VPC 172.31.0.0/16"
            subgraph "Fleet (one per fleet name)"
                alb_api["ALB: API"]
                alb_cloud_api["ALB: Cloud API"]
                alb_ws["ALB: Cloud WebSocket"]
                alb_admin["ALB: Admin API"]
                alb_webapps["ALB: WebApps"]

                api["API Server"]
                cloud_api["Cloud API Server"]
                cloud_ws["Cloud WebSocket Server"]
                admin_api["Admin API Server"]
                webapps["WebApps Server"]
            end

            subgraph "Data Tier"
                rds_primary["Primary PostgreSQL"]
                rds_cloud["Cloud PostgreSQL"]
                redis_v1["Redis v1 Cluster"]
            end

            media_aws["AWS Media Servers"]
        end
    end

    subgraph "DigitalOcean sfo2"
        media_do["DO Media Servers"]
    end

    subgraph "External Services"
        firebase["Firebase / Firestore"]
        s3["S3 (Website)"]
    end

    clients -->|"HTTPS 443"| alb_api
    clients -->|"HTTPS 443"| alb_cloud_api
    clients -->|"WSS 443"| alb_ws
    clients -->|"UDP 10000-65535"| media_aws
    clients -->|"UDP 10000-65535"| media_do

    alb_api -->|"HTTP 80"| api
    alb_cloud_api -->|"HTTP 80"| cloud_api
    alb_ws -->|"HTTP 80"| cloud_ws
    alb_admin -->|"HTTP 80"| admin_api
    alb_webapps -->|"HTTP 80"| webapps

    api -->|"5432"| rds_primary
    api -->|"6379"| redis_v1
    cloud_api -->|"5432"| rds_cloud
    cloud_api -->|"6379"| redis_v1
    cloud_ws -->|"6379"| redis_v1
    admin_api -->|"6379"| redis_v1
    admin_api -.->|"HTTPS"| firebase
```

---

## 2. Network Architecture

### 2.1 Networks and Fleets

The infrastructure uses a two-level hierarchy:

- **Network** — the environment type (production, development, testing)
- **Fleet** — a named deployment group within a network, containing a complete set of services

Each fleet is fully independent: it has its own servers, load balancers, and security groups. Multiple fleets can exist within the same network for blue/green deployments, feature branches, or team isolation.

| Network | Purpose     | Fleet Names           | Notes                             |
|---------|-------------|-----------------------|-----------------------------------|
| `main`  | Production  | rice, ocean, kiwi     | Full-scale sizing, public-facing  |
| `dev`   | Development | gem, elf              | Reduced sizing, team access       |
| `test`  | Testing     | salt, dodo            | Minimal sizing, ephemeral         |

Fleet names are short, memorable English words. The combination `{network}-{fleet}` forms the universal identifier used across all resources (e.g., `main-rice`, `dev-gem`).

```mermaid
graph LR
    subgraph "main (Production)"
        rice["rice"]
        ocean["ocean"]
        kiwi["kiwi"]
    end
    subgraph "dev (Development)"
        gem["gem"]
        elf["elf"]
    end
    subgraph "test (Testing)"
        salt["salt"]
        dodo["dodo"]
    end
```

### 2.2 VPC and Subnet Layout

All infrastructure resides in a single VPC in the `us-west-2` AWS region.

| Resource   | Value              | Notes                            |
|------------|--------------------|----------------------------------|
| VPC        | `172.31.0.0/16`    | AWS default VPC                  |
| Subnet A   | us-west-2a         | Used by ALBs and DB subnet group |
| Subnet B   | us-west-2b         | Used by ALBs and DB subnet group |
| Subnet C   | us-west-2c         | Used by ALBs and DB subnet group |
| Subnet D   | us-west-2d         | Used by ALBs and DB subnet group |

All four subnets are used for Application Load Balancer distribution and database subnet groups. EC2 instances receive public DNS names.

### 2.3 Naming Conventions

| Pattern                                      | Example                        | Used For                |
|----------------------------------------------|--------------------------------|-------------------------|
| `{network}-{fleet}-{service}`                | `main-rice-api`                | Server names, ALBs      |
| `{network}_{fleet}`                          | `main_rice`                    | Placement groups, tags  |
| `[{network}_{fleet}] {resource}`             | `[main_rice] main-rice-api`    | Cloud provider Name tags|
| `{network}-{fleet}-{service}_security_group` | `main-rice-api_security_group` | Security groups         |
| `{network}-{fleet}-{service}-elb`            | `main-rice-api-elb`            | Load balancers          |

---

## 3. Service Roles

Each fleet contains the following services. All application servers run **Ubuntu 24** with **Node.js 24**, **PM2** (cluster mode process manager), and **Nginx** (reverse proxy) unless otherwise noted.

### 3.1 API Server

- **Purpose:** Core Bigscreen API. Handles user authentication, content management, and platform operations.
- **Software Stack:** Node.js 24, PM2 (cluster mode, max CPU cores), Nginx, Python 3.10 (fabricator preprocessor).
- **Network Exposure:** Behind Application Load Balancer. **Public-facing** (HTTPS 443 from all sources). ALB forwards to instance on HTTP 80.
- **Data Dependencies:** Primary PostgreSQL (port 5432), Redis v1 (port 6379).
- **Egress:** Unrestricted — requires access to external APIs and services.
- **Storage:** 128 GB SSD root volume.
- **Placement:** Uses fleet cluster placement group for enhanced network performance.

### 3.2 Admin API Server

- **Purpose:** Firestore-based administration server. Content moderation, administrative data management, Discord bot integration. On production (`main` network), runs 15 PM2 instances (ports 3001-3015) with 4 GB max heap each.
- **Software Stack:** Node.js 24, PM2, Nginx, Python 3.10. Includes Discord bot service.
- **Network Exposure:** Behind ALB. **Not public-facing** — ALB ingress has no open CIDR rule. Access restricted by internal security groups.
- **Data Dependencies:** Redis v1 (port 6379). No direct relational database access. Communicates with Firebase/Firestore over HTTPS.
- **Egress:** **VPC-only** (`172.31.0.0/16`). This is a deliberate security hardening — the Admin API does not require internet egress.
- **Storage:** 128 GB SSD root volume.

### 3.3 Cloud API Server

- **Purpose:** Cloud-facing API serving real-time cloud operations. Handles requests from Bigscreen cloud clients.
- **Software Stack:** Node.js 24, PM2 (cluster mode, max CPU cores), Nginx.
- **Network Exposure:** Behind ALB. **Public-facing** (HTTPS 443 from all sources).
- **Data Dependencies:** Cloud PostgreSQL (port 5432), Redis v1 (port 6379).
- **Egress:** Unrestricted.
- **Storage:** 128 GB SSD root volume.

### 3.4 Cloud WebSocket Server

- **Purpose:** Real-time bidirectional communication for Bigscreen cloud sessions. WebSocket protocol upgraded from HTTPS at the load balancer.
- **Software Stack:** Node.js 24, PM2, Nginx.
- **Network Exposure:** Behind ALB. **Public-facing** (WSS 443 from all sources).
- **Data Dependencies:** Redis v1 (port 6379) only. No relational database access.
- **Egress:** Unrestricted.
- **Storage:** 128 GB SSD root volume.

### 3.5 WebApps Server

- **Purpose:** Hosts internal web applications used by the Bigscreen team for content management, asset processing, and administration.
- **Applications:**
  - **Arda** — Admin/management web application
  - **Fabricator** — Asset processing and server-side rendering
  - **Rohan** — Additional internal application
- **Software Stack:** Node.js 24, PM2, Nginx, Yarn (corepack enabled). Uses monorepo build system with Babel transpilation.
- **Network Exposure:** Behind ALB. **Team-only** — access controlled by the "General Team Access" security group. Direct HTTPS access for team members is also available.
- **Data Dependencies:** None. Does not directly access databases or caches.
- **Egress:** Unrestricted.
- **Storage:** 64 GB SSD root volume (smaller than other services — primarily serves static assets).
- **Placement:** Optional cluster placement group (configurable, off by default).

### 3.6 Website

- **Purpose:** Public-facing Bigscreen website. Serves static content (marketing, documentation, downloads).
- **Deployment:** Pre-built static site deployed to an S3 bucket. No dedicated server instance — served via S3 static website hosting.
- **Targets:** `preview` or `production` S3 buckets depending on environment.
- **Note:** This is not a traditional server but is included for completeness as part of the deployment topology.

### 3.7 Service Summary Matrix

| Service          | ALB | Public | RDS Primary | RDS Cloud | Redis v1 | Firebase   | Storage | Python |
|------------------|-----|--------|-------------|-----------|----------|------------|---------|--------|
| API              | Yes | Yes    | Yes         | No        | Yes      | No         | 128 GB  | Yes    |
| Admin API        | Yes | No     | No          | No        | Yes      | Yes        | 128 GB  | Yes    |
| Cloud API        | Yes | Yes    | No          | Yes       | Yes      | No         | 128 GB  | No     |
| Cloud WebSocket  | Yes | Yes    | No          | No        | Yes      | No         | 128 GB  | No     |
| WebApps          | Yes | No     | No          | No        | No       | No         | 64 GB   | No     |
| Website          | No  | Yes    | No          | No        | No       | No         | S3      | No     |

```mermaid
graph LR
    subgraph "Public Internet"
        C["Clients"]
    end

    subgraph "Load Balancers"
        ALB1["ALB: API<br/>HTTPS 443"]
        ALB2["ALB: Cloud API<br/>HTTPS 443"]
        ALB3["ALB: Cloud WS<br/>WSS 443"]
        ALB4["ALB: Admin API<br/>Restricted"]
        ALB5["ALB: WebApps<br/>Team-only"]
    end

    subgraph "Compute"
        API["API Server"]
        CAPI["Cloud API"]
        CWS["Cloud WebSocket"]
        AAPI["Admin API"]
        WA["WebApps"]
    end

    subgraph "Data Stores"
        RDS1[("Primary<br/>PostgreSQL")]
        RDS2[("Cloud<br/>PostgreSQL")]
        R1[("Redis v1")]
        FS[("Firestore")]
    end

    C --> ALB1 & ALB2 & ALB3
    ALB4 -.->|"restricted"| AAPI
    ALB5 -.->|"team-only"| WA
    ALB1 --> API
    ALB2 --> CAPI
    ALB3 --> CWS

    API --> RDS1
    API --> R1
    CAPI --> RDS2
    CAPI --> R1
    CWS --> R1
    AAPI --> R1
    AAPI -.-> FS
```

---

## 4. Database Tier

### 4.1 Primary PostgreSQL

The primary relational database serves the core Bigscreen API.

- **Engine:** PostgreSQL 15.3
- **Topology:** 1 primary writer + 2 read replicas
- **Subnet Group:** Spans all 4 VPC subnets for high availability
- **Backup:** 7-day automated backup retention with final snapshot on deletion
- **Accessed by:** API Server (port 5432 via security group rule)
- **Naming:** `{network}-{version}-db` (e.g., `main-v1-db`, replicas: `main-v1-db-r1`, `main-v1-db-r2`)

### 4.2 Cloud PostgreSQL

A separate database cluster for cloud-specific services, isolated from the primary database.

- **Engine:** PostgreSQL 15.4
- **Topology:** 1 primary writer + 2 read replicas
- **Encryption:** Storage encrypted at rest (unlike Primary — notable difference)
- **Backup:** 7-day automated backup retention
- **Accessed by:** Cloud API Server (port 5432 via dedicated security group)
- **Naming:** Same pattern as primary, under separate infrastructure module

### 4.3 Replication Topology

Both database clusters follow an identical replication pattern: one primary writer handling all write operations, with two read replicas providing read scaling and failover capability.

```mermaid
graph TB
    subgraph "Primary PostgreSQL (v15.3)"
        P1_W["Primary Writer"]
        P1_R1["Read Replica 1"]
        P1_R2["Read Replica 2"]
        P1_W -->|"streaming replication"| P1_R1
        P1_W -->|"streaming replication"| P1_R2
    end

    subgraph "Cloud PostgreSQL (v15.4, encrypted)"
        P2_W["Primary Writer"]
        P2_R1["Read Replica 1"]
        P2_R2["Read Replica 2"]
        P2_W -->|"streaming replication"| P2_R1
        P2_W -->|"streaming replication"| P2_R2
    end

    API["API Server"] -->|"port 5432"| P1_W
    CAPI["Cloud API"] -->|"port 5432"| P2_W
```

---

## 5. Caching Tier

### 5.1 Redis v1 Cluster

The primary caching layer for real-time data and session management.

- **Engine:** Redis 6.x (managed ElastiCache)
- **Mode:** Cluster mode enabled with automatic failover
- **Configuration:** Configurable number of node groups (shards) and replicas per node group
- **Port:** 6379
- **Clients:** API Server, Admin API Server, Cloud API Server, Cloud WebSocket Server, Jenkins (CI/CD)
- **Naming:** `{network}-{fleet}-redis`

### 5.2 Redis v2 Cluster

An identical cluster configuration, originally created with expanded access for the now-removed Cloud Worker service.

- **Engine:** Redis 6.x (managed ElastiCache) — same engine as v1
- **Mode:** Cluster mode enabled with automatic failover
- **Port:** 6379
- **Clients:** Same as v1 (API Server, Admin API Server, Cloud API Server, Cloud WebSocket Server, Jenkins)
- **Naming:** `{network}-{fleet}-redis-v2`
- **Rationale:** v2 was originally created to add Cloud Worker access without modifying v1's security group rules. The Cloud Worker service has since been removed (October 2025), making v2 functionally identical to v1. Both clusters remain deployed.

### 5.3 Access Control Matrix

| Service          | Redis v1 | Redis v2 |
|------------------|----------|----------|
| API Server       | Yes      | Yes      |
| Admin API Server | Yes      | Yes      |
| Cloud API Server | Yes      | Yes      |
| Cloud WebSocket  | Yes      | Yes      |
| Jenkins (CI/CD)  | Yes      | Yes      |

---

## 6. Media Infrastructure

Media servers handle real-time audio/video streaming for Bigscreen sessions. They are deployed across two cloud providers for geographic diversity and scaling flexibility.

### 6.1 AWS Media Servers

- **Purpose:** Real-time media streaming for Bigscreen sessions.
- **Base Image:** Custom pre-built media server AMI (not a general-purpose Ubuntu image).
- **Storage:** 100 GB high-performance SSD (gp3 with provisioned 1000 IOPS and 200 MB/s throughput).
- **Network:** **No HTTP/HTTPS ingress.** Accepts only UDP traffic on ports 10000-65535 from all sources (media streaming) and ICMP for diagnostics.
- **Count:** Configurable via `server_count` variable — allows horizontal scaling per media group.
- **Region:** AWS us-west-2 (same VPC as fleet services).

### 6.2 DigitalOcean Media Servers

- **Purpose:** Same as AWS media servers. Provides geographic diversity and an alternative cloud provider for media workloads.
- **Provider:** DigitalOcean
- **Region:** `sfo2` (San Francisco), configurable per deployment
- **Compute:** Compute-optimized instances with 8 dedicated Intel CPUs — media streaming is CPU-intensive.
- **OS:** Ubuntu 22.04 x64
- **Network:** Same port profile as AWS: UDP 10000-65535 inbound from all, SSH from Jenkins IPs only, TCP+UDP+ICMP outbound.
- **Provisioning:** Bootstrap script and sysctl performance tuning applied at creation time.

### 6.3 Media Groups

Media servers are organized by **group names** rather than by fleet. Group names (e.g., `rock`, `tea`, `rofl`, `beach`) identify logical clusters of media servers that can be scaled, deployed, and restarted independently. Each group can contain multiple server instances and can be assigned to any network/fleet combination.

```mermaid
graph TB
    subgraph "Clients"
        VR["VR Headsets"]
        WEB["Web Clients"]
    end

    subgraph "AWS us-west-2"
        subgraph "Media Group: rock"
            M1["AWS Media 1"]
            M2["AWS Media 2"]
        end
        subgraph "Media Group: tea"
            M3["AWS Media 1"]
            M4["AWS Media 2"]
        end
    end

    subgraph "DigitalOcean sfo2"
        subgraph "Media Group: beach"
            D1["DO Media 1"]
            D2["DO Media 2"]
        end
    end

    VR -->|"UDP 10000-65535"| M1
    VR -->|"UDP 10000-65535"| D1
    WEB -->|"UDP 10000-65535"| M3
```

---

## 7. Security Model

### 7.1 Security Group Architecture

Each service uses a **two-layer security group model**:

1. **ALB Security Group** — controls what traffic can reach the load balancer (internet-facing or restricted)
2. **Instance Security Group** — controls what traffic can reach the EC2 instance (only from its own ALB + SSH)

This ensures that EC2 instances never receive traffic directly from the internet — all application traffic flows through the load balancer.

```mermaid
graph LR
    Internet["Internet<br/>0.0.0.0/0"] -->|"HTTPS 443"| ALB_SG["ALB Security Group"]
    ALB_SG -->|"HTTP 80"| Instance_SG["Instance Security Group"]
    Jenkins["Jenkins CI/CD"] -->|"SSH 22"| Instance_SG
    DevSSH["Developer SSH<br/>Access Group"] -->|"SSH 22"| Instance_SG
```

**Security Rules by Service:**

| Service         | ALB Ingress                | Instance Ingress (from ALB) | Instance SSH         | Instance Egress          |
|-----------------|----------------------------|-----------------------------|----------------------|--------------------------|
| API             | `0.0.0.0/0` on 443        | Port 80 from ALB SG         | Jenkins + Dev SSH    | `0.0.0.0/0` (all)       |
| Admin API       | **No public rule**         | Port 80 from ALB SG         | Jenkins + Dev SSH    | **`172.31.0.0/16` only** |
| Cloud API       | `0.0.0.0/0` on 443        | Port 80 from ALB SG         | Jenkins + Dev SSH    | `0.0.0.0/0` (all)       |
| Cloud WebSocket | `0.0.0.0/0` on 443        | Port 80 from ALB SG         | Jenkins + Dev SSH    | `0.0.0.0/0` (all)       |
| WebApps         | Team Access SG only on 443 | Port 80 from ALB + Team SG  | Jenkins + Dev SSH    | `0.0.0.0/0` (all)       |
| Media (AWS)     | N/A (no ALB)               | UDP 10000-65535, ICMP       | Jenkins + Dev SSH    | `0.0.0.0/0` (all)       |
| Media (DO)      | N/A (no ALB)               | UDP 10000-65535             | Jenkins IPs only     | TCP/UDP/ICMP all         |

### 7.2 TLS Termination

All Application Load Balancers terminate TLS using:

- **TLS Policy:** TLS 1.3 with TLS 1.2 fallback (forward secrecy enabled)
- **Certificate:** Managed certificate from AWS Certificate Manager (ACM), single regional certificate shared across all ALBs
- **Backend:** Plaintext HTTP on port 80 between ALB and EC2 instances (within the VPC)

### 7.3 Egress Policies

Two egress models are in use:

| Model            | CIDR            | Services                                         | Rationale                                    |
|------------------|-----------------|--------------------------------------------------|----------------------------------------------|
| **Open egress**  | `0.0.0.0/0`    | API, Cloud API, Cloud WebSocket, WebApps, Media  | Requires internet for external APIs, CDNs    |
| **VPC-only**     | `172.31.0.0/16` | Admin API                                        | Security hardening — no internet dependency  |

### 7.4 SSH Access Model

- All fleet EC2 instances include a "Developer SSH Access" security group for interactive access.
- The Jenkins CI/CD server has SSH (port 22) access to all instances for automated deployment.
- SSH keys are stored on the Jenkins build server at predictable filesystem paths organized by network and fleet.
- Interactive SSH to any fleet server requires a two-hop connection: local machine -> Jenkins build server -> target instance.

### 7.5 Authentication and Secrets

- **Client authentication:** Firebase Authentication for client-facing APIs
- **API tokens:** JWT access tokens using public/private key pairs per fleet
- **Secrets storage:** Keys and credentials stored on the Jenkins filesystem, organized by network/fleet directory structure:
  - Firebase service account keys (full access and read-only variants)
  - JWT signing keys (private + public)
  - Brightcove tokens
  - Allowed apps configuration
  - SSH keys for cloud provider access

---

## 8. Environment Sizing

### 8.1 Compute Sizing by Environment

| Service          | Main (Production) | Dev              | Test       |
|------------------|-------------------|------------------|------------|
| API              | c5.4xlarge        | c5.large         | t2.micro   |
| Admin API        | c5.large          | c5.large         | t2.micro   |
| Cloud API        | c5.4xlarge        | c5.large         | t2.micro   |
| Cloud WebSocket  | c5.4xlarge        | c5.large         | t2.micro   |
| WebApps          | c5.large          | t3.medium        | t2.micro   |
| Media (AWS)      | Custom            | Custom           | t2.micro   |
| Media (DO)       | c-8-intel (8 CPU) | c-8-intel (8 CPU)| --         |

### 8.2 Database Sizing by Environment

| Database           | Main (Production)      | Dev                    | Test                   |
|--------------------|------------------------|------------------------|------------------------|
| Primary PostgreSQL | db.t3.xlarge / 1 TB    | db.t3.micro / 100 GB   | db.t3.micro / 100 GB   |
| Cloud PostgreSQL   | db.t3.xlarge / 1 TB    | db.t3.micro / 100 GB   | db.t3.micro / 100 GB   |

### 8.3 Storage Allocation

| Server Type     | Volume Type | Size    | IOPS       | Throughput | Notes                        |
|-----------------|-------------|---------|------------|------------|------------------------------|
| Fleet services  | gp2         | 128 GB  | Burst      | Burst      | API, Admin, Cloud API, WS    |
| WebApps         | gp2         | 64 GB   | Burst      | Burst      | Smaller — static asset focus |
| Media (AWS)     | gp3         | 100 GB  | 1000       | 200 MB/s   | Provisioned for streaming    |

---

## 9. Deployment Model

### 9.1 Pipeline Architecture

Deployment is managed through numbered Jenkins pipelines, each corresponding to a specific infrastructure or application lifecycle operation.

| Pipeline | Name                              | Purpose                                          |
|----------|-----------------------------------|--------------------------------------------------|
| 001      | init                              | Initialize fleet: config files, infrastructure, DNS |
| 002      | init_redis_cluster                | Deploy Redis cluster for a fleet                 |
| 003      | build_all / apis / admin_apis     | Build and deploy application services            |
| 004      | deploy_media_servers_digitalocean | Deploy DigitalOcean media servers                |
| 005      | build_media_servers_aws           | Deploy/manage AWS media servers                  |
| 006      | build_website                     | Deploy public website to S3                      |
| 007      | webapps                           | Deploy internal web applications                 |
| 009      | arda_admin_api                    | Deploy Arda + Admin API (kiwi fleet)             |
| 010      | setup_cloud_database              | Deploy Cloud PostgreSQL database                 |
| 097      | restart_media_servers             | Rolling restart of media servers                 |
| 098      | begin/end_load_test               | Load test lifecycle management                   |
| 099      | info                              | Query cloud provider for server DNS/metadata     |

### 9.2 Fleet Lifecycle

A new fleet is provisioned through a sequence of pipeline runs:

```mermaid
sequenceDiagram
    participant Operator
    participant Jenkins
    participant IaC as Infrastructure Tool
    participant Cloud as Cloud Provider
    participant Server

    Operator->>Jenkins: Trigger 001.init (network, fleet)
    Jenkins->>Jenkins: Generate config files
    Jenkins->>IaC: Apply fleet infrastructure
    IaC->>Cloud: Create placement group
    IaC->>Cloud: Create EC2 instances (API, Admin, Cloud, WS, WebApps)
    IaC->>Cloud: Create ALBs + target groups + listeners
    IaC->>Cloud: Create security groups
    Cloud-->>IaC: Instance IDs, DNS names
    IaC-->>Jenkins: Outputs (DNS, ALB DNS)
    Jenkins->>Server: SSH provisioning (Node.js, PM2, Nginx)
    Jenkins->>Jenkins: Assign DNS via Cloudflare API

    Operator->>Jenkins: Trigger 002.init_redis_cluster
    Jenkins->>IaC: Apply Redis cluster
    IaC->>Cloud: Create ElastiCache cluster

    Operator->>Jenkins: Trigger 003.build_all
    Jenkins->>Jenkins: Checkout cloud repo (branch: {network}-{fleet})
    Jenkins->>Server: Deploy cloud_websocket
    Jenkins->>Server: Deploy cloud_api
    Jenkins->>Server: Deploy api
    Jenkins->>Server: Deploy admin_api
    Jenkins->>Server: Deploy webapp_arda
    Jenkins->>Jenkins: Discord notification (success/failure)
```

### 9.3 Build and Deploy Flow

Two generations of builder scripts exist:

- **Builder v7** — Original builder, used for `main` network fleet initialization and production deployments.
- **Builder v8** — Enhanced builder, used for `dev`/`test` build-all pipelines with additional features.

Each deployment follows this pattern:

1. Checkout `devops` repository on Jenkins
2. Checkout `cloud` repository at the `{network}-{fleet}` branch
3. Verify installers and dependencies
4. Build each service sequentially: WebSocket -> Cloud setup -> API -> Cloud API -> Admin API -> WebApps
5. Post-build Discord notification with deployment summary

### 9.4 Load Testing

Load testing uses dedicated Windows-based User Simulator instances:

- **Platform:** Windows instances with custom load testing AMI
- **Capacity:** Approximately 75 simulated users per instance
- **Default Scale:** 15 instances = ~1,125 simulated users
- **Instance Count:** Configurable (1 to 100+)
- **Access:** RDP via dedicated security group
- **Lifecycle:** Ephemeral — infrastructure is created for the test and destroyed afterward
- **Targets:** API servers, Cloud WebSocket servers, and Media servers

```mermaid
graph LR
    subgraph "Load Test Fleet (ephemeral)"
        LS1["Simulator 1<br/>~75 users"]
        LS2["Simulator 2<br/>~75 users"]
        LSN["Simulator N<br/>~75 users"]
    end

    subgraph "Target Fleet"
        API["API Server"]
        CWS["Cloud WebSocket"]
        Media["Media Servers"]
    end

    LS1 --> API
    LS2 --> CWS
    LSN --> Media
```

---

## 10. Appendix

> The following reference tables contain current implementation-specific identifiers
> for cross-referencing with the existing AWS deployment. These values are specific
> to the current infrastructure and would differ in an alternative deployment.

### A. Port Reference

| Port        | Protocol | Purpose                   | Used By                                   |
|-------------|----------|---------------------------|-------------------------------------------|
| 22          | TCP      | SSH                       | Jenkins CI/CD, Developer access           |
| 80          | TCP      | HTTP (ALB -> instance)    | All ALB-backed services                   |
| 443         | TCP      | HTTPS / WSS (client -> ALB) | All public-facing services             |
| 3001-3015   | TCP      | Admin API PM2 instances   | Admin API (production only)               |
| 5432        | TCP      | PostgreSQL                | API -> Primary RDS, Cloud API -> Cloud RDS|
| 6379        | TCP      | Redis                     | API, Admin API, Cloud API, Cloud WebSocket|
| 10000-65535 | UDP      | Media streaming           | Media servers (AWS + DigitalOcean)        |
| 3389        | TCP      | RDP                       | Load test Windows instances               |

### B. Current Security Group IDs

| ID                     | Purpose                                |
|------------------------|----------------------------------------|
| `sg-0577fa74bb0447c98` | Jenkins SSH access to all servers      |
| `sg-0e0900af538192296` | [Bigscreen] General Team Access        |
| `sg-9b144be5`          | RDS database access (shared)           |

### C. Current Subnet IDs

| Subnet ID          | Used By                  |
|--------------------|--------------------------|
| `subnet-08218e43`  | ALBs, DB subnet groups   |
| `subnet-9e79f4e7`  | ALBs, DB subnet groups   |
| `subnet-bc5736e6`  | ALBs, DB subnet groups   |
| `subnet-41185f69`  | ALBs, DB subnet groups   |

### D. Current AMI Reference

| AMI ID                  | Name                           | Purpose                            |
|-------------------------|--------------------------------|------------------------------------|
| `ami-03aa99ddf5498ceb9` | Ubuntu 24 Base                 | All fleet services                 |
| `ami-0b36db8bdadf848d3` | MediaServerImageSep2022        | AWS media servers                  |
| `ami-0145e9c39d8dee107` | UserSimulatorLoadTest2Feb2023  | Windows load test instances        |