# AWS Terraform Fleet Infrastructure Audit

**Date:** October 2, 2025 (Updated)
**Original Date:** September 2, 2025
**Scope:** Complete fleet infrastructure (`main.tf`, `admin_api`, `api`, `cloud_api`, `cloud_websocket`, `webapps`, `website`)
**Status:** Issues identified, partial remediation completed, new issues identified

## Executive Summary

Comprehensive analysis of the Bigscreen fleet infrastructure reveals systemic security, operational, and maintainability issues across all modules. While the current setup is functional, it demonstrates patterns of legacy configuration management, security vulnerabilities, and architectural technical debt that require immediate attention. The modular approach using `main.tf` provides good separation of concerns but lacks modern Terraform best practices.

**October 2025 Update:** Recent additions of Node.js 24, PM2, and nginx installation provisioners have introduced new issues related to AMI/provisioner mismatch, installation reliability, and provisioner complexity. While the software installation approach is functionally sound, it conflicts with the existing AMI strategy and introduces additional provisioner anti-patterns.

## Infrastructure Context

### Pre-existing VPC Infrastructure
The fleet infrastructure correctly uses **pre-existing VPC and subnet infrastructure**, which is a common and appropriate pattern in enterprise environments. This approach is preferred when:

- **Shared Infrastructure**: Multiple applications/teams share the same network infrastructure
- **Compliance Requirements**: Network architecture requires security review and approval
- **Cost Optimization**: Avoiding duplicate network resources across teams
- **Organizational Boundaries**: Network teams manage VPCs, application teams manage compute resources
- **Complex Networking**: Existing peering connections, transit gateways, and DNS configurations

### Current Implementation Assessment
The fleet currently hardcodes VPC/subnet IDs directly in each module. While the architectural decision to use pre-existing infrastructure is correct, the **implementation approach can be improved** using modern Terraform patterns for better maintainability and environment portability.

## Critical Security Issues

### 1. Outdated SSL Policy
- **Issue:** Using deprecated `ELBSecurityPolicy-2016-08` (from 2016) across all ALB listeners
- **Files:** `admin_api.tf:179`, `api.tf:205`, `cloud_api.tf:189`, `cloud_websocket.tf:187`, `webapps.tf:184`, `website.tf:189`
- **Impact:** Security vulnerabilities, compliance failures, outdated cipher suites
- **Resolution:** Updated all 6 ALB listeners to `ELBSecurityPolicy-TLS13-1-2-2021-06` (modern TLS 1.3 with forward secrecy)
- **Status:** ✅ **FIXED** (September 2, 2025)

### 2. Overly Permissive Security Groups
- **Issue:** ELB security groups allow `0.0.0.0/0` on port 443 (with warning comments acknowledging the risk)
- **Files:** `admin_api.tf:132`, `api.tf:158`, `cloud_api.tf:142`, `cloud_websocket.tf:140`
- **Impact:** Unnecessary exposure to internet traffic, DDoS vulnerability
- **Resolution:** Removed unnecessary internet access from admin_api ELB security group (access controlled by separate security group). Maintained internet access for api, cloud_api, and cloud_websocket as required.
- **Status:** ✅ **FIXED** (September 2, 2025)

### 3. Hardcoded Security Group IDs
- **Issue:** Multiple hardcoded security group IDs across modules:
  - Jenkins: `sg-0577fa74bb0447c98` (all modules)
  - Team Access: `sg-0e0900af538192296` (webapps, website)
  - Master Website: `sg-0ab3f1573cc84c285` (website only)
- **Files:** All modules with security groups
- **Impact:** Environment coupling, security risk, deployment failures in new environments
- **Status:** ❌ Not Fixed

### 4. Overly Permissive Security Groups
- **Issue:** ELB security groups allow `0.0.0.0/0` on port 443 (with warning comments acknowledging the risk)
- **Files:** `admin_api.tf:132`, `api.tf:158`, `cloud_api.tf:142`, `cloud_websocket.tf:140`
- **Impact:** Unnecessary exposure to internet traffic, DDoS vulnerability
- **Status:** ✅ Fixed (September 2025) - admin_api ingress `0.0.0.0/0` rule removed. Remaining services (api, cloud_api, cloud_websocket) require public access for their functionality.

### 5. Unrestricted Outbound Traffic
- **Issue:** Security groups allow unrestricted outbound (`0.0.0.0/0`) on all protocols
- **Files:** All API security group egress rules
- **Impact:** Data exfiltration risk, compliance issues
- **Status:** ✅ Fixed (October 2025) - admin_api restricted to VPC-only egress (172.31.0.0/16). Remaining services (api, cloud_api, cloud_websocket, webapps, website) require internet egress for their functionality (external APIs, CDNs, package registries, etc.).

### 6. SSH Key Management
- **Issue:** Private keys stored in predictable filesystem paths following pattern `/home/ubuntu/networks/${var.network_name}/${var.fleet_name}/aws/${var.key_name}.pem`
- **Files:** All modules with provisioners (admin_api, api, cloud_api, cloud_websocket, webapps, website)
- **Impact:** Security risk, key exposure, path traversal potential
- **Status:** ✅ Not Applicable (October 2025) - SSH keys are only used by Terraform during provisioning on a secure, non-internet-exposed deployment server. This is standard Terraform practice. Keys are not stored on EC2 instances themselves. Application secrets are managed separately via AWS Secrets Manager.

## Configuration Management Issues

### 7. Deprecated Terraform Syntax
- **Issue:** Extensive legacy interpolation syntax `"${var.name}"` instead of `var.name` throughout all modules
- **Files:** All 7 modules (main.tf and 6 submodules)
- **Impact:** Maintainability, future Terraform compatibility, performance degradation
- **Status:** ✅ Fixed (October 2025) - Modernized all interpolation syntax across 8 files (~236 replacements)

### 8. Outdated SSL Policy
- **Issue:** Using deprecated `ELBSecurityPolicy-2016-08` (from 2016) across all ALB listeners
- **Files:** `admin_api.tf:179`, `api.tf:205`, `cloud_api.tf:189`, `cloud_websocket.tf:187`, `webapps.tf:184`, `website.tf:189`
- **Impact:** Security vulnerabilities, compliance failures, outdated cipher suites
- **Status:** ❌ Not Fixed

### 9. Inconsistent Provisioning
- **Issue:** Different approaches to nginx config deployment
- **Details:**
  - `admin_api` uses `mv` command
  - `api` uses `cp` command  
  - `cloud_api` uses `cp` command
- **Impact:** Operational inconsistency, maintenance overhead
- **Status:** ❌ Not Fixed

### 10. Missing Error Handling
- **Issue:** Provisioner scripts lack error handling
- **Files:** All remote-exec provisioners
- **Impact:** Silent failures, unreliable deployments
- **Status:** ❌ Not Fixed

### 11. Resource Naming Inconsistencies
- **Issue:** Target group attachments inconsistently named:
  - `test`: admin_api.tf:168, api.tf:194, cloud_websocket.tf:176, webapps.tf:173, website.tf:178
  - `cloudApiAttachment`: cloud_api.tf:178
- **Impact:** Configuration confusion, debugging difficulty, inconsistent automation
- **Status:** ❌ Not Fixed

### 12. Capitalization Inconsistencies
- **Issue:** Inconsistent naming conventions noted in comments
- **Files:** `cloud_api.tf:202-203`
- **Impact:** Code clarity, maintainability
- **Status:** ❌ Not Fixed

### 13. Infrastructure Reference Implementation (Medium Priority)
- **Issue:** VPC ID `vpc-6284a71b` and subnet IDs hardcoded directly in modules instead of using data sources or centralized variables
- **Files:** `admin_api.tf:85`, `api.tf:83`, `cloud_api.tf:96`, `cloud_websocket.tf:133`, `webapps.tf:86`, `website.tf:84`
- **Impact:** Environment coupling, deployment inflexibility (but appropriate use of pre-existing infrastructure)
- **Recommendation:** Use data sources or centralize in main.tf variables
- **Status:** ❌ Not Fixed

## Operational Issues

### 14. No High Availability
- **Issue:** Single instance deployments for all services (7 total instances)
- **Impact:** Single point of failure for each service, cascading failure risk
- **Status:** ❌ Not Fixed

### 14. Missing Auto-Scaling
- **Issue:** No auto-scaling groups configured
- **Impact:** Cannot handle traffic spikes, poor resource utilization
- **Status:** ❌ Not Fixed

### 15. No Health Checks
- **Issue:** Target groups lack custom health check configurations across all ALB-enabled services
- **Files:** admin_api, api, cloud_api, cloud_websocket, webapps, website target groups
- **Impact:** Traffic routed to unhealthy instances, poor user experience
- **Status:** ❌ Not Fixed

### 16. Missing Monitoring
- **Issue:** No CloudWatch integration or application monitoring
- **Impact:** Limited visibility into system health and performance
- **Status:** ❌ Not Fixed

### 17. Missing Logging
- **Issue:** No centralized logging configuration
- **Impact:** Difficult troubleshooting and audit trail
- **Status:** ❌ Not Fixed

### 18. Outdated Storage Configuration
- **Issue:** All instances using `gp2` volumes instead of more cost-effective `gp3`
- **Files:** All 7 modules (main.tf manages via module calls)
- **Impact:** 20-50% higher storage costs, lower baseline performance
- **Status:** ❌ Not Fixed

### 19. Variable Volume Size Issues
- **Issue:** Inconsistent volume sizing across services:
  - 128GB: admin_api, api, cloud_api, cloud_websocket
  - 64GB: webapps, website
- **Impact:** Potentially over-provisioned storage for some services, inconsistent resource allocation
- **Status:** ❌ Not Fixed

### 20. Missing EBS Encryption
- **Issue:** No encryption specified for EBS volumes
- **Impact:** Data security risk, compliance issues
- **Status:** ❌ Not Fixed

## Code Quality Issues

### 21. Code Duplication
- **Issue:** Massive code duplication across 6 modules - each module repeats similar patterns for ALB, security groups, target groups, and instances
- **Impact:** Maintenance nightmare, inconsistency risk, difficult updates
- **Status:** ❌ Not Fixed

### 22. Missing Module Consolidation
- **Issue:** Should be consolidated into generic reusable modules (e.g., `web_service`, `worker_service`) instead of 6 nearly-identical modules
- **Impact:** Code maintainability, consistency, easier updates and testing
- **Status:** ❌ Not Fixed

### 23. Missing Variable Descriptions and Validation
- **Issue:** Most variables across all modules lack descriptions, types, and validation rules
- **Files:** Variable definitions in all modules
- **Impact:** Poor developer experience, potential misconfigurations, lack of input validation
- **Status:** ❌ Not Fixed

### 24. Insufficient Tagging Strategy
- **Issue:** Only basic Name tags implemented
- **Impact:** Resource management, cost allocation
- **Status:** ❌ Not Fixed

### 25. Missing Lifecycle Management
- **Issue:** No backup, disaster recovery, or instance replacement strategies
- **Impact:** Data loss risk, extended recovery time, no automated recovery
- **Status:** ❌ Not Fixed

## New Issues Identified in October 2025 Update

### 39. AMI/Provisioner Strategy Conflict (CRITICAL)
- **Issue:** AMIs named "NginxServerMar2023" already contain nginx, but provisioners now reinstall nginx during instance creation
- **Files:** All modules now install nginx via provisioner despite AMI presumably containing it
- **Impact:**
  - Wasted provisioning time and network bandwidth
  - Potential version conflicts between AMI nginx and newly installed nginx
  - Unnecessary complexity and duplication
  - Unclear source of truth for software versions
- **Root Cause:** Mismatch between AMI preparation strategy and runtime provisioning
- **Status:** ❌ Not Fixed

### 40. Node.js Installation Without Version Pinning
- **Issue:** Node.js 24 installation uses `setup_24.x` script which installs latest 24.x version without pinning specific minor/patch version
- **Files:** `api.tf:65`, `admin_api.tf:67`, `cloud_api.tf:78`, `cloud_websocket.tf:75`, `webapps.tf:65`, `website.tf:66`
- **Impact:**
  - Different instances may get different Node.js versions if provisioned at different times
  - Difficult to reproduce exact environments
  - Potential compatibility issues between instances
  - No control over when breaking changes are introduced
- **Status:** ❌ Not Fixed

### 41. Missing Provisioner Failure Handling
- **Issue:** Long provisioner scripts with multiple commands lack proper error handling, continue on failure behavior not defined
- **Files:** All remote-exec provisioners
- **Impact:**
  - Silent failures possible (e.g., if npm install fails, PM2 won't be available)
  - Instances may appear healthy but be missing critical software
  - Difficult to diagnose provisioning failures post-deployment
- **Example:** If `curl` for Node.js fails, subsequent `sudo apt-get install -y nodejs` will fail silently
- **Status:** ❌ Not Fixed

### 42. Network Dependency During Provisioning
- **Issue:** Provisioners depend on external network resources (nodesource.com, npmjs.com, Ubuntu repositories) being available
- **Files:** All modules with Node.js/PM2 installation
- **Impact:**
  - Provisioning fails if external sites are down or rate-limiting
  - No offline deployment capability
  - Vulnerable to supply chain attacks via compromised external repositories
  - Unpredictable provisioning times based on network conditions
- **Status:** ❌ Not Fixed

### 43. Redundant apt-get update Calls
- **Issue:** `sudo apt-get update` called twice in most provisioners (once for Node.js setup, once before nginx install)
- **Files:** `api.tf:72`, `admin_api.tf:74`, `cloud_api.tf:85`, `cloud_websocket.tf:82`, `webapps.tf:72`, `website.tf:73`
- **Impact:**
  - Doubled provisioning time for package index updates
  - Unnecessary network traffic
  - Inefficient resource usage
- **Status:** ❌ Not Fixed

### 44. Inconsistent Provisioner Patterns Across Modules
- **Issue:** (Resolved - cloud_worker module removed October 2025)
- **Impact:**
  - Maintenance confusion
  - Copy-paste errors when updating provisioners
  - Unclear whether differences are intentional or oversight
- **Status:** ❌ Not Fixed

### 45. AMI Strategy Undermined
- **Issue:** Original AMI strategy (NginxServerMar2023, WebAppServerMar2023, BasicServerMar2023) is now irrelevant since all software installed via provisioners
- **Files:** `main.tf:22-55` (AMI definitions) vs module provisioners
- **Impact:**
  - Wasted effort in AMI creation and maintenance
  - AMIs become stale and potentially insecure
  - Longer instance launch times (software installation at runtime vs baked into AMI)
  - Loss of golden image benefits (consistency, speed, reliability)
- **Status:** ✅ Fixed (October 2025) - All modules now use base Ubuntu 24 AMI (`ami-03aa99ddf5498ceb9`), eliminating reliance on outdated custom AMIs from March 2023

### 46. Missing Software Version Documentation
- **Issue:** No documentation of which Node.js minor version, PM2 version, or nginx version is being installed
- **Files:** All module provisioners
- **Impact:**
  - Impossible to audit software versions without launching instances
  - No change tracking for software updates
  - Compliance and security audit difficulties
- **Status:** ❌ Not Fixed

### 47. PM2 Global Installation Without Configuration
- **Issue:** PM2 installed globally but no startup configuration, process files, or ecosystem config deployed
- **Files:** All modules with PM2 installation
- **Impact:**
  - PM2 installed but not configured for actual use
  - No process persistence across reboots
  - Manual configuration required post-deployment
- **Status:** ❌ Not Fixed

### 48. cloud_worker Module Inconsistency
- **Issue:** (Resolved - cloud_worker module removed October 2025)
- **Status:** ✅ Fixed - Module removed from fleet
- **Impact:**
  - nginx installed but never configured (no sites-enabled config)
  - Wasted resources on unused nginx installation
  - Unclear service requirements
- **Status:** ❌ Not Fixed

## Previously Identified Issues (September 2025 Review)

### 1. Hardcoded Security Group IDs
- **Issue:** Multiple hardcoded security group IDs across modules:
  - Jenkins: `sg-0577fa74bb0447c98` (all modules)
  - Team Access: `sg-0e0900af538192296` (webapps, website)
  - Master Website: `sg-0ab3f1573cc84c285` (website only)
- **Files:** All modules with security groups
- **Impact:** Environment coupling, security risk, deployment failures in new environments
- **Status:** ❌ Not Fixed

### 2. Unrestricted Outbound Traffic
- **Issue:** Security groups allow unrestricted outbound (`0.0.0.0/0`) on all protocols
- **Files:** All API security group egress rules
- **Impact:** Data exfiltration risk, compliance issues
- **Status:** ❌ Not Fixed

### 3. SSH Key Management
- **Issue:** Private keys stored in predictable filesystem paths following pattern `/home/ubuntu/networks/${var.network_name}/${var.fleet_name}/aws/${var.key_name}.pem`
- **Files:** All modules with provisioners (admin_api, api, cloud_api, cloud_websocket, webapps, website)
- **Impact:** Security risk, key exposure, path traversal potential
- **Status:** ❌ Not Fixed

### 4. Deprecated Terraform Syntax
- **Issue:** Extensive legacy interpolation syntax `"${var.name}"` instead of `var.name` throughout all modules
- **Files:** All 7 modules (main.tf and 6 submodules)
- **Impact:** Maintainability, future Terraform compatibility, performance degradation
- **Status:** ❌ Not Fixed

### 5. Inconsistent Provisioning Commands
- **Issue:** Different approaches to nginx config deployment
- **Details:**
  - `admin_api` uses `mv` command
  - `api` uses `cp` command
  - `cloud_api` uses `cp` command
  - Other modules use `cp` command
- **Impact:** Operational inconsistency, maintenance overhead
- **Status:** ❌ Not Fixed

### 6. Resource Naming Inconsistencies
- **Issue:** Target group attachments inconsistently named:
  - `test`: admin_api.tf:160, api.tf:194, cloud_websocket.tf:176, webapps.tf:173, website.tf:178
  - `cloudApiAttachment`: cloud_api.tf:178
- **Impact:** Configuration confusion, debugging difficulty, inconsistent automation
- **Status:** ❌ Not Fixed

### 7. Capitalization Inconsistencies
- **Issue:** Inconsistent naming conventions noted in comments
- **Files:** `cloud_api.tf:202-203`
- **Impact:** Code clarity, maintainability
- **Status:** ❌ Not Fixed

### 8. Infrastructure Reference Implementation (Medium Priority)
- **Issue:** VPC ID `vpc-6284a71b` and subnet IDs hardcoded directly in modules instead of using data sources or centralized variables
- **Files:** `admin_api.tf:85`, `api.tf:83`, `cloud_api.tf:96`, `cloud_websocket.tf:93`, `webapps.tf:86`, `website.tf:84`
- **Impact:** Environment coupling, deployment inflexibility (but appropriate use of pre-existing infrastructure)
- **Recommendation:** Use data sources or centralize in main.tf variables
- **Status:** ❌ Not Fixed

### 9. No High Availability
- **Issue:** Single instance deployments for all services (7 total instances)
- **Impact:** Single point of failure for each service, cascading failure risk
- **Status:** ❌ Not Fixed

### 10. Missing Auto-Scaling
- **Issue:** No auto-scaling groups configured
- **Impact:** Cannot handle traffic spikes, poor resource utilization
- **Status:** ❌ Not Fixed

### 11. No Health Checks
- **Issue:** Target groups lack custom health check configurations across all ALB-enabled services
- **Files:** admin_api, api, cloud_api, cloud_websocket, webapps, website target groups
- **Impact:** Traffic routed to unhealthy instances, poor user experience
- **Status:** ❌ Not Fixed

### 12. Missing Monitoring
- **Issue:** No CloudWatch integration or application monitoring
- **Impact:** Limited visibility into system health and performance
- **Status:** ❌ Not Fixed

### 13. Missing Logging
- **Issue:** No centralized logging configuration
- **Impact:** Difficult troubleshooting and audit trail
- **Status:** ❌ Not Fixed

### 14. Outdated Storage Configuration
- **Issue:** All instances using `gp2` volumes instead of more cost-effective `gp3`
- **Files:** All 7 modules (main.tf manages via module calls)
- **Impact:** 20-50% higher storage costs, lower baseline performance
- **Status:** ❌ Not Fixed

### 15. Variable Volume Size Issues
- **Issue:** Inconsistent volume sizing across services:
  - 128GB: admin_api, api, cloud_api, cloud_websocket
  - 64GB: webapps, website
- **Impact:** Potentially over-provisioned storage for some services, inconsistent resource allocation
- **Status:** ❌ Not Fixed

### 16. Missing EBS Encryption
- **Issue:** No encryption specified for EBS volumes
- **Impact:** Data security risk, compliance issues
- **Status:** ❌ Not Fixed

### 17. Code Duplication
- **Issue:** Massive code duplication across 6 modules - each module repeats similar patterns for ALB, security groups, target groups, and instances
- **Impact:** Maintenance nightmare, inconsistency risk, difficult updates
- **Status:** ❌ Not Fixed

### 18. Missing Module Consolidation
- **Issue:** Should be consolidated into generic reusable modules (e.g., `web_service`, `worker_service`) instead of 6 nearly-identical modules
- **Impact:** Code maintainability, consistency, easier updates and testing
- **Status:** ❌ Not Fixed

### 19. Missing Variable Descriptions and Validation
- **Issue:** Most variables across all modules lack descriptions, types, and validation rules
- **Files:** Variable definitions in all modules
- **Impact:** Poor developer experience, potential misconfigurations, lack of input validation
- **Status:** ❌ Not Fixed

### 20. Insufficient Tagging Strategy
- **Issue:** Only basic Name tags implemented
- **Impact:** Resource management, cost allocation
- **Status:** ❌ Not Fixed

### 21. Missing Lifecycle Management
- **Issue:** No backup, disaster recovery, or instance replacement strategies
- **Impact:** Data loss risk, extended recovery time, no automated recovery
- **Status:** ❌ Not Fixed

### 22. Main.tf Architectural Issues
- **Issue:** Main.tf contains 164 lines of repetitive variable definitions and default values
- **Files:** `main.tf:1-272`
- **Impact:** Configuration drift, difficult environment management, duplicate variable definitions
- **Status:** ❌ Not Fixed

### 23. AMI Management Issues
- **Issue:** AMI IDs hardcoded in main.tf with comments like "NginxServerMar2023" and "BasicServerMar2023" from March 2023
- **Files:** `main.tf:22-55`
- **Impact:** Outdated base images, security vulnerabilities, manual AMI rotation required
- **Status:** ❌ Not Fixed - **WORSENED** by provisioner additions

### 24. Certificate Management Issues
- **Issue:** Same certificate ARN hardcoded across all services in main.tf
- **Files:** `main.tf:135-163`
- **Impact:** Single point of failure for TLS, certificate rotation complexity
- **Status:** ❌ Not Fixed

### 25. Missing ALB Access Logs
- **Issue:** No access logging configured for any Application Load Balancers
- **Files:** All ALB resources across modules
- **Impact:** No audit trail, difficult troubleshooting, compliance gaps
- **Status:** ❌ Not Fixed

### 26. Inconsistent Security Group Patterns
- **Issue:** Different modules use different approaches:
  - webapps/website: Uses additional hardcoded team access security groups
  - APIs: Standard ALB + SSH pattern
- **Impact:** Inconsistent security posture, confusion in access management
- **Status:** ❌ Not Fixed

### 27. Missing WAF Protection
- **Issue:** No Web Application Firewall configured for any public-facing ALBs
- **Files:** All internet-facing ALBs
- **Impact:** Exposed to application-layer attacks, DDoS vulnerability
- **Status:** ❌ Not Fixed

### 28. No Resource Limits or Quotas
- **Issue:** No guardrails on resource creation or costs
- **Impact:** Potential cost overruns, resource exhaustion
- **Status:** ❌ Not Fixed

### 29. Multiple Output Inconsistencies
- **Issue:** webapps.tf outputs multiple duplicate DNS entries for different webapp names (arda, fabricator, rohan) all pointing to same instance
- **Files:** `webapps.tf:204-226`
- **Impact:** Confusing outputs, misleading service discovery
- **Status:** ❌ Not Fixed

### 30. Provider Configuration Issues
- **Issue:** Provider block repeated in every module instead of centralized configuration
- **Files:** All 6 submodules
- **Impact:** Version drift potential, difficult provider updates
- **Status:** ❌ Not Fixed

### 31. Missing Environment Separation
- **Issue:** No clear distinction between dev/staging/production deployments
- **Files:** main.tf defaults suggest single-environment usage
- **Impact:** Risk of accidental production changes, no environment isolation
- **Status:** ❌ Not Fixed

### 32. Placement Group Issues
- **Issue:** Placement group used inconsistently:
  - Hardcoded in some modules (apis, worker, websocket)
  - Variable-controlled in others (webapps, website)
  - No clear strategy
- **Files:** main.tf:166-169 and module configurations
- **Impact:** Inconsistent performance characteristics, networking confusion
- **Status:** ❌ Not Fixed

### 33. Missing Termination Protection
- **Issue:** No termination protection on any critical instances
- **Impact:** Risk of accidental instance deletion
- **Status:** ❌ Not Fixed

### 34. Provisioner Anti-Patterns (Original)
- **Issue:** Using provisioners for configuration management instead of proper bootstrap methods
- **Files:** All modules with nginx config deployment
- **Impact:** Non-idempotent deployments, difficult troubleshooting, state drift
- **Status:** ❌ Not Fixed - **WORSENED** (October 2025)
- **Update:** Provisioner complexity significantly increased with Node.js 24, PM2, and nginx installation steps

## Recommendations by Priority

### Critical Priority (Security & Compliance)
1. ~~**Upgrade SSL policies** from 2016 to current AWS recommendations~~ ✅ **COMPLETED** (September 2025)
2. ~~**Restrict security group rules** - remove 0.0.0.0/0 access where not required~~ ✅ **COMPLETED** (September 2025)
3. **Resolve AMI/provisioner strategy conflict** - Choose one approach (bake into AMI OR install via provisioner, not both)
4. **Implement secrets management** for SSH keys using AWS Systems Manager Parameter Store or Secrets Manager
5. **Update AMI strategy** - implement automated AMI updates with current security patches (especially critical now that provisioners install software)
6. **Add WAF protection** to all internet-facing ALBs
7. **Enable ALB access logging** for compliance and audit trails

### High Priority (Architecture & Stability)
1. **Fix provisioner reliability issues** - Add error handling, version pinning, and idempotency
2. **Consolidate duplicate modules** into reusable generic modules (web_service, worker_service)
3. **Update to modern Terraform syntax** - remove all legacy interpolation
4. **Add health checks** to all target groups with appropriate paths and timeouts
5. **Implement proper variable validation** and descriptions across all modules
6. **Add termination protection** to critical production instances
7. **Replace provisioners** with proper User Data or cloud-init configurations (or bake into AMI)
8. **Centralize provider configuration** in main.tf only

### Medium Priority (Operational Improvements)
1. **Modernize infrastructure references** - Replace hardcoded VPC/subnet IDs with data sources or centralized variables
2. **Upgrade to gp3 storage** and implement proper volume sizing strategy
3. **Add comprehensive tagging strategy** for cost allocation and resource management
4. **Implement monitoring and alerting** with CloudWatch and SNS
5. **Add centralized logging** with CloudWatch Logs or ELK stack
6. **Enable EBS encryption** by default
7. **Add placement group strategy** - either use consistently or remove entirely
8. **Implement environment separation** (dev/staging/production)

### Low Priority (Advanced Features)
1. **Implement auto-scaling capabilities** with proper scaling policies
2. **Add multi-AZ deployment** for high availability
3. **Implement backup and disaster recovery** procedures
4. **Add blue-green deployment support** with proper traffic shifting
5. **Implement resource limits and cost controls**
6. **Add container orchestration** consideration for microservices architecture

## Implementation Examples

### Fixing Provisioner Issues (October 2025)

#### Option 1: Bake Software into AMI (Recommended)
The best approach is to create new AMIs with Node.js 24, PM2, and nginx pre-installed, then remove provisioners entirely:

```bash
# Using Packer to build AMI
packer build -var 'nodejs_version=24.9.0' bigscreen-app-server.pkr.hcl
```

```hcl
# Packer template (bigscreen-app-server.pkr.hcl)
source "amazon-ebs" "nodejs_server" {
  ami_name      = "bigscreen-nodejs24-{{timestamp}}"
  instance_type = "t2.micro"
  region        = "us-west-2"
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["099720109477"] # Canonical
  }
  ssh_username = "ubuntu"
}

build {
  sources = ["source.amazon-ebs.nodejs_server"]

  provisioner "shell" {
    inline = [
      "curl -fsSL https://deb.nodesource.com/setup_${var.nodejs_version} | sudo -E bash -",
      "sudo apt-get install -y nodejs",
      "sudo npm install -g pm2@latest",
      "sudo apt-get update",
      "sudo apt-get install -y nginx",
      "sudo systemctl enable nginx",
      "sudo systemctl enable pm2"
    ]
  }
}
```

Then simplify Terraform modules to only configure nginx (not install):
```hcl
provisioner "remote-exec" {
  inline = [
    "sudo cp /home/ubuntu/default.nginx.conf /etc/nginx/sites-enabled/default",
    "sudo nginx -t",  # Validate config
    "sudo systemctl restart nginx"
  ]
}
```

#### Option 2: Improve Provisioners (Interim Solution)
If AMI creation is not immediately possible, improve provisioner reliability:

```hcl
provisioner "remote-exec" {
  inline = [
    "set -euo pipefail",  # Exit on error, undefined variables, pipe failures

    # Install Node.js with specific version
    "curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -",
    "sudo apt-get install -y nodejs=24.9.0-1nodesource1",  # Pin exact version

    # Verify Node.js installation
    "node --version | grep -q '^v24'",

    # Install PM2 with specific version
    "sudo npm install -g pm2@5.3.0",  # Pin version

    # Verify PM2 installation
    "pm2 --version | grep -q '5.3.0'",

    # Only install nginx if not already present
    "if ! command -v nginx &> /dev/null; then",
    "  sudo apt-get update",
    "  sudo apt-get install -y nginx=1.18.0-*",  # Pin major version
    "fi",

    # Configure nginx
    "sudo cp /home/ubuntu/default.nginx.conf /etc/nginx/sites-enabled/default",
    "sudo nginx -t",  # Validate before restart
    "sudo systemctl restart nginx"
  ]

  on_failure = "fail"  # Explicitly fail on errors
}

# Add connection retry logic
connection {
  type        = "ssh"
  host        = self.public_dns
  user        = "ubuntu"
  private_key = file("/home/ubuntu/networks/${var.network_name}/${var.fleet_name}/aws/${var.key_name}.pem")
  timeout     = "10m"

  # Retry connection
  retries     = 3
}
```

#### Option 3: Use User Data (Better than Provisioners)
Replace provisioners with user data for better reliability:

```hcl
resource "aws_instance" "api" {
  ami           = var.ami
  instance_type = var.instance_type
  key_name      = var.key_name
  user_data     = templatefile("${path.module}/user_data.sh", {
    network_name = var.network_name
    fleet_name   = var.fleet_name
    nodejs_version = "24.9.0-1nodesource1"
    pm2_version    = "5.3.0"
  })

  user_data_replace_on_change = true

  # ... rest of configuration
}
```

```bash
#!/bin/bash
# user_data.sh
set -euo pipefail

# Log all output
exec > >(tee /var/log/user-data.log)
exec 2>&1

echo "Starting server setup..."

# Install Node.js
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
apt-get install -y nodejs=${nodejs_version}

# Install PM2
npm install -g pm2@${pm2_version}

# Install nginx (only if needed)
if ! command -v nginx &> /dev/null; then
  apt-get update
  apt-get install -y nginx
fi

# Download nginx config from S3 or parameter store
aws s3 cp s3://bigscreen-configs/${network_name}/${fleet_name}/nginx.conf /etc/nginx/sites-enabled/default
nginx -t && systemctl restart nginx

echo "Server setup complete!"
```

### Infrastructure Reference Modernization

#### Option 1: Data Sources Approach (Recommended)
Replace hardcoded VPC/subnet references with dynamic data sources in `main.tf`:

```hcl
# main.tf - Query existing VPC and subnets
data "aws_vpc" "main" {
  id = "vpc-6284a71b"  # or use filters/tags if needed
}

data "aws_subnets" "app_subnets" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.main.id]
  }
  filter {
    name   = "tag:Tier"
    values = ["Application"]  # if subnets are tagged
  }
}

# Pass to modules
module "api" {
  source = "./api"
  vpc_id = data.aws_vpc.main.id
  subnet_ids = data.aws_subnets.app_subnets.ids
  network_name = var.network_name
  fleet_name = var.fleet_name
  # ... other variables
}
```

#### Option 2: Centralized Variables Approach (Interim Solution)
Centralize infrastructure IDs in `main.tf` variables:

```hcl
# main.tf
locals {
  vpc_id = "vpc-6284a71b"
  subnet_ids = ["subnet-08218e43", "subnet-9e79f4e7", "subnet-bc5736e6", "subnet-41185f69"]
}

module "api" {
  source = "./api"
  vpc_id = local.vpc_id
  subnet_ids = local.subnet_ids
  # ... other variables
}
```

#### Module Updates Required
Update each module to accept these as variables instead of hardcoding:

```hcl
# api/api.tf (and similar for other modules)
variable "vpc_id" {
  description = "VPC ID for resources"
  type        = string
}

variable "subnet_ids" {
  description = "Subnet IDs for ALB"
  type        = list(string)
}

resource "aws_security_group" "api_security_group" {
  name        = "${local.api_server_name}_security_group"
  description = "${local.api_server_name} Internal"
  vpc_id      = var.vpc_id  # <-- Use variable instead of hardcoded
}

resource "aws_lb" "api_elb" {
  name               = "${local.api_server_name}-elb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.api_elb_security_group.id]
  subnets            = var.subnet_ids  # <-- Use variable instead of hardcoded
}
```

## Next Steps

1. **Emergency Security Review:** Immediate assessment of hardcoded values and SSL policies
2. **Environment Strategy:** Create separate terraform workspaces/directories for dev/staging/production
3. **Module Refactoring:** Design and implement generic reusable modules
4. **Gradual Migration Plan:** Phase out legacy modules in favor of new consolidated approach
5. **Testing Strategy:** Implement terraform plan/apply in development environment first
6. **Documentation:** Create proper README and deployment guides
7. **Validation:** Test each fix thoroughly before production deployment

## Impact Assessment

**October 2025 Update:**
- **New Critical Issues:** 10 issues identified from recent provisioning changes
- **Worsened Issues:** 2 existing issues made worse by new provisioning approach
- **Recent Fixes:** Issue #4 (Overly Permissive Security Groups), Issue #5 (Unrestricted Outbound Traffic), Issue #7 (Deprecated Terraform Syntax), and Issue #45 (AMI Strategy) resolved
- **Not Applicable:** Issue #6 (SSH Key Management) - determined to be standard practice for secure deployment servers
- **Module Removal:** cloud_worker module removed from fleet (no longer needed)

**Overall Status:**
**Critical Risk Issues:** 14 immediate security and compliance issues (6 resolved/N/A, 10 new)
**High Risk Issues:** 9 architecture and stability issues identified (7 original + 2 worsened)
**Medium Risk Issues:** 14 operational and maintainability issues
**Low Risk Issues:** 11 enhancement and optimization opportunities

**Total Technical Debt:** 48 distinct issues requiring remediation (6 completed/N/A, 10 new, 2 worsened)

### Risk Distribution:
- **Critical**: ~~SSL policies~~ ✅, ~~security groups~~ ✅, ~~SSH key mgmt~~ ✅ N/A, ~~AMI strategy~~ ✅, ~~permissive SGs~~ ✅, ~~outbound traffic~~ ✅, **AMI/provisioner conflict** (NEW), **Node.js version pinning** (NEW), **provisioner failure handling** (NEW), **network dependencies** (NEW), secrets management, AMI updates, WAF, access logging
- **High**: **Provisioner reliability** (NEW), Module consolidation, ~~syntax modernization~~ ✅, health checks, variable validation, termination protection, provisioner replacement, provider centralization
- **Medium**: **Redundant apt-get** (NEW), **provisioner inconsistencies** (NEW), Infrastructure references, storage optimization, tagging, monitoring, logging, encryption, placement groups, environment separation
- **Low**: Auto-scaling, multi-AZ, backup/recovery, blue-green deployment, cost controls, container orchestration

## Best Practices for Enterprise Infrastructure

### When to Create vs Reference Infrastructure

#### Create in Terraform:
- **Application-specific resources**: EC2 instances, security groups, target groups, ALBs
- **Ephemeral infrastructure**: Development/testing environments
- **Microservice-specific**: Resources that scale with individual applications
- **Cost optimization**: Resources that can be destroyed when not needed

#### Reference Existing Infrastructure:
- **Shared network infrastructure**: VPCs, subnets, route tables, NAT gateways
- **Centrally managed resources**: DNS zones, certificates, shared security groups
- **Compliance-controlled**: Resources requiring security/network team approval
- **Cross-team dependencies**: Resources managed by different teams/departments

### Data Source Best Practices

#### Use Cases for Data Sources:
- **Dynamic Infrastructure Discovery**: Query resources by tags rather than hardcoded IDs
- **Multi-Environment Portability**: Same code works across dev/staging/production
- **Dependency Management**: Ensure referenced resources exist before creating dependents
- **Reduced Configuration**: Avoid manual lookups and copy-paste errors

#### Example Patterns:
```hcl
# Query by tags for environment-agnostic code
data "aws_vpc" "main" {
  tags = {
    Environment = var.environment
    Purpose     = "shared"
  }
}

# Query subnets by tier and availability
data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.main.id]
  }
  tags = {
    Tier = "private"
  }
}

# Query most recent AMI
data "aws_ami" "app_server" {
  most_recent = true
  owners      = ["self"]
  
  filter {
    name   = "name"
    values = ["bigscreen-nginx-*"]
  }
}
```

### Environment Separation Strategies

#### 1. Workspace-based (Simple)
```bash
terraform workspace new dev
terraform workspace new staging  
terraform workspace new production
```

#### 2. Directory-based (Recommended)
```
environments/
├── dev/
│   ├── main.tf
│   ├── terraform.tfvars
├── staging/
│   ├── main.tf  
│   ├── terraform.tfvars
└── production/
    ├── main.tf
    ├── terraform.tfvars
```

#### 3. State Management
- **Remote State**: Use S3 + DynamoDB for state locking
- **State Separation**: Different state files per environment
- **Cross-Environment References**: Use terraform_remote_state data source

## October 2025 Provisioner Analysis Summary

### What Changed
All 7 fleet modules were updated to install Node.js 24, PM2, and nginx via Terraform provisioners during instance creation.

### Problems Introduced
1. **Conflicts with existing AMIs** that already contain nginx
2. **No version pinning** - different instances may get different software versions
3. **No error handling** - silent failures possible
4. **Network dependencies** - provisioning fails if external sites unavailable
5. **Inefficient** - redundant apt-get updates and longer launch times
7. **Undermines AMI strategy** - AMIs from March 2023 now irrelevant

### Recommended Path Forward
**Option 1 (Best):** Create new AMIs with Node.js 24, PM2, and nginx pre-installed using Packer, remove provisioners
**Option 2 (Interim):** Add error handling, version pinning, and idempotency checks to provisioners
**Option 3 (Alternative):** Replace provisioners with user_data scripts

### Key Principle
**Choose ONE approach:** Either bake software into AMI (fast, reliable) OR install at runtime (flexible, slower). Don't do both.

## Implementation History

### Completed Fixes

#### Node.js 24 and PM2 Provisioner Addition (October 2, 2025)
- **Scope**: Added Node.js 24, PM2, and nginx installation to all 7 fleet modules via provisioners
- **Modules Modified**: api, admin_api, cloud_api, cloud_websocket, webapps, website
- **Approach**: Runtime installation via remote-exec provisioners
- **Status**: ✅ Functional but introduced new technical debt (see issues #39-48)
- **Next Steps**: Requires remediation to address reliability and consistency issues

#### SSL Policy Modernization (September 2, 2025)
- **Scope**: Updated all 6 ALB listeners from `ELBSecurityPolicy-2016-08` to `ELBSecurityPolicy-TLS13-1-2-2021-06`
- **Security Benefits**: 
  - TLS 1.3 support with forward secrecy
  - Modern cipher suites with deprecated algorithms removed
  - Improved connection establishment performance
  - Compliance with current security standards
- **Files Modified**:
  - `admin_api/admin_api.tf:179`
  - `api/api.tf:205`
  - `cloud_api/cloud_api.tf:189`
  - `cloud_websocket/cloud_websocket.tf:187`
  - `webapps/webapps.tf:184`
  - `website/website.tf:189`

#### Security Group Access Restriction (September 2, 2025 & October 2025)
- **Scope**: Removed overly permissive security group rules where inappropriate
- **Security Benefits**:
  - Reduced attack surface for admin API (administrative interface)
  - Maintained necessary internet access for public APIs
  - Better defense-in-depth security posture
  - Restricted outbound traffic to prevent data exfiltration
- **Changes Made**:
  - **September 2, 2025**: Removed `0.0.0.0/0` ingress rule from admin_api (access controlled by separate security group)
  - **October 2025**: Restricted admin_api egress traffic to VPC-only (172.31.0.0/16) for both instance and ELB security groups
  - **api, cloud_api, cloud_websocket**: Maintained internet access as required for public services
- **Files Modified**:
  - `admin_api/admin_api.tf:116-120` (instance security group egress)
  - `admin_api/admin_api.tf:140-144` (ELB security group egress)
- **Next Steps**: Run `terraform plan` and `terraform apply` to implement changes

## Notes

- Current setup is functional but represents significant technical debt
- Architecture shows signs of organic growth without governance
- Many issues stem from copy-paste development patterns
- Modular structure is good foundation but needs consolidation
- **Two critical security fixes completed** - SSL policy modernization and security group access restriction
- Changes should be tested in development environment first
- Consider infrastructure-as-code governance and review processes
- Plan for gradual rollout to minimize service disruption
- Document all changes and establish ongoing maintenance procedures