# AWS Secrets Manager Migration Plan

## Overview

This document outlines the migration plan for transitioning from filesystem-based secret management to AWS Secrets Manager for the `init_server_fleet_config_files` function in `server_builder_v8.sh`.

## Current State Analysis

The `init_server_fleet_config_files` function (located at `C:\Bigscreen\devops\Jenkins\server_builder_v8.sh:169`) currently handles several types of sensitive data directly on the filesystem:

1. **AWS EC2 Key Pairs** (lines 193-212): PEM files stored in `$NETWORK_FLEET_DIR/aws/`
2. **Config Files** (lines 214-232): JSON config with URLs and service names
3. **Firebase Credentials** (lines 234-237): Multiple Firebase JSON files
4. **Access Token Keys** (lines 239-243): Private/public key pairs
5. **Brightcove Token** (line 245-246): PEM file
6. **Allowed Apps** (lines 248-249): JSON file
7. **DigitalOcean Keys** (lines 251-274): SSH key pairs and PAT tokens

---

## Migration Strategy

### Phase 1: Secret Storage Design

**Organize secrets hierarchically in AWS Secrets Manager:**

```
/{NETWORK_NAME}/{FLEET_NAME}/aws/keypair       # EC2 PEM key material
/{NETWORK_NAME}/{FLEET_NAME}/firebase/admin    # Firebase admin credentials
/{NETWORK_NAME}/{FLEET_NAME}/firebase/readonly # Firebase readonly credentials
/{NETWORK_NAME}/{FLEET_NAME}/firebase/analytics # Firebase analytics credentials
/{NETWORK_NAME}/{FLEET_NAME}/access-token/private # Access token private key
/{NETWORK_NAME}/{FLEET_NAME}/access-token/public  # Access token public key
/{NETWORK_NAME}/{FLEET_NAME}/brightcove/token     # Brightcove PEM
/{NETWORK_NAME}/{FLEET_NAME}/allowed-apps         # Allowed apps JSON
/{NETWORK_NAME}/{FLEET_NAME}/digitalocean/pat     # DigitalOcean PAT
/{NETWORK_NAME}/{FLEET_NAME}/digitalocean/ssh-private # DO SSH private key
/{NETWORK_NAME}/{FLEET_NAME}/digitalocean/ssh-public  # DO SSH public key
/{NETWORK_NAME}/{FLEET_NAME}/config               # Base configuration JSON
```

### Phase 2: Network-Level Secrets

Some secrets are shared across fleets (currently in `$HOME_FOLDER/$NETWORK_NAME/.keys/`):

```
/{NETWORK_NAME}/shared/firebase/admin
/{NETWORK_NAME}/shared/firebase/readonly
/{NETWORK_NAME}/shared/firebase/analytics
/{NETWORK_NAME}/shared/access-token/private
/{NETWORK_NAME}/shared/access-token/public
/{NETWORK_NAME}/shared/brightcove/token
/{NETWORK_NAME}/shared/allowed-apps
/{NETWORK_NAME}/shared/config/template
```

**Decision Point:** Do you want fleet-specific secrets or shared network secrets? Current code copies from network-level to fleet-level, suggesting you might want:
- Store once at network level for shared secrets
- Store at fleet level only for fleet-specific overrides

### Phase 3: Function Transformation

**Current Flow:**
1. Create local directories
2. Generate/fetch keys → Save to filesystem
3. Copy config files from templates
4. Modify configs with jq
5. Use filesystem paths throughout deployment

**New Flow:**
1. ~~Create local directories~~ (optional, only if temp files needed)
2. Generate/fetch keys → **Store in AWS Secrets Manager**
3. ~~Copy config files~~ → **Build config in memory from Secrets Manager**
4. Modify configs in memory
5. **Store final config back to Secrets Manager**
6. **Deployment scripts fetch from Secrets Manager at runtime**

### Phase 4: Key Changes Required

#### A. AWS Key Pair Handling (lines 193-212)
**Current:** Creates EC2 key pair, saves PEM to file
**New Approach:**
- Still create key pair via AWS CLI (or check if exists)
- Instead of `jq --raw-output ".KeyMaterial" > file`, capture KeyMaterial
- Store directly: `aws secretsmanager create-secret --name "/{NETWORK}/{FLEET}/aws/keypair" --secret-string "$KEY_MATERIAL"`
- **Critical:** You only get KeyMaterial once during creation - must capture and store immediately

#### B. Config File Management (lines 214-232)
**Current:** Copy template, modify with jq-replace, save to file
**New Approach:**
- Retrieve template from Secrets Manager
- Perform jq operations in memory
- Store result: `aws secretsmanager create-secret --name "/{NETWORK}/{FLEET}/config" --secret-string "$CONFIG_JSON"`

#### C. Firebase & Other Key Files (lines 234-250)
**Current:** Copy from network `.keys` folder
**New Approach:**
- **One-time migration:** Upload existing keys to Secrets Manager at network level
- Function retrieves from `/{NETWORK}/shared/*` paths
- If fleet needs copies, create fleet-specific secrets pointing to same values, OR just reference network-level secrets

#### D. DigitalOcean Keys (lines 251-274)
**Current:** Reads PAT from file, generates SSH keys, saves locally
**New Approach:**
- PAT token should already be in Secrets Manager (pre-populated)
- Retrieve PAT: `aws secretsmanager get-secret-value --secret-id "/{NETWORK}/{FLEET}/digitalocean/pat"`
- Generate SSH key in memory (use `ssh-keygen` with stdout redirection or scripting)
- Upload both private/public keys to Secrets Manager
- Upload public key to DigitalOcean API (same as current)

---

### Phase 5: Terraform Integration

**Critical Change:** Terraform needs to fetch secrets instead of reading local files.

**Current Terraform Approach:**
```hcl
# Reads local file
private_key = file("${path.module}/../../keys/key.pem")
```

**New Approach - Option 1 (External Data Source):**
```hcl
data "aws_secretsmanager_secret_version" "ec2_key" {
  secret_id = "/${var.network_name}/${var.fleet_name}/aws/keypair"
}

resource "aws_instance" "example" {
  key_name = aws_key_pair.fleet_key.key_name
  # Use data.aws_secretsmanager_secret_version.ec2_key.secret_string
}
```

**New Approach - Option 2 (Pass from Bash):**
- Fetch secrets in bash script
- Pass as terraform variables: `terraform apply -var="ec2_private_key=$KEY_MATERIAL"`
- ⚠️ **Security risk:** Secrets visible in process list and terraform state

**Recommended: Option 1** - Let Terraform fetch secrets directly using data sources.

---

### Phase 6: Downstream Script Changes

**Scripts that currently read from `$NETWORK_FLEET_DIR` will need updates:**

1. **ServerEnvironmentBuilder (lines 277-284):**
   - Currently reads `--fleet-config-file` and `--allowed-apps-file` from filesystem
   - **Change:** Modify `app_v6.js` to accept secret IDs and fetch from Secrets Manager
   - OR fetch secrets in bash and write to temp files before calling node script

2. **Deployment/Bootstrap Scripts:**
   - Any scripts using `$NETWORK_FLEET_DIR/keys/*` or `$NETWORK_FLEET_DIR/config/*`
   - Need AWS SDK/CLI integration to fetch secrets
   - Consider using **EC2 instance IAM roles** so instances can fetch their own secrets at runtime

---

### Phase 7: Migration Execution Plan

**Step 1: Pre-Migration (No Breaking Changes)**
1. Create AWS Secrets Manager structure/naming convention
2. Upload existing keys from filesystem to Secrets Manager (one-time bulk upload)
3. Verify all secrets are accessible

**Step 2: Parallel Operation**
1. Modify `init_server_fleet_config_files` to:
   - Continue writing to filesystem (backwards compatibility)
   - **ALSO** write to AWS Secrets Manager
2. Test thoroughly with new fleet creation
3. Verify secrets are correctly stored in both locations

**Step 3: Update Consumers**
1. Modify Terraform scripts to fetch from Secrets Manager
2. Update ServerEnvironmentBuilder to fetch from Secrets Manager
3. Update any deployment/bootstrap scripts
4. Test end-to-end with a test fleet

**Step 4: Remove Filesystem Operations**
1. Remove directory creation logic
2. Remove file write operations
3. Remove file copy operations
4. Keep only Secrets Manager writes

**Step 5: Cleanup**
1. Secure/archive old filesystem-based keys
2. Document new secret management procedures
3. Update runbooks and documentation

---

### Phase 8: Security Considerations

**IAM Permissions Required:**
- Jenkins role needs: `secretsmanager:CreateSecret`, `secretsmanager:PutSecretValue`, `secretsmanager:GetSecretValue`
- EC2 instances need: `secretsmanager:GetSecretValue` (scoped to their fleet path)
- Terraform execution role needs: `secretsmanager:GetSecretValue`

**Best Practices:**
1. **Least Privilege:** Scope permissions by path: `/{NETWORK_NAME}/${FLEET_NAME}/*`
2. **Encryption:** Use KMS keys for secret encryption (per-network or per-fleet keys)
3. **Rotation:** Implement secret rotation for long-lived credentials
4. **Versioning:** Leverage Secrets Manager versioning for rollback capability
5. **Audit:** Enable CloudTrail logging for secret access

**Secrets That Should Never Be in Secrets Manager:**
- EC2 key pairs are **registered with AWS** - the secret is just the PEM material
- DigitalOcean SSH public keys are **uploaded to DO** - only private key is sensitive

---

### Phase 9: Potential Gotchas

1. **EC2 Key Pair Creation:** You can only retrieve KeyMaterial once during creation. Must be captured immediately.

2. **File Format vs. Secret String:** Some secrets are multi-line (PEM files). Store as string with `\n` or use base64 encoding.

3. **DigitalOcean PAT:** Currently read from `$NETWORK_FLEET_DIR/digitalocean/$NETWORK_NAME-$FLEET_NAME.pat` (line 252). This path suggests it's fleet-specific, but might be pre-populated. Clarify if this is generated or pre-existing.

4. **jq-replace Function:** Custom function not shown. Need to ensure it works with in-memory JSON strings, not just files.

5. **Terraform State:** Secrets fetched by Terraform will appear in state files. Use backend encryption and restricted access.

6. **Cost:** Secrets Manager charges $0.40/secret/month + API calls. Budget accordingly for potentially hundreds of secrets.

---

### Recommended Quick Win Alternative: AWS Systems Manager Parameter Store

If cost is a concern, consider **SSM Parameter Store** instead:
- **Free tier:** 10,000 parameters
- Standard parameters: Free
- Advanced parameters (>4KB, policies, longer history): $0.05/parameter/month
- Same IAM integration as Secrets Manager
- No built-in rotation (but you can script it)

API is nearly identical:
```bash
# Store
aws ssm put-parameter --name "/{NETWORK}/{FLEET}/key" --value "$SECRET" --type SecureString

# Retrieve
aws ssm get-parameter --name "/{NETWORK}/{FLEET}/key" --with-decryption
```

---

## Implementation Timeline

**High-level changes:**
1. Replace all `mkdir`, `cp`, `>` file writes with `aws secretsmanager create-secret` or `put-secret-value`
2. Modify downstream consumers (Terraform, Node.js scripts) to fetch from Secrets Manager
3. Update IAM permissions for Jenkins + EC2 instances + Terraform
4. Consider SSM Parameter Store for cost savings
5. Migrate in phases: parallel operation → consumer updates → remove filesystem ops

**Timeline estimate:**
- Phase 1-2 (Design): 1-2 days
- Phase 3-4 (Function changes): 3-5 days
- Phase 5-6 (Terraform & consumers): 5-7 days
- Phase 7 (Migration): 2-3 days
- **Total: ~3 weeks** with testing

---

## Code Examples

### Example: Storing EC2 Key Pair in Secrets Manager

**Before:**
```bash
aws --region $DEFAULT_AWS_REGION ec2 create-key-pair --key-name $KEY_NAME > "$TMP_OUTPUT_FILE"
jq --raw-output ".KeyMaterial" "$TMP_OUTPUT_FILE" > "$NETWORK_FLEET_DIR/aws/$KEY_NAME.pem"
rm "$TMP_OUTPUT_FILE"
chmod 400 "$NETWORK_FLEET_DIR/aws/$KEY_NAME.pem"
```

**After:**
```bash
KEY_MATERIAL=$(aws --region $DEFAULT_AWS_REGION ec2 create-key-pair --key-name $KEY_NAME | jq --raw-output ".KeyMaterial")
aws secretsmanager create-secret \
    --name "/${NETWORK_NAME}/${FLEET_NAME}/aws/keypair" \
    --description "EC2 key pair for $NETWORK_NAME-$FLEET_NAME" \
    --secret-string "$KEY_MATERIAL" \
    --region $DEFAULT_AWS_REGION
```

### Example: Retrieving Secret in Terraform

```hcl
data "aws_secretsmanager_secret_version" "ec2_keypair" {
  secret_id = "/${var.network_name}/${var.fleet_name}/aws/keypair"
}

data "aws_secretsmanager_secret_version" "firebase_admin" {
  secret_id = "/${var.network_name}/shared/firebase/admin"
}

# Use in resources
locals {
  ec2_private_key = data.aws_secretsmanager_secret_version.ec2_keypair.secret_string
  firebase_config = jsondecode(data.aws_secretsmanager_secret_version.firebase_admin.secret_string)
}
```

### Example: Retrieving Secret in Node.js (ServerEnvironmentBuilder)

```javascript
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

async function getSecret(secretId) {
  const client = new SecretsManagerClient({ region: process.env.AWS_REGION || 'us-west-2' });
  const command = new GetSecretValueCommand({ SecretId: secretId });
  const response = await client.send(command);
  return response.SecretString;
}

// Usage
const configJson = await getSecret(`/${networkName}/${fleetName}/config`);
const allowedAppsJson = await getSecret(`/${networkName}/${fleetName}/allowed-apps`);
```

---

## Questions to Answer Before Implementation

1. **Shared vs. Fleet-Specific Secrets:** Should Firebase credentials and other keys be truly shared at the network level, or duplicated per-fleet?

2. **Cost Analysis:** How many fleets exist? Calculate total secret count to estimate monthly Secrets Manager costs vs. SSM Parameter Store.

3. **Backwards Compatibility:** Do existing fleets need to continue working with filesystem-based secrets during migration?

4. **Secret Rotation:** Which secrets should have automatic rotation enabled (e.g., database passwords, API tokens)?

5. **DigitalOcean PAT Source:** Is the PAT pre-existing or generated? Where should it be initially stored?

6. **Terraform State Backend:** Is the Terraform state backend encrypted? This is critical since secrets will be in state files.

---

## Success Criteria

- [ ] All secrets stored in AWS Secrets Manager with proper naming convention
- [ ] Zero secrets stored on Jenkins filesystem
- [ ] Terraform successfully fetches all secrets from Secrets Manager
- [ ] ServerEnvironmentBuilder successfully fetches all secrets from Secrets Manager
- [ ] EC2 instances can fetch their own secrets using IAM roles
- [ ] Test fleet deployed end-to-end using new secret management
- [ ] Production fleet deployed successfully with zero downtime
- [ ] Old filesystem-based secrets archived securely
- [ ] Documentation updated for new procedures
- [ ] IAM policies implemented with least privilege access

---

## Rollback Plan

If migration encounters critical issues:

1. **Phase 2 (Parallel Operation):** Continue using filesystem, ignore Secrets Manager writes
2. **Phase 3+ (Consumers Updated):** Revert code changes to consumers, redeploy from git history
3. **Emergency:** Keep filesystem-based secrets available in secure backup for 90 days post-migration

---

## References

- AWS Secrets Manager Documentation: https://docs.aws.amazon.com/secretsmanager/
- AWS Systems Manager Parameter Store: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
- Terraform AWS Provider - Secrets Manager: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret_version
- Source Function: `C:\Bigscreen\devops\Jenkins\server_builder_v8.sh:169`