# Arda App Update Roadmap

This document outlines planned features and improvements for the Arda admin application. Each section describes the feature in plain language, followed by specific file changes required for implementation.

---

## Table of Contents

1. [Batch Shipment Page](#1-batch-shipment-page)
2. [Immediate Bugs](#2-immediate-bugs)
3. [DHL Charges Tracking](#3-dhl-charges-tracking)
4. [Lost Shipments](#4-lost-shipments)
5. [Shopify to BigOrder Grouping](#5-shopify-to-bigorder-grouping)
6. [Shipping UI Changes](#6-shipping-ui-changes)
7. [Analytics](#7-analytics)
8. [International Shipping](#8-international-shipping)
9. [Shipment Summary Page](#9-shipment-summary-page)
10. [Better Selection of Shipment Prices](#10-better-selection-of-shipment-prices)
11. [QC Changes](#11-qc-changes)
12. [Factory QA Tools for BigOrders](#12-factory-qa-tools-for-bigorders)
13. [Customs and Shipping Data Refactoring](#13-customs-and-shipping-data-refactoring)

---

## 1. Batch Shipment Page

**Status: COMPLETED AND SHIPPED**

### What It Does

The Batch Shipment page allows warehouse staff to process multiple orders at once instead of handling them one by one. Users can search for orders ready to ship, select multiple orders using checkboxes, and then have the system automatically process each one in sequence. For each order, the system creates the shipment, picks inventory items, generates the shipping label, and prints it automatically.

### Current Implementation

The feature is fully implemented in **BigBatchShipper.jsx** with two processing modes:
- **Labels Only**: Generates and prints shipping labels without marking orders as fulfilled in Shopify
- **Generate All**: Complete workflow including Shopify fulfillment

### Key Files

| File | Purpose |
|------|---------|
| `webapps/src/components/BigLogistics/BigBatchShipper.jsx` | Main batch processing UI with order selection and sequential processing |
| `webapps/src/components/BigLogistics/BigShipperNextOrder.jsx` | Order query functionality reused for finding shippable orders |
| `api/src/fabricator/ShippingAdminApi.ts` | Backend API handling shipment creation, picking, and label generation |

### Processing Flow

1. Query orders via `/api/admin/shipper/next3`
2. Display orders with checkboxes and "select all" functionality
3. For each selected order (processed in series):
   - Create shipment via `POST /api/admin/shipper/shipment`
   - Auto-pick inventory items via `PickInventoryItem` action
   - Generate label via `GenerateShippingLabel` action
   - Print label using hidden iframe with PDF
   - Complete packing via `StartPacking` and `FinishPacking` actions
   - (Optional) Mark as shipped via `ShipIt` action

---

## 2. Immediate Bugs

### What It Does

Address immediate usability issues, specifically around audio feedback in the application.

### Feature Description

The app uses audio cues (success sounds, error sounds) when scanning barcodes or completing actions. These sounds need to be updated with new audio files from Jeff, and users should be able to toggle sounds on/off based on their preference.

### Files to Change

| File | Change Required |
|------|-----------------|
| `webapps/src/components/BigLogistics/BigShipmentFulfillmentQueue.jsx` | Add sound toggle control; currently plays audio on scan success/failure |
| `webapps/src/components/BigLogistics/BigShipment.jsx` | Add sound toggle if scanner audio is used here |
| `webapps/public/audio/` | Add new audio files from Jeff (success.mp3, error.mp3, etc.) |
| `webapps/src/components/common/UserPreferences.jsx` | Create new component for storing sound preference in localStorage |

### Implementation Notes

- Store sound preference in browser localStorage
- Add a toggle switch in the shipper navigation or settings area
- Replace existing audio file references with new files from Jeff
- Ensure sound toggle state persists across page refreshes

---

## 3. DHL Charges Tracking

**Status: PARTIALLY IMPLEMENTED (UI Only)**

### What It Does

Create a system to track and validate DHL shipping charges by cross-referencing invoice PDFs against expected shipping costs. This helps identify incorrect or inflated charges ("bullshit charges") that DHL may bill incorrectly.

### Feature Description

DHL sends periodic invoices that may contain charges that don't match what was quoted at label creation time. This feature will:
- Parse DHL invoice PDFs to extract individual charges
- Match each charge against the corresponding shipment in our system
- Flag discrepancies where the invoiced amount differs significantly from the quoted rate
- Display a prioritized list of problematic charges sorted by date
- Pull new invoice data from DHL automatically (daily)

### Current State

The system currently stores shipping rates in `currentShippingLabel.rate` on each BigShipment record, including:
- `amount` - The quoted cost at label creation
- `provider` - Carrier name (DHL, UPS, etc.)
- `servicelevel_name` - Service type

However, there is no invoice tracking or reconciliation system.

### Current Implementation

An experimental UI page has been created at `/experimental/dhl-charges` with the following features:
- Summary statistics (new flags count, total discrepancy, unmatched charges)
- Invoice list sidebar with filtering
- Flagged charges table showing tracking numbers, invoiced vs expected amounts, discrepancy percentages
- Status management (new/reviewed/disputed/resolved) with notes
- Shipment lookup modal that queries the real `/api/admin/shipper/shipments` API by tracking number
- Displays quoted rate from `currentShippingLabel.rate` for comparison

**Implemented Files:**
- `webapps/src/components/Experimental/DHLChargesPage.jsx` - Main UI page (uses mock data for invoices/charges)
- `webapps/src/components/Experimental/ExperimentalWrapper.jsx` - Navigation link added
- `webapps/src/components/Experimental/ExperimentalHome.jsx` - Card added to experimental home
- `webapps/arda/app/App.jsx` - Route added at `/experimental/dhl-charges`

**Still Needed:**
- Backend API endpoints for invoice management
- Database tables for invoices and charge flags
- PDF parsing functionality
- Automatic reconciliation logic

### Files to Create

| File | Purpose |
|------|---------|
| `api/src/fabricator/DHLInvoiceApi.ts` | New API for DHL invoice management and parsing |
| `api/src/fabricator/DHLInvoiceSchemas.ts` | Schema definitions for invoice records and charge items |
| ~~`webapps/src/components/BigLogistics/DHLChargesPage.jsx`~~ | ~~Main UI for viewing and managing flagged charges~~ (Created as experimental page)
| `webapps/src/components/BigLogistics/DHLChargesList.jsx` | Table component displaying charge discrepancies |

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/FabricatorDatabase.ts` | Add methods for storing/querying DHL invoice records |
| `api/src/fabricator/FabricatorSchemas.ts` | Add `BigDHLInvoice` and `BigDHLCharge` schema types |
| `apps/admin_api/admin_api.ts` | Register new DHL invoice API routes |
| `webapps/src/components/BigLogistics/BigShipperWrapper.jsx` | Add navigation link to DHL Charges page |
| `apps/admin_api/db_setup.ts` | Add database table for `big_dhl_invoices` |

### New Database Table

```sql
CREATE TABLE big_dhl_invoices (
    id TEXT PRIMARY KEY,
    invoiceNumber TEXT NOT NULL,
    invoiceDate BIGINT NOT NULL,
    totalAmount NUMERIC,
    currency TEXT,
    pdfUrl TEXT,
    charges JSONB,  -- Array of individual charge line items
    reconciliationStatus TEXT,  -- 'pending', 'reviewed', 'disputed'
    createdAt BIGINT,
    updatedAt BIGINT
);

CREATE TABLE big_dhl_charge_flags (
    id TEXT PRIMARY KEY,
    invoiceId TEXT REFERENCES big_dhl_invoices(id),
    bigShipmentId TEXT,
    trackingNumber TEXT,
    invoicedAmount NUMERIC,
    expectedAmount NUMERIC,
    discrepancyAmount NUMERIC,
    discrepancyPercent NUMERIC,
    flagReason TEXT,
    status TEXT,  -- 'new', 'reviewed', 'disputed', 'resolved'
    notes TEXT,
    createdAt BIGINT
);
```

### API Endpoints to Add

| Method | Endpoint | Purpose |
|--------|----------|---------|
| POST | `/api/admin/dhl/invoices/upload` | Upload and parse DHL invoice PDF |
| GET | `/api/admin/dhl/invoices` | List all invoices with pagination |
| GET | `/api/admin/dhl/invoices/:id` | Get specific invoice details |
| GET | `/api/admin/dhl/charges/flagged` | Get all flagged charge discrepancies |
| PUT | `/api/admin/dhl/charges/:id` | Update charge status (mark as reviewed, disputed) |
| POST | `/api/admin/dhl/reconcile` | Run reconciliation for an invoice against shipments |

---

## 4. Lost Shipments

### What It Does

Streamline the process of handling lost-in-transit shipments, including customer service ticket integration and document submission for carrier claims.

### Feature Description

When a customer reports a package as lost, customer service creates a ticket. This feature will:
- Link CS tickets to shipment records in Arda
- Track the claim status through the carrier dispute process
- Store submitted documents (proof of shipment, value declarations, etc.)
- Automate parts of the claim filing process where possible
- Provide visibility into all open lost shipment cases

### Current State

The system tracks shipment status including `DeliveryFailed` but has no:
- Customer service ticket integration
- Document upload capability
- Claim filing workflow
- Lost shipment case management

### Files to Create

| File | Purpose |
|------|---------|
| `api/src/fabricator/LostShipmentApi.ts` | API for lost shipment case management |
| `api/src/fabricator/LostShipmentSchemas.ts` | Schema for lost shipment cases and documents |
| `webapps/src/components/BigLogistics/LostShipmentsPage.jsx` | Main page listing all lost shipment cases |
| `webapps/src/components/BigLogistics/LostShipmentCase.jsx` | Individual case detail view with documents |
| `webapps/src/components/BigLogistics/LostShipmentDocuments.jsx` | Document upload and management component |

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/FabricatorDatabase.ts` | Add methods for lost shipment case CRUD |
| `api/src/fabricator/FabricatorSchemas.ts` | Add `BigLostShipmentCase` schema type |
| `api/src/fabricator/ShippingAdminApi.ts` | Add action to mark shipment as lost and create case |
| `apps/admin_api/admin_api.ts` | Register lost shipment API routes |
| `apps/admin_api/db_setup.ts` | Add database table for lost shipment cases |
| `webapps/src/components/BigLogistics/BigShipperWrapper.jsx` | Add navigation link to Lost Shipments |
| `webapps/src/components/BigLogistics/BigShipment.jsx` | Add button to report shipment as lost |

### New Database Table

```sql
CREATE TABLE big_lost_shipment_cases (
    id TEXT PRIMARY KEY,
    bigShipmentId TEXT NOT NULL,
    trackingNumber TEXT,
    carrier TEXT,
    customerEmail TEXT,
    csTicketId TEXT,  -- External CS system ticket ID
    csTicketUrl TEXT,
    claimStatus TEXT,  -- 'reported', 'claim_filed', 'pending_carrier', 'approved', 'denied', 'resolved'
    claimAmount NUMERIC,
    claimCurrency TEXT,
    carrierClaimId TEXT,
    documents JSONB,  -- Array of uploaded document references
    notes TEXT,
    reportedAt BIGINT,
    claimFiledAt BIGINT,
    resolvedAt BIGINT,
    createdAt BIGINT,
    updatedAt BIGINT
);
```

### Implementation Notes

- Consider integration with Zendesk, Freshdesk, or other CS platforms via API
- Documents should be uploaded to S3 with signed URLs for secure access
- Carrier claim filing may need carrier-specific API integrations (DHL, UPS claim APIs)

---

## 5. Shopify to BigOrder Grouping

### What It Does

Group all items across active Shopify orders that belong to the same customer or shipping destination, making it easier to see the complete picture of what a customer has ordered.

### Feature Description

Currently, each Shopify order creates one BigOrder, and there's no easy way to see all orders from the same customer. This feature will:
- Automatically link BigOrders that share the same Shopify customer
- Display grouped orders on the order detail page
- Allow staff to see if a customer has multiple pending orders
- Help coordinate shipments that could potentially be combined

### Current State

BigOrders are linked to Shopify via:
- `shopifyOrderId` - The Shopify order ID
- `shopifyCustomerId` - The Shopify customer ID
- `shopifyOrderName` - Display name (e.g., "#BS12345")

The `shopifyCustomerId` field enables grouping but is not currently utilized for this purpose.

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/FabricatorAdminApi.ts` | Add endpoint to get related orders by customer ID |
| `api/src/fabricator/FabricatorDatabase.ts` | Add query method `getBigOrdersByShopifyCustomerId` |
| `webapps/src/components/BigOrders/BigOrderComponent.jsx` | Add "Related Orders" section showing other orders from same customer |
| `webapps/src/components/BigOrders/BigOrderTableCells.jsx` | Add indicator if customer has multiple orders |
| `webapps/src/components/BigOrders/BigOrdersList.jsx` | Add filter/grouping option by customer |

### API Endpoints to Add

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/admin/shop/orders/by-customer/:customerId` | Get all BigOrders for a Shopify customer |
| GET | `/api/admin/shop/orders/:id/related` | Get related orders for a specific BigOrder |

### UI Changes

On the BigOrder detail page, add a new section:

```
Related Orders from This Customer
---------------------------------
| Order      | Date       | Status              | Items           |
|------------|------------|---------------------|-----------------|
| #BS12340   | 2024-01-10 | Shipped             | Beyond 2        |
| #BS12345   | 2024-01-15 | ReadyToShip         | Custom Cushion  |
| #BS12350   | 2024-01-18 | WaitingForInventory | Prescription Lens|
```

---

## 6. Shipping UI Changes

### What It Does

Add a direct link from the BigOrder page to the corresponding Shopify order page, making it easier to access Shopify-specific information.

### Feature Description

When viewing a BigOrder, staff often need to access the original Shopify order for customer communication, refunds, or order modifications. Currently, they must manually navigate to Shopify. This change adds a clickable link directly to the Shopify admin page for the order.

### Current State

The BigOrder page displays the `shopifyOrderName` (e.g., "#BS12345") but does not link to Shopify's admin interface.

### Files to Modify

| File | Change Required |
|------|-----------------|
| `webapps/src/components/BigOrders/BigOrderComponent.jsx` | Add Shopify admin link next to order number |
| `webapps/src/components/BigOrders/BigOrderTableCells.jsx` | Add Shopify link icon in order tables |
| `api/src/fabricator/FabricatorSchemas.ts` | Ensure `shopifyOrderId` is always available for URL construction |

### Implementation Details

Shopify admin order URLs follow the pattern:
```
https://{shop-name}.myshopify.com/admin/orders/{shopify_order_id}
```

Add a button/link that opens in a new tab:
```jsx
<Button
    as="a"
    href={`https://bigscreenvr.myshopify.com/admin/orders/${bigOrder.shopifyOrderId}`}
    target="_blank"
    icon="external"
    content="View in Shopify"
/>
```

---

## 7. Analytics

### What It Does

Integrate Darshan's existing analytics dashboard into Arda and add a new page showing reasons why orders have not shipped yet.

### Feature Description

This feature has two parts:

**Part A: Integrate Darshan's Dashboard**
Embed or recreate the analytics dashboard that Darshan has built, making it accessible directly within Arda instead of a separate tool.

**Part B: Unshipped Orders Analysis**
Create a new report showing why orders are stuck and haven't shipped. Based on Max's spreadsheet, categorize orders by their blocking reason:
- Waiting for face scan
- Waiting for manufacturing
- Waiting for inventory (out of stock)
- On shipping hold
- Address issues
- Payment issues

### Current State

Existing analytics in Arda:
- `/analytics` - Basic reports for jobs, orders, IPD breakdown
- `/shipper/stats` - Shipment statistics with charts (recently enhanced with stacked charts and moving averages)
- `/shop/orders` with report views - Order filtering and querying
- `/inventory/report` - Inventory levels by product type

### Files to Create

| File | Purpose |
|------|---------|
| `webapps/src/components/Analytics/DarshanDashboard.jsx` | Embedded or recreated version of Darshan's dashboard |
| `webapps/src/components/Analytics/UnshippedOrdersReport.jsx` | New page showing why orders haven't shipped |
| `webapps/src/components/Analytics/UnshippedOrdersChart.jsx` | Visualization of order blockers by category |

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/FabricatorAdminApi.ts` | Add endpoint for unshipped orders analysis |
| `api/src/fabricator/FabricatorDatabase.ts` | Add query to categorize unshipped orders by blocker reason |
| `webapps/src/components/Analytics/AnalyticsWrapper.jsx` | Add navigation links to new pages |
| `webapps/app/App.jsx` | Add routes for new analytics pages |

### API Endpoints to Add

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/admin/analytics/unshipped-reasons` | Get counts of orders by blocking reason |
| GET | `/api/admin/analytics/unshipped-orders` | Get list of unshipped orders with reasons |

### Unshipped Orders Categories

Query orders and categorize by state:
- `WaitingForScanRequest` + no scan submitted = "Waiting for face scan"
- `WaitingForManufacturing` = "In manufacturing"
- `WaitingForInventory` = "Out of stock"
- `ReadyToShip` + `shipmentHold` set = "On hold"
- `ShopifyProblem` = "Address/payment issues"
- `Error` = "System errors"

---

## 8. International Shipping

### What It Does

Improve international shipping capabilities, particularly around VAT handling for EU shipments where product combinations may cause issues.

### Feature Description

There is a known VAT problem where certain combinations of products in a single shipment may break the order due to customs/VAT calculation issues. Additionally, decisions about warehouse space (possibly EU-based fulfillment) will affect how international orders are processed.

### Current State

The system has extensive international shipping support:
- VAT rates defined for 29 EU countries in `ShippingAdminApi.ts`
- Customs declarations with tariff numbers per product
- Multiple distribution centers (LA, Australia, Japan, Netherlands)
- DHL Express integration for international shipments

### Known Issues

1. **VAT Combination Problem**: When an order contains a mix of certain product types, the VAT calculation may fail or produce incorrect values
2. **Warehouse Space Decision**: Pending decision about EU warehouse may change international fulfillment flow

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/ShippingAdminApi.ts` | Fix VAT calculation for mixed product orders (lines 2289-2320) |
| `api/src/fabricator/FabricatorSchemas.ts` | Add new distribution center if EU warehouse is approved |
| `api/src/fabricator/ShippingAdminApi.ts` | Update `createCustomsDeclaration` to handle product combinations |

### Implementation Notes

The `createCustomsDeclaration` function needs to:
1. Detect problematic product combinations
2. Either split into separate customs declarations or calculate combined VAT correctly
3. Consider the customs value overrides already in place (`customsValueOverrides`)

Current VAT calculation location: `ShippingAdminApi.ts` lines 2289-2320 (`EuroCountryTaxRates`)

### Pending Decisions

- EU warehouse location and setup timeline
- Whether to route EU orders through Netherlands Flexport facility
- Carrier selection for EU-to-EU shipments (currently using DHL Express)

---

## 9. Shipment Summary Page

### What It Does

Create a page that shows the daily shipment summary that Max currently posts to Discord manually. This summary breaks down shipments by product type and shows what was shipped each day.

### Feature Description

At the end of each day, a summary of shipments is shared on Discord. This feature brings that summary into Arda where it can be:
- Generated automatically
- Viewed for any date or date range
- Broken down by product type (Beyond headsets, cushions, lenses, etc.)
- Compared across days to spot trends

### Current State

The backend API already exists and is used by the stats page:
- `/shipper/stats` shows shipment counts by status and product type with charts
- The API endpoint `/api/admin/shipper/shipments/stats` returns statistics with time resolution (hourly, daily, weekly, monthly)
- `BigShipmentsStats.jsx` has visualization with stacked charts and moving averages
- The API supports grouping by status or product type via the `groupBy` parameter

**No new API is needed** - the existing `/api/admin/shipper/shipments/stats` endpoint provides all the data required. The new page just needs to present this data in a Discord-friendly text format.

### Files to Create

| File | Purpose |
|------|---------|
| `webapps/src/components/BigLogistics/DailyShipmentSummary.jsx` | New page with Discord-style daily summary |

### Files to Modify

| File | Change Required |
|------|-----------------|
| `webapps/src/components/BigLogistics/BigShipperWrapper.jsx` | Add navigation link to Daily Summary |
| `webapps/app/App.jsx` | Add route for `/shipper/summary` |

### Existing API to Use

The page should call the existing endpoint with appropriate parameters:

```
GET /api/admin/shipper/shipments/stats
    ?minCreatedAt={startOfDay}
    &maxCreatedAt={endOfDay}
    &resolution=day
    &groupBy=productType
```

This returns data in the format:
```javascript
{
    items: [{ period: "2025-01-14", productType: "Beyond2", count: 23 }, ...],
    resolution: "day",
    totals: { total: 47, byProductType: { Beyond2: 23, Beyond2Eye: 12, ... } }
}
```

### UI Implementation

The new component should:
1. Use a date picker to select the summary date (default: today)
2. Call the existing stats API with `groupBy=productType`
3. Format the response into a text summary
4. Provide a "Copy to Clipboard" button for easy Discord posting

### Summary Format

```
Daily Shipment Summary - January 14, 2025
=========================================

Total Shipments: 47

By Product Type:
  Beyond 2 Headsets:     23
  Beyond 2 Eye Tracking: 12
  Custom Cushions:        8
  Prescription Lenses:    3
  Audio Straps:           1
```

### Optional Enhancements

- Add carrier breakdown (may require additional API parameter or separate call)
- Add country breakdown (may require additional API parameter or separate call)
- Add "Compare to yesterday" or "Compare to last week" views
- Add export to CSV/PDF options

---

## 10. Better Selection of Shipment Prices

### What It Does

Reinstate the multi-carrier rate comparison functionality that automatically selects the cheapest shipping option across all available carriers.

### Feature Description

The system previously had (or was designed to have) functionality that:
1. Queries shipping rates from multiple carriers (UPS, DHL, USPS, Sendle)
2. Compares all available rates
3. Automatically selects the cheapest option that meets delivery requirements
4. Allows manual override if needed

This feature needs to be reinstated or completed to reduce shipping costs.

### Current State

The infrastructure exists:
- `createShipmentRateEstimate` in `ShippingAdminApi.ts` queries Shippo for rates from multiple carriers
- Rates are returned sorted by price (lowest first)
- The UI in `BigShipment.jsx` can display and select from available rates
- Carrier accounts are configured for UPS, DHL, USPS, and Sendle

However, the automatic selection of the cheapest rate may not be working, or the rate display/selection flow may be disabled.

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/ShippingAdminApi.ts` | Review and fix `createShipmentRateEstimate` to return all carrier rates |
| `api/src/fabricator/ShippingAdminApi.ts` | Add or enable auto-selection of cheapest rate in `generateShippingLabelInternal` |
| `webapps/src/components/BigLogistics/BigShipment.jsx` | Ensure rate comparison UI is visible and functional |
| `webapps/src/components/BigLogistics/BigBatchShipper.jsx` | Add option to use cheapest rate in batch processing |

### Implementation Notes

Key code locations:
- Rate calculation: `ShippingAdminApi.ts` lines 3695-3830 (`createShipmentRateEstimate`)
- Carrier account IDs:
  - UPS Main: `5162778754904d31a0a61c919ba0b66a`
  - UPS EU: `278f870edc4d4e66b4abe90361282607`
  - DHL Export: `bd29c54a1c3049aaa9b3b28d1661ab38`
  - DHL Import: `1b92670b789846ff9f1dc712b7e54d30`

The `CalculateShippingRates` action should:
1. Query all active carrier accounts
2. Filter out incompatible service levels
3. Sort by price
4. Return full list for manual selection OR auto-select cheapest

---

## 11. QC Changes

### What It Does

Evaluate bringing the Quality Control (QC) system into Arda for better integration with the rest of the order and manufacturing workflow.

### Feature Description

Currently, QC processes exist within the fabricator/manufacturing flow but may benefit from deeper integration with Arda. This is an exploratory item to:
- Review current QC workflow
- Identify pain points
- Design improved QC tracking within Arda
- Consider QC dashboards and metrics

### Current State

QC is partially implemented in the job system:

**Job States for QC:**
- `AwaitingQA` (state 17)
- `QAInProgress` (state 18)
- `QAFailed` (state 19)
- `AwaitingManufacturingQC` (state 103)
- `ManufacturingQCFailed` (state 104)

**Job Actions for QC:**
- `TryQA`, `QASuccess`, `QAFailed`
- `ManufacturingQCFailed`, `ManufacturingQCSuccess`

**Inventory Status:**
- `RequiresQA` - Item needs QC steps
- `Damaged` - Item flagged as damaged during QC

### Questions to Answer

1. Should QC have its own dedicated section in Arda?
2. Are there QC metrics that should be tracked (pass rate, failure reasons)?
3. Should QC failures automatically create support cases?
4. Is there a need for QC photo documentation?

### Potential Files to Create

| File | Purpose |
|------|---------|
| `webapps/src/components/QC/QCDashboard.jsx` | Overview of QC status and metrics |
| `webapps/src/components/QC/QCQueue.jsx` | Items awaiting QC review |
| `webapps/src/components/QC/QCItem.jsx` | Individual item QC workflow |

### Potential Files to Modify

| File | Change Required |
|------|-----------------|
| `webapps/app/ArdaWrapper.jsx` | Add QC section to main navigation |
| `api/src/fabricator/FabricatorSchemas.ts` | Add QC-specific schemas if needed |
| `api/src/fabricator/InventoryAdminApi.ts` | Add QC-related endpoints |

### Implementation Notes

This item requires further discussion with the team to understand:
- Current QC workflow pain points
- What data needs to be captured during QC
- Integration with manufacturing jobs vs. inventory items
- Whether QC should tie into the fabricator or be standalone

---

## 12. Factory QA Tools for BigOrders

**Status: PARTIALLY IMPLEMENTED (UI with Basic Workflow)**

### What It Does

Create a new page in Arda that allows staff to run a full end-to-end test of the order workflow using production systems but with test-flagged data. This tool helps verify that the entire order-to-shipment pipeline is working correctly without affecting real customer orders.

### Feature Description

The Factory QA Tools page will:
- Allow users to create test orders with configurable initial conditions (product types, IPD values, priority, etc.)
- Automatically progress the test order through each stage of the workflow
- At each step, validate that the order state matches expected assumptions
- Run checks against production data and services (Shopify, Shippo, inventory)
- Display real-time progress and any failures or warnings
- Provide a comprehensive test report at the end

This differs from the existing integration tests because:
- Integration tests run against test fixtures and mocked data
- This tool runs against **live production systems** with test-flagged orders
- It provides a UI for non-developers to run verification checks
- It can catch issues that only appear with real production data

### Current State

The infrastructure for test orders already exists:

**Test Order Creation:**
- Endpoint: `POST /api/admin/shop/test_order`
- Creates orders with `origin: TestData` to distinguish from real orders
- Test orders follow the exact same state machine as production orders

**Order State Machine:**
```
WaitingForScanRequest
    ↓
WaitingForManufacturing
    ↓
WaitingForInventory
    ↓
ReadyToShip
    ↓
Shipped
```

**Job State Machine (for custom cushions):**
```
AwaitingInit → AwaitingScanVerification → AwaitingToolPathCreation
    → AwaitingManufacture → AwaitingNFC → AwaitingQA → AwaitingFulfilment → JobComplete
```

**Shipment State Machine:**
```
Init → Picking → PrepareShippingLabel → ReadyToPack
    → Packing → PreparingToShip → WaitingForPickup → Shipped
```

**Existing Test Utilities:**
- `tests/fabricator/004.AddTestOrders.ts` - Creates test orders
- `tests/fabricator/005.BigOrderFlow.ts` - Full workflow test (220+ lines)
- `tests/utils/TestUtils.Fabricator.ts` - Helper functions

### Current Implementation

An experimental UI page has been created at `/experimental/qa-tools` with the following features:
- Test configuration form (product type, customer email, IPD, priority, country)
- Test options checkboxes (create scan request, simulate manufacturing, auto-pick inventory, generate label, Shopify fulfillment)
- Step-by-step test runner with visual progress indicators
- Real-time logs panel
- Test results table showing pass/fail status for each step
- Links to created test orders and shipments

**Implemented Test Steps:**
1. Create Test Order - calls `POST /api/admin/shop/test_order`
2. Verify Order Created - validates `origin: TestData`
3. Send Scan Request - calls order action `SendScanRequest`
4. Simulate Scan Completion - marks scan as `ReadyForFabricator`
5. Check Inventory - verifies order status and line items
6. Create Shipment - calls `POST /api/admin/shipper/shipment`
7. Pick Inventory Items - calls shipment action `PickInventoryItem`
8. Generate Shipping Label - calls shipment action `GenerateShippingLabel`
9. Verify Final State - checks order and shipment status

**Implemented Files:**
- `webapps/src/components/Experimental/FactoryQATools.jsx` - Main QA tools page with full test runner
- `webapps/src/components/Experimental/ExperimentalWrapper.jsx` - Navigation wrapper
- `webapps/src/components/Experimental/ExperimentalHome.jsx` - Experimental features home page
- `webapps/arda/app/App.jsx` - Route added at `/experimental/qa-tools`

**Still Needed:**
- Backend API for orchestrated test sequences (currently runs client-side)
- Test cleanup functionality (delete test orders)
- More comprehensive validation at each step
- Historical test run logging

### Files to Create

| File | Purpose |
|------|---------|
| ~~`webapps/src/components/QA/FactoryQATools.jsx`~~ | ~~Main QA tools page with test configuration~~ (Created as experimental page)
| `webapps/src/components/QA/QATestRunner.jsx` | Component that executes and displays test progress |
| `webapps/src/components/QA/QATestStep.jsx` | Individual test step display with pass/fail status |
| `webapps/src/components/QA/QATestReport.jsx` | Summary report after test completion |
| `api/src/fabricator/QAToolsApi.ts` | Backend API for running QA test sequences |

### Files to Modify

| File | Change Required |
|------|-----------------|
| ~~`webapps/app/App.jsx`~~ | ~~Add route for `/qa/tools`~~ (Added at `/experimental/qa-tools`)
| `webapps/app/ArdaWrapper.jsx` | Add QA Tools navigation link (optional - currently in Experimental section)
| `apps/admin_api/admin_api.ts` | Register QA tools API routes |
| `api/src/fabricator/FabricatorAdminApi.ts` | Potentially add batch action execution endpoint |

### Test Configuration Options

The UI should allow users to configure:

```
Test Order Configuration
========================

Product Type:    [Beyond 2 ▼] [Beyond 2 Eye ▼] [Custom Cushion ▼]
Customer Email:  [test-qa-{timestamp}@bigscreenvr.com]
IPD Value:       [63] mm
Priority:        [50] (0-100)
Country:         [US ▼]

Test Options:
☑ Create scan request (for cushion orders)
☑ Simulate manufacturing completion
☑ Auto-pick inventory items
☑ Generate shipping label (test mode)
☐ Complete Shopify fulfillment (careful!)

[Start Full Test]  [Run Step-by-Step]
```

### Test Steps to Execute

1. **Create Test Order**
   - Call `POST /api/admin/shop/test_order`
   - Verify order created with correct origin (`TestData`)
   - Verify initial state is `WaitingForScanRequest`

2. **Send Scan Request** (if applicable)
   - Call `PUT /api/admin/shop/order/{id}` with action `SendScanRequest`
   - Verify scan request record created

3. **Simulate Scan Completion**
   - Update scan request to `ReadyForFabricator` state
   - Verify order transitions to `WaitingForManufacturing`

4. **Create Manufacturing Job**
   - Call action `CreateJob`
   - Verify job created and linked to order

5. **Progress Job Through States**
   - Simulate toolpath creation success
   - Simulate manufacturing completion
   - Simulate QA pass
   - Verify job reaches `AwaitingFulfilment`

6. **Check Inventory Availability**
   - Query inventory slots for order
   - Verify inventory is available or note shortage

7. **Create Shipment**
   - Call `POST /api/admin/shipper/shipment`
   - Verify shipment created in `Init` status

8. **Pick Inventory Items**
   - Execute `PickInventoryItem` for each slot
   - Verify all items picked successfully

9. **Generate Shipping Label**
   - Execute `GenerateShippingLabel` action
   - Verify label URL returned
   - Check tracking number assigned

10. **Complete Packing**
    - Execute `StartPacking` and `FinishPacking`
    - Verify status transitions correctly

11. **Verify Final State**
    - Order should be `ReadyToShip` or `Shipped`
    - All inventory items should be in correct status
    - Shipment should have tracking info

12. **Cleanup** (optional)
    - Delete test order and related records
    - Return inventory items to stock

### API Endpoints to Add

| Method | Endpoint | Purpose |
|--------|----------|---------|
| POST | `/api/admin/qa/test-order` | Create test order with full configuration |
| POST | `/api/admin/qa/run-step` | Execute a single test step |
| POST | `/api/admin/qa/run-full-test` | Execute complete test sequence |
| GET | `/api/admin/qa/test-status/:id` | Get current test run status |
| DELETE | `/api/admin/qa/cleanup/:orderId` | Clean up test order and related data |

### Implementation Notes

- All test orders MUST have `origin: TestData` to prevent confusion with real orders
- The shipping label generation should use Shippo's test mode to avoid charges
- Consider adding a "dry run" mode that validates without making changes
- Test results should be logged for historical tracking
- Add rate limiting to prevent accidental mass test creation

---

## 13. Customs and Shipping Data Refactoring

### What It Does

Move hardcoded customs and shipping configuration data from TypeScript source files into a database-driven configuration system that can be modified through the Arda UI without code deployments.

### Feature Description

Currently, critical shipping and customs data is hardcoded across multiple files, requiring code changes and deployments to update things like:
- Shopify product ID to Arda product type mappings
- Product customs information (tariff numbers, descriptions, weights)
- Shipping package configurations (dimensions, declared values)
- Shipment grouping rules

This refactoring will:
- Create database tables to store all configuration data
- Build an admin UI in Arda to view and edit configurations
- Migrate existing hardcoded data to the database
- Update the shipping/customs code to read from the database instead of code

### Current State

There is an explicit comment in the codebase acknowledging this need:

```typescript
// At line 651 of FabricatorSchemas.ts:
/**
 * There are five Five FIVE FIVE!!! enums dealing with Shopify products.
 *
 * THIS MUST BE TURNED INTO A DATABASE SOMEWHERE.
 */
```

### Data Structures to Migrate

#### 1. Shopify Product Variant Mapping

**Current Location:** `api/src/fabricator/ShopifyData.ts` (lines 26-650)

**Structure:** Maps Shopify `product_id:variant_id` to internal product types

```typescript
// Current hardcoded format:
"8681410330841:45083659665625" → {
    title: "Fibre Optic Cable",
    sku: "RBS1C",
    productTitle: "Beyond Fibre Optic Cable",
    productType: BigProductType.BeyondFibreOpticCableV1
}
```

**Contains:** 140+ product variant mappings

#### 2. Product Type Enum

**Current Location:** `api/src/fabricator/FabricatorSchemas.ts` (lines 657-771)

**Contains:** 75+ product type codes including:
- VR Headsets: `Beyond2`, `Beyond2Eye`, `BigscreenBeyondV1`
- Accessories: `AudioStrapV1`, `HaloMount`, `PrescriptionLenses`
- Cyberbox variants: `B2ClearCyberbox`, `B2BlackCyberbox`, etc. (20+ variants)
- Shells, cushions, merchandise

#### 3. Product Customs Information

**Current Location:** `api/src/fabricator/FabricatorSchemas.ts` (lines 3100-3605)

**Structure:**
```typescript
BigProductTypeCustomsInfo[productType] = {
    tariffNumber: "8528.5210.00",
    description: "VR Headset",
    weight: "2.0",
    massUnit: "lb",
    defaultValue: 999.0,
    customsValueOverrides: { JPY: 159500, EUR: 999, GBP: 899 }
}
```

**Contains:** 60+ product customs configurations

#### 4. Shipping Package Configurations

**Current Location:** `api/src/fabricator/FabricatorSchemas.ts` (lines 3640-4011)

**Structure:**
```typescript
BigShippingConfigs[packageType] = {
    shippoParcelRequest: {
        length: "10", width: "7", height: "6",
        distanceUnit: "in", weight: "2.5", massUnit: "lb"
    },
    tarriffNumber: "8528.5210.00",
    customsDescription: "VR Headset",
    itemDescription: "VR Headset",
    includeExporterIdentification: true,
    valueAmount: 999.0,
    valueCurrency: "USD",
    customsItems: [...]
}
```

**Contains:** 20+ package type configurations

#### 5. Shipment Grouping Logic

**Current Location:** `api/src/fabricator/ShippingAdminApi.ts` (lines 930-1247)

**Logic includes:**
- How line items are grouped into shipments
- Which products can ship together
- Package type selection based on product combinations
- Inventory type determination for pre-packed items

### New Database Tables

```sql
-- Product type definitions
CREATE TABLE big_product_types (
    id TEXT PRIMARY KEY,
    code TEXT UNIQUE NOT NULL,
    displayName TEXT NOT NULL,
    category TEXT,  -- 'headset', 'accessory', 'cushion', 'lens', etc.
    isActive BOOLEAN DEFAULT true,
    createdAt BIGINT,
    updatedAt BIGINT
);

-- Shopify to product type mapping
CREATE TABLE big_shopify_product_mappings (
    id TEXT PRIMARY KEY,
    shopifyProductId TEXT NOT NULL,
    shopifyVariantId TEXT NOT NULL,
    productTypeId TEXT REFERENCES big_product_types(id),
    sku TEXT,
    title TEXT,
    productTitle TEXT,
    isActive BOOLEAN DEFAULT true,
    createdAt BIGINT,
    updatedAt BIGINT,
    UNIQUE(shopifyProductId, shopifyVariantId)
);

-- Product customs information
CREATE TABLE big_product_customs_info (
    id TEXT PRIMARY KEY,
    productTypeId TEXT REFERENCES big_product_types(id),
    tariffNumber TEXT NOT NULL,
    description TEXT NOT NULL,
    weight NUMERIC NOT NULL,
    massUnit TEXT DEFAULT 'lb',
    defaultValue NUMERIC NOT NULL,
    valueCurrency TEXT DEFAULT 'USD',
    createdAt BIGINT,
    updatedAt BIGINT
);

-- Currency-specific value overrides
CREATE TABLE big_customs_value_overrides (
    id TEXT PRIMARY KEY,
    productCustomsInfoId TEXT REFERENCES big_product_customs_info(id),
    currency TEXT NOT NULL,
    value NUMERIC NOT NULL,
    UNIQUE(productCustomsInfoId, currency)
);

-- Package type definitions
CREATE TABLE big_package_types (
    id TEXT PRIMARY KEY,
    code TEXT UNIQUE NOT NULL,
    displayName TEXT NOT NULL,
    lengthInches NUMERIC,
    widthInches NUMERIC,
    heightInches NUMERIC,
    weightLbs NUMERIC,
    tariffNumber TEXT,
    customsDescription TEXT,
    itemDescription TEXT,
    includeExporterIdentification BOOLEAN DEFAULT true,
    declaredValueUsd NUMERIC,
    isActive BOOLEAN DEFAULT true,
    createdAt BIGINT,
    updatedAt BIGINT
);

-- Shipment grouping rules
CREATE TABLE big_shipment_grouping_rules (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    priority INT DEFAULT 0,  -- Higher priority rules evaluated first
    productTypeIds TEXT[],   -- Array of product types this rule applies to
    resultingPackageTypeId TEXT REFERENCES big_package_types(id),
    resultingInventoryTypeId TEXT REFERENCES big_product_types(id),
    conditions JSONB,  -- Flexible conditions (e.g., "hasAudioStrap": true)
    isActive BOOLEAN DEFAULT true,
    createdAt BIGINT,
    updatedAt BIGINT
);
```

### Files to Create

| File | Purpose |
|------|---------|
| `api/src/fabricator/ConfigurationApi.ts` | API for managing configuration data |
| `api/src/fabricator/ConfigurationSchemas.ts` | Schema definitions for config tables |
| `api/src/fabricator/ConfigurationDatabase.ts` | Database queries for config data |
| `webapps/src/components/Configuration/ProductTypesPage.jsx` | Manage product types |
| `webapps/src/components/Configuration/ShopifyMappingsPage.jsx` | Manage Shopify mappings |
| `webapps/src/components/Configuration/CustomsInfoPage.jsx` | Manage customs info |
| `webapps/src/components/Configuration/PackageTypesPage.jsx` | Manage package configs |
| `webapps/src/components/Configuration/GroupingRulesPage.jsx` | Manage grouping rules |
| `webapps/src/components/Configuration/ConfigurationWrapper.jsx` | Navigation wrapper |

### Files to Modify

| File | Change Required |
|------|-----------------|
| `api/src/fabricator/ShopifyData.ts` | Replace hardcoded map with database lookup |
| `api/src/fabricator/FabricatorSchemas.ts` | Keep enums for backward compatibility but load values from DB |
| `api/src/fabricator/ShippingAdminApi.ts` | Update `getBigOrderShipmentGroups` to use DB rules |
| `api/src/fabricator/ShippingAdminApi.ts` | Update customs declaration to use DB config |
| `apps/admin_api/admin_api.ts` | Register configuration API routes |
| `apps/admin_api/db_setup.ts` | Add new configuration tables |
| `webapps/app/App.jsx` | Add routes for configuration pages |
| `webapps/app/ArdaWrapper.jsx` | Add Configuration section to navigation |

### API Endpoints to Add

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/api/admin/config/product-types` | List all product types |
| POST | `/api/admin/config/product-types` | Create product type |
| PUT | `/api/admin/config/product-types/:id` | Update product type |
| GET | `/api/admin/config/shopify-mappings` | List Shopify mappings |
| POST | `/api/admin/config/shopify-mappings` | Create mapping |
| PUT | `/api/admin/config/shopify-mappings/:id` | Update mapping |
| DELETE | `/api/admin/config/shopify-mappings/:id` | Delete mapping |
| GET | `/api/admin/config/customs-info` | List customs info |
| POST | `/api/admin/config/customs-info` | Create customs info |
| PUT | `/api/admin/config/customs-info/:id` | Update customs info |
| GET | `/api/admin/config/package-types` | List package types |
| POST | `/api/admin/config/package-types` | Create package type |
| PUT | `/api/admin/config/package-types/:id` | Update package type |
| GET | `/api/admin/config/grouping-rules` | List grouping rules |
| POST | `/api/admin/config/grouping-rules` | Create rule |
| PUT | `/api/admin/config/grouping-rules/:id` | Update rule |
| POST | `/api/admin/config/migrate` | Run migration from code to DB |
| GET | `/api/admin/config/validate` | Validate current configuration |

### Migration Strategy

1. **Phase 1: Create Tables and API**
   - Create all new database tables
   - Build the configuration API
   - Build the admin UI pages

2. **Phase 2: Data Migration**
   - Write migration script to copy hardcoded data to database
   - Run migration in staging environment
   - Validate all data transferred correctly

3. **Phase 3: Dual-Read Mode**
   - Update code to read from database first, fall back to hardcoded
   - Monitor for any discrepancies
   - Fix any edge cases

4. **Phase 4: Database Primary**
   - Switch to database as primary source
   - Keep hardcoded data as emergency fallback
   - Add caching layer for performance

5. **Phase 5: Cleanup**
   - Remove hardcoded data (or mark as deprecated)
   - Update documentation
   - Train team on using configuration UI

### Implementation Notes

- **Caching:** Database config should be cached (Redis) since it changes rarely
- **Validation:** UI should validate entries (e.g., tariff number format, weight > 0)
- **Audit Log:** Track who changed what configuration and when
- **Import/Export:** Allow exporting config as JSON for backup/migration
- **Version Control:** Consider storing config snapshots for rollback capability
- **Testing:** New Shopify products will still need code review before mapping

### Risk Mitigation

- Keep existing hardcoded data as fallback during transition
- Add comprehensive validation before saving configuration changes
- Implement "preview" mode to see how changes would affect shipments
- Create alerts if database config is missing expected entries
- Document the configuration schema thoroughly

---

## Appendix: Key File Reference

### Backend API Files

| File | Description |
|------|-------------|
| `api/src/fabricator/ShippingAdminApi.ts` | Main shipping API (4800+ lines) - labels, rates, carriers |
| `api/src/fabricator/FabricatorAdminApi.ts` | Order management API - BigOrders, Shopify integration |
| `api/src/fabricator/InventoryAdminApi.ts` | Inventory management API |
| `api/src/fabricator/FabricatorDatabase.ts` | Database queries for all fabricator entities |
| `api/src/fabricator/FabricatorSchemas.ts` | TypeScript schemas for all data types |
| `apps/admin_api/admin_api.ts` | API route registration |

### Frontend Component Directories

| Directory | Purpose |
|-----------|---------|
| `webapps/src/components/BigLogistics/` | Shipping and fulfillment UI |
| `webapps/src/components/BigOrders/` | Order management UI |
| `webapps/src/components/Analytics/` | Analytics and reporting UI |
| `webapps/src/components/Inventory/` | Inventory management UI |

### Key Frontend Components

| Component | Route | Purpose |
|-----------|-------|---------|
| `BigBatchShipper.jsx` | `/shipper/batch` | Batch shipment processing |
| `BigShipperNextOrder.jsx` | `/shipper/next` | Find next order to ship |
| `BigShipment.jsx` | `/shipper/shipment/:id` | Individual shipment workflow |
| `BigShipmentsStats.jsx` | `/shipper/stats` | Shipment statistics dashboard |
| `BigOrderComponent.jsx` | `/shop/order/:id` | Order detail page |
| `BigOrdersList.jsx` | `/shop/orders` | Order list with filtering |

---

*Last Updated: January 2025*