# WITwhat Data Flow Architecture **Version:** 1.0 **Created:** 2026-03-22 **Phase:** 13 — Data Flow Architecture Document This document is the authoritative reference for all data entities, field sources, sync directions, storage layers, and transformation steps in WITwhat. It is prescriptive — not descriptive of aspirations. Rules in the AGENT RULES section are binding. --- ## AGENT RULES **Read this section before touching any data struct, sync logic, storage, or field definition.** --- ### RULE-01: `github_profile_url` does not exist **Status:** PROHIBITED **Background:** Early agents introduced `github_profile_url` assuming it was a GitHub Project column. It is not, and has never been. The field is present in some structs (`Recipient`, `RecipientCardSnapshot`, `DashboardCardViewModel`, service `RecipientSnapshot`) as technical debt from early incorrect design. **Action:** Never add, read, or query this field. Do not treat it as a data source. When refactoring structs that contain it, remove it. No GH Project column named `github_profile_url` exists or should ever be created. --- ### RULE-02: Shipment status belongs on Cards, not Recipients **Status:** PROHIBITED on Recipients **Background:** `shipment_status` and `tracking_state` exist on the `Recipient` domain model as migration debt. Shipment data comes from Shopify fulfillments, which are order-level (card- level), not person-level. A recipient is a person; a card is a shipment. **Action:** All shipment status reads must come from card-level data (Shopify order + fulfillment). Never add new shipment fields to `Recipient`. When refactoring `Recipient`, remove `shipment_status` and `tracking_state`. --- ### RULE-03: SQLite is the single read source **Status:** MANDATORY (once implemented) **Background:** The current in-memory `Repository` (HashMap-backed) is a transitional artifact. Once SQLite is implemented, it becomes the authoritative local cache. All sync writes go to SQLite; all app reads come from SQLite. **Action:** All production reads go through SQLite. Never read from in-memory `Repository` in production code after SQLite is implemented. The in-memory Repository is test/transitional only. --- ### RULE-04: Cards exist only from Shopify orders **Status:** MANDATORY **Background:** The pipeline is Shopify-first (established in Phase 12.1.1). Shopify orders tagged `"wit-what"` are the sole card source. A GitHub Project recipient without a matching Shopify order does NOT produce a card. Cards without a matching recipient show as unassigned (keyed `unassigned:order:{id}`). **Action:** Never create a card from a GH-only recipient. The sync pipeline starts with Shopify orders and looks up GH recipients — not the other way around. --- ### RULE-05: GH Issues are the cloud of record **Status:** MANDATORY (once implemented) **Background:** Cards are stored as `ww-card` issues and products as `ww-product` issues in the `BigscreenVR/beyond-outgoing` GitHub repository. This is the authoritative remote store for all card and product data. **Action:** When implementing cloud persistence, use GH Issues as the authoritative remote store. SQLite is the local mirror of GH Issues. GH writes are the source; SQLite reads are the cache. --- ### RULE-06: No new data fields without a documented source **Status:** MANDATORY **Background:** Undocumented fields accumulate as technical debt and confuse future agents. Every field in every struct must have a named, verifiable source. **Action:** Before adding any field to any data struct, document its source in this file. If you cannot name the source (GH Project column name, Shopify API field, GH Issue body field, etc.), the field should not exist. **Update obligation:** When you add or change data architecture (fields, sources, sync flow, storage schema), update DATA-FLOW.md in the same commit. --- ### RULE-07: `item_summary` is deprecated **Status:** DEPRECATED — do not build on it **Background:** `item_summary` is a pipe-separated string (`"Item A | Item B"`) used throughout the current pipeline as a shortcut for multi-item representation. It is fragile, loses ordering guarantees, and cannot carry product IDs or serial numbers. **Action:** Do not build new features on `item_summary`. New product-aware code must use product references (`Vec` with product ID, name, and optional serial number). `item_summary` will be removed when the product reference model is fully implemented. --- ### RULE-08: `latest_note` is a convenience field, not the notes history **Status:** DEPRECATED as canonical store **Background:** `latest_note` is a scalar `Option` (the most recent note, truncated to 80 chars for preview). It is a UI convenience field. The authoritative notes are `Vec` sourced from GH Issue comments (each comment = one note). **Action:** Do not treat `latest_note` as the full notes history. Do not write new notes by setting only `latest_note`. New notes code must read/write the full `Vec` sourced from GH Issue comments. --- ### RULE-09: Sync-merge contract **Status:** ENFORCED **Background:** Sync passes that overwrite SQLite rows have historically destroyed local-only fields (unit assignments, archive state, notes metadata) when the merge logic missed a preservation step. See RETRO-AGENT-FAILURE-PATTERNS.md for the full failure taxonomy (F1, F2). **Action:** Before modifying any sync entry point or adding a field to a synced struct, read `.planning/SYNC-MERGE.md`. After modifying, update SYNC-MERGE.md in the same commit. Every new struct field must declare `local_only: true|false` in that document. --- ## Overview Four data layers and their external systems: ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ EXTERNAL DATA SOURCES │ │ │ │ ┌──────────────────────┐ ┌──────────────────────┐ ┌───────────────────┐ │ │ │ GitHub Project │ │ GitHub Issues │ │ Shopify Admin │ │ │ │ (BigscreenVR/#11) │ │ (beyond-outgoing) │ │ (store/{slug}) │ │ │ │ │ │ │ │ │ │ │ │ Recipients │ │ ww-card issues │ │ Orders tagged │ │ │ │ (people + vision │ │ ww-product issues │ │ "wit-what" │ │ │ │ Rx + purpose) │ │ │ │ Fulfillments │ │ │ └──────────┬───────────┘ └──────────┬───────────┘ └────────┬──────────┘ │ └─────────────┼────────────────────────── ┼────────────────────────┼─────────────-┘ │ GH CLI (read) │ GH API (read/write) │ HTTP (read) v v v ┌─────────────────────────────────────────────────────────────────────────────────┐ │ SYNC PIPELINE │ │ │ │ run_sync_cycle() ──> SQLite full-mirror cache <── write-back paths │ │ (live_client.rs) (witwhat.db, target) (GH write-back) │ └──────────────────────────────────────┬──────────────────────────────────────────┘ │ reads v ┌─────────────────────────────────────────────────────────────────────────────────┐ │ APPLICATION TRANSFORMATION PIPELINE │ │ │ │ RecipientCardSnapshot ──> DashboardCardViewModel ──> Slint CardData │ │ (service_client.rs) (view_model.rs) (dashboard.slint) │ └─────────────────────────────────────────────────────────────────────────────────┘ ``` **Four data layers:** | Layer | Entity | Source | Cloud Store | Status | |-------|--------|--------|-------------|--------| | 1 | Recipients | GitHub Project columns | GitHub Project (live) | Current | | 2 | Cards | Shopify orders tagged "wit-what" | GH Issues (`ww-card`) | Partial (no GH Issue write yet) | | 3 | Products | Manual / Shopify-linked | GH Issues (`ww-product`) | Target (not yet implemented) | | 4 | Serial Instances | Sub-entity of products | Within product's GH Issue | Target (not yet implemented) | --- ## Layer 1: Recipients **Source of truth:** GitHub Project (BigscreenVR, Project #11, view #19) **Sync direction:** GH Project --> App (read-only, except Shopify Profile URL write-back on assignment) **Data ownership:** Recipients are **people**. They carry identity, vision data, and contact info. They do NOT carry shipment status. See RULE-02. ### Field Table | Field | Type | Source | Status | Notes | |-------|------|--------|--------|-------| | `recipient_id` | `String` | GH Title (or Shopify URL username) | KEEP | Human-readable key; no UUID yet. Same as `recipient_key` conceptually. | | `github_item_id` | `Option` | GH Project item node ID | KEEP | Required for GH Project write-back via `update_field_text` | | `shopify_customer_id` | `Option` | Extracted from `Shopify Profile URL` GH field | KEEP | Last numeric path segment of the Shopify admin URL | | `shopify_order_id` | `Option` | Shopify order | MOVE to Card | Belongs on card, not recipient; migrate out | | `github_profile_url` | `Option` | **DOES NOT EXIST** | **REMOVE** | See RULE-01. Technical debt. Never query or add. | | `shipment_status` | `Option` | Shopify fulfillment | **MOVE to Card** | See RULE-02. On Recipient as migration debt only. | | `tracking_state` | `Option` | Shopify tracking | **MOVE to Card** | See RULE-02. On Recipient as migration debt only. | | `discord_username` | `Option` | GH Project "discord_username" column (to be added) | KEEP | Recipient-level contact info | | `discord_user_id` | `Option` | GH Project "discord_user_id" column (to be added) | KEEP | Used for Discord avatar fetch | | `avatar_hash` | `Option` | Discord API `GET /users/{id}` -> `.avatar` field | IMPLEMENTED (Phase 16.1) | Written after Discord user fetch; used for staleness comparison against Discord CDN | | `avatar_gh_sha` | `Option` | GitHub Contents API -> `.sha` field | IMPLEMENTED (Phase 16.1) | Written after avatar upload to `recipient-avatars` branch; used for skip-if-current optimization during sync | | `unresolved_fields` | `Vec` | Ingest validation | KEEP | Warning log for unknown/unresolvable GH column values | | `discrepancy` | `RecipientDiscrepancy` | Computed during sync | KEEP | Drives UI warning badge | | `manual_override` | `RecipientOverride` | User action (Settings) | KEEP | User-set `discord_username` override preserves manual entries through re-sync | | `version` | `i64` | Sync counter | KEEP | Optimistic concurrency for in-memory store | | `purpose` | `Option` | GH Project "Purpose" column | IMPLEMENTED (Phase 16) | Drives colored ring around profile picture in UI | | `purpose_color` | `Option` | Derived from GH API Purpose option color enum via `gh_color_to_hex()` | IMPLEMENTED (Phase 16) | Hex color string e.g. "#4a7cff"; stored on recipients table for persistence; NOT a direct GH column | | `vision_rx_od` | `Option` | GH Project "Vision Rx - OD" column | IMPLEMENTED (Phase 16) | Right eye prescription (OD = oculus dexter) | | `vision_rx_os` | `Option` | GH Project "Vision Rx - OS" column | IMPLEMENTED (Phase 16) | Left eye prescription (OS = oculus sinister) | | `product_names` | `Vec` | GH Project "product_names" column (comma-separated) | IMPLEMENTED (Phase 16) | Product names assigned to this recipient; index-aligned with `product_shopify_urls` | | `product_shopify_urls` | `Vec` | GH Project "product_shopify_urls" column (comma-separated) | IMPLEMENTED (Phase 16) | Shopify product URLs; index-aligned with `product_names` | ### GH Project Column Mapping | GH Project Column | Maps To | How | |-------------------|---------|-----| | `Title` | `recipient_id` (fallback) | Used as recipient key when Shopify URL is absent | | `Shopify Profile URL` | `shopify_profile_url` on `GithubMappedRecipient`; `shopify_customer_id` extracted | Last numeric path segment extracted via `extract_customer_id_from_shopify_profile_url()` | | `Purpose` | `purpose` | Direct string read | | `Vision Rx - OD` | `vision_rx_od` | Direct string read | | `Vision Rx - OS` | `vision_rx_os` | Direct string read | | `discord_username` | `discord_username` | Direct field read (column to be added to GH Project) | | `discord_user_id` | `discord_user_id` | Direct field read (column to be added to GH Project) | | Product names column | `product_names` (parallel array) | Index-aligned strings (column to be added) | | Product Shopify URLs column | `product_shopify_urls` (parallel array) | Index-aligned strings (column to be added) | | ~~`github_profile_url`~~ | **NOT MAPPED** | This column does not exist. See RULE-01. | ### Sync Direction ``` GitHub Project (source) │ │ GH CLI (gh api graphql) │ read-only v GithubMappedRecipient │ │ map_rows() in project_mapping.rs v Recipient domain model │ │ (write-back only: Shopify Profile URL update after recipient assignment) └─────────────────────────> GitHub Project (write-back via update_field_text) ``` --- ## Layer 2: Cards **Primary source:** Shopify orders tagged `"wit-what"` (RULE-04) **Cloud storage:** GitHub Issues in `BigscreenVR/beyond-outgoing`, label `ww-card` **Data ownership:** Cards represent shipments. Shipment status, tracking, and product assignments live here — not on Recipients. ### Field Table | Field | Type | Source | Status | Notes | |-------|------|--------|--------|-------| | `recipient_id` / `recipient_key` | `String` | GH Title or Shopify URL | KEEP (rename) | Human-readable key for GH recipient. Rename to `recipient_key` conceptually. `recipient_id` and `recipient_key` are currently the same field. SQLite will add a UUID PK alongside it. | | `recipient_name` | `String` | GH Title | KEEP | Display name. Different from `recipient_key` once UUID is introduced. | | `discord_username` | `Option` | Recipient (GH Project) | KEEP | Propagated from recipient for display. | | `discord_user_id` | `Option` | Recipient (GH Project) | KEEP | Propagated from recipient for avatar fetch. | | `github_profile_url` | `Option` | **DOES NOT EXIST** | **REMOVE** | See RULE-01. Propagated from incorrect Recipient field. | | `shopify_customer_id` | `Option` | Shopify order `customer.id` | KEEP | Card-level Shopify identity. | | `shopify_order_id` | `Option` | Shopify order `id` | KEEP | Card-level Shopify order reference. | | `email` | `Option` | Shopify order `email` | KEEP | Card-level. Never on Recipient. As of Phase 16.1, email is popover-only (not shown on card face). | | `shipment_status` | `Option` | Shopify fulfillment tracking events via `shopify_projection::derive_shipment_status_for_order()` | KEEP | Card-level. State enum (Phase 20.2): **Preparing** (no fulfillments), **Packing** (fulfillment exists, label_created or no tracking events), **In Transit** (in_transit tracking), **Delivered** (delivered tracking), **Return Underway** (set by `sync_return_states()` via GraphQL reverse fulfillment API), **Returned** (set by `sync_return_states()`). V008 migration renames legacy values. | | `shipment_status_date` | `Option` | Shopify fulfillment updated_at | KEEP | Card-level. | | `item_summary` | `String` | Items (pipe-separated) | **REPLACE** | See RULE-07. Replace with `product_refs: Vec`. | | `latest_note` | `Option` | Manual note (truncated) | **REPLACE** | See RULE-08. Replace with `notes: Vec`. | | `first_item_image_hint` | `Option` | Items | **REMOVE** | Retired. Products shown as individual image squares, not a single hint URL. | | `partial_data` | `bool` | Sync state | KEEP | True when sync data is incomplete for this card. | | `last_updated_at` | `Option` | Sync clock | KEEP | Used to compute staleness badge (>12h threshold). | | `is_unassigned` | `bool` | Sync pipeline | KEEP | True when no GH recipient matched the Shopify order's customer ID. | | `unassigned_customer_name` | `Option` | Shopify order `customer.first_name + last_name` | KEEP | Display name for unassigned cards. | | `recipient_key` | `String` | GH Project | NEW | Explicit rename target for `recipient_id`. Cards reference a valid recipient key. | | `product_refs` | `Vec` | GH Issue body | NEW | Replaces `item_summary`. Each entry: `{ product_id, name, serial_id? }`. | | `notes` | `Vec` | GH Issue comments | NEW | Replaces `latest_note`. Each entry: `{ date: String (ISO 8601), content: String, author: Option (GH handle from comment author.login; None for optimistic local writes) }`. Latest note = first element (read_notes returns newest-first). | | `github_issue_number` | `Option` | GH Issue create_issue() return value | IMPLEMENTED (Phase 18) | GH Issue number for the `ww-card` issue backing this card. Set via `set_card_issue_number()` — never part of `upsert_card`. Stored in `cards.github_issue_number` (V006 migration). | | `has_unassigned_units` | `bool` | Unit assignment state cross-check | IMPLEMENTED (Phase 19) | True when any product_ref on this card has a serial unit in Available state (unit.assigned_card_id is None or points to a different card). Populated during sync projection from SQLite unit table. Default: false. Source: `serial_instances.assigned_card_id` column cross-check. | | `partial_return_warning` | `bool` | Shopify sync partial return detection | IMPLEMENTED (Phase 19) | True when card has mixed return/non-return unit states (some units in Return Initiated, Return Underway, or Returned while others are not). Populated during sync projection from SQLite unit state data. Default: false. Source: mixed `serial_instances.state` values on same card's product_refs. | ### Notes Sync State The `notes` table has a `synced_at TEXT` column (V006 migration, D-13) for write-back tracking: - `NULL` — note has not yet been synced to GH as an Issue comment - ISO 8601 timestamp — note was successfully posted as a GH Issue comment at this time - Source: App clock at successful GH comment post (via `mark_note_synced()`) - Design intent: write-only in Phase 18; future phases can add bidirectional sync by reading `ww-note`-tagged comments back - `author TEXT` column (V012 migration, Phase 20.1.1) — GH handle from `gh issue view --comments --json comments` `.comments[].author.login`. NULL for optimistic local notes until the flusher confirms + targeted refresh reads back the real author. UI renders NULL as "You". ### GH Issue Storage (`ww-card`) GH Issues in `BigscreenVR/beyond-outgoing` with label `ww-card` are the cloud store for cards. **Issue title format:** `{Recipient Name} - {First Product Name} {Serial ID} (+{Extra Count}) [#{Order ID}]` - Example: `John Doe - Frying Pan FP3067 (+2) [#6012345]` - Formatted by `format_card_title()` in `issues_client.rs` **Issue body structure** (JSON code block, schema_version: 1): - `card_id` — stable local card identifier - `recipient_key` — the GH recipient identifier - `shopify_order_id` — Shopify order reference (optional) - `product_refs` — list of `{ product_id, serial_id? }` assignments - `shipment_status` — current shipment status string (optional) - `shipment_status_date` — date of last shipment status update (optional) - Formatted/parsed by `format_card_body()` / `parse_card_body()` in `issues_client.rs` - Struct: `CardIssueBody` (with `CardProductRef` for product entries) **Issue comments:** Each comment = one note entry. Format: `` `ww-note` `` code tag + italic quote block. - Formatted by `format_note_comment()` in `issues_client.rs` - Posted via `create_issue_comment()` on `GhIssuesClient` - `synced_at` column on `notes` table tracks which notes have been posted - Read-back path: `GhIssuesClient::list_issue_comments(issue_number)` parses ww-note-tagged comments into NoteEntry rows (author = comment.author.login, date = createdAt, content = parse_note_comment(body)). Triggered on notes-popover open (Phase 20.1.1, D-11). SQLite notes are reconciled via diff-insert (`upsert_notes_for_card`) — never DELETE+re-INSERT (see SQLITE_TIPS.md). ### Data Constraints - Unassigned cards use `is_unassigned=true` and key `unassigned:order:{shopify_order_id}`. - Unassigned cards show `unassigned_customer_name` as the display name. - `recipient_id` and `recipient_key` are currently the same field (human-readable GH Title). SQLite will introduce a true UUID PK alongside. --- ## Layer 3: Products **Cloud storage:** GitHub Issues in `BigscreenVR/beyond-outgoing`, label `ww-product` **Status:** Target architecture — not yet implemented in code. Products are a standalone catalog. They can be created manually (no Shopify link required) or linked to a Shopify product URL. All products are assumed serializable — individual units are tracked as separate `ww-product-unit` GH Issues (see Layer 4). ### Field Table | Field | Type | Source | Status | Notes | |-------|------|--------|--------|-------| | `product_id` | `String` | Generated (UUID or slug) | REQUIRED | Stable identifier across all layers | | `name` | `String` | GH Issue body / manual entry | REQUIRED | Display name (also GH Issue title) | | `image_url` | `Option` | GH Issue body | OPTIONAL | Product image; auto-fetched from Shopify if linked and no custom image set | | `shopify_product_url` | `Option` | GH Issue body | OPTIONAL | Shopify product admin link; absent for manually created products | | `github_issue_number` | `i64` | GH Issue number | REQUIRED | Reference to the `ww-product` GH Issue | ### GH Issue Storage (`ww-product`) GH Issues in `BigscreenVR/beyond-outgoing` with label `ww-product`: - Issue title = product name - Issue body = JSON code block with `schema_version: 1`: - `product_id`, `image_url`, `shopify_product_url`, `unit_refs` - `unit_refs`: array of child unit issue references in `owner/repo#number` format (e.g., `"BigscreenVR/beyond-outgoing#42"`) — renders as clickable links in GitHub UI and creates automatic back-references - Issue comments/timeline used for product-level notes - Product unit data lives in separate `ww-product-unit` issues (see Layer 4); parent body only holds the reference list ### GH Project Parallel-Array Columns GH Project rows (Recipients) encode product data per-recipient as parallel-array columns: | GH Project Column | Content | Encoding | |-------------------|---------|----------| | Product names column | One product name per row entry | Newline-separated or comma-separated list | | Product Shopify URLs column | One Shopify URL per row entry | Index-aligned with product names column | Indexes align: `product_names[0]` pairs with `product_shopify_urls[0]`. Digested during sync — unmatched products (by Shopify URL) create new catalog entries in SQLite. --- ## Layer 4: Product Units (Serial Instances) **Sub-entity of:** Products **Cloud storage:** Each unit has its own GH Issue in `BigscreenVR/beyond-outgoing`, label `ww-product-unit` **Status:** Target architecture — not yet implemented in code. Each product unit gets a dedicated GH Issue so that the issue timeline can be used **outside of WITwhat** for comments, documentation, and history of what happens to that specific physical unit. This is a critical architectural decision — units are not embedded in the parent product issue. ### Field Table | Field | Type | Notes | |-------|------|-------| | `serial_id` | `String` | Human-readable serial number, e.g. `"PC-001"`. Unique within a product. GH Issue title. | | `product_id` | `String` | FK to parent product. | | `state` | `SerialInstanceState` | Current lifecycle state. See state machine below. | | `assigned_card_id` | `Option` | FK to card. Null when state is `Available`. | | `assigned_at` | `Option` | Timestamp of most recent assignment. | | `parent_ref` | `String` | GitHub issue reference to parent `ww-product` in `owner/repo#number` format. Stored in unit's issue body JSON. Clickable in GitHub UI. | | `github_issue_number` | `i64` | GH Issue number for this unit's own `ww-product-unit` issue. | ### GH Issue Storage (`ww-product-unit`) GH Issues in `BigscreenVR/beyond-outgoing` with label `ww-product-unit`: - Issue title = serial ID (e.g., "HMD-001") - Issue body = JSON code block with `schema_version: 1`: - `serial_id`, `product_id`, `parent_ref`, `state`, `assigned_card_id` - `parent_ref`: parent product issue in `owner/repo#number` format (e.g., `"BigscreenVR/beyond-outgoing#37"`) — renders as clickable link back to parent in GitHub UI - **Cross-linking rule:** All parent↔child references use `owner/repo#number` notation so GitHub renders them as clickable links and auto-creates back-references in issue timelines - **Issue timeline is the primary documentation trail** — comments, observations, incident notes about this specific physical unit are added by humans outside of WITwhat - WITwhat reads and updates the JSON body; it does NOT manage timeline comments (those are user-owned) ### Lifecycle State Machine ``` ┌─────────┐ ┌──────>│ Created │ │ └────┬────┘ │ │ assign to card │ v │ ┌──────────┐ │ │ Assigned │ │ └────┬─────┘ │ │ shipped │ v │ ┌────────────┐ │ │ In Transit │ │ └─────┬──────┘ │ │ delivered │ v │ ┌───────────┐ │ │ Delivered │ │ └─────┬─────┘ │ │ return initiated │ v │ ┌──────────────────┐ │ │ Return Underway│ │ └────────┬─────────┘ │ │ returned │ v │ ┌──────────┐ └───────│ Returned │ └────┬─────┘ │ processed v ┌───────────┐ │ Available │<──────── ready for next assignment └───────────┘ ``` **Active states:** Any state between `Assigned` and `Returned` (exclusive of `Available`). An instance in an active state cannot be reassigned. **Constraint:** Only one active assignment per serial instance at any time. The `assigned_card_id` field is set when transitioning into `Assigned` and cleared when transitioning into `Available`. --- ## Sync Pipeline The sync pipeline is **Shopify-first** (established in Phase 12.1.1). Shopify orders drive card creation. GitHub Project recipients are a lookup table for matching. ### Flow Diagram ``` ┌──────────────────────────┐ ┌──────────────────────────────┐ │ GitHub Project │ │ Shopify Admin │ │ (BigscreenVR/#11) │ │ orders_by_tag("wit-what") │ └────────────┬─────────────┘ └──────────────┬───────────────┘ │ GH CLI graphql │ HTTP GET v v map_rows() Vec project_mapping.rs │ v Vec │ └──> HashMap │ │ for each ShopifyOrder: │ order.customer_id in map? │ ┌───────┴──────────┐ │ YES │ NO v v RecipientCardSnapshot RecipientCardSnapshot (recipient_key = GH) (is_unassigned = true) (matched card) (key: unassigned:order:{id}) │ │ └────────┬─────────┘ v Vec + Vec │ v slint::invoke_from_event_loop │ v UI update (window.set_cards()) ``` ### run_sync_cycle Steps 1. Fetch GH Project rows via GH CLI graphql → `Vec` 2. Map rows via `map_rows()` → `Vec` 3. Build lookup map: `HashMap` (recipient index by Shopify customer ID) 4. Fetch Shopify orders tagged `"wit-what"` via HTTP → `Vec` 5. For each order: lookup `customer_id` in map - Match found → create assigned `RecipientCardSnapshot` with recipient fields + shipment data - No match → create unassigned `RecipientCardSnapshot` (`is_unassigned=true`, key `unassigned:order:{id}`) 6. Collect `(Vec, Vec)` tuple 7. Push to UI via `slint::invoke_from_event_loop` ### Shipment Status Mapping Source: `run_sync_cycle()` in `crates/app/src/live_client.rs` | Shopify `fulfillment_status` | Displayed As | |------------------------------|--------------| | `"fulfilled"` | `"Delivered"` | | `"partial"` | `"In Transit"` | | `"restocked"` | `"Returned"` | | Any other string | That string verbatim | | `null` / absent | `"Unfulfilled"` | ### `customer_id` Extraction Shopify Profile URL format: `https://admin.shopify.com/store/{store_slug}/customers/{id}` ``` extract_customer_id_from_shopify_profile_url(url): 1. Trim trailing "/" 2. Split on "/" 3. Take last segment 4. Return Some(segment) if all ASCII digits, else None ``` Example: `https://admin.shopify.com/store/mystore/customers/12345` → `Some("12345")` ### Write-Back Paths | Action | API Call | Target | |--------|----------|--------| | Assign recipient to unassigned card | `update_field_text(item_id, field_id, customer_url)` | GH Project "Shopify Profile URL" column | | Save note | `save_note(package_id, note)` → `service::api::items::save_package_note` | In-memory Repository (current) / GH Issue comment (target) | | Add item | `add_item(package_id, display_name, image_hint, shopify_product_id)` | Repository + PendingEditQueue | | Remove item | `remove_item(item_id)` | Repository + PendingEditQueue | | Rename item | `rename_item(item_id, new_display_name)` | Repository + PendingEditQueue | --- ## Transformation Pipeline Data flows through seven layers from raw API to UI. ### Layer-by-Layer Diagram ``` GH Project row (HashMap) [External API] │ │ map_rows() │ crates/integrations/src/github/project_mapping.rs v GithubMappedRecipient [Integration layer] │ │ via in-memory Repository + Service layer │ crates/service/src/db/repository.rs v Recipient domain model [Domain layer] │ crates/core/src/domain/recipient.rs │ │ + Package + Item aggregation │ crates/service/src/db/models.rs v RecipientAggregate [Service aggregate] │ crates/service/src/db/models.rs │ │ get_recipient_snapshot() │ crates/service/src/api/recipients.rs v service RecipientSnapshot [Service DTO] │ crates/service/src/api/recipients.rs │ │ map_snapshot() in live_client.rs │ + Shopify order data merged here │ crates/app/src/live_client.rs v app RecipientCardSnapshot [App DTO — primary data shape] │ crates/app/src/service_client.rs │ │ project_snapshot() │ crates/app/src/dashboard/projection.rs v DashboardCardViewModel [View model — UI-ready] │ crates/app/src/dashboard/view_model.rs │ │ main.rs callback binding v Slint CardData [UI layer — terminal] crates/app/ui/dashboard.slint ``` ### Layer Responsibilities | Layer | Key Struct | File | Responsibility | |-------|-----------|------|----------------| | External API | `GithubProjectRow` | `project_client.rs` | Raw GH API response | | Integration | `GithubMappedRecipient` | `project_mapping.rs` | Maps GH columns to typed fields | | Domain | `Recipient` | `core/domain/recipient.rs` | Business entity; validates, normalizes | | Service aggregate | `RecipientAggregate` | `service/db/models.rs` | Combines recipient + packages + items | | Service DTO | `RecipientSnapshot` | `service/api/recipients.rs` | Serializable service contract | | App DTO | `RecipientCardSnapshot` | `app/service_client.rs` | Card data shape; merges Shopify data | | View model | `DashboardCardViewModel` | `app/dashboard/view_model.rs` | UI-ready; computes pills, badges | | UI | `CardData` (Slint struct) | `app/ui/dashboard.slint` | Terminal layer; display only | ### Slint CardData Field Mapping | Slint Field | Rust Source | Notes | |-------------|-------------|-------| | `recipient-name` | `DashboardCardViewModel.recipient_name` | Display name | | `status-pill` | `DashboardCardViewModel.status_pill` | Formatted shipment status string | | `status-date` | `DashboardCardViewModel.status_date_inline` | Formatted date string | | `item-summary` | `DashboardCardViewModel.item_summary` | Pipe-separated (deprecated — see RULE-07) | | `item-squares` | `[ItemSquareData]` | One entry per product; replaces single image hint | | `note-preview` | `DashboardCardViewModel.note_preview` | First 80 chars of latest note | | `archive-state` | `DashboardCardViewModel.archive_state` as `int` | 0=Active, 1=TBA, 2=Archived | | `discord-user-id` | `DashboardCardViewModel.discord_user_id` | For Discord avatar fetch | | `shopify-customer-url` | Built from `shopify_customer_id` + `store_slug` | Menu link | | `shopify-order-url` | Built from `shopify_order_id` + `store_slug` | Menu link | | `package-id` | `DashboardCardViewModel.package_id` | For item CRUD (transitional) | | `active-item-id` | `DashboardCardViewModel.active_item_id` | Currently unused | | `is-unassigned` | `DashboardCardViewModel.is_unassigned` | Drives unassigned badge | | `unassigned-customer-name` | `DashboardCardViewModel.unassigned_customer_name` | Display name for unassigned card | | `discord-username` | `DashboardCardViewModel.discord_username` (from `RecipientCardSnapshot.discord_username`) | Propagated from GH Project `discord_username` column; shown/edited in summary popover (Phase 16.1) | | `avatar-image` | `slint::Image` loaded from avatar cache on disk | Discord avatar loaded via `Image::load_from_path()`; fetched by Discord API using `discord_user_id` (Phase 16.1) | | `last-activity-label` | Computed in `main.rs` projection | Pre-formatted string e.g. "Shipped — 2026-03-01"; derived from card shipment data; empty string when absent (Phase 16.1) | | `last-activity-products` | Computed in `main.rs` projection | Product names from most recent qualifying card event; empty array when absent (Phase 16.1) | | `recipient-id` | `DashboardCardViewModel.recipient_id` | Human-readable recipient key (same as `recipient_key`); used for GH Project write-back callbacks (Phase 16.1) | | `partial-return-warning` | `DashboardCardViewModel.partial_return_warning` | `true` when card has mixed return/non-return unit states (some units in Return Initiated/In Transit/Returned, others not). **Source:** Shopify sync partial return detection — cross-checks unit states from SQLite during sync projection. Direction: read-only computed field (Phase 19, D-13). | ### ItemSquareData Field Mapping | Slint Field | Rust Source | Notes | |-------------|-------------|-------| | `item-id` | `ItemSquareData.item_id` | Product name (display identity for this square) | | `display-name` | `ItemSquareData.display_name` | Human-readable product name | | `initials` | `ItemSquareData.initials` | Up to 2 uppercase initials from display-name words | | `bg-color-index` | Computed from `item_square_color_index(name)` | 0–7 index into fixed palette | | `has-image` | `ItemSquareData.has_image` | True when a locally cached image file exists for this product | | `image-url` | `ItemSquareData.image_url` | Product CDN URL (Shopify-sourced); stored in SQLite `products.image_url` | | `image` | `slint::Image` loaded from product image cache on disk | Product image loaded via `Image::load_from_path()` from `%APPDATA%/WITwhat/product-images/{product_id}.{ext}`; source URL from Shopify CDN stored in SQLite `products.image_url` (Phase 20.1, D-08) | | `is-dimmed` | `ItemSquareData.is_dimmed` | `true` when the serial unit assigned to this slot has been unassigned from this card (unit.assigned_card_id != this card's id OR unit.assigned_card_id is None). **Source:** Unit assignment state cross-check during sync projection. Direction: read-only computed field (Phase 19, D-08). | The Slint layer is **terminal**: it reads from ViewModel, never from raw API data, snapshots, or domain models. --- ### RecipientTileData Field Mapping `RecipientTileData` is defined in `crates/app/ui/option-grid.slint` and populated by `build_recipient_tiles()` in `crates/app/src/main.rs`. | Slint Field | Rust Source | Notes | |-------------|-------------|-------| | `name` | `CardData.recipient_name` | Recipient display name | | `initial` | First char of `recipient_name`, uppercased | First letter of name for avatar fallback | | `section-header` | Computed from first letter grouping | Section label (e.g. "A", "B") | | `grid-row` | Computed layout position | Row index in grid | | `grid-col` | Computed layout position | Column index in grid | | `dimmed` | `dimmed_names.contains(name)` | True when tile is not selectable | | `contact-secondary` | `CardData.email` (preferred) or `CardData.contact_secondary` | Secondary contact info shown under name | | `has-avatar-image` | `CardData.avatar_image.size().width > 0` | True when local avatar cache file was loaded into the card's image slot (Phase 20.1, D-03) | | `avatar-image` | `CardData.avatar_image.clone()` | Discord avatar reused from CardData — loaded from `%APPDATA%/WITwhat/avatars/{discord_user_id}.png` via `slint::Image::load_from_path()` at card build time (Phase 20.1, D-03) | | `purpose-color` | `CardData.purpose_color` | Purpose-specific ring color for avatar border; derived hex string stored in SQLite recipients table (Phase 20.1, D-03) | --- ## Storage Architecture ### Current State (Transitional) **In-memory (non-persistent):** - `Repository` — HashMap-backed store: `recipients`, `packages`, `items`, `conflict_notes` - File: `crates/service/src/db/repository.rs` - Thread-safe access: wrapped in `Arc>` at LiveClient level - `LiveClient.unassigned_cards` — `Arc>>` (background-thread written) - `LiveClient.recipient_list` — `Arc>>` (background-thread written) **Persistent files** (all in `%APPDATA%\WITwhat\`): - `config.toml` — `AppConfig` struct (shopify_store_slug, shopify_secret_ref, github_project_node_id, github_project_url, discord_bot_secret_ref, discord_guild_id) - `archive_store.json` — `ArchiveStore`: `HashMap` (Active/TBA/Archived + timestamps) - `pending_edits.json` — `PendingEditQueue`: `Vec` (SaveNote | AddItem | RemoveItem | RenameItem, serialized with `serde(tag="kind")`) - `avatars/{discord_user_id}.png` — Local performance cache for Discord avatar PNGs (see two-tier avatar storage below) **Secrets (Windows Credential Manager):** | WCM Key | Content | Notes | |---------|---------|-------| | `wincred:shopify/token` | Shopify access token | Required for Shopify API calls | | `wincred:discord/bot-token` | Discord bot token for API calls | Optional; enables avatar fetch and username resolution (Phase 16.1) | ### Two-Tier Avatar Storage (Phase 16.1) Discord avatar PNGs use a two-tier caching strategy: | Tier | Location | Purpose | When Used | |------|----------|---------|-----------| | Durable store | GitHub branch `recipient-avatars` in `BigscreenVR/beyond-outgoing`, files at `avatars/{discord_user_id}.png` | Survives machine wipe; shared across devices | Upload after Discord CDN download; download on new machine or after cache miss | | Performance cache | Local disk `%APPDATA%/WITwhat/avatars/{discord_user_id}.png` | Fast local reads without GitHub API calls | Written after every CDN download; read by Slint `Image::load_from_path()` | **Flow on username change / first avatar fetch:** 1. Discord CDN download → `fetch_and_cache_avatar()` → local cache write 2. `AvatarBranchClient.upload_avatar()` → GitHub branch write → update `avatar_gh_sha` in SQLite 3. Discord API user fetch → update `avatar_hash` in SQLite (for future staleness comparison) **Flow on sync/startup (cache warmup):** 1. Check if local cache file exists (`avatar_cache_path(discord_user_id)`) 2. If not: `AvatarBranchClient.download_avatar()` → write to local cache 3. If avatar_hash changed (Discord CDN differs from stored hash): re-fetch from CDN, re-upload to branch ### Target State: SQLite Full-Mirror Cache **Location:** `%APPDATA%\WITwhat\witwhat.db` **Schema philosophy:** 1. Every entity gets its own normalized table — no JSON blob columns. 2. Foreign keys enforced with `ON DELETE CASCADE` for dependent entities (notes → cards, serial_instances → products, card_products → cards/products). 3. App reads exclusively from SQLite. Background sync writes to SQLite from GH/Shopify sources. 4. JSON files (`archive_store.json`, `pending_edits.json`) migrate into SQLite tables. 5. Conflict resolution: last-write-wins. The higher `synced_at` timestamp wins. Losing values logged to `sync_audit_log`. **Table Listing:** ``` recipients id TEXT PRIMARY KEY -- UUID (new in SQLite phase) recipient_key TEXT NOT NULL -- human-readable GH Title / username title TEXT NOT NULL -- display name purpose TEXT -- colored ring in UI vision_rx_od TEXT -- right eye prescription vision_rx_os TEXT -- left eye prescription discord_username TEXT discord_user_id TEXT shopify_customer_id TEXT shopify_profile_url TEXT github_item_id TEXT -- GH Project item node ID for write-back synced_at INTEGER -- Unix timestamp of last sync cards id TEXT PRIMARY KEY -- UUID recipient_key TEXT -- FK to recipients.recipient_key (nullable = unassigned) shopify_order_id TEXT NOT NULL shopify_customer_id TEXT is_unassigned INTEGER NOT NULL -- 0/1 unassigned_customer_name TEXT shipment_status TEXT shipment_status_date TEXT email TEXT github_issue_id TEXT -- ww-card GH Issue number synced_at INTEGER card_products card_id TEXT NOT NULL -- FK cards.id ON DELETE CASCADE product_id TEXT NOT NULL -- FK products.id ON DELETE CASCADE serial_id TEXT -- FK serial_instances.id (nullable for non-serializable) PRIMARY KEY (card_id, product_id, serial_id) notes id TEXT PRIMARY KEY card_id TEXT NOT NULL -- FK cards.id ON DELETE CASCADE date TEXT NOT NULL -- ISO 8601 content TEXT NOT NULL synced_at INTEGER products id TEXT PRIMARY KEY name TEXT NOT NULL image_url TEXT shopify_product_url TEXT is_serializable INTEGER NOT NULL -- 0/1 github_issue_id TEXT -- ww-product GH Issue number synced_at INTEGER serial_instances id TEXT PRIMARY KEY -- e.g. "PC-001" product_id TEXT NOT NULL -- FK products.id ON DELETE CASCADE serial_number TEXT NOT NULL state TEXT NOT NULL -- SerialInstanceState enum value assigned_card_id TEXT -- FK cards.id (nullable when Available) assigned_at TEXT -- ISO 8601 (nullable) synced_at INTEGER archive_records recipient_id TEXT PRIMARY KEY state INTEGER NOT NULL -- 0=Active, 1=TBA, 2=Archived archived_at TEXT -- ISO 8601 auto_archived INTEGER -- 0/1 manual_unarchive_override INTEGER -- 0/1 pending_edits id INTEGER PRIMARY KEY AUTOINCREMENT entity_type TEXT NOT NULL -- e.g. "card" entity_id TEXT NOT NULL -- card_id or other entity key edit_type TEXT NOT NULL -- SaveNote | UpdateCardBody | UpdateCardTitle payload TEXT NOT NULL -- JSON payload for the edit created_at TEXT NOT NULL -- ISO 8601 UTC timestamp retry_count INTEGER NOT NULL DEFAULT 0 -- V006: incremented on each failed flush attempt last_attempted_at TEXT -- V006: ISO 8601 UTC timestamp of last flush attempt sync_audit_log id TEXT PRIMARY KEY entity_type TEXT NOT NULL -- recipients | cards | products | etc. entity_id TEXT NOT NULL field_name TEXT NOT NULL old_value TEXT new_value TEXT written_at INTEGER -- Unix timestamp source TEXT NOT NULL -- GH | Shopify | Manual ``` ### Offline Mode When network sync fails or GH/Shopify is unreachable: ``` ┌──────────────────────────────────────────────────────┐ │ OFFLINE MODE │ │ │ │ App reads from SQLite cache (last successful sync) │ │ Persistent offline indicator shown in UI │ │ │ │ User edits queue to pending_edits table: │ │ Pending -> Flushing -> Failed │ │ No functionality disabled — full read/write │ │ │ │ On reconnect: │ │ 1. Flush pending_edits in order │ │ 2. Run sync cycle to refresh SQLite │ │ 3. Clear offline indicator │ └──────────────────────────────────────────────────────┘ ``` Pending edit statuses: - `Pending` — queued, not yet attempted - `Flushing` — in-flight on reconnect - `Failed` — attempted and failed; user notified; requires manual retry or discard --- ## Deprecated Fields and Migration Debt These fields exist in code but must not be extended. They will be removed during future refactoring phases. | Field | Location | Replacement | Rule | |-------|----------|-------------|------| | `github_profile_url` | `Recipient`, `RecipientCardSnapshot`, `DashboardCardViewModel`, service `RecipientSnapshot` | **None** — field should not exist | RULE-01 | | `shipment_status` on `Recipient` | `core/domain/recipient.rs` | `shipment_status` on Card (already correct there) | RULE-02 | | `tracking_state` on `Recipient` | `core/domain/recipient.rs` | `shipment_status` / `tracking_state` on Card | RULE-02 | | `item_summary` (pipe-separated string) | `RecipientCardSnapshot`, `DashboardCardViewModel` | `product_refs: Vec` | RULE-07 | | `latest_note` (scalar string) | `RecipientCardSnapshot`, `DashboardCardViewModel` | `notes: Vec` | RULE-08 | | `first_item_image_hint` | `RecipientCardSnapshot` | Individual `ItemSquareData` entries in `item_squares` | Retired | | `email` on `Recipient` domain model | `core/domain/recipient.rs` | `email` on Card (`RecipientCardSnapshot.email`) | Lives on Shopify orders | | `package_id` / `active_item_id` on `CardData` | `dashboard.slint` | Product reference model (`card_products` table) | Transitional placeholder for item CRUD | | `shopify_order_id` on `Recipient` | `core/domain/recipient.rs` | `shopify_order_id` on Card | Belongs on card | --- ## External Systems | System | URL / Reference | Role | |--------|----------------|------| | GitHub Project | `https://github.com/orgs/BigscreenVR/projects/11/views/19` | Recipient source of truth; product parallel-array columns | | GitHub Issues repo | `BigscreenVR/beyond-outgoing` | Cloud store for `ww-card` and `ww-product` issues | | Shopify Admin | `https://admin.shopify.com/store/{store_slug}` | Order + fulfillment + tracking source | --- ## Open Questions These items are undecided and will be resolved during their respective implementation phases. Do not make assumptions — treat them as TBD. ### OQ-01: Serial instance storage location within GH Issue **Context:** Serial instances are stored within the parent product's GH Issue, but the exact encoding is undecided. **Options:** - Issue body block (structured JSON/YAML) — simple bulk read, harder to audit individual transitions - Individual comments (one per lifecycle event) — audit trail, but complex bulk read **Resolution:** TBD during GH Issues implementation phase. ### OQ-02: `recipient_key` vs `recipient_id` naming **Context:** CONTEXT.md specifies `recipient_name` renamed to `recipient_key` on cards. However, the codebase uses `recipient_id` as the primary identifier on `Recipient` (currently set to the GH Title, i.e., the human-readable key). **Current state:** `recipient_id` and `recipient_key` are the same field — both are the human- readable GH username or Title. There is no UUID. **Resolution:** SQLite will introduce a true UUID primary key. `recipient_key` = human-readable GH identifier. `recipient_id` = UUID primary key (SQLite only). Until then, they are equivalent. ### OQ-03: GH Issue body format for `ww-card` and `ww-product` **Context:** Issues must encode structured metadata (product assignments, serial numbers, shipment references). The exact format is not yet decided. **Options:** JSON front-matter block, YAML block, custom delimiter format. **Resolution:** TBD during GH Issues implementation phase. Document must describe what fields must be representable; encoding is an implementation detail. --- *Created: 2026-03-22 | Phase 13 | Maintain this document when changing data architecture.*