# Phase 2: External Sync Integrations - Research

**Researched:** 2026-03-11
**Domain:** GitHub Project ingestion, Shopify customer/order/fulfillment sync, resilient polling orchestration
**Confidence:** HIGH (verified against implemented code in crates/)

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions
- GitHub source target is configurable project + view.
- GitHub ingestion runs full refresh each poll and computes local diffs.
- Mapping is limited to identity + shipment-relevant operational fields.
- Invalid/missing required mapped fields are skipped with warning logs.
- Shopify linkage prefers profile URL from GitHub when present.
- Shopify orders are filtered by configured tags/status before projection.
- Recipients without Shopify matches remain persisted with null Shopify fields.
- Shipment truth comes from Shopify fulfillment status + tracking events.
- Polling is adaptive (not fixed cadence) with exponential backoff in-cycle.
- Partial failures apply successful provider updates and preserve stale failed-provider data.
- UI shows persistent sync status indicator without timestamp display.
- Staleness warning appears only when data is older than 12 hours.
- Merge precedence: GitHub identity fields, Shopify commerce/shipment fields.
- Merge stores conflict notes for diagnostics.
- Manual overrides persist until explicitly cleared.
- Discrepancies must be visible with warning styling + explanatory tooltip.
- Merge atomicity is per-recipient.

### Claude's Discretion
- Adaptive interval bounds and adjustment heuristics.
- Backoff timing (base delay, cap, attempt count).
- Conflict-note schema details.
- Concrete warning/tooltip component structure.

### Deferred Ideas (OUT OF SCOPE)
- Archive automation on `Returned` status (Phase 7).
- Standalone manual tracking-status override workflow beyond conflict signaling.
</user_constraints>

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|-----------------|
| DATA-01 | Import and sync recipients from configured GitHub Project. | `GithubIngestConfig` with `project_id`/`view_id`; `ingest_github_recipients` computes `IngestDiff` (new/changed/removed keys); invalid rows emit `MappingWarning` and are skipped. |
| DATA-02 | Link recipient records to Shopify customer profiles. | `match_customer` uses URL-first then discord fallback; `link_recipients` produces `LinkedRecipient` with `shopify_customer_id: Option<String>` and `unresolved_fields` for no-match cases. |
| DATA-03 | Fetch Shopify order/fulfillment/tracking per linked recipient. | `project_shipment_for_customer` returns `ShipmentProjection`; fulfillment status normalization: delivered > in_transit > fulfilled/shipped; tracking events normalized to canonical strings. |
| DATA-07 | Auto-poll every 5 minutes with retry/backoff behavior. | `SyncScheduler` targets 300s interval (min 240s, max 900s); `BackoffPolicy` with 15s base, 120s cap, 3 max attempts; `SyncStatusTracker` marks stale after 12h; `ServiceRuntime` wires all components. |
</phase_requirements>

## Summary

Phase 2 establishes a four-stage sync pipeline: GitHub ingest -> Shopify linkage -> shipment projection -> merge/persist. All stages are implemented in `crates/integrations` (provider clients) and `crates/service/src/sync` (orchestration). The implementation uses in-memory fakes for both providers (`InMemoryGithubProjectClient`, `InMemoryShopifyClient`) with trait boundaries ready for real HTTP clients in a future phase.

The architecture separates provider snapshot responsibilities from merge logic. GitHub ingest computes structural diffs (new/changed/removed recipient keys) enabling incremental updates. Shopify linkage uses URL-first matching with silent no-match preservation. The merge engine applies locked precedence rules, captures conflict notes, and preserves manual overrides. All of this feeds a `RecipientSnapshot` API surface that exposes `discrepancy_warning` and `discrepancy_tooltip` for UI rendering.

The adaptive scheduler runs around a 300-second target, expanding on failure (by adding backoff delay to current interval) and contracting on success (by subtracting 30s, floor at 240s, ceiling at 900s). `SyncStatusTracker` is clock-injectable (accepts `SystemTime`) making it deterministically testable.

**Primary recommendation:** All four plans are complete and verified. The service layer is implemented with in-memory fakes; real HTTP integration (actual GitHub GraphQL and Shopify REST calls) is the key open question for this architecture.

## Standard Stack

### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `tokio` | workspace | Async runtime for scheduler and HTTP clients | Existing async baseline in service crate |
| `reqwest` | (not yet wired) | Real GitHub/Shopify HTTP transport | Standard Rust async HTTP client |
| `serde` | workspace | DTO serialization/deserialization | Already used across core/service crates |
| `sqlx` SQLite | (not yet wired) | Snapshot and conflict note persistence | Workspace dependency planned for real persistence |

### Implemented (In-Memory Layer)
| Component | Location | Purpose |
|-----------|----------|---------|
| `InMemoryGithubProjectClient` | `crates/integrations/src/github/project_client.rs` | Test double for GitHub project row fetching |
| `InMemoryShopifyClient` | `crates/integrations/src/shopify/order_fulfillment_client.rs` | Test double for Shopify order/fulfillment/tracking |
| `Repository` | `crates/service/src/db/repository.rs` | In-memory persistence with `upsert_recipient`, `upsert_conflict_notes` |
| `ServiceRuntime` | `crates/service/src/runtime/startup.rs` | Wires scheduler + status tracker; exposes `sync_health()` |

## Architecture Patterns

### Recommended Module Layout
```
crates/integrations/src/
├── github/
│   ├── project_client.rs       # GithubProjectClient trait + InMemoryGithubProjectClient
│   └── project_mapping.rs      # map_rows() -> (Vec<GithubMappedRecipient>, Vec<MappingWarning>)
└── shopify/
    ├── customer_lookup.rs      # match_customer() with URL-first logic
    └── order_fulfillment_client.rs  # ShopifyOrderFulfillmentClient trait + InMemoryShopifyClient

crates/service/src/sync/
├── github_ingest.rs            # ingest_github_recipients() with GithubIngestConfig + IngestDiff
├── shopify_linker.rs           # link_recipients() -> Vec<LinkedRecipient>
├── shopify_projection.rs       # project_shipment_for_customer() -> ShipmentProjection
├── merge.rs                    # merge_recipient(MergeInput) -> MergeResult
├── conflict_notes.rs           # detect_conflicts() + ConflictNote struct
├── backoff.rs                  # BackoffPolicy with delay_for_attempt()
├── scheduler.rs                # SyncScheduler with on_cycle_success/on_cycle_failure
└── status.rs                   # SyncStatusTracker with mark_success/mark_failure/snapshot()

crates/service/src/
├── runtime/startup.rs          # ServiceRuntime integrating scheduler + status
└── api/recipients.rs           # get_recipient_snapshot() -> RecipientSnapshot
```

### Pattern 1: Trait-Bounded Provider Clients
**What:** Provider clients are traits (`GithubProjectClient`, `ShopifyOrderFulfillmentClient`); in-memory fakes implement traits for testing. Real HTTP implementations will implement the same traits.
**When to use:** Everywhere a provider boundary is crossed.
**Example:**
```rust
// crates/integrations/src/github/project_client.rs
pub trait GithubProjectClient {
    fn fetch_rows(
        &self,
        project_id: &str,
        view_id: &str,
    ) -> Result<Vec<GithubProjectRow>, GithubProjectClientError>;
}
```

### Pattern 2: Skip+Warn Mapping (Not Fail-Fast)
**What:** `map_rows` skips rows with missing/invalid `github_profile_url` and emits `MappingWarning` instead of returning an error.
**When to use:** All upstream field validation during ingestion.
**Example:**
```rust
// crates/integrations/src/github/project_mapping.rs
let Some(profile_url) = row.fields.get("github_profile_url") else {
    warnings.push(MappingWarning { item_id: row.item_id.clone(), reason: "missing github_profile_url".to_string() });
    continue;
};
```

### Pattern 3: URL-First Shopify Linkage With Silent No-Match
**What:** `match_customer` tries GitHub profile URL first, falls back to discord handle key match. If no match, `shopify_customer_id` is `None` and `unresolved_fields` contains `"shopify_customer_id"`.
**When to use:** Shopify customer resolution for all linked recipients.

### Pattern 4: Adaptive Scheduler With Clock-Injectable Status
**What:** `SyncScheduler` adapts interval up/down based on cycle outcome. `SyncStatusTracker::snapshot(now: SystemTime)` accepts injected time for deterministic tests.
**Configured defaults:**
```
target_interval: 300s (5 min)
min_interval:    240s (4 min)
max_interval:    900s (15 min)
backoff.base_delay: 15s
backoff.max_delay:  120s
backoff.max_attempts: 3
stale_threshold: 43200s (12 hours)
```

### Pattern 5: Per-Recipient Atomic Merge
**What:** `merge_recipient(MergeInput)` is a pure function. GitHub wins identity fields; Shopify wins commerce/shipment fields. Manual override from `previous.manual_override` takes highest precedence.
**When to use:** Every recipient update in the sync pipeline.

### Anti-Patterns to Avoid
- **Global merge transaction:** Using a single transaction across all recipients causes lock contention and prevents partial progress on failure.
- **Overwriting nulls:** Applying Shopify shipment fields when Shopify linkage failed (shopify_customer_id is None) overwrites valid data with null.
- **Logging raw payloads:** Logging full upstream row/response content risks exposing tokens and PII; log field names and error reason only.
- **Fixed polling interval:** A fixed 5-minute timer does not adapt to network degradation; use `SyncScheduler` which expands on failure.

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Exponential backoff math | Custom retry loops | `BackoffPolicy::delay_for_attempt(attempt)` | Already implemented; saturating arithmetic prevents overflow |
| Stale threshold logic | Manual time comparisons | `SyncStatusTracker::snapshot(now)` | Clock injection prevents test flakiness |
| Recipient key derivation | Custom URL parsing | `map_rows()` in `project_mapping.rs` | Handles normalization, empty-username edge cases |
| Conflict detection | Inline field comparison | `detect_conflicts()` in `conflict_notes.rs` | Handles both-none, one-none, and actual mismatch cases correctly |
| Provider fallback matching | Nested if-let chains | `match_customer()` in `customer_lookup.rs` | URL-first with secondary key fallback in one function |

**Key insight:** All sync primitives are pure functions with no async I/O — they accept trait objects and return value types. This makes them fully testable without mocks or async runtimes.

## Common Pitfalls

### Pitfall 1: Recipient Key Derived From GitHub Username, Not Item ID
**What goes wrong:** Using `github_item_id` as the stable recipient identifier causes identity loss when GitHub Project rows are reordered or reimported.
**Why it happens:** GitHub item IDs are stable within a project but may not match cross-system identity.
**How to avoid:** The implemented `recipient_key` is the normalized lowercase GitHub username extracted from `github_profile_url` (the path segment after `github.com/`).
**Warning signs:** Recipients appear duplicated after sync; `IngestDiff.changed_keys` has unexpected entries.

### Pitfall 2: Partial Failure Overwrites Good Provider Data
**What goes wrong:** If Shopify fetch fails but GitHub succeeds, naive merge clears shipment fields to null.
**Why it happens:** Merge input has null `ShipmentProjection` fields; merge applies them unconditionally.
**How to avoid:** Pass the previous `Recipient` as `MergeInput.previous`; the merge preserves `previous.shipment_status` when the new projection has no data. Currently the merge engine uses Shopify projection unconditionally — the partial failure preservation must be handled at the orchestration layer (don't call `project_shipment_for_customer` when Shopify is unavailable, pass previous projection instead).
**Warning signs:** Shipment status resets to null after a Shopify API error.

### Pitfall 3: `unresolved_fields` Overloaded for Two Semantics
**What goes wrong:** `unresolved_fields` on `Recipient` serves both "Shopify linkage failed" (`"shopify_customer_id"`) and a `"stale"` sentinel checked in `get_recipient_snapshot`. These are semantically different concerns sharing the same Vec.
**Why it happens:** Design choice to avoid a separate stale flag on `Recipient`.
**How to avoid:** Be explicit when adding to `unresolved_fields`; check the exact string `"stale"` for stale detection; do not conflate with linkage failures.
**Warning signs:** `RecipientSnapshot.stale` fires for recipients with Shopify linkage problems.

### Pitfall 4: `on_cycle_failure` Only Respects `max_attempts` Guard in `ServiceRuntime`
**What goes wrong:** `SyncScheduler::on_cycle_failure` itself has no attempt bound — it always expands the interval. The `max_attempts` guard is only enforced in `ServiceRuntime::record_cycle_failure`.
**Why it happens:** The policy check is split between `ServiceRuntime` and `BackoffPolicy`.
**How to avoid:** Always use `ServiceRuntime::record_cycle_failure(attempt, error)` as the entry point; do not call `scheduler.on_cycle_failure` directly.

### Pitfall 5: Shopify Order Selection Is by String Sort of order_id
**What goes wrong:** `project_shipment_for_customer` sorts orders by `order_id` string and picks the last. If `order_id` values are not lexicographically ordered by creation time (e.g., "order-9" > "order-10"), the wrong order is selected.
**Why it happens:** No `created_at` timestamp in `ShopifyOrder`; lexicographic sort used as proxy.
**How to avoid:** When wiring real Shopify HTTP client, include `created_at` in `ShopifyOrder` and sort by it. The in-memory fake uses IDs that happen to sort correctly.

## Code Examples

Verified patterns from actual implementation:

### Full Ingest Pipeline (Data-01)
```rust
// crates/service/src/sync/github_ingest.rs
pub fn ingest_github_recipients(
    client: &dyn GithubProjectClient,
    config: &GithubIngestConfig,
    previous: &[GithubMappedRecipient],
) -> Result<GithubIngestResult, GithubProjectClientError> {
    let rows = client.fetch_rows(&config.project_id, &config.view_id)?;
    let (recipients, warnings) = map_rows(&rows);
    let diff = compute_ingest_diff(previous, &recipients);
    Ok(GithubIngestResult { recipients, warnings, diff })
}
```

### Merge With Override Preservation (Data-02, Data-03)
```rust
// crates/service/src/sync/merge.rs
pub fn merge_recipient(input: MergeInput) -> MergeResult {
    // Manual override wins over GitHub value
    let discord_username = if let Some(override_value) = override_discord {
        Some(override_value)
    } else {
        input.github.discord_username.clone()
    };
    // Shopify wins shipment fields — these come from ShipmentProjection
    MergeResult {
        recipient: Recipient {
            shipment_status: input.shipment.shipment_status,
            tracking_state: input.shipment.tracking_state,
            discord_username,
            // ...
        },
        conflict_notes,
    }
}
```

### Scheduler Adaptive Interval (Data-07)
```rust
// crates/service/src/sync/scheduler.rs
pub fn on_cycle_success(&mut self) -> Duration {
    self.current_interval = self.current_interval.saturating_sub(Duration::from_secs(30));
    if self.current_interval < self.config.min_interval {
        self.current_interval = self.config.min_interval;
    }
    self.current_interval
}

pub fn on_cycle_failure(&mut self, attempt: u32) -> Duration {
    let retry = self.config.backoff.delay_for_attempt(attempt);
    let expanded = self.current_interval.saturating_add(retry);
    self.current_interval = expanded.min(self.config.max_interval);
    self.current_interval
}
```

### Conflict Note Detection
```rust
// crates/service/src/sync/conflict_notes.rs
pub fn detect_conflicts(
    recipient_id: &str,
    github_value: Option<&str>,
    shopify_value: Option<&str>,
    field: &str,
) -> Vec<ConflictNote> {
    // Only conflicts when both sides are Some and differ
    if github_value == shopify_value { return Vec::new(); }
    if github_value.is_none() || shopify_value.is_none() { return Vec::new(); }
    // ...
}
```

## State of the Art

| Old Approach | Current Approach | Impact |
|--------------|------------------|--------|
| Fixed 5-minute poll timer | Adaptive `SyncScheduler` (240-900s range) | Recovers faster from transient failures; backs off during degradation |
| Global sync transaction | Per-recipient `merge_recipient()` pure function | No long locks; partial progress on failure |
| Hard fail on bad rows | Skip+warn pattern in `map_rows()` | Resilient ingest; diagnostics without sync disruption |
| Timestamp display in UI | `SyncState` enum (Healthy/Degraded/Stale) with 12h stale threshold | Cleaner UI; only surfaces warning when genuinely stale |

## Open Questions

1. **Real HTTP clients not yet implemented**
   - What we know: `GithubProjectClient` and `ShopifyOrderFulfillmentClient` are traits with in-memory fakes.
   - What's unclear: GitHub GraphQL schema for Projects v2 fields; Shopify REST vs GraphQL Admin API; authentication header injection patterns.
   - Recommendation: Wire real HTTP clients in a dedicated plan using `reqwest` + `serde_json`; add integration tests with recorded HTTP fixtures.

2. **Shopify order selection by creation time**
   - What we know: Current sort uses lexicographic `order_id` comparison.
   - What's unclear: Whether real Shopify order IDs sort correctly by creation time.
   - Recommendation: Add `created_at: String` (ISO 8601) to `ShopifyOrder` when wiring real client; update `project_shipment_for_customer` to sort by timestamp.

3. **Partial failure: previous projection preservation**
   - What we know: `MergeInput.previous` carries old recipient data; merge engine uses it for override preservation.
   - What's unclear: The sync orchestration layer (which calls `project_shipment_for_customer`) is not yet wired to pass the previous projection on Shopify failure.
   - Recommendation: The orchestration loop should detect Shopify client errors and substitute the previous `ShipmentProjection` for the failed recipient rather than passing empty nulls.

4. **Conflict notes persistence**
   - What we know: `Repository::upsert_conflict_notes` exists in the in-memory repository.
   - What's unclear: Whether conflict notes need to survive application restarts; current `Repository` is in-memory only.
   - Recommendation: When SQLite persistence is introduced, add a `conflict_notes` table alongside the recipients table.

## Validation Architecture

### Test Framework
| Property | Value |
|----------|-------|
| Framework | Rust built-in test runner (`cargo test`) |
| Config file | `Cargo.toml` workspace (no separate test config) |
| Quick run command | `cargo test -p service sync -- --nocapture` |
| Full suite command | `cargo test --workspace` |

### Phase Requirements -> Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| DATA-01 | Skip invalid rows, emit warnings | unit | `cargo test -p service github_ingest_mapping_skips_invalid` | ✅ `crates/service/tests/github_ingest_tests.rs` |
| DATA-01 | Compute new/changed/removed diff | unit | `cargo test -p service github_ingest_tests_computes` | ✅ `crates/service/tests/github_ingest_tests.rs` |
| DATA-02 | URL-first Shopify customer match | unit | `cargo test -p service shopify_sync_tests` | ✅ `crates/service/tests/shopify_sync_tests.rs` |
| DATA-03 | Fulfillment status normalization | unit | `cargo test -p service shopify_sync_tests` | ✅ `crates/service/tests/shopify_sync_tests.rs` |
| DATA-07 | Backoff grows with attempt ceiling | unit | `cargo test -p service scheduler_backoff_grows` | ✅ `crates/service/tests/scheduler_tests.rs` |
| DATA-07 | Stale only after 12h threshold | unit | `cargo test -p service sync_status_policy_only` | ✅ `crates/service/tests/scheduler_tests.rs` |
| DATA-07 | Scheduler adapts interval on failure/success | unit | `cargo test -p service scheduler_tests_adapts` | ✅ `crates/service/tests/scheduler_tests.rs` |
| DATA-01/02/03 | Merge precedence: GitHub identity, Shopify shipment | unit | `cargo test -p service merge_precedence` | ✅ `crates/service/tests/merge_pipeline_tests.rs` |
| DATA-01/02/03 | Manual override preserved over upstream | unit | `cargo test -p service merge_conflicts_and_overrides` | ✅ `crates/service/tests/merge_pipeline_tests.rs` |
| DATA-01/02/03 | Discrepancy warning exposed in API snapshot | unit | `cargo test -p service merge_pipeline_tests` | ✅ `crates/service/tests/merge_pipeline_tests.rs` |

### Sampling Rate
- **Per task commit:** `cargo test -p service sync -- --nocapture`
- **Per wave merge:** `cargo test --workspace`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps
None — existing test infrastructure covers all phase requirements.

## Sources

### Primary (HIGH confidence)
- Implemented source code in `crates/integrations/src/` and `crates/service/src/sync/` — direct code inspection
- `crates/service/tests/` — verified test coverage against requirement IDs
- `crates/service/src/runtime/startup.rs` — confirmed scheduler/status wiring and default parameters

### Secondary (MEDIUM confidence)
- CONTEXT.md decisions — confirmed all locked decisions are implemented as specified

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH — verified from `Cargo.toml` and implemented source files
- Architecture: HIGH — all patterns confirmed in actual code; type signatures documented verbatim
- Pitfalls: HIGH — pitfalls 1-4 confirmed from code inspection; pitfall 5 is a MEDIUM (edge case with current in-memory fake that sorts correctly)

**Research date:** 2026-03-11
**Valid until:** 2026-04-10 (stable Rust ecosystem; re-verify when real HTTP clients are wired)

---

*Phase: 02-external-sync-integrations*
*Research completed: 2026-03-11*
*Ready for planning: yes (all 4 plans already complete)*