# MESH-RECOVERY — milestone triage (doyle, 2026-07-16)

<!-- [doc->REQ-HAZARD-MESH-BOOTSTRAP-TRAP] -->

**Milestone:** close the peer-address bootstrap deletion trap that let two
nodes sit fully sequestered from their subnets — with green status — while
their rosters held correct direct addresses the entire time.

**Canonical RCA:** `.claude/reports/2026-07-10-hertz-session/02-mesh-isolation-rca.md`
(hertz 2026-07-10; doyle seam-verified on main @37f0b38, 2026-07-16).
**Decision record:** ADR-0039. **Hazard:** KNOWN-HAZARDS 7.42.
**Field incident:** HFENDULEAM + ENLYZEAM symmetric sequester (2026-07-10),
both sides `PUMP_PEER_FAIL:<peer>:submit-dial exceeded the 10s bound` for all
targets, Tailscale ping ~12ms both directions, both `net_up: true`.

## Seam census (verified current, main @37f0b38)

| Seam | Location | Defect |
|---|---|---|
| Unconditional drop-on-fail | `spt-daemon/src/pump/mod.rs:869-881` (`presence_state_effect`, `PRESENCE_DIAL_FAILED` arm → `PeerAddrStore::drop_seed`) | deletes the only warm route after ONE failed dial |
| Cache-else-id-only resolver | `pump/mod.rs:950-959` (`resolve_submit_addr`) | never consults `RosterEntry.address` |
| Refill requires success | `seedproofx.rs:870-877` (`gapfill_peeraddrs`, fill-only) + `:933-937` (runs on seed-proof exchange only) | recovery requires the connectivity the drop destroyed |
| Unvalidated cache write | `spt-store/src/peeraddrs.rs:82-90` (`put`) | accepts outer-key/`address.id` mismatch; poison rows persist (live on both incident nodes) |
| Falsified invariant text | `peeraddrs.rs:14-15,96`, `pump/mod.rs:945` ("a stale addr never strands a peer") | premise falsified in field; amend with the fix |
| Green-while-dead status | `net_up` = local NetHost only; heartbeat written pre-fanout (`pump/mod.rs` round loop); counts = durable membership | incident invisible to every existing surface |

## Build shape — ONE wave (build/mesh-recovery-w1), legs A-D

**Leg A — the route chain + nondestructive retention (REQ-PEER-ROUTE-CHAIN,
REQ-HAZARD-MESH-BOOTSTRAP-TRAP).** `resolve_submit_addr` becomes the chain:
exact cache → validated `RosterEntry.address` (id-match required) → id-only
discovery. `PRESENCE_DIAL_FAILED` stops deleting: demote (suspect-mark or
equivalent) instead of forget; a suspect row is superseded by any validated
fresher address (connect write-back, roster reconcile) and skipped in favor
of the roster leg while suspect — never destroyed while it is the only route.
Startup + roster-merge reconcile validated roster addresses into the cache
(upgrade beyond fill-only: validated-fresher replaces failed/suspect).
Recovery must never require prior success or operator state surgery.

**Leg B — cache invariant + migration (REQ-PEERADDR-INVARIANT).**
Enforce `outer key == address.id` on load and write: repair from current
roster when possible, reject otherwise, loudly. Migration = rebuild invalid
rows from roster on first post-upgrade load — NEVER bare-delete the file
(cold recovery depends on the id-only path staying behind warm routes).

**Leg C — stage truth + user-meaningful health (REQ-PUMP-STAGE-TRUTH).**
Split the 10s failure token into stages: address-resolution, QUIC connect,
ALPN, seed-proof send, seed-proof receive/verify, roster exchange — stamped
(wall+mono) and peer-attributed (folds the 2026-07-14 PUMP_PEER_FAIL-unstamped
seed). `spt daemon status` / `spt subnet status` report: live peer count,
last successful peer dial, last admitted registry update, all-peer-failure
duration. Health = the milestone's acceptance surface, same contract.

**Leg D — regression evidence (production-path, the RCA's list).**
At the REAL pump resolver/failure lifecycle seam:
1. Valid roster + matching cache; one transient dial failure; id-only
   discovery DISABLED → next attempt still holds the roster-derived route.
2. Fail every peer once, restore listeners → automatic convergence, zero
   state surgery.
3. Seed an outer-key/nested-id mismatch → repaired-or-rejected, never used
   as a route.
4. Health degrades during all-peer failure; healthy again only after real
   peer progress (not on heartbeat, not on time).

## Gate rulings (pre-made, binding)

1. **Production-path only.** Regressions ride the real pump round /
   `resolve_submit_addr` / presence-event seam. A test that hand-injects a
   route or calls the store directly proves nothing about the trap.
2. **The chain is static.** Cache → validated-roster → id-only, always fully
   consulted in order. No failure-count heuristics gating which legs exist
   (rejected in ADR-0039); suspect-skip within the cache leg is the only
   state-sensitive behavior.
3. **No deletion of a sole route, ever.** The only removal paths: superseded
   by validated fresher address, or roster tombstone. Show it in a unit: a
   suspect row with no roster/discovery replacement survives N failures.
4. **Validation is id-match.** A roster or cache address whose `id` ≠ the
   peer key resolves NOTHING (poison row can never become a route) — unit
   discriminator required.
5. **Migration preserves fallback.** The W2 rebuild touches invalid rows
   only; valid rows and the absent-file/corrupt-degrades-empty behavior are
   untouched (existing REQ-CONV-1 units must stay green unmodified where
   behavior is unchanged, amended where drop-on-fail semantics changed —
   list the amended ones explicitly in the PR).
6. **Health tells the truth.** No green without real peer progress; the
   incident's exact fingerprint (all dials failing, heartbeat fresh,
   net_up true) must render degraded. New status fields additive (N-1
   readers unaffected).
7. **Doc-truth rides the change.** peeraddrs.rs/pump doc-comments +
   REQ-CONV-1 registry annotation amended in the SAME commits that change
   the mechanism — no stale falsified invariant text at merge.
8. **New daemon-tree e2es enter the HEAVY list at birth** (3 identical
   strings: ci.yml Phase A + Phase B + nextest.toml).
9. **Gate mechanics:** isolated worktree of the gate sha + fresh
   `CARGO_TARGET_DIR` + nextest + workspace clippy + `traceable-reqs check`;
   shared-seam sweep (pump/store seams touch sync/update workers — grep +
   run every seam-sibling suite); cross-platform: both CI runners green.

## Explicitly out of scope (do not build)

- Alternative-address rotation machinery (ADR-0039 rejection).
- F-043 NO_PERCH presence-lag verdict (send layer, root unproven — its own
  triage when perri's falsification pass lands).
- The remaining RCA discriminator (where id-only stalls: QUIC vs seed-proof)
  — leg C's stage split IS the instrumentation that answers it on the next
  field occurrence; no packet-capture work in this milestone.

## Field acceptance (hertz, after release)

1. **Poison-row repair:** both incident nodes' caches held the `5ff…`-outer
   poison row class — post-upgrade first load repairs/rejects it, loudly;
   hertz verifies on the real files.
2. **Health truth:** induced peer outage (stop a remote daemon) renders
   degraded status naming the stage; recovery renders healthy on real
   progress.
3. **Trap regression (best-effort in field):** on a test subnet, drop a
   cache row + block discovery → roster route converges without surgery.

Build-time cross-check (RCA note): compare the 2026-07-14 HFENDULEAM
pump-stall evidence (`C:/spt-diag/pump-stall-2026-07-14/`) against leg D's
lifecycle assertions — restart-cured behavior may be this class.

## Lane

doyle triaged (this doc) → todlando builds (branch `build/mesh-recovery-w1`,
all four REQs, flip `required_stages` at build start per
[[traceable-per-wave-activation]]) → doyle gates (rulings above) → deployah
releases (counter from published metadata, bump-in-PR) → hertz field-verify.