---
name: mesh-recovery-progress
description: "MESH-RECOVERY milestone ledger: triage @0bb37f3 DONE 2026-07-16, todlando dispatched W1 (4 REQs), hertz field-verify partner; drive to release without operator waits."
metadata: 
  node_type: memory
  type: project
  originSessionId: a2fdb848-6643-4e10-b221-6baee7e91b7b
---

**MESH-RECOVERY** — peer-address bootstrap deletion trap: `drop_seed` on every failed dial + resolver that never consults `RosterEntry.address` + cache that refills only on successful seed-proof = self-sustaining green-status isolation (hertz RCA 2026-07-10, HFENDULEAM+ENLYZEAM symmetric sequester; doyle seam-verified on main 2026-07-16). REQ-CONV-1's "stale addr never strands" premise FALSIFIED in its drop-on-fail shape. Plus independent poison-row defect: `put` validates nothing, outer key ≠ address.id rows persisted live on both incident nodes.

**Status 2026-07-16:**
- ✅ TRIAGE @0bb37f3 on main: docs/MESH-RECOVERY-TRIAGE.md (seam census verified current + build shape legs A-D + gate rulings 1-9 pre-made + field acceptance) + ADR-0039 (route chain / nondestructive retention / cache invariant / stage truth; rejections: rotation machinery, bare-delete migration, failure-count chain heuristics) + KNOWN-HAZARDS 7.42 + CONTEXT peer-route-chain glossary + 4 REQs registry-first (required_stages=[], flip at build): REQ-HAZARD-MESH-BOOTSTRAP-TRAP (invariant + production-path int), REQ-PEER-ROUTE-CHAIN (A: chain+demote+reconcile), REQ-PEERADDR-INVARIANT (B: enforce+repair+migration, never bare-delete), REQ-PUMP-STAGE-TRUTH (C: stage split + truthful health = acceptance surface; subsumes PUMP_PEER_FAIL-unstamped seed). REQ-CONV-1 annotation amended (falsified mechanism pointer).
- ✅ todlando DISPATCHED W1 (SENT live 2026-07-16): branch build/mesh-recovery-w1, single build wave, all 4 REQs.
- GOTCHA this session: found HEAD on fix/changelog-0330-heading when committing triage (gh pr merge #15 left checkout on the PR branch) — cherry-picked onto real main, pushed, stray branch deleted. CHECK `git branch --show-current` before committing to main.
- ✅ **W1 BUILT (todlando 2026-07-16): PR #16 @18489c5, PR-READY, doyle pinged SENT, CI in flight.** All 4 REQs flipped+satisfied, single commit. Build shape: A = resolve_submit_addr chain (valid_route cache → roster_route id-match → resolver) + demote_seed on DIAL_FAILED (PeerAddrStore.suspect BTreeSet, additive serde) + reconcile_peeraddrs (gapfill upgraded: fills absent, replaces suspect/invalid from validated roster) at pump startup (startup_reconcile_peeraddrs, idempotent per supervised restart) + roster-merge sink; wansend seed_first_addr → valid_route too. B = put refuses id-mismatch loudly (PEERADDR_INVARIANT_REJECT) + startup migration repair-else-drop-loud, never bare-delete. C = seedproofx::DialStage cell through connect+prove (stages: address-resolution/quic-connect/alpn/seed-proof-send/seed-proof-recv/roster-exchange; outer timeout reads the cell) + stamped PUMP_PEER_FAIL wall+mono + pump/health.rs PumpHealth (identity/pump-health.json beside heartbeat; RegistryHost::last_admit_ms AtomicU64) + daemon/subnet status render (render_peer_health_line + verdict token, additive JSON both surfaces). D = tests/mesh_recovery.rs 3/3 HEAVY-at-birth ×3 strings, real pump + real QUIC + B production dispatcher.
- **BUILD GOTCHAS:** (1) pump legs PULL against the peer — a dispatcher-less peer broker WEDGES the round mid-legs (health file froze; looked like note_connected never ran) → int rigs MUST run run_dispatch_loop on the serving side (pump.rs precedent). (2) **iroh accumulates dial candidates per endpoint id**: stale cache addr at a LIVE wrong-identity socket poisons later dials for that id (UnknownIssuer even on the correct roster addr) — deterministic repro in the D1 rig's first shape; SEED for own triage (kin poison-row class), disclosed in PR + to doyle. Dead-socket transient + set_quic_op_timeout(1.5s) = the clean test shape. (3) Tightened converge predicates caught two trivially-true assertions (route-present-at-start; repair-without-connect) — require health.live_peers>0 (real QUIC connect) as convergence proof.
- Amended tests (ruling 5, disclosed): resolve_submit_addr_prefers_seed_then_heals_stale → resolve_submit_addr_walks_the_route_chain; gapfill unit → reconcile triple; wansend seed_first unit +suspect-defers.
- Preflight @18489c5: spt-daemon 669/670 (wan_arrival standing class only), spt 408/408, spt-store 246/246, mesh_recovery 3/3, redispatch 5/5, clippy 0, traceable 0, xtask 0.
- ⚠ **DOYLE GATE ROUND 1 (2026-07-16): source review PASSED, ONE small must-fix** (PR #16 comment issuecomment-4997398035, todlando pinged SENT): `note_registry_admit(None)` clobbers the persisted last_registry_admit_ms every post-restart round (RegistryHost::last_admit_ms = process-lifetime AtomicU64, 0→None; skip_serializing then drops it from disk) — DEGRADED render shows 'last admitted: never' exactly when operator diagnoses post-restart; contradicts health.rs's own survive-restart header. FIX ASK: keep prior on None (or max(prior,incoming)) + restart-shape unit. ACCEPTED everything else: chain static+id-match every leg, demote+suspect persist+old-file compat, reconcile fill/replace-suspect/repair-invalid, migration never-bare-delete, DialStage cell w/ timeout reading it, stamped stage-attributed failures, PumpHealth pure verdict unit-proven, additive JSON both surfaces + subnet human line, wansend valid_route, D-legs real-pump/real-QUIC/retention-every-sample, HEAVY ×3, flips precise, roster-path = round-ctx precedent.
- ✅ **ROUND-1 FIX PUSHED @66d9dc6 (todlando 2026-07-16), doyle pinged SENT.** note_registry_admit: None keeps prior stamp, Some takes max (never regresses); fn doc carries the round-1 pointer; new unit registry_admit_stamp_survives_restart_rounds_and_never_regresses (persist→reload→None-rounds keep on disk→fresh advances→older refused). Health units 6/6, clippy 0, traceable 0.
- ⚠ **CI red ×2 @66d9dc6 = GITHUB INCIDENT, not code (doyle grounded 2026-07-16):** both platforms' suites fully green both runs (kitsubito 1729+120, hfenduleam 1742+121; mesh_recovery green Phase B both); failing step = actions/checkout SaberMage/spt-shell-notify (dogfood rung) 'Retrieving the default branch name' → GitHub HTML 50x, 3 retries, both boxes; repo verified healthy from dev box (API+web 200); githubstatus.com = ACTIVE 'Partially Degraded Service'. Rerun deferred until doyle heavy leg done + incident clears. SEED noted: dogfood rung still checkouts SaberMage org post-FORKENING — move to BigscreenVR or vendor (CI-reliability rider).
- ✅ **GATE CLOSED, PR #16 MERGED @7f5ead8 (doyle 2026-07-16).** Heavy leg @66d9dc6 ALL GREEN (isolated worktree, fresh target): clippy 0, traceable OK, spt-store 15/15, spt-daemon units 45/45 (reconcile triple + suspect-supersede), wansend 9/9, heavy ints 29/29 (mesh_recovery + pump family + sync + redispatch shared-seam sweep). CI green both runners on rerun 3 after GitHub incident cleared (reds ×2 = dogfood checkout only; suites green every run). Gate worktree removed clean. GOTCHA ×2 now: local checkout AGAIN found on the PR branch at merge time (gh/hook side-effect unresolved) — always `git branch --show-current` before/after gh pr merge.
- ✅ **GO SENT deployah (2026-07-16 SENT live): counter 62**, bs-releases, bump-in-PR, CHANGELOG scope vs v0.34.0 (FIXED: no self-strand after transient dial fail — demote+roster-fallback+zero-surgery recovery; poison-row startup repair. ADDED: peer-reachability health both status surfaces w/ DEGRADED-naming-stage, stamped stage-attributed dial failures. Additive, no wire changes). Ladder self-check + GitHub-residue rerun warning included. todlando pinged gate-closed.
- ✅ todlando post-merge housekeeping (2026-07-16): main synced @7f5ead8, mirror pushed (SaberMage/spt-core 01d2d18→7f5ead8), W1 branch gone. **Build lane COMPLETE**; staying reachable for deployah/hertz reader questions.
- ✅ **v0.35.0 PUBLISHED counter 62 (deployah 2026-07-16)** — latest on bs-releases, CI green first try, ladder self-check passed, signed from isolated worktree (shared tree held todlando's redispatch-stall-w1 checkout, untouched). See [[v0350-published]].
- ✅ **hertz field-verify GO SENT (doyle 2026-07-16)** with the acceptance triple + known-benign counter-reset note.
- ✅ **MILESTONE CLOSED 2026-07-17 on hertz 3/3 field PASS (doyle adjudicated same day):** Leg 1 poison-row selective repair both incident nodes (5ff HFENDULEAM / 39c KITSUBITO, valid rows untouched, no bare-delete; pre/post json artifacts). Leg 2 health truth — all-peer outage rendered DEGRADED naming quic-connect w/ fresh heartbeat (the incident fingerprint), healthy ONLY on real progress; stall_evict=1 = induced residue, benign class. Leg 3 trap — absent-cache convergence zero surgery, validated write-back; field caveat ACCEPTED per pre-made ruling (no production switch to disable id-only discovery — the strict discovery-disabled arm lives in the hermetic mesh_recovery int, D-leg 1, gated green). RESOLVED en route: HFENDULEAM daemon-stop during tests = hertz's controlled leg-2 suspension, benign/induced (todlando told). SEED (low-pri, update-tooling): remote update systemd stop leg exit 1 then start-reports-already-running — self-healed, no half-update; fold at next release-tooling wave.
- Next: REDISPATCH-STALL W1 in flight ([[redispatch-stall-progress]], counter 63). Then DAEMON-LIFECYCLE cluster (3-report hertz set, ranked first).

**Key seams:** pump/mod.rs:869-881 (DIAL_FAILED drop_seed), :950-959 (resolve_submit_addr cache-else-id-only), seedproofx.rs:870-877+933-937 (gapfill fill-only, seed-proof-gated), peeraddrs.rs:82-90 (put unvalidated), :14-15/96 + pump/mod.rs:945 (falsified doc-comments — amend rides build, gate ruling 7).

**Explicitly out of scope:** rotation machinery (ADR-0039 rejected); F-043 NO_PERCH presence-lag (send layer, own triage after perri falsification); id-only stall discriminator (leg C's stages ARE the future instrumentation).

**Field acceptance (hertz, triple):** (1) poison-row (5ff…-outer class) repair on real caches at first post-upgrade load, loud; (2) induced peer outage renders degraded naming the stage, healthy only on real progress; (3) best-effort: cache-row drop + blocked discovery converges via roster route, zero surgery. Build-time cross-check: compare C:/spt-diag/pump-stall-2026-07-14/ evidence vs leg D assertions.