# ADR-0039: Peer-route resolution truth — the route chain, nondestructive retention, and the cache invariant - **Status:** accepted (MESH-RECOVERY triage, 2026-07-16) - **Deciders:** doyle (gater/owner), shaped with hertz (RCA owner, field partner) - **Context:** hertz mesh-isolation RCA 2026-07-10 (`.claude/reports/2026-07-10-hertz-session/02-mesh-isolation-rca.md`), doyle seam-verified on main 2026-07-16; KNOWN-HAZARDS 7.42 ## Context Two nodes (HFENDULEAM, ENLYZEAM) became fully sequestered from every remote member of their subnets — symmetric, self-sustaining, and invisible: both reported `running: true`, `net_up: true`, fresh pump heartbeat, and normal durable subnet counts, while every dial failed on the 10-second bound. The pinned mechanism is a **bootstrap deletion trap** in peer-route resolution: 1. The pump resolves a dial address from the exact `peer-addrs.json` cache entry, else falls back to id-only discovery. It **never consults the valid `RosterEntry.address`** sitting in `roster.json`. 2. Every failed presence dial **unconditionally deletes** the cached entry (`PeerAddrStore::drop_seed` on `PRESENCE_DIAL_FAILED`). 3. The cache refills **only after a future successful seed-proof connection** — which requires the very connectivity the deletion just destroyed. One transient failure therefore converts a warm route into id-only-or-nothing; when id-only discovery is unavailable or stalled, **isolation prevents its own repair**. Both incident rosters held correct, current direct addresses the whole time. REQ-CONV-1's documented premise — *"a stale addr never strands a peer"* — is **falsified in the field** by this shape: the drop-on-fail mechanism intended as a self-heal is the strand. This ADR amends that invariant's mechanism; the sentence stays true only under the new route chain. An independent defect compounds it: `PeerAddrStore::put` validates neither the outer key nor `outer key == address.id`, and `gapfill_peeraddrs` fill-only behavior preserves historical corruption after the roster is corrected — persisted poison rows were live on both incident nodes (outer key of one member mapping another member's address). ## Decision Peer-route resolution and retention carry explicit **route truth**: 1. **The route chain.** Dial-address resolution consults, in order: the exact peer cache entry → the **validated** `RosterEntry.address` for that peer → id-only discovery. A node holding a valid roster address for a peer is never route-less. ("Validated" = the entry's address `id` matches the peer key being resolved — poison rows never become routes.) 2. **Retention is nondestructive.** A failed dial NEVER deletes the only bootstrap route. A cached address that failed is *demoted* (marked suspect / superseded when a fresher source disagrees), not forgotten; the last-known route stays reconstructible. Deletion of a cache row is legitimate only when replaced by a validated fresher address or when the peer leaves the roster (tombstone). 3. **Reconciliation is connection-independent.** Validated roster addresses are reconciled into the cache at daemon startup and after every roster merge — recovery must not require an already-successful connection. (The existing gapfill runs on seed-proof exchange only, and fill-only; reconcile upgrades it: a validated roster address newer than a failed/suspect cache row replaces it.) 4. **The cache invariant is enforced.** `outer peer key == address.id`, on load and on write. Invalid rows are repaired from the current roster when possible, rejected otherwise — never silently kept, and the file is never bare-deleted as "migration" (cold recovery must keep failing over the id-only path, not lose every warm route at once). 5. **Failure truth is staged and health is user-meaningful.** The single 10-second `PUMP_PEER_FAIL` token splits into stages (address resolution, QUIC connect, ALPN, seed-proof send, seed-proof receive/verify, roster exchange), each failure stamped and attributable; daemon/subnet status reports live peer count, last successful peer dial, last admitted registry update, and the duration of any all-peer failure. Truthful health is the acceptance surface of this contract, not a follow-up: the incident was invisible precisely because `net_up`, the heartbeat, and durable counts all answer a different question than "can I reach anyone?". ### Amendment (2026-08-04, releases#75) — Decision 5 gains a THIRD word, and its counts get one population Decision 5's health vocabulary had no way to say **"that machine is off."** `PRESENCE_DIAL_FAILED` fires for every reason there is, including the ordinary powered-down peer, so an operator whose one subnet peer was switched off read `DEGRADED` — a word about *their own* node. The verdict set gains a third word, `peers-absent` (rendered `peers unreachable:`), which is neither a widening of `healthy` nor a softening of `degraded`: - **The discriminator is POSITIVE, and it is about the failing peer.** A failing target counts as absent only when the registry holds rows for its NODE and not one of them is routable. Silence about a node — no rows at all — is not evidence and keeps the conservative verdict. This is deliberately *not* the tempting inference that a live peer proves this node's egress and therefore acquits it of the others' failures: that is one peer's success testifying about a different peer's failure, which is exactly the outvoting logic #41 exists to forbid. Egress to X says nothing about the route to Y. - **The evidence must be fresh.** The registry's `Offline` rows count only while `last_registry_admit_ms` is within the registry's own silent-peer eviction window of the instant the rows were sampled. Past it the registry is already reaping rows for silence, so what it still says is memory rather than observation. No fresh gossip = no basis to call anyone absent. - **The 2026-07-10 sequester fingerprint survives without a special case.** A node that can reach nobody admits no inbound registry feed either, so its admit stamp goes stale and the freshness gate returns it to `DEGRADED` on its own. The guard is structural, not a carve-out. - **Scope.** `peers-absent` replaces a degraded verdict and only ever a degraded verdict. It cannot be reached with an empty failing set (so `healthy` is unreachable from it) and it never replaces `connecting` (a round still working through its first dials has established nothing yet). On the machine surface the token is additive, on the same terms as `degraded-partial`: an older reader meets an unfamiliar token rather than a `degraded` that blames the wrong machine. Decision 5's "reports live peer count" is amended to say **over which population**. The rendered pair was a live count taken from the pump's connection cache over a roster-derived denominator — two different sets, read as one fraction. Leaving a subnet disconnects nobody, so ex-members went on counting in the numerator while the denominator followed the leave: the field specimen was `7/1`. Both ends now count the fan-out targets, the live count being the intersection of the live connections with the current target set, re-derived whenever the targets move (the only path a leave travels). The fix is an **intersection, never a teardown at leave** — a peer can be justified by a second subnet, and dropping its connection would make the count right for the wrong reason. Connections to non-targets remain real and remain reported; they are simply not part of the answer to "how many of the peers I am trying to reach are up". Explicitly **rejected**: - **Alternative-address rotation machinery** (trying N historical addresses per peer): retention + the roster fallback closes the pinned trap; rotation is speculative policy until a post-fix incident demonstrates need. - **Bare deletion of `peer-addrs.json` as the W2 migration**: cold recovery today depends on failing id-only discovery; nuking every warm route trades one trap for another. - **Filtering the route chain by dial-failure origin** (only-roster-after-N -failures heuristics): the chain order is static and always fully consulted — simplicity is the guard against the next ordering trap. ## Consequences - `REQ-CONV-1`'s registry annotation gains the ruling pointer; its "stale addr never strands" sentence is guaranteed by the chain + retention, no longer by drop-on-fail. The `peeraddrs.rs` / `pump/mod.rs` doc-comments that state the falsified mechanism are amended in the same change that lands the chain. - New requirements: `REQ-HAZARD-MESH-BOOTSTRAP-TRAP` (the invariant), `REQ-PEER-ROUTE-CHAIN` (chain + retention + reconcile), `REQ-PEERADDR-INVARIANT` (enforce/repair/migrate), `REQ-PUMP-STAGE-TRUTH` (stage split + truthful health). - The regression surface rides the PRODUCTION pump resolver/failure lifecycle seam (KNOWN-HAZARDS 7.42): transient dial failure with id-only discovery disabled must still converge via the roster route, with no state surgery. ## Kin (verified distinct, not folded) - **F-043 presence-lag NO_PERCH silent loss** (perri field): the *send verdict* layer, not the pump route chain — root unproven, falsification pass pending; mint at its own triage. - **0.30.1 peer-pump UPDATE-leg reply-read poison** (fixed): same pump, a different leg — the reply-read path, not route resolution. - **PUMP_PEER_FAIL unstamped** (2026-07-14 seed): folded here — the W3 stage split subsumes it.