---
name: subnet-visibility-lease-regression
description: HFENDULEAM cross-site invisibility RCA — lease-epoch regression freezes roster self-rows fleet-wide; route chain never reaches discovery leg (suspect/reconcile livelock); relay path non-functional
metadata: 
  node_type: memory
  type: project
  originSessionId: 6c0a72dc-b7e0-4516-9fc1-23f61e6c2c50
  modified: 2026-07-31T03:14:12.775Z
---

RCA 2026-07-30 (reported to doyle): HFENDULEAM sees only its own site because the cross-site address-refresh pipeline is dead end to end:

1. **Lease regression**: merge_entry = strictly-greater lease_epoch; nodes' EpochSource counters regressed below their own fleet-replicated self rows (hfenduleam 314616 < 803460, enlyzeam 168 < 690512) → self-advertisements merged Stale forever → roster addresses frozen at dead ports fleet-wide. A node can never repair its own stale advertised address.
2. **Livelock**: resolve chain leg2 (roster) has no failure memory; reconcile_peeraddrs reinstalls the just-failed stale roster addr over a suspect row and clears the mark each exchange → leg3 id-only discovery NEVER runs. Proven live: injected fresh addr into peer-addrs.json, daemon dialed it, failed, and self-reverted to stale within ~2 min.
3. **No fallback**: whole fleet homes on iroh-CANARY relays (config says n0 default — mismatch un-RCA'd); relay path produces no connections. Cross-site inbound UDP also filtered on some hosts (gravity firewall, droplet DO firewall). Connectivity survives only via same-LAN mDNS or live-conn-warmed cache rows.

**Why:** any "why can't node X see node Y" report — check lease_epoch vs epoch counter and whether roster row port matches the peer's live port before suspecting network.
Epoch-regression cause CLOSED (same day, #40 comment 5138846335): next_epoch persists via NON-durable atomic write whose atomic.rs doc explicitly (falsely) classifies epoch counters as reconstructible; hard reset lands in the rename-to-flush window (~every 3s at ~19 consumes/min) → NUL husk → load_from silent 0. Crash-correlated on both nodes via Kernel-Power Event 41 (enlyzeam 11:11:57 Jul 30 = 72s after last winning row; hfenduleam Jul 19 00:21 after Jul 18 20:16 row). No remediation performed — F1 rollout heals; manual counter surgery would mask field verification.

Teardown-oracle lane: fix/teardown-oracle-soundness @ fc2820e pushed (no PR; next thin-lane batch after golden ffs). Deviations ratified: announced-PID keying (spec's session id never reaches SessionsReply) + unified 60s BRINGUP_OBSERVE.

Triage (doyle 2026-07-30): releases#40 = F1-F4 fix set (evidence comment 5138684607 has full trail; peer-addrs backup at %LOCALAPPDATA%\Temp\hertz-peer-addrs-backup-20260730.json until #40 closes), #41 = F5 observability, #42 = relay RCA (blocked-by #41). Product fixes → todlando post-v0.47.0; hertz lanes = teardown-authority test fix first, then epoch-regression-cause RCA + #42.

**How to apply:** stores under %LOCALAPPDATA%\spt-core\identity\ (roster.json, peer-addrs.json, epoch); cross-check from enlyzeam via `ssh decid@enlyzeam`; enlyzeam's peer-addrs holds live observed addrs for its site. Fix proposals F1-F6 in scratchpad hertz-to-doyle-subnet-visibility-rca.md. Related [[cross-machine-test-rig]].