1	//! Outbound peer pump (D9-1-2) — the production trigger loops every D5–D8
2	//! slice deferred ("rides D9 with the daemon lifecycle").
3	//!
4	//! One daemon-hosted loop drives three cadenced pumps against every trusted
5	//! peer of every member subnet, each a thin call over an already-shipped,
6	//! idempotent driver — the pump owns *when*, the drivers own *how*:
7	//!
8	//! - **registry push** ([`RegistryHost::advertise_local`] → one feed stream
9	//!   per peer): the node re-advertises its own rows; peers reconverge within
10	//!   one cadence (replay-is-recovery, the D4d/D8 push posture). Own rows
11	//!   only — no transitive gossip in v1: trust is pairwise per subnet, and a
12	//!   relayed third-party row would carry an origin the receiver cannot gate
13	//!   honestly (KNOWN-HAZARDS 7.5).
14	//! - **notif push** ([`crate::notifsync::emit_notif_feed`]): the full
15	//!   semilattice spool per subnet; the receiver's join dedups.
16	//! - **sync pull** ([`crate::sync::request_sync`]): anti-entropy for the
17	//!   minds. Want-refs are the union of [`crate::sync::select_refs`] (locally
18	//!   tracked branches the policy allows toward the peer) and
19	//!   **registry-derived** `a-<id>` refs (endpoints known in the shared
20	//!   subnets — how a fresh instance bootstraps a mind it has never held),
21	//!   both gated by the same [`SyncPolicy::allows`] the responder enforces
22	//!   (symmetric scoping).
23	//!
24	//! Conflicts a pull surfaces are reported through [`PumpHooks::on_pull`] —
25	//! the default production hook logs them loud. The bounded Psyche reconcile
26	//! turn itself still runs at the activation paths (ADR-0013): the daemon
27	//! does not yet host per-agent manifest runtimes (the 7.4 fan-out), so an
28	//! in-pump turn has no runtime to ride; conflict artifacts persist safely
29	//! meanwhile (REQ-HAZARD-CONFLICT-BOTH-PRESERVED).
30	//!
31	//! ## Wire discipline (inherited from the test suites, now production)
32	//!
33	//! Every pump write is fire-and-forget (`op_id: None`) — a journaled
34	//! `net_stream_send` blocks on its ack and DISCARDS interleaved events
35	//! (gotcha #1). Stream opens for push feeds carry no op either: the feeds
36	//! are idempotent by construction. The sync pull's `open_op` IS durable
37	//! (exactly-once open identity), minted from a dedicated [`EpochSource`]
38	//! file so a daemon restart can never reuse one.
39	// [impl->REQ-INST-5]
40	// [impl->REQ-INST-7]
41	// [impl->REQ-NOTIF-1]
42
43	use std::collections::HashMap;
44	use std::io;
45	use std::path::PathBuf;
46	use std::sync::atomic::{AtomicBool, Ordering};
47	use std::sync::Arc;
48	use std::thread;
49	use std::time::{Duration, Instant};
50
51	use spt_net::net::endpoint::addr_for_node_hex;
52	use spt_store::contextstore::ContextStore;
53	use spt_store::epoch::EpochSource;
54	use spt_store::notif::NotifStore;
55	use spt_store::perch;
56	use spt_store::subnet::SubnetStore;
57	use spt_store::trust::TrustStore;
58	use spt_store::visibility::VisibilityStore;
59
60	use crate::brain::Brain;
61	use crate::config::DaemonConfig;
62	use crate::consent::{decide, most_recently_active};
63	use crate::notif::{produce_consent_notif, NotifSurfacePolicy};
64	use crate::notifsync::emit_notif_feed;
65	use crate::propagate::{request_update, UpdatePullOutcome};
66	use crate::registryhost::RegistryHost;
67	use crate::relcache::ReleaseCache;
68	use crate::release::VerifyPolicy;
69	use crate::sync::{request_sync, select_refs, SyncPolicy, SyncPullReport};
70	use crate::update::BrokerAbi;
71
72	/// The pump's tick granularity — each cadence fires when its period elapsed.
73	const TICK: Duration = Duration::from_millis(200);
74
75	/// Heartbeat write throttle (M8 decision 23, REQ-DAEMON-5): the stall
76	/// warning reads 3-cadence staleness, so a few seconds of heartbeat
77	/// granularity is invisible to it — and 5 writes/sec of disk churn is not.
78	const HEARTBEAT_PERIOD: Duration = Duration::from_secs(5);
79
80	/// Supervisor backoff floor: the first restart after a pump death.
81	const SUPERVISE_BACKOFF_BASE: Duration = Duration::from_secs(5);
82
83	/// Supervisor backoff cap (M8 decision 23): a persistently-crashing pump
84	/// retries at most every 5 minutes — visible, bounded, never silent.
85	const SUPERVISE_BACKOFF_CAP: Duration = Duration::from_secs(300);
86
87	/// A run that survived this long before dying counts as healthy — the next
88	/// backoff resets to the floor instead of doubling (a once-a-day panic must
89	/// not ratchet the pump toward permanent 5-minute outages).
90	const SUPERVISE_HEALTHY_RUN: Duration = Duration::from_secs(60);
91
92	/// Resolve a trusted peer's stable name (lowercase-hex pubkey) to a dialable
93	/// address (the serde JSON `net-dial` carries). Production resolution lets
94	/// discovery do the work ([`production_resolver`]); hermetic tests inject a
95	/// concrete address map (loopback endpoints run no discovery).
96	pub type PeerResolver = Box<dyn Fn(&str) -> Option<serde_json::Value> + Send>;
97
98	/// The production resolver: an id-only iroh address — mDNS on the LAN, n0
99	/// DNS on the WAN (`addr_for_node_hex`).
100	pub fn production_resolver() -> PeerResolver {
101	    Box::new(|hex| addr_for_node_hex(hex).and_then(|a| serde_json::to_value(a).ok()))
102	}
103
104	/// Where this node's pump-side stores live. Production = the canonical
105	/// SPT_HOME layout; the loopback E2E gives its in-process "node" distinct
106	/// roots so two nodes can share one test process.
107	pub struct PumpPaths {
108	    /// The perch tree (registry advertisement scan).
109	    pub owlery: PathBuf,
110	    /// The context store root ([`ContextStore::open_or_init_in`]).
111	    pub ctx_root: PathBuf,
112	    /// The notif spool DB ([`NotifStore::open_at`]).
113	    pub notif_db: PathBuf,
114	    /// Transient pull scratch.
115	    pub scratch: PathBuf,
116	    /// `subnet.json` ([`SubnetStore::load_from`]).
117	    pub subnets: PathBuf,
118	    /// `trust/peers.json` ([`TrustStore::load_from`]).
119	    pub trust: PathBuf,
120	    /// `visibility.json` ([`VisibilityStore::load_from`]).
121	    pub vis: PathBuf,
122	    /// The pump's durable open-op counter (its own file — colliding with the
123	    /// registry's advertisement epochs would break neither, but a shared
124	    /// in-memory cursor pair over one file double-mints; separate files keep
125	    /// each `EpochSource` the sole writer of its counter).
126	    pub ops: PathBuf,
127	    /// The staged-release cache dir ([`ReleaseCache::open`]).
128	    pub releases: PathBuf,
129	    /// `release-keys.json` ([`VerifyPolicy::load_from`]).
130	    pub release_keys: PathBuf,
131	    /// The wake-edge freshness-pull marker (D9-2, REQ-INST-3): a
132	    /// dormant→active transition drops it; the pump consumes it and forces a
133	    /// sync round NOW instead of waiting out `cadence.sync`
134	    /// ([`crate::resting::PULL_MARKER_FILE`]).
135	    pub pull_marker: PathBuf,
136	    /// The event-driven advertisement marker (M8 decision 15, REQ-CONV-2):
137	    /// an endpoint online/offline transition drops it; the pump consumes it
138	    /// and forces a registry round NOW instead of waiting out
139	    /// `cadence.registry` ([`crate::registryhost::ADVERTISE_MARKER_FILE`]).
140	    pub advertise_marker: PathBuf,
141	    /// `peer-addrs.json` — last-known peer addresses (M8 decisions 14/20,
142	    /// REQ-CONV-1): consulted before id-only discovery, written back after
143	    /// every successful dial ([`spt_store::peeraddrs::PeerAddrStore`]).
144	    pub peer_addrs: PathBuf,
145	    /// The pump's last-tick heartbeat (M8 decision 23, REQ-DAEMON-5):
146	    /// epoch-ms written while the loop is alive, rendered by
147	    /// `spt daemon status` / the `subnet status` stall warning
148	    /// ([`crate::endpoint::pump_heartbeat_path`]).
149	    pub heartbeat: PathBuf,
150	}
151
152	impl PumpPaths {
153	    /// The canonical SPT_HOME layout.
154	    pub fn production() -> PumpPaths {
155	        let home = perch::spt_home();
156	        let identity = perch::identity_dir();
157	        PumpPaths {
158	            owlery: perch::owlery_dir(),
159	            ctx_root: home.join("context"),
160	            notif_db: identity.join("notifs.db"),
161	            scratch: home.join("scratch").join("pump"),
162	            subnets: identity.join("subnet.json"),
163	            trust: identity.join("trust").join("peers.json"),
164	            vis: identity.join("visibility.json"),
165	            ops: identity.join("pump-ops.json"),
166	            releases: home.join("releases"),
167	            release_keys: identity.join("release-keys.json"),
168	            pull_marker: identity.join(crate::resting::PULL_MARKER_FILE),
169	            advertise_marker: identity.join(crate::registryhost::ADVERTISE_MARKER_FILE),
170	            peer_addrs: identity.join("peer-addrs.json"),
171	            heartbeat: crate::endpoint::pump_heartbeat_path(),
172	        }
173	    }
174	}
175
176	/// Pump cadences (config-driven in production: [`DaemonConfig`]).
177	#[derive(Debug, Clone, Copy)]
178	pub struct PumpCadence {
179	    pub registry: Duration,
180	    pub notif: Duration,
181	    pub sync: Duration,
182	    pub update: Duration,
183	    /// Silent-peer registry eviction window (KNOWN-HAZARDS 4.10): swept each
184	    /// registry tick, before re-advertisement.
185	    pub registry_evict_after: Duration,
186	}
187
188	impl PumpCadence {
189	    /// From the daemon config knobs.
190	    pub fn from_config(cfg: &DaemonConfig) -> PumpCadence {
191	        PumpCadence {
192	            registry: cfg.registry_pump_period,
193	            notif: cfg.notif_pump_period,
194	            sync: cfg.sync_pull_period,
195	            update: cfg.update_check_period,
196	            registry_evict_after: cfg.registry_evict_after,
197	        }
198	    }
199	}
200
201	/// The pump's resolved knobs: cadences + the REQ-UPD-4 consent stance.
202	#[derive(Debug, Clone, Copy)]
203	pub struct PumpConfig {
204	    pub cadence: PumpCadence,
205	    /// `true` = the explicit opt-in to prompt-free updates; default `false`
206	    /// surfaces every staged release as a consent notif.
207	    pub full_auto_update: bool,
208	}
209
210	impl PumpConfig {
211	    /// From the daemon config knobs.
212	    pub fn from_config(cfg: &DaemonConfig) -> PumpConfig {
213	        PumpConfig {
214	            cadence: PumpCadence::from_config(cfg),
215	            full_auto_update: cfg.full_auto_update,
216	        }
217	    }
218	}
219
220	/// A pull observer: `(peer_hex, report)` after every completed sync pull.
221	pub type OnPull = Box<dyn FnMut(&str, &SyncPullReport) + Send>;
222
223	/// Observation hooks. `on_pull` fires after every completed sync pull —
224	/// the conflict surface (ADR-0013) and the E2E's probe.
225	pub struct PumpHooks {
226	    pub on_pull: OnPull,
227	}
228
229	impl Default for PumpHooks {
230	    /// The production hook: conflicts are loud, never silent — the artifacts
231	    /// already persist (hazard 6.6); the reconcile turn runs at activation.
232	    fn default() -> PumpHooks {
233	        PumpHooks {
234	            on_pull: Box::new(|peer, report| {
235	                for (branch, file) in report.conflicts() {
236	                    eprintln!("SYNC_CONFLICT:{peer}:{branch}:{file}");
237	                }
238	            }),
239	        }
240	    }
241	}
242
243	/// Run the pump until `stop` is set. Owns one brain IPC connection and a
244	/// per-peer connection cache (an IO failure against a peer drops its cache
245	/// row; the next tick redials — conns are rebuilt freely, nothing leaks
246	/// because the broker's closed-watcher reaps dead conns).
247	pub fn run_peer_pump(
248	    broker_name: &str,
249	    registry: Arc<RegistryHost>,
250	    paths: &PumpPaths,
251	    resolver: PeerResolver,
252	    config: PumpConfig,
253	    mut hooks: PumpHooks,
254	    stop: &AtomicBool,
255	) -> io::Result<()> {
256	    let PumpConfig {
257	        cadence,
258	        full_auto_update,
259	    } = config;
260	    let mut brain = connect(broker_name)?;
261	    let mut ops = EpochSource::load_from(&paths.ops);
262	    let mut conns: HashMap<String, u64> = HashMap::new();
263	    // Stagger from "everything due now" — the first tick primes peers.
264	    // `None` = never run yet → due immediately. Avoids the `Instant -
265	    // Duration` underflow panic on hosts whose uptime is below the offset
266	    // (Windows monotonic clock can't represent an instant before boot).
267	    let (mut last_reg, mut last_notif, mut last_sync, mut last_upd): (
268	        Option<Instant>,
269	        Option<Instant>,
270	        Option<Instant>,
271	        Option<Instant>,
272	    ) = (None, None, None, None);
273	    let mut last_hb: Option<Instant> = None;
274
275	    while !stop.load(Ordering::Relaxed) {
276	        let now = Instant::now();
277	        // Liveness heartbeat (M8 decision 23, REQ-DAEMON-5): written while
278	        // the loop is ALIVE — including idle ticks — never per work round,
279	        // so a wedged peer_io and a healthy quiet pump are distinguishable.
280	        // Throttled: the stall warning reads 3-cadence staleness, sub-second
281	        // precision buys nothing but disk churn.
282	        // [impl->REQ-DAEMON-5]
283	        if due(last_hb, now, HEARTBEAT_PERIOD) {
284	            write_heartbeat(&paths.heartbeat);
285	            last_hb = Some(now);
286	        }
287	        // Event-driven advertisement (M8 decision 15, REQ-CONV-2): an
288	        // endpoint online/offline transition dropped the marker — consume it
289	        // and run the registry leg NOW. A wake of THIS loop, not a second
290	        // advertisement path: the forced round is the same advertise_local +
291	        // push code, so the epoch lease and visibility gates ride unchanged.
292	        // [impl->REQ-CONV-2]
293	        let wake_adv = crate::registryhost::take_advertise_now(&paths.advertise_marker);
294	        let due_reg = wake_adv || due(last_reg, now, cadence.registry);
295	        let due_notif = due(last_notif, now, cadence.notif);
296	        // The wake edge's freshness pull (D9-2, REQ-INST-3): a dormant→active
297	        // transition dropped the marker — consume it and pull NOW, the
298	        // catch-up-on-activation latency is the point. The take is the remove,
299	        // so one wake claims exactly one forced round.
300	        // [impl->REQ-INST-3]
301	        let wake_pull = crate::resting::take_freshness_pull(&paths.pull_marker);
302	        let due_sync = wake_pull || due(last_sync, now, cadence.sync);
303	        let due_upd = due(last_upd, now, cadence.update);
304	        if !(due_reg || due_notif || due_sync || due_upd) {
305	            thread::sleep(TICK);
306	            continue;
307	        }
308
309	        // Re-load config-ish stores per round (the access-gate freshness
310	        // stance — edits take effect without a restart).
311	        let subnets = SubnetStore::load_from(&paths.subnets);
312	        // Serve-state (M8 decision 6, REQ-SUBNET-5): a detached subnet is
313	        // neither advertised into nor connected to — skipped wholesale in
314	        // the per-subnet loop below. Re-read per round like every other
315	        // config-ish store, so a flip takes effect within one tick.
316	        // [impl->REQ-SUBNET-5]
317	        let attachment = spt_store::attachment::AttachmentStore::load_from(
318	            &paths
319	                .subnets
320	                .parent()
321	                .map(|p| p.join("attachment.json"))
322	                .unwrap_or_else(|| PathBuf::from("attachment.json")),
323	        );
324	        // The push target stays the directly-paired trusted set at D5 (the
325	        // fan-out widens to all roster members only at D6); the sync *gate*,
326	        // however, now authorizes on roster membership (Mesh-D5, REQ-MESH-5).
327	        let trust = TrustStore::load_from(&paths.trust);
328	        let roster_path = paths
329	            .subnets
330	            .parent()
331	            .map(|p| p.join("roster.json"))
332	            .unwrap_or_else(|| PathBuf::from("roster.json"));
333	        let policy = SyncPolicy {
334	            subnets: subnets
335	                .subnets
336	                .iter()
337	                .map(|s| (s.name.clone(), s.hide_new_endpoints))
338	                .collect(),
339	            roster: spt_store::roster::RosterStore::load_from(&roster_path),
340	            vis: VisibilityStore::load_from(&paths.vis),
341	        };
342	        let adverts = if due_reg {
343	            // Consume the re-pair eviction marker FIRST (M8 decision 13,
344	            // REQ-SUBNET-7): a superseded identity's in-memory rows go
345	            // before this tick's snapshot mirror can resurrect the pairing
346	            // host's disk-side eviction.
347	            // [impl->REQ-SUBNET-7]
348	            registry.consume_repair_evictions();
349	            // Sweep silent-node ghost rows BEFORE re-advertising (KNOWN-
350	            // HAZARDS 4.10): a dead node identity's rows decay instead of
351	            // poisoning bare-id resolution forever; a live peer re-stamps its
352	            // heard entry with every admitted feed and is never touched.
353	            // [impl->REQ-HAZARD-REGISTRY-GHOST-ROWS]
354	            registry.evict_silent_peers(cadence.registry_evict_after);
355	            registry.advertise_local(&paths.owlery)
356	        } else {
357	            Vec::new()
358	        };
359
360	        for sub in &subnets.subnets {
361	            if attachment.is_detached(&sub.name) {
362	                continue; // detached: neither advertise into nor connect to (REQ-SUBNET-5)
363	            }
364	            for peer in trust.peers_in(&sub.name) {
365	                if peer.pubkey_hex == registry.node_hex() {
366	                    continue; // never pump at ourselves
367	                }
368	                let Some(conn_id) = ensure_conn(
369	                    &mut brain,
370	                    &mut conns,
371	                    &resolver,
372	                    &peer.pubkey_hex,
373	                    &paths.peer_addrs,
374	                ) else {
375	                    continue; // unresolvable/undialable now — next tick retries
376	                };
377
378	                let mut peer_io = || -> io::Result<()> {
379	                    if due_reg {
380	                        let lines: Vec<u8> = adverts
381	                            .iter()
382	                            .filter(|u| u.subnet() == sub.name)
383	                            .flat_map(|u| u.encode_line())
384	                            .collect();
385	                        if !lines.is_empty() {
386	                            push_feed(&mut brain, conn_id, &lines)?;
387	                        }
388	                    }
389	                    if due_notif {
390	                        let store = NotifStore::open_at(&paths.notif_db)
391	                            .map_err(|e| io::Error::other(e.to_string()))?;
392	                        let feed = emit_notif_feed(&store, &sub.name).map_err(io::Error::other)?;
393	                        if !feed.is_empty() {
394	                            push_feed(&mut brain, conn_id, &feed)?;
395	                        }
396	                    }
397	                    if due_sync {
398	                        let cs = ContextStore::open_or_init_in(&paths.ctx_root)?;
399	                        let mut want = select_refs(&policy, &cs, &peer.pubkey_hex)?;
400	                        // Registry-derived bootstrap refs: minds named in the
401	                        // shared subnet that this node never held locally.
402	                        if let Some(snap) = registry.snapshot(&sub.name) {
403	                            for id in snap.endpoint_ids() {
404	                                let r = format!("a-{id}");
405	                                if !want.contains(&r) && policy.allows(&r, &peer.pubkey_hex) {
406	                                    want.push(r);
407	                                }
408	                            }
409	                        }
410	                        if !want.is_empty() {
411	                            let open_op = ops.next_epoch()?;
412	                            let report = request_sync(
413	                                &mut brain,
414	                                conn_id,
415	                                &want,
416	                                open_op,
417	                                &cs,
418	                                &paths.scratch,
419	                            )?;
420	                            (hooks.on_pull)(&peer.pubkey_hex, &report);
421	                        }
422	                    }
423	                    if due_upd {
424	                        // The peer-propagation pull (REQ-UPD-1) under the
425	                        // PRODUCTION gate: keys + channel pin from
426	                        // release-keys.json (fail-closed when unprovisioned),
427	                        // version floor = the staged release (rollback gate),
428	                        // wall clock for expiry only.
429	                        let cache = ReleaseCache::open(&paths.releases);
430	                        let base_policy = VerifyPolicy::load_from(&paths.release_keys, 0, now_ms());
431	                        let current = if cache.staged_channel().as_deref()
432	                            == Some(base_policy.pinned_channel.as_str())
433	                        {
434	                            cache.staged_version().unwrap_or(0)
435	                        } else {
436	                            0
437	                        };
438	                        let policy = VerifyPolicy {
439	                            current_version: current,
440	                            ..base_policy
441	                        };
442	                        if !policy.trusted_keys.is_empty() {
443	                            let open_op = ops.next_epoch()?;
444	                            match request_update(
445	                                &mut brain,
446	                                conn_id,
447	                                open_op,
448	                                &BrokerAbi::current(),
449	                                &policy,
450	                                &cache,
451	                                &paths.scratch,
452	                            )? {
453	                                UpdatePullOutcome::Updated(plan) => {
454	                                    let version = cache
455	                                        .staged_version()
456	                                        .map(|v| v.to_string())
457	                                        .unwrap_or_else(|| "?".to_string());
458	                                    eprintln!(
459	                                        "UPDATE_STAGED:{version}:{:?} (from {})",
460	                                        plan.class, peer.pubkey_hex
461	                                    );
462	                                    // Convergence fact (M8 decision 19): the
463	                                    // status query reports this as the last
464	                                    // typed outcome.
465	                                    // [impl->REQ-UPD-6]
466	                                    let _ = cache.record_last_outcome("staged");
467	                                    // REQ-UPD-4: gated on user confirmation by
468	                                    // default — the staged release surfaces as
469	                                    // a consent notif (D8's producer); apply
470	                                    // rides the existing self-update machinery
471	                                    // once confirmed. Full-auto writes nothing.
472	                                    let decision = decide(
473	                                        full_auto_update,
474	                                        most_recently_active(&paths.owlery),
475	                                    );
476	                                    let surface = NotifSurfacePolicy {
477	                                        subnets: subnets
478	                                            .subnets
479	                                            .iter()
480	                                            .map(|s| (s.name.clone(), s.hide_new_endpoints))
481	                                            .collect(),
482	                                        vis: VisibilityStore::load_from(&paths.vis),
483	                                    };
484	                                    let store = NotifStore::open_at(&paths.notif_db)
485	                                        .map_err(|e| io::Error::other(e.to_string()))?;
486	                                    // The consent notif's id MUST mint from the
487	                                    // node's CANONICAL epoch (shared with the
488	                                    // registry leg + the CLI notify front
489	                                    // door), NOT `ops` (the stream-open op
490	                                    // counter, a separate file) — else two
491	                                    // counters issue the same `node:epoch`
492	                                    // notif_id and collide (the rig
493	                                    // presence-rung UNIQUE failure).
494	                                    // [impl->REQ-HAZARD-REGISTRY-EPOCH-LEASE]
495	                                    let produced = registry.with_epoch(|ep| {
496	                                        produce_consent_notif(
497	                                            &decision,
498	                                            &version,
499	                                            &store,
500	                                            registry.node_hex(),
501	                                            ep,
502	                                            &sub.name,
503	                                            &surface,
504	                                            &paths.owlery,
505	                                            now_ms(),
506	                                        )
507	                                    });
508	                                    if let Err(e) = produced {
509	                                        eprintln!("UPDATE_NOTIF_FAIL:{e}");
510	                                    }
511	                                }
512	                                UpdatePullOutcome::UpToDate => {}
513	                                UpdatePullOutcome::Rejected(reason) => {
514	                                    // Loud + typed, never silent (ADR-0004 §D).
515	                                    eprintln!("UPDATE_REJECTED:{}:{reason}", peer.pubkey_hex);
516	                                    // Convergence fact (M8 decision 19) —
517	                                    // `Rejected{reason}` in the table.
518	                                    // [impl->REQ-UPD-6]
519	                                    let _ =
520	                                        cache.record_last_outcome(&format!("rejected:{reason}"));
521	                                }
522	                            }
523	                        }
524	                    }
525	                    Ok(())
526	                };
527	                if let Err(e) = peer_io() {
528	                    eprintln!("PUMP_PEER_FAIL:{}:{e}", peer.pubkey_hex);
529	                    conns.remove(&peer.pubkey_hex); // redial next tick
530	                }
531	            }
532	        }
533
534	        let now = Instant::now();
535	        if due_reg {
536	            last_reg = Some(now);
537	        }
538	        if due_notif {
539	            last_notif = Some(now);
540	        }
541	        if due_sync {
542	            last_sync = Some(now);
543	        }
544	        if due_upd {
545	            last_upd = Some(now);
546	        }
547	    }
548	    Ok(())
549	}
550
551	/// Dial (or reuse) the connection for a peer. `None` = unresolvable or
552	/// undialable right now — skip, don't fail the round.
553	///
554	/// Address-seeded resolution order (M8 decisions 14/20, REQ-CONV-1):
555	///
556	/// 1. the durable last-known address (`peer-addrs.json`) — the cold-start
557	///    fast path that skips the id-only discovery round-trip;
558	/// 2. on a miss OR a failed seeded dial, the injected resolver (production:
559	///    id-only discovery) — a stale seed costs one failed dial, never strands
560	///    a peer.
561	///
562	/// Every successful dial writes the OBSERVED address back (the `net-dialed`
563	/// reply's `remote_addr`), keeping the seed warm for the next cold start.
564	// [impl->REQ-CONV-1]
565	fn ensure_conn(
566	    brain: &mut Brain,
567	    conns: &mut HashMap<String, u64>,
568	    resolver: &PeerResolver,
569	    peer_hex: &str,
570	    peer_addrs: &std::path::Path,
571	) -> Option<u64> {
572	    if let Some(id) = conns.get(peer_hex) {
573	        return Some(*id);
574	    }
575	    let conn_id = dial_seeded(resolver, peer_hex, peer_addrs, |addr| {
576	        brain
577	            .net_dial(addr, None)
578	            .map(|d| (d.conn_id, d.remote_addr))
579	    })?;
580	    conns.insert(peer_hex.to_string(), conn_id);
581	    Some(conn_id)
582	}
583
584	/// The seeded dial plan (M8 decisions 14/20): try the durable last-known
585	/// address, then the injected resolver — first success wins, persisting the
586	/// dial reply's OBSERVED address back into the seed file. `dial` returns the
587	/// broker-minted conn id + the observed remote addr (`null` = nothing worth
588	/// seeding). Factored from [`ensure_conn`] so the resolution ORDER is a unit
589	/// fact, not a prose promise.
590	// [impl->REQ-CONV-1]
591	fn dial_seeded(
592	    resolver: &PeerResolver,
593	    peer_hex: &str,
594	    peer_addrs: &std::path::Path,
595	    mut dial: impl FnMut(serde_json::Value) -> io::Result<(u64, serde_json::Value)>,
596	) -> Option<u64> {
597	    let seeded = spt_store::peeraddrs::PeerAddrStore::load_from(peer_addrs)
598	        .get(peer_hex)
599	        .cloned();
600	    for addr in seeded.into_iter().chain(resolver(peer_hex)) {
601	        if let Ok((conn_id, observed)) = dial(addr) {
602	            // Write-back: persist what the live connection actually shows
603	            // (null when no path is readable — nothing worth seeding).
604	            if !observed.is_null() {
605	                let _ = spt_store::peeraddrs::PeerAddrStore::record(peer_addrs, peer_hex, observed);
606	            }
607	            return Some(conn_id);
608	        }
609	        // Seeded dial failed → fall through to the id-only resolver;
610	        // resolver dial failed → peer offline/unreachable, next tick retries.
611	    }
612	    None
613	}
614
615	/// Open a fresh stream on `conn_id`, write one whole feed fire-and-forget,
616	/// and finish it (the receiver's dispatcher classifies + applies). One
617	/// stream per feed: streams are single-purpose by the demux contract.
618	fn push_feed(brain: &mut Brain, conn_id: u64, feed: &[u8]) -> io::Result<()> {
619	    let opened = brain.net_open_stream(conn_id, None)?;
620	    brain.net_stream_send(opened.stream_id, feed, None, true)?;
621	    Ok(())
622	}
623
624	/// Connect a fresh brain IPC client (retrying through broker startup).
625	fn connect(broker_name: &str) -> io::Result<Brain> {
626	    let mut last = io::Error::other("broker never answered");
627	    for _ in 0..300 {
628	        match Brain::cold_start(broker_name, now_ms()) {
629	            Ok(b) => return Ok(b),
630	            Err(e) => last = e,
631	        }
632	        thread::sleep(Duration::from_millis(10));
633	    }
634	    Err(last)
635	}
636
637	fn now_ms() -> u64 {
638	    std::time::SystemTime::now()
639	        .duration_since(std::time::UNIX_EPOCH)
640	        .map(|d| d.as_millis() as u64)
641	        .unwrap_or(0)
642	}
643
644	/// Cadence gate: a never-run leg (`last == None`) is always due so the first
645	/// tick primes peers; thereafter due once `cadence` has elapsed. Uses only
646	/// forward `duration_since` — NEVER `Instant - Duration`, which underflow-
647	/// panics on a monotonic clock younger than the offset (a freshly-booted
648	/// Windows host, KNOWN-HAZARDS 5.9).
649	/// [impl->REQ-HAZARD-INSTANT-UNDERFLOW]
650	fn due(last: Option<Instant>, now: Instant, cadence: Duration) -> bool {
651	    last.is_none_or(|t| now.duration_since(t) >= cadence)
652	}
653
654	/// Write the pump's last-tick heartbeat (epoch ms) — the liveness fact
655	/// `spt daemon status` renders and the `subnet status` stall warning reads
656	/// (M8 decision 23). Best-effort: a failed write costs honesty, never the
657	/// pump (and the reader treats an absent/unreadable file as "no heartbeat
658	/// recorded", not as healthy).
659	// [impl->REQ-DAEMON-5]
660	fn write_heartbeat(path: &std::path::Path) {
661	    if let Some(parent) = path.parent() {
662	        let _ = std::fs::create_dir_all(parent);
663	    }
664	    let _ = spt_store::atomic::atomic_write_string(path, &now_ms().to_string());
665	}
666
667	/// The supervisor's next backoff after a death: a healthy run (survived
668	/// [`SUPERVISE_HEALTHY_RUN`]) resets to the floor `base`; a fast death
669	/// doubles, capped at [`SUPERVISE_BACKOFF_CAP`] (M8 decision 23 — a
670	/// persistently-crashing pump stays visible and bounded, never silent and
671	/// never a hot crash loop). Pure for the unit table.
672	// [impl->REQ-DAEMON-5]
673	fn next_backoff(prev: Duration, ran_for: Duration, base: Duration) -> Duration {
674	    if ran_for >= SUPERVISE_HEALTHY_RUN {
675	        base
676	    } else {
677	        prev.saturating_mul(2).min(SUPERVISE_BACKOFF_CAP)
678	    }
679	}
680
681	/// Supervise repeated runs of a pump body until `stop` is raised (M8
682	/// decision 23, REQ-DAEMON-5): a panic is CAUGHT and logged loud
683	/// (`PEER_PUMP_PANIC`), an error return logged (`PEER_PUMP_FAIL`), and the
684	/// body restarts after a capped, doubling backoff — a 5.9-class pump death
685	/// self-heals visibly instead of silently halving the daemon for 24 h.
686	/// `base` is the backoff floor (production: [`SUPERVISE_BACKOFF_BASE`];
687	/// the unit harness shrinks it). Backoff sleeps in [`TICK`] slices so a
688	/// stop request lands promptly. Generic over the body for the unit
689	/// harness; production binds [`run_peer_pump`].
690	// [impl->REQ-DAEMON-5]
691	pub fn supervise_pump(stop: &AtomicBool, base: Duration, mut body: impl FnMut() -> io::Result<()>) {
692	    let mut backoff = base;
693	    while !stop.load(Ordering::Relaxed) {
694	        let started = Instant::now();
695	        let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(&mut body));
696	        if stop.load(Ordering::Relaxed) {
697	            return; // a clean stop — whatever the body returned
698	        }
699	        let ran_for = started.elapsed();
700	        match outcome {
701	            // The body only returns Ok when stop is raised; reaching here
702	            // without it is a died-loop fact — same restart as an error.
703	            Ok(Ok(())) => eprintln!("PEER_PUMP_EXIT: loop returned without stop — restarting"),
704	            Ok(Err(e)) => eprintln!("PEER_PUMP_FAIL: {e}"),
705	            Err(panic) => {
706	                let msg = panic
707	                    .downcast_ref::<&str>()
708	                    .map(|s| s.to_string())
709	                    .or_else(|| panic.downcast_ref::<String>().cloned())
710	                    .unwrap_or_else(|| "<non-string panic>".to_string());
711	                eprintln!("PEER_PUMP_PANIC: {msg}");
712	            }
713	        }
714	        backoff = next_backoff(backoff, ran_for, base);
715	        eprintln!(
716	            "PEER_PUMP_RESTART: supervised restart in {}s",
717	            backoff.as_secs()
718	        );
719	        let mut slept = Duration::ZERO;
720	        while slept < backoff && !stop.load(Ordering::Relaxed) {
721	            thread::sleep(TICK.min(backoff - slept));
722	            slept += TICK;
723	        }
724	    }
725	}
726
727	/// Spawn the pump on its own thread with the production wiring — the
728	/// daemon-boot entry ([`crate::daemon::Daemon::run`]). The thread body is
729	/// the SUPERVISOR ([`supervise_pump`]), not the bare loop: the pump
730	/// restarts after a panic or error with capped backoff (M8 decision 23).
731	// [impl->REQ-DAEMON-5]
732	pub fn spawn_peer_pump(
733	    broker_name: String,
734	    registry: Arc<RegistryHost>,
735	    cfg: DaemonConfig,
736	    stop: Arc<AtomicBool>,
737	) -> thread::JoinHandle<()> {
738	    thread::spawn(move || {
739	        let paths = PumpPaths::production();
740	        supervise_pump(&stop, SUPERVISE_BACKOFF_BASE, || {
741	            run_peer_pump(
742	                &broker_name,
743	                Arc::clone(&registry),
744	                &paths,
745	                production_resolver(),
746	                PumpConfig::from_config(&cfg),
747	                PumpHooks::default(),
748	                &stop,
749	            )
750	        });
751	    })
752	}
753
754	#[cfg(test)]
755	mod tests {
756	    use super::*;
757
758	    // The pump's cadence gate must prime on the first tick AND never compute
759	    // an instant before now (the underflow panic that killed the peer pump on
760	    // a sub-24h-uptime Windows host). `due` takes an Option and only ever runs
761	    // `now.duration_since(past)` forward — there is no `Instant - Duration`
762	    // arithmetic left to underflow.
763	    // [unit->REQ-HAZARD-INSTANT-UNDERFLOW]
764	    #[test]
765	    fn first_tick_is_due_and_never_underflows() {
766	        let now = Instant::now();
767	        // A day-long cadence on a process that may have started milliseconds
768	        // ago: the never-run leg is due regardless, with no subtraction-from-
769	        // instant anywhere in the path.
770	        assert!(due(None, now, Duration::from_secs(86_400)));
771	        // A leg that just ran is not due until its cadence elapses.
772	        assert!(!due(Some(now), now, Duration::from_secs(86_400)));
773	        // And it becomes due once enough has elapsed.
774	        let earlier = now - Duration::from_millis(50);
775	        assert!(due(Some(earlier), now, Duration::from_millis(10)));
776	        assert!(!due(Some(earlier), now, Duration::from_millis(500)));
777	    }
778
779	    // [unit->REQ-DAEMON-5] the backoff ladder: fast deaths double from the
780	    // floor and saturate at the 5-minute cap (decision 23's bound); a run
781	    // that survived the healthy window resets to the floor — a once-a-day
782	    // panic never ratchets the pump toward permanent 5-minute outages.
783	    #[test]
784	    fn supervisor_backoff_doubles_caps_and_resets() {
785	        let base = SUPERVISE_BACKOFF_BASE;
786	        let fast = Duration::from_millis(10); // died immediately
787	        let mut b = base;
788	        b = next_backoff(b, fast, base);
789	        assert_eq!(b, base * 2, "first fast death doubles");
790	        b = next_backoff(b, fast, base);
791	        assert_eq!(b, base * 4, "and keeps doubling");
792	        for _ in 0..16 {
793	            b = next_backoff(b, fast, base);
794	        }
795	        assert_eq!(b, SUPERVISE_BACKOFF_CAP, "saturates at the 5-minute cap");
796	        assert_eq!(
797	            next_backoff(b, SUPERVISE_HEALTHY_RUN, base),
798	            base,
799	            "a healthy run resets to the floor"
800	        );
801	    }
802
803	    // [unit->REQ-DAEMON-5] the supervisor catches a PANICKING body and
804	    // restarts it (with backoff) instead of dying silent — the 5.9-class
805	    // half-death self-heals; raising stop ends supervision promptly.
806	    #[test]
807	    fn supervisor_restarts_a_panicking_pump_until_stop() {
808	        let stop = AtomicBool::new(false);
809	        let runs = std::cell::Cell::new(0u32);
810	        supervise_pump(&stop, Duration::from_millis(5), || {
811	            runs.set(runs.get() + 1);
812	            match runs.get() {
813	                1 => panic!("injected 5.9-class pump death"),
814	                2 => Err(io::Error::other("injected pump error")),
815	                _ => {
816	                    // Third life: a healthy pump that runs until stopped.
817	                    stop.store(true, Ordering::Relaxed);
818	                    Ok(())
819	                }
820	            }
821	        });
822	        assert_eq!(
823	            runs.get(),
824	            3,
825	            "panic + error each restarted; the healthy run ended supervision"
826	        );
827	    }
828
829	    // [unit->REQ-DAEMON-5] the heartbeat writer lands a parseable epoch-ms
830	    // stamp at the given path (creating parents) and a later write advances
831	    // it — what the D1 `daemon status` reader + stall warning consume.
832	    #[test]
833	    fn heartbeat_writes_and_advances() {
834	        let dir = tempfile::tempdir().unwrap();
835	        let path = dir.path().join("identity").join("pump-heartbeat.json");
836	        write_heartbeat(&path);
837	        let first: u64 = std::fs::read_to_string(&path)
838	            .expect("heartbeat written")
839	            .trim()
840	            .parse()
841	            .expect("epoch ms parses");
842	        assert!(first > 0, "a real wall-clock stamp");
843	        thread::sleep(Duration::from_millis(15));
844	        write_heartbeat(&path);
845	        let second: u64 = std::fs::read_to_string(&path)
846	            .unwrap()
847	            .trim()
848	            .parse()
849	            .unwrap();
850	        assert!(second > first, "a later tick advances the stamp");
851	    }
852
853	    // [unit->REQ-CONV-1] the seeded dial plan: a recorded address is tried
854	    // FIRST (no discovery round-trip); a stale seed falls back to the
855	    // id-only resolver (one wasted dial, never a stranded peer); a miss goes
856	    // straight to the resolver; and a successful dial writes the OBSERVED
857	    // address back so the next cold start is seeded.
858	    #[test]
859	    fn seeded_dial_order_fallback_and_writeback() {
860	        let dir = tempfile::tempdir().unwrap();
861	        let seeds = dir.path().join("peer-addrs.json");
862	        let seeded_addr = serde_json::json!({"id": "ab", "addrs": ["192.168.1.7:4711"]});
863	        let id_only = serde_json::json!({"id": "ab", "addrs": []});
864	        let observed = serde_json::json!({"id": "ab", "addrs": ["10.0.0.9:4711"]});
865	        let resolver: PeerResolver = {
866	            let id_only = id_only.clone();
867	            Box::new(move |hex| (hex == "ab").then(|| id_only.clone()))
868	        };
869
870	        // Miss: no seed file → the resolver's id-only addr is the only try.
871	        let mut tried: Vec<serde_json::Value> = Vec::new();
872	        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
873	            tried.push(addr);
874	            Ok((1, serde_json::Value::Null))
875	        });
876	        assert_eq!(got, Some(1));
877	        assert_eq!(
878	            tried,
879	            vec![id_only.clone()],
880	            "miss goes straight to id-only"
881	        );
882	        assert!(
883	            spt_store::peeraddrs::PeerAddrStore::load_from(&seeds)
884	                .get("ab")
885	                .is_none(),
886	            "a null observed addr is not worth seeding"
887	        );
888
889	        // Seeded: the recorded addr is tried FIRST; success writes back the
890	        // OBSERVED addr (what the live connection showed, not what we dialed).
891	        spt_store::peeraddrs::PeerAddrStore::record(&seeds, "ab", seeded_addr.clone()).unwrap();
892	        let mut tried = Vec::new();
893	        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
894	            tried.push(addr);
895	            Ok((2, observed.clone()))
896	        });
897	        assert_eq!(got, Some(2));
898	        assert_eq!(
899	            tried,
900	            vec![seeded_addr.clone()],
901	            "seed tried first, resolver never needed"
902	        );
903	        assert_eq!(
904	            spt_store::peeraddrs::PeerAddrStore::load_from(&seeds).get("ab"),
905	            Some(&observed),
906	            "observed addr written back"
907	        );
908
909	        // Stale seed: the seeded dial fails → id-only fallback dials next
910	        // (a stale addr costs one failed dial, never strands the peer).
911	        let mut tried = Vec::new();
912	        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
913	            tried.push(addr.clone());
914	            if addr == observed {
915	                Err(io::Error::other("stale addr — host moved"))
916	            } else {
917	                Ok((3, serde_json::Value::Null))
918	            }
919	        });
920	        assert_eq!(got, Some(3));
921	        assert_eq!(
922	            tried,
923	            vec![observed.clone(), id_only.clone()],
924	            "stale seed → id-only fallback"
925	        );
926
927	        // Both fail: undialable now — the round skips, the next tick retries.
928	        let got = dial_seeded(&resolver, "ab", &seeds, |_| {
929	            Err(io::Error::other("offline"))
930	        });
931	        assert_eq!(got, None);
932	    }
933
934	    // [unit->REQ-CONV-2] the advertise-now marker is consumed exactly once —
935	    // one online/offline transition forces exactly one registry round, and
936	    // an absent marker forces nothing.
937	    #[test]
938	    fn advertise_marker_is_taken_exactly_once() {
939	        let dir = tempfile::tempdir().unwrap();
940	        let marker = dir.path().join("advertise-now");
941	        assert!(
942	            !crate::registryhost::take_advertise_now(&marker),
943	            "nothing to take yet"
944	        );
945	        std::fs::write(&marker, b"").unwrap();
946	        assert!(
947	            crate::registryhost::take_advertise_now(&marker),
948	            "first take claims the transition"
949	        );
950	        assert!(
951	            !crate::registryhost::take_advertise_now(&marker),
952	            "second take gets nothing"
953	        );
954	    }
955	}
956

//! Outbound peer pump (D9-1-2) — the production trigger loops every D5–D8
//! slice deferred ("rides D9 with the daemon lifecycle").
//!
//! One daemon-hosted loop drives three cadenced pumps against every trusted
//! peer of every member subnet, each a thin call over an already-shipped,
//! idempotent driver — the pump owns *when*, the drivers own *how*:
//!
//! - **registry push** ([`RegistryHost::advertise_local`] → one feed stream
//!   per peer): the node re-advertises its own rows; peers reconverge within
//!   one cadence (replay-is-recovery, the D4d/D8 push posture). Own rows
//!   only — no transitive gossip in v1: trust is pairwise per subnet, and a
//!   relayed third-party row would carry an origin the receiver cannot gate
//!   honestly (KNOWN-HAZARDS 7.5).
//! - **notif push** ([`crate::notifsync::emit_notif_feed`]): the full
//!   semilattice spool per subnet; the receiver's join dedups.
//! - **sync pull** ([`crate::sync::request_sync`]): anti-entropy for the
//!   minds. Want-refs are the union of [`crate::sync::select_refs`] (locally
//!   tracked branches the policy allows toward the peer) and
//!   **registry-derived** `a-<id>` refs (endpoints known in the shared
//!   subnets — how a fresh instance bootstraps a mind it has never held),
//!   both gated by the same [`SyncPolicy::allows`] the responder enforces
//!   (symmetric scoping).
//!
//! Conflicts a pull surfaces are reported through [`PumpHooks::on_pull`] —
//! the default production hook logs them loud. The bounded Psyche reconcile
//! turn itself still runs at the activation paths (ADR-0013): the daemon
//! does not yet host per-agent manifest runtimes (the 7.4 fan-out), so an
//! in-pump turn has no runtime to ride; conflict artifacts persist safely
//! meanwhile (REQ-HAZARD-CONFLICT-BOTH-PRESERVED).
//!
//! ## Wire discipline (inherited from the test suites, now production)
//!
//! Every pump write is fire-and-forget (`op_id: None`) — a journaled
//! `net_stream_send` blocks on its ack and DISCARDS interleaved events
//! (gotcha #1). Stream opens for push feeds carry no op either: the feeds
//! are idempotent by construction. The sync pull's `open_op` IS durable
//! (exactly-once open identity), minted from a dedicated [`EpochSource`]
//! file so a daemon restart can never reuse one.
// [impl->REQ-INST-5]
// [impl->REQ-INST-7]
// [impl->REQ-NOTIF-1]

use std::collections::HashMap;
use std::io;
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::thread;
use std::time::{Duration, Instant};

use spt_net::net::endpoint::addr_for_node_hex;
use spt_store::contextstore::ContextStore;
use spt_store::epoch::EpochSource;
use spt_store::notif::NotifStore;
use spt_store::perch;
use spt_store::subnet::SubnetStore;
use spt_store::trust::TrustStore;
use spt_store::visibility::VisibilityStore;

use crate::brain::Brain;
use crate::config::DaemonConfig;
use crate::consent::{decide, most_recently_active};
use crate::notif::{produce_consent_notif, NotifSurfacePolicy};
use crate::notifsync::emit_notif_feed;
use crate::propagate::{request_update, UpdatePullOutcome};
use crate::registryhost::RegistryHost;
use crate::relcache::ReleaseCache;
use crate::release::VerifyPolicy;
use crate::sync::{request_sync, select_refs, SyncPolicy, SyncPullReport};
use crate::update::BrokerAbi;

/// The pump's tick granularity — each cadence fires when its period elapsed.
const TICK: Duration = Duration::from_millis(200);

/// Heartbeat write throttle (M8 decision 23, REQ-DAEMON-5): the stall
/// warning reads 3-cadence staleness, so a few seconds of heartbeat
/// granularity is invisible to it — and 5 writes/sec of disk churn is not.
const HEARTBEAT_PERIOD: Duration = Duration::from_secs(5);

/// Supervisor backoff floor: the first restart after a pump death.
const SUPERVISE_BACKOFF_BASE: Duration = Duration::from_secs(5);

/// Supervisor backoff cap (M8 decision 23): a persistently-crashing pump
/// retries at most every 5 minutes — visible, bounded, never silent.
const SUPERVISE_BACKOFF_CAP: Duration = Duration::from_secs(300);

/// A run that survived this long before dying counts as healthy — the next
/// backoff resets to the floor instead of doubling (a once-a-day panic must
/// not ratchet the pump toward permanent 5-minute outages).
const SUPERVISE_HEALTHY_RUN: Duration = Duration::from_secs(60);

/// Resolve a trusted peer's stable name (lowercase-hex pubkey) to a dialable
/// address (the serde JSON `net-dial` carries). Production resolution lets
/// discovery do the work ([`production_resolver`]); hermetic tests inject a
/// concrete address map (loopback endpoints run no discovery).
pub type PeerResolver = Box<dyn Fn(&str) -> Option<serde_json::Value> + Send>;

/// The production resolver: an id-only iroh address — mDNS on the LAN, n0
/// DNS on the WAN (`addr_for_node_hex`).
pub fn production_resolver() -> PeerResolver {
    Box::new(|hex| addr_for_node_hex(hex).and_then(|a| serde_json::to_value(a).ok()))
}

/// Where this node's pump-side stores live. Production = the canonical
/// SPT_HOME layout; the loopback E2E gives its in-process "node" distinct
/// roots so two nodes can share one test process.
pub struct PumpPaths {
    /// The perch tree (registry advertisement scan).
    pub owlery: PathBuf,
    /// The context store root ([`ContextStore::open_or_init_in`]).
    pub ctx_root: PathBuf,
    /// The notif spool DB ([`NotifStore::open_at`]).
    pub notif_db: PathBuf,
    /// Transient pull scratch.
    pub scratch: PathBuf,
    /// `subnet.json` ([`SubnetStore::load_from`]).
    pub subnets: PathBuf,
    /// `trust/peers.json` ([`TrustStore::load_from`]).
    pub trust: PathBuf,
    /// `visibility.json` ([`VisibilityStore::load_from`]).
    pub vis: PathBuf,
    /// The pump's durable open-op counter (its own file — colliding with the
    /// registry's advertisement epochs would break neither, but a shared
    /// in-memory cursor pair over one file double-mints; separate files keep
    /// each `EpochSource` the sole writer of its counter).
    pub ops: PathBuf,
    /// The staged-release cache dir ([`ReleaseCache::open`]).
    pub releases: PathBuf,
    /// `release-keys.json` ([`VerifyPolicy::load_from`]).
    pub release_keys: PathBuf,
    /// The wake-edge freshness-pull marker (D9-2, REQ-INST-3): a
    /// dormant→active transition drops it; the pump consumes it and forces a
    /// sync round NOW instead of waiting out `cadence.sync`
    /// ([`crate::resting::PULL_MARKER_FILE`]).
    pub pull_marker: PathBuf,
    /// The event-driven advertisement marker (M8 decision 15, REQ-CONV-2):
    /// an endpoint online/offline transition drops it; the pump consumes it
    /// and forces a registry round NOW instead of waiting out
    /// `cadence.registry` ([`crate::registryhost::ADVERTISE_MARKER_FILE`]).
    pub advertise_marker: PathBuf,
    /// `peer-addrs.json` — last-known peer addresses (M8 decisions 14/20,
    /// REQ-CONV-1): consulted before id-only discovery, written back after
    /// every successful dial ([`spt_store::peeraddrs::PeerAddrStore`]).
    pub peer_addrs: PathBuf,
    /// The pump's last-tick heartbeat (M8 decision 23, REQ-DAEMON-5):
    /// epoch-ms written while the loop is alive, rendered by
    /// `spt daemon status` / the `subnet status` stall warning
    /// ([`crate::endpoint::pump_heartbeat_path`]).
    pub heartbeat: PathBuf,
}

impl PumpPaths {
    /// The canonical SPT_HOME layout.
    pub fn production() -> PumpPaths {
        let home = perch::spt_home();
        let identity = perch::identity_dir();
        PumpPaths {
            owlery: perch::owlery_dir(),
            ctx_root: home.join("context"),
            notif_db: identity.join("notifs.db"),
            scratch: home.join("scratch").join("pump"),
            subnets: identity.join("subnet.json"),
            trust: identity.join("trust").join("peers.json"),
            vis: identity.join("visibility.json"),
            ops: identity.join("pump-ops.json"),
            releases: home.join("releases"),
            release_keys: identity.join("release-keys.json"),
            pull_marker: identity.join(crate::resting::PULL_MARKER_FILE),
            advertise_marker: identity.join(crate::registryhost::ADVERTISE_MARKER_FILE),
            peer_addrs: identity.join("peer-addrs.json"),
            heartbeat: crate::endpoint::pump_heartbeat_path(),
        }
    }
}

/// Pump cadences (config-driven in production: [`DaemonConfig`]).
#[derive(Debug, Clone, Copy)]
pub struct PumpCadence {
    pub registry: Duration,
    pub notif: Duration,
    pub sync: Duration,
    pub update: Duration,
    /// Silent-peer registry eviction window (KNOWN-HAZARDS 4.10): swept each
    /// registry tick, before re-advertisement.
    pub registry_evict_after: Duration,
}

impl PumpCadence {
    /// From the daemon config knobs.
    pub fn from_config(cfg: &DaemonConfig) -> PumpCadence {
        PumpCadence {
            registry: cfg.registry_pump_period,
            notif: cfg.notif_pump_period,
            sync: cfg.sync_pull_period,
            update: cfg.update_check_period,
            registry_evict_after: cfg.registry_evict_after,
        }
    }
}

/// The pump's resolved knobs: cadences + the REQ-UPD-4 consent stance.
#[derive(Debug, Clone, Copy)]
pub struct PumpConfig {
    pub cadence: PumpCadence,
    /// `true` = the explicit opt-in to prompt-free updates; default `false`
    /// surfaces every staged release as a consent notif.
    pub full_auto_update: bool,
}

impl PumpConfig {
    /// From the daemon config knobs.
    pub fn from_config(cfg: &DaemonConfig) -> PumpConfig {
        PumpConfig {
            cadence: PumpCadence::from_config(cfg),
            full_auto_update: cfg.full_auto_update,
        }
    }
}

/// A pull observer: `(peer_hex, report)` after every completed sync pull.
pub type OnPull = Box<dyn FnMut(&str, &SyncPullReport) + Send>;

/// Observation hooks. `on_pull` fires after every completed sync pull —
/// the conflict surface (ADR-0013) and the E2E's probe.
pub struct PumpHooks {
    pub on_pull: OnPull,
}

impl Default for PumpHooks {
    /// The production hook: conflicts are loud, never silent — the artifacts
    /// already persist (hazard 6.6); the reconcile turn runs at activation.
    fn default() -> PumpHooks {
        PumpHooks {
            on_pull: Box::new(|peer, report| {
                for (branch, file) in report.conflicts() {
                    eprintln!("SYNC_CONFLICT:{peer}:{branch}:{file}");
                }
            }),
        }
    }
}

/// Run the pump until `stop` is set. Owns one brain IPC connection and a
/// per-peer connection cache (an IO failure against a peer drops its cache
/// row; the next tick redials — conns are rebuilt freely, nothing leaks
/// because the broker's closed-watcher reaps dead conns).
pub fn run_peer_pump(
    broker_name: &str,
    registry: Arc<RegistryHost>,
    paths: &PumpPaths,
    resolver: PeerResolver,
    config: PumpConfig,
    mut hooks: PumpHooks,
    stop: &AtomicBool,
) -> io::Result<()> {
    let PumpConfig {
        cadence,
        full_auto_update,
    } = config;
    let mut brain = connect(broker_name)?;
    let mut ops = EpochSource::load_from(&paths.ops);
    let mut conns: HashMap<String, u64> = HashMap::new();
    // Stagger from "everything due now" — the first tick primes peers.
    // `None` = never run yet → due immediately. Avoids the `Instant -
    // Duration` underflow panic on hosts whose uptime is below the offset
    // (Windows monotonic clock can't represent an instant before boot).
    let (mut last_reg, mut last_notif, mut last_sync, mut last_upd): (
        Option<Instant>,
        Option<Instant>,
        Option<Instant>,
        Option<Instant>,
    ) = (None, None, None, None);
    let mut last_hb: Option<Instant> = None;

    while !stop.load(Ordering::Relaxed) {
        let now = Instant::now();
        // Liveness heartbeat (M8 decision 23, REQ-DAEMON-5): written while
        // the loop is ALIVE — including idle ticks — never per work round,
        // so a wedged peer_io and a healthy quiet pump are distinguishable.
        // Throttled: the stall warning reads 3-cadence staleness, sub-second
        // precision buys nothing but disk churn.
        // [impl->REQ-DAEMON-5]
        if due(last_hb, now, HEARTBEAT_PERIOD) {
            write_heartbeat(&paths.heartbeat);
            last_hb = Some(now);
        }
        // Event-driven advertisement (M8 decision 15, REQ-CONV-2): an
        // endpoint online/offline transition dropped the marker — consume it
        // and run the registry leg NOW. A wake of THIS loop, not a second
        // advertisement path: the forced round is the same advertise_local +
        // push code, so the epoch lease and visibility gates ride unchanged.
        // [impl->REQ-CONV-2]
        let wake_adv = crate::registryhost::take_advertise_now(&paths.advertise_marker);
        let due_reg = wake_adv || due(last_reg, now, cadence.registry);
        let due_notif = due(last_notif, now, cadence.notif);
        // The wake edge's freshness pull (D9-2, REQ-INST-3): a dormant→active
        // transition dropped the marker — consume it and pull NOW, the
        // catch-up-on-activation latency is the point. The take is the remove,
        // so one wake claims exactly one forced round.
        // [impl->REQ-INST-3]
        let wake_pull = crate::resting::take_freshness_pull(&paths.pull_marker);
        let due_sync = wake_pull || due(last_sync, now, cadence.sync);
        let due_upd = due(last_upd, now, cadence.update);
        if !(due_reg || due_notif || due_sync || due_upd) {
            thread::sleep(TICK);
            continue;
        }

        // Re-load config-ish stores per round (the access-gate freshness
        // stance — edits take effect without a restart).
        let subnets = SubnetStore::load_from(&paths.subnets);
        // Serve-state (M8 decision 6, REQ-SUBNET-5): a detached subnet is
        // neither advertised into nor connected to — skipped wholesale in
        // the per-subnet loop below. Re-read per round like every other
        // config-ish store, so a flip takes effect within one tick.
        // [impl->REQ-SUBNET-5]
        let attachment = spt_store::attachment::AttachmentStore::load_from(
            &paths
                .subnets
                .parent()
                .map(|p| p.join("attachment.json"))
                .unwrap_or_else(|| PathBuf::from("attachment.json")),
        );
        // The push target stays the directly-paired trusted set at D5 (the
        // fan-out widens to all roster members only at D6); the sync *gate*,
        // however, now authorizes on roster membership (Mesh-D5, REQ-MESH-5).
        let trust = TrustStore::load_from(&paths.trust);
        let roster_path = paths
            .subnets
            .parent()
            .map(|p| p.join("roster.json"))
            .unwrap_or_else(|| PathBuf::from("roster.json"));
        let policy = SyncPolicy {
            subnets: subnets
                .subnets
                .iter()
                .map(|s| (s.name.clone(), s.hide_new_endpoints))
                .collect(),
            roster: spt_store::roster::RosterStore::load_from(&roster_path),
            vis: VisibilityStore::load_from(&paths.vis),
        };
        let adverts = if due_reg {
            // Consume the re-pair eviction marker FIRST (M8 decision 13,
            // REQ-SUBNET-7): a superseded identity's in-memory rows go
            // before this tick's snapshot mirror can resurrect the pairing
            // host's disk-side eviction.
            // [impl->REQ-SUBNET-7]
            registry.consume_repair_evictions();
            // Sweep silent-node ghost rows BEFORE re-advertising (KNOWN-
            // HAZARDS 4.10): a dead node identity's rows decay instead of
            // poisoning bare-id resolution forever; a live peer re-stamps its
            // heard entry with every admitted feed and is never touched.
            // [impl->REQ-HAZARD-REGISTRY-GHOST-ROWS]
            registry.evict_silent_peers(cadence.registry_evict_after);
            registry.advertise_local(&paths.owlery)
        } else {
            Vec::new()
        };

        for sub in &subnets.subnets {
            if attachment.is_detached(&sub.name) {
                continue; // detached: neither advertise into nor connect to (REQ-SUBNET-5)
            }
            for peer in trust.peers_in(&sub.name) {
                if peer.pubkey_hex == registry.node_hex() {
                    continue; // never pump at ourselves
                }
                let Some(conn_id) = ensure_conn(
                    &mut brain,
                    &mut conns,
                    &resolver,
                    &peer.pubkey_hex,
                    &paths.peer_addrs,
                ) else {
                    continue; // unresolvable/undialable now — next tick retries
                };

                let mut peer_io = || -> io::Result<()> {
                    if due_reg {
                        let lines: Vec<u8> = adverts
                            .iter()
                            .filter(|u| u.subnet() == sub.name)
                            .flat_map(|u| u.encode_line())
                            .collect();
                        if !lines.is_empty() {
                            push_feed(&mut brain, conn_id, &lines)?;
                        }
                    }
                    if due_notif {
                        let store = NotifStore::open_at(&paths.notif_db)
                            .map_err(|e| io::Error::other(e.to_string()))?;
                        let feed = emit_notif_feed(&store, &sub.name).map_err(io::Error::other)?;
                        if !feed.is_empty() {
                            push_feed(&mut brain, conn_id, &feed)?;
                        }
                    }
                    if due_sync {
                        let cs = ContextStore::open_or_init_in(&paths.ctx_root)?;
                        let mut want = select_refs(&policy, &cs, &peer.pubkey_hex)?;
                        // Registry-derived bootstrap refs: minds named in the
                        // shared subnet that this node never held locally.
                        if let Some(snap) = registry.snapshot(&sub.name) {
                            for id in snap.endpoint_ids() {
                                let r = format!("a-{id}");
                                if !want.contains(&r) && policy.allows(&r, &peer.pubkey_hex) {
                                    want.push(r);
                                }
                            }
                        }
                        if !want.is_empty() {
                            let open_op = ops.next_epoch()?;
                            let report = request_sync(
                                &mut brain,
                                conn_id,
                                &want,
                                open_op,
                                &cs,
                                &paths.scratch,
                            )?;
                            (hooks.on_pull)(&peer.pubkey_hex, &report);
                        }
                    }
                    if due_upd {
                        // The peer-propagation pull (REQ-UPD-1) under the
                        // PRODUCTION gate: keys + channel pin from
                        // release-keys.json (fail-closed when unprovisioned),
                        // version floor = the staged release (rollback gate),
                        // wall clock for expiry only.
                        let cache = ReleaseCache::open(&paths.releases);
                        let base_policy = VerifyPolicy::load_from(&paths.release_keys, 0, now_ms());
                        let current = if cache.staged_channel().as_deref()
                            == Some(base_policy.pinned_channel.as_str())
                        {
                            cache.staged_version().unwrap_or(0)
                        } else {
                            0
                        };
                        let policy = VerifyPolicy {
                            current_version: current,
                            ..base_policy
                        };
                        if !policy.trusted_keys.is_empty() {
                            let open_op = ops.next_epoch()?;
                            match request_update(
                                &mut brain,
                                conn_id,
                                open_op,
                                &BrokerAbi::current(),
                                &policy,
                                &cache,
                                &paths.scratch,
                            )? {
                                UpdatePullOutcome::Updated(plan) => {
                                    let version = cache
                                        .staged_version()
                                        .map(|v| v.to_string())
                                        .unwrap_or_else(|| "?".to_string());
                                    eprintln!(
                                        "UPDATE_STAGED:{version}:{:?} (from {})",
                                        plan.class, peer.pubkey_hex
                                    );
                                    // Convergence fact (M8 decision 19): the
                                    // status query reports this as the last
                                    // typed outcome.
                                    // [impl->REQ-UPD-6]
                                    let _ = cache.record_last_outcome("staged");
                                    // REQ-UPD-4: gated on user confirmation by
                                    // default — the staged release surfaces as
                                    // a consent notif (D8's producer); apply
                                    // rides the existing self-update machinery
                                    // once confirmed. Full-auto writes nothing.
                                    let decision = decide(
                                        full_auto_update,
                                        most_recently_active(&paths.owlery),
                                    );
                                    let surface = NotifSurfacePolicy {
                                        subnets: subnets
                                            .subnets
                                            .iter()
                                            .map(|s| (s.name.clone(), s.hide_new_endpoints))
                                            .collect(),
                                        vis: VisibilityStore::load_from(&paths.vis),
                                    };
                                    let store = NotifStore::open_at(&paths.notif_db)
                                        .map_err(|e| io::Error::other(e.to_string()))?;
                                    // The consent notif's id MUST mint from the
                                    // node's CANONICAL epoch (shared with the
                                    // registry leg + the CLI notify front
                                    // door), NOT `ops` (the stream-open op
                                    // counter, a separate file) — else two
                                    // counters issue the same `node:epoch`
                                    // notif_id and collide (the rig
                                    // presence-rung UNIQUE failure).
                                    // [impl->REQ-HAZARD-REGISTRY-EPOCH-LEASE]
                                    let produced = registry.with_epoch(|ep| {
                                        produce_consent_notif(
                                            &decision,
                                            &version,
                                            &store,
                                            registry.node_hex(),
                                            ep,
                                            &sub.name,
                                            &surface,
                                            &paths.owlery,
                                            now_ms(),
                                        )
                                    });
                                    if let Err(e) = produced {
                                        eprintln!("UPDATE_NOTIF_FAIL:{e}");
                                    }
                                }
                                UpdatePullOutcome::UpToDate => {}
                                UpdatePullOutcome::Rejected(reason) => {
                                    // Loud + typed, never silent (ADR-0004 §D).
                                    eprintln!("UPDATE_REJECTED:{}:{reason}", peer.pubkey_hex);
                                    // Convergence fact (M8 decision 19) —
                                    // `Rejected{reason}` in the table.
                                    // [impl->REQ-UPD-6]
                                    let _ =
                                        cache.record_last_outcome(&format!("rejected:{reason}"));
                                }
                            }
                        }
                    }
                    Ok(())
                };
                if let Err(e) = peer_io() {
                    eprintln!("PUMP_PEER_FAIL:{}:{e}", peer.pubkey_hex);
                    conns.remove(&peer.pubkey_hex); // redial next tick
                }
            }
        }

        let now = Instant::now();
        if due_reg {
            last_reg = Some(now);
        }
        if due_notif {
            last_notif = Some(now);
        }
        if due_sync {
            last_sync = Some(now);
        }
        if due_upd {
            last_upd = Some(now);
        }
    }
    Ok(())
}

/// Dial (or reuse) the connection for a peer. `None` = unresolvable or
/// undialable right now — skip, don't fail the round.
///
/// Address-seeded resolution order (M8 decisions 14/20, REQ-CONV-1):
///
/// 1. the durable last-known address (`peer-addrs.json`) — the cold-start
///    fast path that skips the id-only discovery round-trip;
/// 2. on a miss OR a failed seeded dial, the injected resolver (production:
///    id-only discovery) — a stale seed costs one failed dial, never strands
///    a peer.
///
/// Every successful dial writes the OBSERVED address back (the `net-dialed`
/// reply's `remote_addr`), keeping the seed warm for the next cold start.
// [impl->REQ-CONV-1]
fn ensure_conn(
    brain: &mut Brain,
    conns: &mut HashMap<String, u64>,
    resolver: &PeerResolver,
    peer_hex: &str,
    peer_addrs: &std::path::Path,
) -> Option<u64> {
    if let Some(id) = conns.get(peer_hex) {
        return Some(*id);
    }
    let conn_id = dial_seeded(resolver, peer_hex, peer_addrs, |addr| {
        brain
            .net_dial(addr, None)
            .map(|d| (d.conn_id, d.remote_addr))
    })?;
    conns.insert(peer_hex.to_string(), conn_id);
    Some(conn_id)
}

/// The seeded dial plan (M8 decisions 14/20): try the durable last-known
/// address, then the injected resolver — first success wins, persisting the
/// dial reply's OBSERVED address back into the seed file. `dial` returns the
/// broker-minted conn id + the observed remote addr (`null` = nothing worth
/// seeding). Factored from [`ensure_conn`] so the resolution ORDER is a unit
/// fact, not a prose promise.
// [impl->REQ-CONV-1]
fn dial_seeded(
    resolver: &PeerResolver,
    peer_hex: &str,
    peer_addrs: &std::path::Path,
    mut dial: impl FnMut(serde_json::Value) -> io::Result<(u64, serde_json::Value)>,
) -> Option<u64> {
    let seeded = spt_store::peeraddrs::PeerAddrStore::load_from(peer_addrs)
        .get(peer_hex)
        .cloned();
    for addr in seeded.into_iter().chain(resolver(peer_hex)) {
        if let Ok((conn_id, observed)) = dial(addr) {
            // Write-back: persist what the live connection actually shows
            // (null when no path is readable — nothing worth seeding).
            if !observed.is_null() {
                let _ = spt_store::peeraddrs::PeerAddrStore::record(peer_addrs, peer_hex, observed);
            }
            return Some(conn_id);
        }
        // Seeded dial failed → fall through to the id-only resolver;
        // resolver dial failed → peer offline/unreachable, next tick retries.
    }
    None
}

/// Open a fresh stream on `conn_id`, write one whole feed fire-and-forget,
/// and finish it (the receiver's dispatcher classifies + applies). One
/// stream per feed: streams are single-purpose by the demux contract.
fn push_feed(brain: &mut Brain, conn_id: u64, feed: &[u8]) -> io::Result<()> {
    let opened = brain.net_open_stream(conn_id, None)?;
    brain.net_stream_send(opened.stream_id, feed, None, true)?;
    Ok(())
}

/// Connect a fresh brain IPC client (retrying through broker startup).
fn connect(broker_name: &str) -> io::Result<Brain> {
    let mut last = io::Error::other("broker never answered");
    for _ in 0..300 {
        match Brain::cold_start(broker_name, now_ms()) {
            Ok(b) => return Ok(b),
            Err(e) => last = e,
        }
        thread::sleep(Duration::from_millis(10));
    }
    Err(last)
}

fn now_ms() -> u64 {
    std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .map(|d| d.as_millis() as u64)
        .unwrap_or(0)
}

/// Cadence gate: a never-run leg (`last == None`) is always due so the first
/// tick primes peers; thereafter due once `cadence` has elapsed. Uses only
/// forward `duration_since` — NEVER `Instant - Duration`, which underflow-
/// panics on a monotonic clock younger than the offset (a freshly-booted
/// Windows host, KNOWN-HAZARDS 5.9).
/// [impl->REQ-HAZARD-INSTANT-UNDERFLOW]
fn due(last: Option<Instant>, now: Instant, cadence: Duration) -> bool {
    last.is_none_or(|t| now.duration_since(t) >= cadence)
}

/// Write the pump's last-tick heartbeat (epoch ms) — the liveness fact
/// `spt daemon status` renders and the `subnet status` stall warning reads
/// (M8 decision 23). Best-effort: a failed write costs honesty, never the
/// pump (and the reader treats an absent/unreadable file as "no heartbeat
/// recorded", not as healthy).
// [impl->REQ-DAEMON-5]
fn write_heartbeat(path: &std::path::Path) {
    if let Some(parent) = path.parent() {
        let _ = std::fs::create_dir_all(parent);
    }
    let _ = spt_store::atomic::atomic_write_string(path, &now_ms().to_string());
}

/// The supervisor's next backoff after a death: a healthy run (survived
/// [`SUPERVISE_HEALTHY_RUN`]) resets to the floor `base`; a fast death
/// doubles, capped at [`SUPERVISE_BACKOFF_CAP`] (M8 decision 23 — a
/// persistently-crashing pump stays visible and bounded, never silent and
/// never a hot crash loop). Pure for the unit table.
// [impl->REQ-DAEMON-5]
fn next_backoff(prev: Duration, ran_for: Duration, base: Duration) -> Duration {
    if ran_for >= SUPERVISE_HEALTHY_RUN {
        base
    } else {
        prev.saturating_mul(2).min(SUPERVISE_BACKOFF_CAP)
    }
}

/// Supervise repeated runs of a pump body until `stop` is raised (M8
/// decision 23, REQ-DAEMON-5): a panic is CAUGHT and logged loud
/// (`PEER_PUMP_PANIC`), an error return logged (`PEER_PUMP_FAIL`), and the
/// body restarts after a capped, doubling backoff — a 5.9-class pump death
/// self-heals visibly instead of silently halving the daemon for 24 h.
/// `base` is the backoff floor (production: [`SUPERVISE_BACKOFF_BASE`];
/// the unit harness shrinks it). Backoff sleeps in [`TICK`] slices so a
/// stop request lands promptly. Generic over the body for the unit
/// harness; production binds [`run_peer_pump`].
// [impl->REQ-DAEMON-5]
pub fn supervise_pump(stop: &AtomicBool, base: Duration, mut body: impl FnMut() -> io::Result<()>) {
    let mut backoff = base;
    while !stop.load(Ordering::Relaxed) {
        let started = Instant::now();
        let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(&mut body));
        if stop.load(Ordering::Relaxed) {
            return; // a clean stop — whatever the body returned
        }
        let ran_for = started.elapsed();
        match outcome {
            // The body only returns Ok when stop is raised; reaching here
            // without it is a died-loop fact — same restart as an error.
            Ok(Ok(())) => eprintln!("PEER_PUMP_EXIT: loop returned without stop — restarting"),
            Ok(Err(e)) => eprintln!("PEER_PUMP_FAIL: {e}"),
            Err(panic) => {
                let msg = panic
                    .downcast_ref::<&str>()
                    .map(|s| s.to_string())
                    .or_else(|| panic.downcast_ref::<String>().cloned())
                    .unwrap_or_else(|| "<non-string panic>".to_string());
                eprintln!("PEER_PUMP_PANIC: {msg}");
            }
        }
        backoff = next_backoff(backoff, ran_for, base);
        eprintln!(
            "PEER_PUMP_RESTART: supervised restart in {}s",
            backoff.as_secs()
        );
        let mut slept = Duration::ZERO;
        while slept < backoff && !stop.load(Ordering::Relaxed) {
            thread::sleep(TICK.min(backoff - slept));
            slept += TICK;
        }
    }
}

/// Spawn the pump on its own thread with the production wiring — the
/// daemon-boot entry ([`crate::daemon::Daemon::run`]). The thread body is
/// the SUPERVISOR ([`supervise_pump`]), not the bare loop: the pump
/// restarts after a panic or error with capped backoff (M8 decision 23).
// [impl->REQ-DAEMON-5]
pub fn spawn_peer_pump(
    broker_name: String,
    registry: Arc<RegistryHost>,
    cfg: DaemonConfig,
    stop: Arc<AtomicBool>,
) -> thread::JoinHandle<()> {
    thread::spawn(move || {
        let paths = PumpPaths::production();
        supervise_pump(&stop, SUPERVISE_BACKOFF_BASE, || {
            run_peer_pump(
                &broker_name,
                Arc::clone(&registry),
                &paths,
                production_resolver(),
                PumpConfig::from_config(&cfg),
                PumpHooks::default(),
                &stop,
            )
        });
    })
}

#[cfg(test)]
mod tests {
    use super::*;

    // The pump's cadence gate must prime on the first tick AND never compute
    // an instant before now (the underflow panic that killed the peer pump on
    // a sub-24h-uptime Windows host). `due` takes an Option and only ever runs
    // `now.duration_since(past)` forward — there is no `Instant - Duration`
    // arithmetic left to underflow.
    // [unit->REQ-HAZARD-INSTANT-UNDERFLOW]
    #[test]
    fn first_tick_is_due_and_never_underflows() {
        let now = Instant::now();
        // A day-long cadence on a process that may have started milliseconds
        // ago: the never-run leg is due regardless, with no subtraction-from-
        // instant anywhere in the path.
        assert!(due(None, now, Duration::from_secs(86_400)));
        // A leg that just ran is not due until its cadence elapses.
        assert!(!due(Some(now), now, Duration::from_secs(86_400)));
        // And it becomes due once enough has elapsed.
        let earlier = now - Duration::from_millis(50);
        assert!(due(Some(earlier), now, Duration::from_millis(10)));
        assert!(!due(Some(earlier), now, Duration::from_millis(500)));
    }

    // [unit->REQ-DAEMON-5] the backoff ladder: fast deaths double from the
    // floor and saturate at the 5-minute cap (decision 23's bound); a run
    // that survived the healthy window resets to the floor — a once-a-day
    // panic never ratchets the pump toward permanent 5-minute outages.
    #[test]
    fn supervisor_backoff_doubles_caps_and_resets() {
        let base = SUPERVISE_BACKOFF_BASE;
        let fast = Duration::from_millis(10); // died immediately
        let mut b = base;
        b = next_backoff(b, fast, base);
        assert_eq!(b, base * 2, "first fast death doubles");
        b = next_backoff(b, fast, base);
        assert_eq!(b, base * 4, "and keeps doubling");
        for _ in 0..16 {
            b = next_backoff(b, fast, base);
        }
        assert_eq!(b, SUPERVISE_BACKOFF_CAP, "saturates at the 5-minute cap");
        assert_eq!(
            next_backoff(b, SUPERVISE_HEALTHY_RUN, base),
            base,
            "a healthy run resets to the floor"
        );
    }

    // [unit->REQ-DAEMON-5] the supervisor catches a PANICKING body and
    // restarts it (with backoff) instead of dying silent — the 5.9-class
    // half-death self-heals; raising stop ends supervision promptly.
    #[test]
    fn supervisor_restarts_a_panicking_pump_until_stop() {
        let stop = AtomicBool::new(false);
        let runs = std::cell::Cell::new(0u32);
        supervise_pump(&stop, Duration::from_millis(5), || {
            runs.set(runs.get() + 1);
            match runs.get() {
                1 => panic!("injected 5.9-class pump death"),
                2 => Err(io::Error::other("injected pump error")),
                _ => {
                    // Third life: a healthy pump that runs until stopped.
                    stop.store(true, Ordering::Relaxed);
                    Ok(())
                }
            }
        });
        assert_eq!(
            runs.get(),
            3,
            "panic + error each restarted; the healthy run ended supervision"
        );
    }

    // [unit->REQ-DAEMON-5] the heartbeat writer lands a parseable epoch-ms
    // stamp at the given path (creating parents) and a later write advances
    // it — what the D1 `daemon status` reader + stall warning consume.
    #[test]
    fn heartbeat_writes_and_advances() {
        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("identity").join("pump-heartbeat.json");
        write_heartbeat(&path);
        let first: u64 = std::fs::read_to_string(&path)
            .expect("heartbeat written")
            .trim()
            .parse()
            .expect("epoch ms parses");
        assert!(first > 0, "a real wall-clock stamp");
        thread::sleep(Duration::from_millis(15));
        write_heartbeat(&path);
        let second: u64 = std::fs::read_to_string(&path)
            .unwrap()
            .trim()
            .parse()
            .unwrap();
        assert!(second > first, "a later tick advances the stamp");
    }

    // [unit->REQ-CONV-1] the seeded dial plan: a recorded address is tried
    // FIRST (no discovery round-trip); a stale seed falls back to the
    // id-only resolver (one wasted dial, never a stranded peer); a miss goes
    // straight to the resolver; and a successful dial writes the OBSERVED
    // address back so the next cold start is seeded.
    #[test]
    fn seeded_dial_order_fallback_and_writeback() {
        let dir = tempfile::tempdir().unwrap();
        let seeds = dir.path().join("peer-addrs.json");
        let seeded_addr = serde_json::json!({"id": "ab", "addrs": ["192.168.1.7:4711"]});
        let id_only = serde_json::json!({"id": "ab", "addrs": []});
        let observed = serde_json::json!({"id": "ab", "addrs": ["10.0.0.9:4711"]});
        let resolver: PeerResolver = {
            let id_only = id_only.clone();
            Box::new(move |hex| (hex == "ab").then(|| id_only.clone()))
        };

        // Miss: no seed file → the resolver's id-only addr is the only try.
        let mut tried: Vec<serde_json::Value> = Vec::new();
        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
            tried.push(addr);
            Ok((1, serde_json::Value::Null))
        });
        assert_eq!(got, Some(1));
        assert_eq!(
            tried,
            vec![id_only.clone()],
            "miss goes straight to id-only"
        );
        assert!(
            spt_store::peeraddrs::PeerAddrStore::load_from(&seeds)
                .get("ab")
                .is_none(),
            "a null observed addr is not worth seeding"
        );

        // Seeded: the recorded addr is tried FIRST; success writes back the
        // OBSERVED addr (what the live connection showed, not what we dialed).
        spt_store::peeraddrs::PeerAddrStore::record(&seeds, "ab", seeded_addr.clone()).unwrap();
        let mut tried = Vec::new();
        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
            tried.push(addr);
            Ok((2, observed.clone()))
        });
        assert_eq!(got, Some(2));
        assert_eq!(
            tried,
            vec![seeded_addr.clone()],
            "seed tried first, resolver never needed"
        );
        assert_eq!(
            spt_store::peeraddrs::PeerAddrStore::load_from(&seeds).get("ab"),
            Some(&observed),
            "observed addr written back"
        );

        // Stale seed: the seeded dial fails → id-only fallback dials next
        // (a stale addr costs one failed dial, never strands the peer).
        let mut tried = Vec::new();
        let got = dial_seeded(&resolver, "ab", &seeds, |addr| {
            tried.push(addr.clone());
            if addr == observed {
                Err(io::Error::other("stale addr — host moved"))
            } else {
                Ok((3, serde_json::Value::Null))
            }
        });
        assert_eq!(got, Some(3));
        assert_eq!(
            tried,
            vec![observed.clone(), id_only.clone()],
            "stale seed → id-only fallback"
        );

        // Both fail: undialable now — the round skips, the next tick retries.
        let got = dial_seeded(&resolver, "ab", &seeds, |_| {
            Err(io::Error::other("offline"))
        });
        assert_eq!(got, None);
    }

    // [unit->REQ-CONV-2] the advertise-now marker is consumed exactly once —
    // one online/offline transition forces exactly one registry round, and
    // an absent marker forces nothing.
    #[test]
    fn advertise_marker_is_taken_exactly_once() {
        let dir = tempfile::tempdir().unwrap();
        let marker = dir.path().join("advertise-now");
        assert!(
            !crate::registryhost::take_advertise_now(&marker),
            "nothing to take yet"
        );
        std::fs::write(&marker, b"").unwrap();
        assert!(
            crate::registryhost::take_advertise_now(&marker),
            "first take claims the transition"
        );
        assert!(
            !crate::registryhost::take_advertise_now(&marker),
            "second take gets nothing"
        );
    }
}


Now roster API + the cutover sites. Read in parallel.


<owl_messages>
<owl_message from="doyle-psyche" priority="normal"><EVENT type="echo_commune" from="doyle-psyche" timestamp="2026-06-08T15:20:16-07:00" machine="HFENDULEAM" project="spt-core" branch="dev-freeform" head_sha="9b09b11a9709965f86e4c12247bcc9237d8f6575" head_subject="docs(mesh): MESH-D6-PLAN.md — fan-out widen + cutover → mesh on" note="Echo commune brief — auto-fired">&lt;live-context&gt;
Delta since last commune (you're stale at &quot;D5a shipped, D5b=cutover NEXT&quot;): D5b SHIPPED but RE-CUT, and the original D5b plan was unachievable.

The design call that changed the phase boundary. Reviewing the original D5b (&quot;delete TrustStore, keep push narrow&quot;) against the docs (user asked me to use judgment on SUBNET-MESH-PLAN + ADR-0017), I found it's internally impossible: peerloop's narrow push reads `trust.peers_in`. Stripping pairing's trust write empties `peers.json`; with no narrow source, the only repair points peerloop at `roster.members_in` — but the roster is WIDE (D4 made A learn C transitively), so that swap IS the D6 fan-out widen. Deletion and widen are inseparable — there is no &quot;delete TrustStore but stay narrow&quot; intermediate. So the cutover+widen fold into D6 (one mesh-on commit), and D5b shrank to the one genuinely separable, no-reach half.

That separable half was a latent bug D5a silently shipped. D5a moved every inbound gate to `roster.is_member`, but `repair_evict_superseded` still only dropped the (no-longer-consulted) trust row. So a re-keyed identity kept its roster entry and stayed a member — the REQ-SUBNET-7 re-pair overwrite was ineffective against the new gate. The eviction test only asserted `trust.is_trusted` cleared, never the roster, so it didn't catch it. D5b (`91083ef`) fixes it: tombstone the superseded roster entry (D3 grow-set primitive) alongside the trust drop. Test now asserts `!is_member` in the ceremony subnet + still-member elsewhere.

Reflection: the re-cut is the call I'd flag to a reviewer — I deviated from the written plan, but the plan was wrong (the coupling makes its D5b literally uncompilable), and the deviation surfaced a real correctness gap. Both the plan doc (MESH-D5-PLAN re-cut note) and the new MESH-D6-PLAN spell out the coupling so it doesn't read as scope drift. The grep-false-green lesson you already hold paid off — used `; echo ===EXIT:$?===` throughout; daemon suite + clippy + traceable-reqs all genuinely exit 0.
&lt;/live-context&gt;
&lt;project-context&gt;
name=spt-core:

Mesh-D5b SHIPPED — `91083ef` on `dev-freeform` (unpushed). D6 plan written (`9b09b11`).

D5b ≠ the cutover you have queued. Re-cut (documented in `MESH-D5-PLAN.md`): D5b = roster-anchor the REQ-SUBNET-7 eviction only (the latent-gap fix above). `repair_evict_superseded` now takes `&amp;mut RosterStore` too and tombstones the superseded entry; caller `pairhost.rs` passes `&amp;mut roster`. TrustStore/pairing-write/propagate/peerloop-narrow/CLI all still on trust (≡ roster at D5). Green: 14 registryhost + full daemon suite + clippy + traceable-reqs all exit 0. REQ-MESH-5 still inactive `[]` (tags forward evidence; activates at D6).

IMMEDIATE NEXT = Mesh-D6 — NOT the old &quot;D5b&quot;. Full JIT plan in `MESH-D6-PLAN.md`. The single mesh-ON commit (big, harder-to-reverse: deletes `peers.json` fleet-wide, no migration). Steps: (1) strip pairing-trust write (pairhost both legs + `run_responder`/`run_initiator` sigs in `spt_net pairing/wire.rs`); (2) propagate serve gate `&amp;TrustStore`→`&amp;RosterStore` + `dispatch.rs:326` caller; (3) demote warn-on-change → non-blocking machine_id notif at the `repair_evict_superseded` detection point; (4) THE WIDEN — peerloop push `trust.peers_in`→`roster.members_in` (reuse the roster already loaded for the sync gate; drop `PumpPaths.trust`); (5) CLI repoint status/list/prune→roster (prune = local tombstone; propagating revoke is D7); (6) delete `trust.rs` + `perch::trust_file` + `lib.rs pub mod trust`, KEEP `perch::trust_dir()` (still homes access/grants — the plan I wrote corrects the psyche's &quot;delete trust_dir&quot;), grep-verify zero `TrustStore`/`is_trusted`/`trust::`/`peers.json` in non-comment code; (7) migrate twohost/propagate/wire ceremony tests trust→roster (notifsync/sync/peerloop/dispatch already done at D5a); (8) un-ignore `mesh.rs` D0 staggered+star harness (flips GREEN — A↔C reach, B never relays) + add the no-third-party-row regression assert (every applied row origin == author); (9) activate REQ-MESH-3 `[impl,unit,int]` + complete REQ-MESH-5 `[impl,unit]`. Commit: `feat(mesh)!: fan-out widen + delete TrustStore → mesh on — Mesh-D6`.

Mesh arc: D0–D4 ✅ · D5a ✅ · D5b ✅ (`91083ef`) · D6 ← NEXT (mesh on) · D7 revoke/rotate/re-seed + deferred per-conn proven_subnets epoch cross-check · D8 status-probe concurrency.

`dev-freeform` now carries the mesh set through D5b + the D6 plan doc (+ earlier offline-name fix + daemon-start alias) — user controls the push. Queued untouched on disk: `DAEMON-LIFECYCLE-PLAN.md`.
&lt;/project-context&gt;</EVENT></owl_message>
</owl_messages>