Full seam understood. Three QUIC await sites in nethost: `dial` (`endpoint.connect` + `prove_membership`), `open_stream` Quic (`conn.open_bi`), `send_stream` Quic (`write_all`/`finish`). All `block_on` bridges. Now the critical invariant #1 — verify how the pump classifies TimedOut vs ordinary broker error.


Found 22 files
crates\spt\tests\livehost_bootrace_e2e.rs
crates\spt\tests\livehost_nonresident_e2e.rs
crates\spt-daemon\src\livehost.rs
crates\spt-daemon\src\shellwake.rs
crates\spt\tests\livehost_psyche_fail_e2e.rs
crates\spt\src\cli.rs
crates\spt-daemon\src\lifecycle.rs
crates\spt-daemon\src\endpoint.rs
crates\spt-daemon\src\daemon.rs
crates\spt-daemon\src\brainproc.rs
crates\spt-msg\src\ring.rs
crates\spt\src\rc.rs
crates\spt\tests\brain_survive.rs
crates\spt-daemon\src\brain.rs
crates\spt-daemon\src\applyhost.rs
crates\spt-daemon\tests\pumpdeadline.rs
crates\spt-daemon\src\pump\mod.rs
crates\spt\tests\n1_compat.rs
crates\spt\tests\brain_split.rs
crates\spt\tests\n1_pairing.rs
crates\spt-store\src\gitrun.rs
crates\spt-daemon\src\config.rs


54://! file so a daemon restart can never reuse one.
100:/// escalated to a supervised restart, NEVER a per-peer retry.
101:const PUMP_PEER_IO_TIMEOUT: Duration = Duration::from_secs(30);
103:/// Supervisor backoff floor: the first restart after a pump death.
298:    fn peer_step(
326:    /// takes effect within one tick, no restart).
408:    /// open identity — a restart can never reuse one).
425:pub fn run_peer_pump(
439:    // channel) so every reply read is bounded by PUMP_PEER_IO_TIMEOUT — a
441:    // hfenduleam stall). A timed-out read escalates to a supervised restart
525:            // Recovery tier-split: a TimedOut brain-IPC read bubbles OUT of the
526:            // round (supervised restart); any other error is an ordinary peer
528:            peer_outcome(outcome, &peer_hex, &mut conns)?;
583:/// DEADLINE). A `TimedOut` is NOT an ordinary peer failure: a brain-IPC read
591:fn peer_outcome(
598:        Err(e) if e.kind() == io::ErrorKind::TimedOut => Err(e), // poisoned → supervised restart
735:        match Brain::cold_start_pump(broker_name, now_ms(), PUMP_PEER_IO_TIMEOUT) {
794:/// body restarts after a capped, doubling backoff — a 5.9-class pump death
801:pub fn supervise_pump(stop: &AtomicBool, base: Duration, mut body: impl FnMut() -> io::Result<()>) {
812:            // without it is a died-loop fact — same restart as an error.
813:            Ok(Ok(())) => eprintln!("PEER_PUMP_EXIT: loop returned without stop — restarting"),
826:            "PEER_PUMP_RESTART: supervised restart in {}s",
839:/// the SUPERVISOR ([`supervise_pump`]), not the bare loop: the pump
840:/// restarts after a panic or error with capped backoff (M8 decision 23).
850:        supervise_pump(&stop, SUPERVISE_BACKOFF_BASE, || {
978:        fn peer_step(
1155:    // [unit->REQ-HAZARD-PUMP-IPC-DEADLINE] the recovery tier-split: a TimedOut
1157:    // wrong stream id later) → peer_outcome propagates Err so the round bubbles
1160:    // the conn. This split is what makes a 5s-backoff restart replace the 2.2h
1163:    fn peer_outcome_timeout_bubbles_ordinary_aborts() {
1170:        assert!(peer_outcome(Ok(()), "aa", &mut conns).is_ok());
1175:        assert!(peer_outcome(Err(ordinary), "bb", &mut conns).is_ok());
1178:        // TimedOut → bubbles out (Err): the supervisor restarts the pump with a
1179:        // fresh client; the conn cache is rebuilt by the restart, not here.
1180:        let timed = io::Error::new(io::ErrorKind::TimedOut, "brain IPC deadline");
1181:        let bubbled = peer_outcome(Err(timed), "cc", &mut conns);
1184:            "a poisoned client escalates to a supervised restart, never a per-peer retry"
1186:        assert_eq!(bubbled.unwrap_err().kind(), io::ErrorKind::TimedOut);
1214:    // restarts it (with backoff) instead of dying silent — the 5.9-class
1217:    fn supervisor_restarts_a_panicking_pump_until_stop() {
1220:        supervise_pump(&stop, Duration::from_millis(5), || {
1235:            "panic + error each restarted; the healthy run ended supervision"


290
291	    /// Node-local work run ONCE per due round, before any `peer_step`.
292	    /// Default: nothing.
293	    fn pre_round(&mut self, _ctx: &RoundCtx) {}
294
295	    /// The per-peer step for one `(subnet, peer)` under this round's context. A
296	    /// failure aborts the remaining due workers for THIS peer and drops its
297	    /// conn (redial next tick) — abort-on-first-fail, current behaviour.
298	    fn peer_step(
299	        &mut self,
300	        io: &mut PeerIo,
301	        subnet: &str,
302	        peer_hex: &str,
303	        ctx: &RoundCtx,
304	    ) -> io::Result<()>;
305	}
306
307	/// Per-round shared loads (§Frozen-3): the config-ish stores the shell reads
308	/// ONCE per due round and hands every worker by `&RoundCtx`. The single-read
309	/// invariant — push targets and the sync gate see the SAME roster — is made a
310	/// structural fact by loading the roster-backed [`SyncPolicy`] here, once.
311	/// Carries ONLY per-round shared loads; never long-lived deps (those are
312	/// construction-captured by each worker).
313	struct RoundCtx {
314	    /// `subnet.json` — the member subnets iterated for fan-out.
315	    subnets: SubnetStore,
316	    /// `attachment.json` — a detached subnet is skipped wholesale (REQ-SUBNET-5).
317	    attachment: spt_store::attachment::AttachmentStore,
318	    /// The roster-backed sync gate — one store feeding both the push targets
319	    /// and the sync authorization (symmetric scoping).
320	    policy: SyncPolicy,
321	}
322
323	impl RoundCtx {
324	    /// Load the per-round shared stores from the pump's paths (the access-gate
325	    /// freshness stance — every config-ish store re-read per round so an edit
326	    /// takes effect within one tick, no restart).
327	    fn load(paths: &PumpPaths) -> RoundCtx {
328	        let subnets = SubnetStore::load_from(&paths.subnets);
329	        // Serve-state (M8 decision 6, REQ-SUBNET-5): a detached subnet is
330	        // neither advertised into nor connected to — skipped wholesale in the
331	        // per-subnet loop below.
332	        // [impl->REQ-SUBNET-5]
333	        let attachment = spt_store::attachment::AttachmentStore::load_from(
334	            &paths
335	                .subnets
336	                .parent()
337	                .map(|p| p.join("attachment.json"))
338	                .unwrap_or_else(|| PathBuf::from("attachment.json")),
339	        );
340	        // THE WIDEN (Mesh-D6, REQ-MESH-3): the push target is the full roster
341	        // (all members), not the directly-paired set. The one roster loaded
342	        // here feeds both the push (who we dial) and the sync gate (who we
343	        // authorize), so they read one store.
344	        let roster_path = paths
345	            .subnets
346	            .parent()
347	            .map(|p| p.join("roster.json"))
348	            .unwrap_or_else(|| PathBuf::from("roster.json"));
349	        let policy = SyncPolicy {
350	            subnets: subnets
351	                .subnets
352	                .iter()
353	                .map(|s| (s.name.clone(), s.hide_new_endpoints))
354	                .collect(),
355	            roster: spt_store::roster::RosterStore::load_from(&roster_path),
356	            vis: VisibilityStore::load_from(&paths.vis),
357	        };
358	        RoundCtx {
359	            subnets,
360	            attachment,
361	            policy,
362	        }
363	    }
364	}
365
366	#[cfg(test)]
367	impl RoundCtx {
368	    /// A minimal empty round context for worker unit tests whose `pre_round` /
369	    /// `peer_step` ignore it (the registry leg keys off its own state, not the
370	    /// shared loads).
371	    fn for_test() -> RoundCtx {
372	        RoundCtx {
373	            subnets: SubnetStore::default(),
374	            attachment: spt_store::attachment::AttachmentStore::default(),
375	            policy: SyncPolicy {
376	                subnets: Vec::new(),
377	                roster: spt_store::roster::RosterStore::default(),
378	                vis: VisibilityStore::default(),
379	            },
380	        }
381	    }
382	}
383
384	/// The shell-owned per-peer IO surface (§Frozen-5): the ONE brain IPC handle,
385	/// the established conn id, and the SOLE open-op [`EpochSource`]
386	/// (`pump-ops.json`). A worker mints durable open-ops only through this handle
387	/// — a worker-owned `EpochSource` over the shared counter is the KH
388	/// epoch-lease double-mint and is forbidden.
389	struct PeerIo<'a> {
390	    /// The one brain IPC handle.
391	    brain: &'a mut Brain,
392	    /// The established peer conn id for this step.
393	    conn_id: u64,
394	    /// The pump's sole durable open-op counter. Workers mint open-ops only
395	    /// through [`PeerIo::open_op`] (the sync + update legs); a worker-owned
396	    /// `EpochSource` over this counter would be the KH epoch-lease double-mint.
397	    ops: &'a mut EpochSource,
398	}
399
400	impl PeerIo<'_> {
401	    /// Open a fresh stream and write one whole feed fire-and-forget (the
402	    /// push-feed discipline — `op_id: None`, finished on send).
403	    fn push_feed(&mut self, feed: &[u8]) -> io::Result<()> {
404	        push_feed(self.brain, self.conn_id, feed)
405	    }
406
407	    /// Mint the next durable open-op from the pump's sole counter (exactly-once
408	    /// open identity — a restart can never reuse one).
409	    fn open_op(&mut self) -> io::Result<u64> {
410	        self.ops.next_epoch()
411	    }
412	}
413
414	/// Run the pump until `stop` is set. Owns one brain IPC connection and a
415	/// per-peer connection cache (an IO failure against a peer drops its cache
416	/// row; the next tick redials — conns are rebuilt freely, nothing leaks
417	/// because the broker's closed-watcher reaps dead conns).
418	///
419	/// The pure scheduler + fan-out (pump-seam P4, all four legs behind the seam):

    /// Node-local work run ONCE per due round, before any `peer_step`.
    /// Default: nothing.
    fn pre_round(&mut self, _ctx: &RoundCtx) {}

    /// The per-peer step for one `(subnet, peer)` under this round's context. A
    /// failure aborts the remaining due workers for THIS peer and drops its
    /// conn (redial next tick) — abort-on-first-fail, current behaviour.
    fn peer_step(
        &mut self,
        io: &mut PeerIo,
        subnet: &str,
        peer_hex: &str,
        ctx: &RoundCtx,
    ) -> io::Result<()>;
}

/// Per-round shared loads (§Frozen-3): the config-ish stores the shell reads
/// ONCE per due round and hands every worker by `&RoundCtx`. The single-read
/// invariant — push targets and the sync gate see the SAME roster — is made a
/// structural fact by loading the roster-backed [`SyncPolicy`] here, once.
/// Carries ONLY per-round shared loads; never long-lived deps (those are
/// construction-captured by each worker).
struct RoundCtx {
    /// `subnet.json` — the member subnets iterated for fan-out.
    subnets: SubnetStore,
    /// `attachment.json` — a detached subnet is skipped wholesale (REQ-SUBNET-5).
    attachment: spt_store::attachment::AttachmentStore,
    /// The roster-backed sync gate — one store feeding both the push targets
    /// and the sync authorization (symmetric scoping).
    policy: SyncPolicy,
}

impl RoundCtx {
    /// Load the per-round shared stores from the pump's paths (the access-gate
    /// freshness stance — every config-ish store re-read per round so an edit
    /// takes effect within one tick, no restart).
    fn load(paths: &PumpPaths) -> RoundCtx {
        let subnets = SubnetStore::load_from(&paths.subnets);
        // Serve-state (M8 decision 6, REQ-SUBNET-5): a detached subnet is
        // neither advertised into nor connected to — skipped wholesale in the
        // per-subnet loop below.
        // [impl->REQ-SUBNET-5]
        let attachment = spt_store::attachment::AttachmentStore::load_from(
            &paths
                .subnets
                .parent()
                .map(|p| p.join("attachment.json"))
                .unwrap_or_else(|| PathBuf::from("attachment.json")),
        );
        // THE WIDEN (Mesh-D6, REQ-MESH-3): the push target is the full roster
        // (all members), not the directly-paired set. The one roster loaded
        // here feeds both the push (who we dial) and the sync gate (who we
        // authorize), so they read one store.
        let roster_path = paths
            .subnets
            .parent()
            .map(|p| p.join("roster.json"))
            .unwrap_or_else(|| PathBuf::from("roster.json"));
        let policy = SyncPolicy {
            subnets: subnets
                .subnets
                .iter()
                .map(|s| (s.name.clone(), s.hide_new_endpoints))
                .collect(),
            roster: spt_store::roster::RosterStore::load_from(&roster_path),
            vis: VisibilityStore::load_from(&paths.vis),
        };
        RoundCtx {
            subnets,
            attachment,
            policy,
        }
    }
}

#[cfg(test)]
impl RoundCtx {
    /// A minimal empty round context for worker unit tests whose `pre_round` /
    /// `peer_step` ignore it (the registry leg keys off its own state, not the
    /// shared loads).
    fn for_test() -> RoundCtx {
        RoundCtx {
            subnets: SubnetStore::default(),
            attachment: spt_store::attachment::AttachmentStore::default(),
            policy: SyncPolicy {
                subnets: Vec::new(),
                roster: spt_store::roster::RosterStore::default(),
                vis: VisibilityStore::default(),
            },
        }
    }
}

/// The shell-owned per-peer IO surface (§Frozen-5): the ONE brain IPC handle,
/// the established conn id, and the SOLE open-op [`EpochSource`]
/// (`pump-ops.json`). A worker mints durable open-ops only through this handle
/// — a worker-owned `EpochSource` over the shared counter is the KH
/// epoch-lease double-mint and is forbidden.
struct PeerIo<'a> {
    /// The one brain IPC handle.
    brain: &'a mut Brain,
    /// The established peer conn id for this step.
    conn_id: u64,
    /// The pump's sole durable open-op counter. Workers mint open-ops only
    /// through [`PeerIo::open_op`] (the sync + update legs); a worker-owned
    /// `EpochSource` over this counter would be the KH epoch-lease double-mint.
    ops: &'a mut EpochSource,
}

impl PeerIo<'_> {
    /// Open a fresh stream and write one whole feed fire-and-forget (the
    /// push-feed discipline — `op_id: None`, finished on send).
    fn push_feed(&mut self, feed: &[u8]) -> io::Result<()> {
        push_feed(self.brain, self.conn_id, feed)
    }

    /// Mint the next durable open-op from the pump's sole counter (exactly-once
    /// open identity — a restart can never reuse one).
    fn open_op(&mut self) -> io::Result<u64> {
        self.ops.next_epoch()
    }
}

/// Run the pump until `stop` is set. Owns one brain IPC connection and a
/// per-peer connection cache (an IO failure against a peer drops its cache
/// row; the next tick redials — conns are rebuilt freely, nothing leaks
/// because the broker's closed-watcher reaps dead conns).
///
/// The pure scheduler + fan-out (pump-seam P4, all four legs behind the seam):


425	pub fn run_peer_pump(
426	    broker_name: &str,
427	    registry: Arc<RegistryHost>,
428	    paths: &PumpPaths,
429	    resolver: PeerResolver,
430	    config: PumpConfig,
431	    hooks: PumpHooks,
432	    stop: &AtomicBool,
433	) -> io::Result<()> {
434	    let PumpConfig {
435	        cadence,
436	        full_auto_update,
437	    } = config;
438	    // PUMP MODE: a brain whose carrier is SPLIT at construction (reader thread +
439	    // channel) so every reply read is bounded by PUMP_PEER_IO_TIMEOUT — a
440	    // black-holed peer can never wedge this single-threaded loop (the 2.2h
441	    // hfenduleam stall). A timed-out read escalates to a supervised restart
442	    // below, never a per-peer retry (REQ-HAZARD-PUMP-IPC-DEADLINE).
443	    // [impl->REQ-HAZARD-PUMP-IPC-DEADLINE]
444	    let mut brain = connect(broker_name)?;
445	    let mut ops = EpochSource::load_from(&paths.ops);
446	    let mut conns: HashMap<String, u64> = HashMap::new();
447	    // The four cadenced legs behind the worker seam, in fan-out order
448	    // (registry → notif → sync → update). The shell drives them generically;
449	    // each leg's *what* lives in its module (§Frozen). Construction capture
450	    // (§Frozen-4): each worker owns the registry Arc / path slices / config
451	    // knob it needs — the sync worker additionally owns the conflict-surface
452	    // hooks (ADR-0013).
453	    let mut registry_worker = RegistryWorker::new(Arc::clone(&registry), paths, &cadence);
454	    let mut notif_worker = NotifWorker::new(paths, &cadence);
455	    let mut sync_worker = SyncWorker::new(Arc::clone(&registry), paths, &cadence, hooks);
456	    let mut update_worker = UpdateWorker::new(Arc::clone(&registry), paths, &cadence, full_auto_update);
457	    let mut workers: [&mut dyn PumpWorker; 4] = [
458	        &mut registry_worker,
459	        &mut notif_worker,
460	        &mut sync_worker,
461	        &mut update_worker,
462	    ];
463	    // WorkerLasts (ADR-0018 §V4): RAM-only last-ran instants, parallel to
464	    // `workers`. `None` = never run yet → due immediately, so the first tick
465	    // primes every leg. Never `Instant - Duration` — the underflow panic on a
466	    // sub-uptime monotonic clock (KH 5.9) lives nowhere in this path.
467	    let mut last: [Option<Instant>; 4] = [None; 4];
468	    let mut last_hb: Option<Instant> = None;
469
470	    while !stop.load(Ordering::Relaxed) {
471	        let now = Instant::now();
472	        // Shell-owned liveness heartbeat (M8 decision 23, REQ-DAEMON-5): written
473	        // while the loop is ALIVE — including idle ticks — never per work round,
474	        // so a wedged step and a healthy quiet pump are distinguishable.
475	        // Throttled; per-tick, never a worker (REQ-DAEMON-5), no wake.
476	        // [impl->REQ-DAEMON-5]
477	        if due(last_hb, false, now, HEARTBEAT_PERIOD) {
478	            write_heartbeat(&paths.heartbeat);
479	            last_hb = Some(now);
480	        }
481	        // Poll every worker's wake and decide due (the pure scheduler step):
482	        // every wake is consumed EVERY tick so a marker is never stranded (the
483	        // registry advertise-now, REQ-CONV-2; the sync freshness pull,
484	        // REQ-INST-3). The take IS the remove — one wake claims exactly one
485	        // forced round.
486	        let (due_flags, any_due) = compute_due(&mut workers, &last, now);
487	        if !any_due {
488	            thread::sleep(TICK);
489	            continue;
490	        }
491
492	        // Per-round shared loads (§Frozen-3): subnets / attachment /
493	        // roster-backed policy, read ONCE and handed to every worker.
494	        let ctx = RoundCtx::load(paths);
495
496	        // Each due worker's node-local pre_round, once per round before ANY
497	        // peer_step (only the registry leg does work here — its sweep +
498	        // advertise; the rest default to a no-op).
499	        pre_rounds(&mut workers, &due_flags, &ctx);
500
501	        // Fan out to every (subnet, peer) target (detached subnets skipped,
502	        // self excluded): establish the conn, then run each due worker's
503	        // peer_step in fan-out order. A failure aborts the remaining due
504	        // workers for THIS peer and drops its conn (abort-on-first-fail — the
505	        // broker reaps the dead conn; the next tick redials). Each worker mints
506	        // its open-ops only through the shell-owned `PeerIo`.
507	        for (sub_name, peer_hex) in fan_targets(&ctx, registry.node_hex()) {
508	            let Some(conn_id) = ensure_conn(
509	                &mut brain,
510	                &mut conns,
511	                &resolver,
512	                &peer_hex,
513	                &paths.peer_addrs,
514	            ) else {
515	                continue; // unresolvable/undialable now — next tick retries
516	            };
517	            let outcome = run_due(&due_flags, |i| {
518	                let mut io = PeerIo {
519	                    brain: &mut brain,
520	                    conn_id,
521	                    ops: &mut ops,
522	                };
523	                workers[i].peer_step(&mut io, &sub_name, &peer_hex, &ctx)
524	            });
525	            // Recovery tier-split: a TimedOut brain-IPC read bubbles OUT of the
526	            // round (supervised restart); any other error is an ordinary peer
527	            // failure (drop the conn, redial next tick).
528	            peer_outcome(outcome, &peer_hex, &mut conns)?;
529	        }
530
531	        // Mark-after-round with a POST-round instant (ADR-0018 §V4: the
532	        // stagger — ran = ATTEMPTED, not succeeded; the next due is measured
533	        // from here, not from the round's start).
534	        mark_ran(&mut last, &due_flags, Instant::now());
535	    }
536	    Ok(())
537	}
538
539	/// Compute each worker's due flag for this tick (the pure scheduler step):
540	/// poll EVERY worker's wake first — even a not-otherwise-due leg, so a marker
541	/// is never stranded — then the [`due`] kernel over its RAM-only last-ran
542	/// instant. Returns the per-worker flags (parallel to `workers`) and whether
543	/// ANY leg is due (an all-idle tick sleeps without loading a round).
544	fn compute_due(
545	    workers: &mut [&mut dyn PumpWorker],
546	    last: &[Option<Instant>],
547	    now: Instant,
548	) -> (Vec<bool>, bool) {
549	    let mut flags = vec![false; workers.len()];
550	    let mut any = false;
551	    for i in 0..workers.len() {
552	        let wake = workers[i].poll_wake();
553	        flags[i] = due(last[i], wake, now, workers[i].cadence());
554	        any |= flags[i];
555	    }
556	    (flags, any)
557	}
558
559	/// Run each due worker's node-local `pre_round` ONCE for the round, before any
560	/// peer_step (the registry sweep + advertise; the other legs default no-op).
561	fn pre_rounds(workers: &mut [&mut dyn PumpWorker], due_flags: &[bool], ctx: &RoundCtx) {
562	    for i in 0..workers.len() {
563	        if due_flags[i] {
564	            workers[i].pre_round(ctx);
565	        }
566	    }
567	}
568
569	/// Run `step` for each due worker index in fan-out order, aborting on the
570	/// FIRST error (abort-on-first-fail — the caller drops the peer's conn and
571	/// redials next tick). The peer-step IO itself lives in `step`; this is the
572	/// pure ordering/abort kernel, so the choreography is a unit fact.
573	fn run_due(due_flags: &[bool], mut step: impl FnMut(usize) -> io::Result<()>) -> io::Result<()> {
574	    for (i, &is_due) in due_flags.iter().enumerate() {
575	        if is_due {
576	            step(i)?;
577	        }
578	    }
579	    Ok(())
580	}
581
582	/// The recovery tier-split for a per-peer fan-out outcome (REQ-HAZARD-PUMP-IPC-
583	/// DEADLINE). A `TimedOut` is NOT an ordinary peer failure: a brain-IPC read
584	/// that timed out may have desynced the stream (a late reply could bind the
585	/// wrong stream id in a later call), so the client is POISONED — propagate it
586	/// (`Err`) to bubble out of the round and trigger a SUPERVISED RESTART (fresh
587	/// brain client + conn cache + WorkerLasts re-prime). Any other error is an
588	/// ordinary peer failure: drop the conn (redial next tick) and keep the round
589	/// going (`Ok`). A success keeps the conn.
590	// [impl->REQ-HAZARD-PUMP-IPC-DEADLINE]
591	fn peer_outcome(
592	    result: io::Result<()>,
593	    peer_hex: &str,
594	    conns: &mut HashMap<String, u64>,
595	) -> io::Result<()> {
596	    match result {
597	        Ok(()) => Ok(()),
598	        Err(e) if e.kind() == io::ErrorKind::TimedOut => Err(e), // poisoned → supervised restart
599	        Err(e) => {
600	            eprintln!("PUMP_PEER_FAIL:{peer_hex}:{e}");
601	            conns.remove(peer_hex); // ordinary failure → redial next tick
602	            Ok(())
603	        }
604	    }

pub fn run_peer_pump(
    broker_name: &str,
    registry: Arc<RegistryHost>,
    paths: &PumpPaths,
    resolver: PeerResolver,
    config: PumpConfig,
    hooks: PumpHooks,
    stop: &AtomicBool,
) -> io::Result<()> {
    let PumpConfig {
        cadence,
        full_auto_update,
    } = config;
    // PUMP MODE: a brain whose carrier is SPLIT at construction (reader thread +
    // channel) so every reply read is bounded by PUMP_PEER_IO_TIMEOUT — a
    // black-holed peer can never wedge this single-threaded loop (the 2.2h
    // hfenduleam stall). A timed-out read escalates to a supervised restart
    // below, never a per-peer retry (REQ-HAZARD-PUMP-IPC-DEADLINE).
    // [impl->REQ-HAZARD-PUMP-IPC-DEADLINE]
    let mut brain = connect(broker_name)?;
    let mut ops = EpochSource::load_from(&paths.ops);
    let mut conns: HashMap<String, u64> = HashMap::new();
    // The four cadenced legs behind the worker seam, in fan-out order
    // (registry → notif → sync → update). The shell drives them generically;
    // each leg's *what* lives in its module (§Frozen). Construction capture
    // (§Frozen-4): each worker owns the registry Arc / path slices / config
    // knob it needs — the sync worker additionally owns the conflict-surface
    // hooks (ADR-0013).
    let mut registry_worker = RegistryWorker::new(Arc::clone(&registry), paths, &cadence);
    let mut notif_worker = NotifWorker::new(paths, &cadence);
    let mut sync_worker = SyncWorker::new(Arc::clone(&registry), paths, &cadence, hooks);
    let mut update_worker = UpdateWorker::new(Arc::clone(&registry), paths, &cadence, full_auto_update);
    let mut workers: [&mut dyn PumpWorker; 4] = [
        &mut registry_worker,
        &mut notif_worker,
        &mut sync_worker,
        &mut update_worker,
    ];
    // WorkerLasts (ADR-0018 §V4): RAM-only last-ran instants, parallel to
    // `workers`. `None` = never run yet → due immediately, so the first tick
    // primes every leg. Never `Instant - Duration` — the underflow panic on a
    // sub-uptime monotonic clock (KH 5.9) lives nowhere in this path.
    let mut last: [Option<Instant>; 4] = [None; 4];
    let mut last_hb: Option<Instant> = None;

    while !stop.load(Ordering::Relaxed) {
        let now = Instant::now();
        // Shell-owned liveness heartbeat (M8 decision 23, REQ-DAEMON-5): written
        // while the loop is ALIVE — including idle ticks — never per work round,
        // so a wedged step and a healthy quiet pump are distinguishable.
        // Throttled; per-tick, never a worker (REQ-DAEMON-5), no wake.
        // [impl->REQ-DAEMON-5]
        if due(last_hb, false, now, HEARTBEAT_PERIOD) {
            write_heartbeat(&paths.heartbeat);
            last_hb = Some(now);
        }
        // Poll every worker's wake and decide due (the pure scheduler step):
        // every wake is consumed EVERY tick so a marker is never stranded (the
        // registry advertise-now, REQ-CONV-2; the sync freshness pull,
        // REQ-INST-3). The take IS the remove — one wake claims exactly one
        // forced round.
        let (due_flags, any_due) = compute_due(&mut workers, &last, now);
        if !any_due {
            thread::sleep(TICK);
            continue;
        }

        // Per-round shared loads (§Frozen-3): subnets / attachment /
        // roster-backed policy, read ONCE and handed to every worker.
        let ctx = RoundCtx::load(paths);

        // Each due worker's node-local pre_round, once per round before ANY
        // peer_step (only the registry leg does work here — its sweep +
        // advertise; the rest default to a no-op).
        pre_rounds(&mut workers, &due_flags, &ctx);

        // Fan out to every (subnet, peer) target (detached subnets skipped,
        // self excluded): establish the conn, then run each due worker's
        // peer_step in fan-out order. A failure aborts the remaining due
        // workers for THIS peer and drops its conn (abort-on-first-fail — the
        // broker reaps the dead conn; the next tick redials). Each worker mints
        // its open-ops only through the shell-owned `PeerIo`.
        for (sub_name, peer_hex) in fan_targets(&ctx, registry.node_hex()) {
            let Some(conn_id) = ensure_conn(
                &mut brain,
                &mut conns,
                &resolver,
                &peer_hex,
                &paths.peer_addrs,
            ) else {
                continue; // unresolvable/undialable now — next tick retries
            };
            let outcome = run_due(&due_flags, |i| {
                let mut io = PeerIo {
                    brain: &mut brain,
                    conn_id,
                    ops: &mut ops,
                };
                workers[i].peer_step(&mut io, &sub_name, &peer_hex, &ctx)
            });
            // Recovery tier-split: a TimedOut brain-IPC read bubbles OUT of the
            // round (supervised restart); any other error is an ordinary peer
            // failure (drop the conn, redial next tick).
            peer_outcome(outcome, &peer_hex, &mut conns)?;
        }

        // Mark-after-round with a POST-round instant (ADR-0018 §V4: the
        // stagger — ran = ATTEMPTED, not succeeded; the next due is measured
        // from here, not from the round's start).
        mark_ran(&mut last, &due_flags, Instant::now());
    }
    Ok(())
}

/// Compute each worker's due flag for this tick (the pure scheduler step):
/// poll EVERY worker's wake first — even a not-otherwise-due leg, so a marker
/// is never stranded — then the [`due`] kernel over its RAM-only last-ran
/// instant. Returns the per-worker flags (parallel to `workers`) and whether
/// ANY leg is due (an all-idle tick sleeps without loading a round).
fn compute_due(
    workers: &mut [&mut dyn PumpWorker],
    last: &[Option<Instant>],
    now: Instant,
) -> (Vec<bool>, bool) {
    let mut flags = vec![false; workers.len()];
    let mut any = false;
    for i in 0..workers.len() {
        let wake = workers[i].poll_wake();
        flags[i] = due(last[i], wake, now, workers[i].cadence());
        any |= flags[i];
    }
    (flags, any)
}

/// Run each due worker's node-local `pre_round` ONCE for the round, before any
/// peer_step (the registry sweep + advertise; the other legs default no-op).
fn pre_rounds(workers: &mut [&mut dyn PumpWorker], due_flags: &[bool], ctx: &RoundCtx) {
    for i in 0..workers.len() {
        if due_flags[i] {
            workers[i].pre_round(ctx);
        }
    }
}

/// Run `step` for each due worker index in fan-out order, aborting on the
/// FIRST error (abort-on-first-fail — the caller drops the peer's conn and
/// redials next tick). The peer-step IO itself lives in `step`; this is the
/// pure ordering/abort kernel, so the choreography is a unit fact.
fn run_due(due_flags: &[bool], mut step: impl FnMut(usize) -> io::Result<()>) -> io::Result<()> {
    for (i, &is_due) in due_flags.iter().enumerate() {
        if is_due {
            step(i)?;
        }
    }
    Ok(())
}

/// The recovery tier-split for a per-peer fan-out outcome (REQ-HAZARD-PUMP-IPC-
/// DEADLINE). A `TimedOut` is NOT an ordinary peer failure: a brain-IPC read
/// that timed out may have desynced the stream (a late reply could bind the
/// wrong stream id in a later call), so the client is POISONED — propagate it
/// (`Err`) to bubble out of the round and trigger a SUPERVISED RESTART (fresh
/// brain client + conn cache + WorkerLasts re-prime). Any other error is an
/// ordinary peer failure: drop the conn (redial next tick) and keep the round
/// going (`Ok`). A success keeps the conn.
// [impl->REQ-HAZARD-PUMP-IPC-DEADLINE]
fn peer_outcome(
    result: io::Result<()>,
    peer_hex: &str,
    conns: &mut HashMap<String, u64>,
) -> io::Result<()> {
    match result {
        Ok(()) => Ok(()),
        Err(e) if e.kind() == io::ErrorKind::TimedOut => Err(e), // poisoned → supervised restart
        Err(e) => {
            eprintln!("PUMP_PEER_FAIL:{peer_hex}:{e}");
            conns.remove(peer_hex); // ordinary failure → redial next tick
            Ok(())
        }
    }


`peer_outcome` keys off `io::ErrorKind::TimedOut` exactly. So my broker timeout must reach the brain as a NON-TimedOut error frame. The broker maps QUIC errors to String (kind lost), brain reconstructs. Must verify brain's error-frame→io::Error reconstruction isn't TimedOut. Check brain.rs request path.


Grep

fn net_dial|fn net_open_stream|fn net_stream_send|ErrorKind::TimedOut|fn request|error frame|KIND_ERROR|fn recv_reply|ErrorKind::Other|io::Error::other


44:    KIND_BRAIN_RESTARTED, KIND_DISPLACED, KIND_ERROR, KIND_EXIT, KIND_INPUT, KIND_KILL, KIND_NET_DIAL,
417:                    return Err(io::Error::other(message));
501:                    return Err(io::Error::other(message));
527:    /// [`io::ErrorKind::TimedOut`] — the pump escalates that to a SUPERVISED
731:                KIND_ERROR => {
826:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
856:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
868:    pub fn net_dial(
881:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
896:    pub fn net_dial_loopback(&mut self) -> io::Result<NetDialed> {
902:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
913:    pub fn net_open_stream(
927:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
938:    pub fn net_stream_send(
962:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
975:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
995:    pub fn request_brain_restart(&mut self) -> io::Result<bool> {
1004:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
1017:                BrokerEvent::Error { message } => return Err(io::Error::other(message)),
1123:    ///   on the remaining budget. A `Timeout` surfaces [`io::ErrorKind::TimedOut`]
1142:                            io::ErrorKind::TimedOut,


Confirmed invariant #1 is automatic: brain maps `BrokerEvent::Error` → `io::Error::other` (ErrorKind::Other, line 881/927/962); only the brain's OWN channel recv-deadline (line 1142) yields TimedOut. A ~10s broker bound replies an error frame well before the 30s brain deadline → non-TimedOut arm. And the QUIC ops are already called *inside* the `apply_once` closures, so a timeout Err propagates as closure-Err. Now verify invariant #3: apply_once must NOT record on closure-Err.


fn apply_once|fn apply\b|Outcome::Applied|record|already applied|Deduped


crates\spt-daemon\src\effect.rs:31://! applied-set, runs the effect, and records it **all under one lock in the
crates\spt-daemon\src\effect.rs:33://! the broker, that dies), so the effect-applied-but-not-recorded window — the
crates\spt-daemon\src\effect.rs:76:    /// The stable journal token for this kind. (The kind is recorded on the
crates\spt-daemon\src\effect.rs:99:    /// The effect was not in the applied-set; it ran and is now recorded.
crates\spt-daemon\src\effect.rs:101:    /// The effect was already applied; the closure was **not** run (the no-dup
crates\spt-daemon\src\effect.rs:103:    Deduped,
crates\spt-daemon\src\effect.rs:107:/// effect, and the record are one indivisible step.
crates\spt-daemon\src\effect.rs:110:    /// The authoritative applied-set: keys whose effect has run + been recorded.
crates\spt-daemon\src\effect.rs:113:    /// not record completion (a broker crash mid-effect; surfaced for the future
crates\spt-daemon\src\effect.rs:157:    /// Run `effect` exactly once for `key`. If `key` is already applied, the
crates\spt-daemon\src\effect.rs:158:    /// closure is **not** run and [`Outcome::Deduped`] is returned (the no-dup
crates\spt-daemon\src\effect.rs:161:    /// ([`Outcome::Applied`]). On closure error the `PENDING` record is left in
crates\spt-daemon\src\effect.rs:168:    pub fn apply_once<F>(&self, key: EffectKey, kind: EffectKind, effect: F) -> io::Result<Outcome>
crates\spt-daemon\src\effect.rs:174:            return Ok(Outcome::Deduped); // dedup-at-effect: already landed, do nothing
crates\spt-daemon\src\effect.rs:187:        Ok(Outcome::Applied)
crates\spt-daemon\src\effect.rs:190:    /// Whether `key`'s effect has already been applied + recorded.
crates\spt-daemon\src\effect.rs:309:            Outcome::Applied
crates\spt-daemon\src\effect.rs:335:        assert_eq!(o1, Outcome::Applied);
crates\spt-daemon\src\effect.rs:338:            Outcome::Deduped,
crates\spt-daemon\src\effect.rs:348:    fn failed_effect_is_not_recorded_applied() {
crates\spt-daemon\src\effect.rs:362:        assert_eq!(ok, Outcome::Applied);
crates\spt-daemon\src\effect.rs:391:        assert_eq!(o, Outcome::Deduped);
crates\spt-daemon\src\effect.rs:432:                return; // crashed before recording intent for this op
crates\spt-daemon\src\effect.rs:442:            debug_assert!(outcome == Outcome::Applied || outcome == Outcome::Deduped);
crates\spt-daemon\src\effect.rs:481:            assert!(j.is_applied((1, op)), "op {op} recorded applied");


155	    }
156
157	    /// Run `effect` exactly once for `key`. If `key` is already applied, the
158	    /// closure is **not** run and [`Outcome::Deduped`] is returned (the no-dup
159	    /// guard). Otherwise a `PENDING` line is fsync'd, the closure runs, and on
160	    /// success a `DONE` line is fsync'd and `key` joins the applied-set
161	    /// ([`Outcome::Applied`]). On closure error the `PENDING` record is left in
162	    /// place (the effect did not complete) and the error is propagated.
163	    ///
164	    /// The whole sequence holds the inner lock, so for any one broker process it
165	    /// is atomic against a concurrent re-drive — the property that makes a brain
166	    /// crash unable to open a duplicate window (see the module docs).
167	    // [impl->REQ-HAZARD-RESTART-IDEMPOTENT]
168	    pub fn apply_once<F>(&self, key: EffectKey, kind: EffectKind, effect: F) -> io::Result<Outcome>
169	    where
170	        F: FnOnce() -> io::Result<()>,
171	    {
172	        let mut g = self.inner.lock().expect("effect journal lock poisoned");
173	        if g.applied.contains(&key) {
174	            return Ok(Outcome::Deduped); // dedup-at-effect: already landed, do nothing
175	        }
176	        write_line(
177	            &mut g.file,
178	            &format!("PENDING {} {} {}", key.0, key.1, kind.as_str()),
179	        )?;
180	        g.pending.insert(key);
181	        // Run the real effect. A brain crash cannot interleave here — this is the
182	        // broker's thread, and the broker survives the brain.
183	        effect()?;
184	        write_line(&mut g.file, &format!("DONE {} {}", key.0, key.1))?;
185	        g.pending.remove(&key);
186	        g.applied.insert(key);
187	        Ok(Outcome::Applied)
188	    }
189
190	    /// Whether `key`'s effect has already been applied + recorded.
191	    pub fn is_applied(&self, key: EffectKey) -> bool {
192	        self.inner
193	            .lock()
194	            .expect("effect journal lock poisoned")

    }

    /// Run `effect` exactly once for `key`. If `key` is already applied, the
    /// closure is **not** run and [`Outcome::Deduped`] is returned (the no-dup
    /// guard). Otherwise a `PENDING` line is fsync'd, the closure runs, and on
    /// success a `DONE` line is fsync'd and `key` joins the applied-set
    /// ([`Outcome::Applied`]). On closure error the `PENDING` record is left in
    /// place (the effect did not complete) and the error is propagated.
    ///
    /// The whole sequence holds the inner lock, so for any one broker process it
    /// is atomic against a concurrent re-drive — the property that makes a brain
    /// crash unable to open a duplicate window (see the module docs).
    // [impl->REQ-HAZARD-RESTART-IDEMPOTENT]
    pub fn apply_once<F>(&self, key: EffectKey, kind: EffectKind, effect: F) -> io::Result<Outcome>
    where
        F: FnOnce() -> io::Result<()>,
    {
        let mut g = self.inner.lock().expect("effect journal lock poisoned");
        if g.applied.contains(&key) {
            return Ok(Outcome::Deduped); // dedup-at-effect: already landed, do nothing
        }
        write_line(
            &mut g.file,
            &format!("PENDING {} {} {}", key.0, key.1, kind.as_str()),
        )?;
        g.pending.insert(key);
        // Run the real effect. A brain crash cannot interleave here — this is the
        // broker's thread, and the broker survives the brain.
        effect()?;
        write_line(&mut g.file, &format!("DONE {} {}", key.0, key.1))?;
        g.pending.remove(&key);
        g.applied.insert(key);
        Ok(Outcome::Applied)
    }

    /// Whether `key`'s effect has already been applied + recorded.
    pub fn is_applied(&self, key: EffectKey) -> bool {
        self.inner
            .lock()
            .expect("effect journal lock poisoned")


Invariant #3 confirmed: `effect()?` (line 183) propagates **before** `applied.insert` (186) and the closure calls `host.record_dial_op` only after the QUIC op succeeds — so a timed-out op records no phantom conn/stream id. Existing test `failed_effect_is_not_recorded_applied` already proves the journal semantics.

Design is fully grounded. Now the test harness — read the A-half model + the int infra.


1	//! Brain-IPC deadline (REQ-HAZARD-PUMP-IPC-DEADLINE): a pump-mode brain whose
2	//! broker accepts a request but NEVER replies must surface
3	//! [`io::ErrorKind::TimedOut`] from the bounded read — it must never block
4	//! forever. This is the unit that replaces the retired codec poll test, raised
5	//! to the brain level so it exercises the real carrier: a split stream
6	//! (reader thread + channel) over a REAL local socket, the exact mechanism the
7	//! pump runs on (and the only one that works on Windows named pipes, where the
8	//! non-blocking toggle corrupts mid-stream).
9	//!
10	//! The wedge this guards: the 2.2h hfenduleam pump stall on
11	//! `net_open_stream`'s unbounded `loop { read_event }` against a black-holed
12	//! peer the broker never answered for.
13
14	use std::io;
15	use std::sync::atomic::{AtomicU32, Ordering};
16	use std::thread;
17	use std::time::{Duration, Instant};
18
19	use spt_daemon::brain::Brain;
20	use spt_daemon::codec::read_frame;
21	use spt_daemon::transport::{recv_hello, DaemonTransport, LocalSocketTransport};
22	use spt_daemon::frame::Role;
23
24	static SEQ: AtomicU32 = AtomicU32::new(0);
25	fn unique_name() -> String {
26	    let n = SEQ.fetch_add(1, Ordering::Relaxed);
27	    format!("spt-daemon-ipc-deadline-{}-{}.sock", std::process::id(), n)
28	}
29
30	// [unit->REQ-HAZARD-PUMP-IPC-DEADLINE] a pump-mode brain reading a reply the
31	// broker never sends times out at the per-call deadline (TimedOut) instead of
32	// wedging forever — and within a small multiple of the budget, not the 30s
33	// production value. The stub broker holds the conn OPEN for the whole window
34	// (a blocking read that never returns), so the result is a genuine read-
35	// deadline timeout, not an EOF from a dropped connection.
36	#[test]
37	fn pump_brain_times_out_when_broker_never_replies() {
38	    let name = unique_name();
39	    let broker = LocalSocketTransport::bind(&name).expect("bind stub broker");
40
41	    // A short deadline so the test is fast; the production value is 30s.
42	    let timeout = Duration::from_millis(200);
43
44	    // Stub broker: accept, complete the hello handshake, read the brain's
45	    // open-stream request, then NEVER reply — hold the conn OPEN for well past
46	    // the deadline window (so the client times out on the MISSING reply, not on
47	    // an EOF), then return and drop the conn. We deliberately do NOT wait on the
48	    // client closing its handle: the pump carrier's abandoned `pump-ipc-reader`
49	    // thread keeps the client's `RecvHalf` open (split halves share one OS
50	    // handle), so an EOF-driven join would hang — the very KH 7.6 leak this fix
51	    // accepts. The bounded sleep sidesteps it; the stub owns its own exit.
52	    let server = thread::spawn(move || {
53	        let mut conn = broker.accept().expect("accept brain");
54	        recv_hello(&mut conn, Role::Brain).expect("hello handshake");
55	        // The KIND_NET_STREAM_OPEN request the brain sends before it waits.
56	        let _req = read_frame(&mut conn).expect("read open request");
57	        // Keep the half-open alive across the client's whole deadline window,
58	        // then drop it (unblocking the client's reader thread to exit cleanly).
59	        thread::sleep(timeout * 3);
60	    });
61	    let mut brain =
62	        Brain::cold_start_pump(&name, 0, timeout).expect("connect pump-mode brain");
63
64	    let started = Instant::now();
65	    let err = brain
66	        .net_open_stream(1, None)
67	        .expect_err("a never-answered open must error, never block forever");
68	    let elapsed = started.elapsed();
69
70	    assert_eq!(
71	        err.kind(),
72	        io::ErrorKind::TimedOut,
73	        "the bounded read surfaces TimedOut (the pump's supervised-restart signal)"
74	    );
75	    assert!(
76	        elapsed >= timeout,
77	        "it waited at least the deadline ({elapsed:?} >= {timeout:?})"
78	    );
79	    assert!(
80	        elapsed < timeout * 10,
81	        "and not wildly past it ({elapsed:?}) — the deadline actually fired"
82	    );
83
84	    // Dropping the brain closes the client half, unblocking the stub's parked
85	    // read so its thread exits cleanly.
86	    drop(brain);
87	    server.join().expect("stub broker thread");
88	}
89

//! Brain-IPC deadline (REQ-HAZARD-PUMP-IPC-DEADLINE): a pump-mode brain whose
//! broker accepts a request but NEVER replies must surface
//! [`io::ErrorKind::TimedOut`] from the bounded read — it must never block
//! forever. This is the unit that replaces the retired codec poll test, raised
//! to the brain level so it exercises the real carrier: a split stream
//! (reader thread + channel) over a REAL local socket, the exact mechanism the
//! pump runs on (and the only one that works on Windows named pipes, where the
//! non-blocking toggle corrupts mid-stream).
//!
//! The wedge this guards: the 2.2h hfenduleam pump stall on
//! `net_open_stream`'s unbounded `loop { read_event }` against a black-holed
//! peer the broker never answered for.

use std::io;
use std::sync::atomic::{AtomicU32, Ordering};
use std::thread;
use std::time::{Duration, Instant};

use spt_daemon::brain::Brain;
use spt_daemon::codec::read_frame;
use spt_daemon::transport::{recv_hello, DaemonTransport, LocalSocketTransport};
use spt_daemon::frame::Role;

static SEQ: AtomicU32 = AtomicU32::new(0);
fn unique_name() -> String {
    let n = SEQ.fetch_add(1, Ordering::Relaxed);
    format!("spt-daemon-ipc-deadline-{}-{}.sock", std::process::id(), n)
}

// [unit->REQ-HAZARD-PUMP-IPC-DEADLINE] a pump-mode brain reading a reply the
// broker never sends times out at the per-call deadline (TimedOut) instead of
// wedging forever — and within a small multiple of the budget, not the 30s
// production value. The stub broker holds the conn OPEN for the whole window
// (a blocking read that never returns), so the result is a genuine read-
// deadline timeout, not an EOF from a dropped connection.
#[test]
fn pump_brain_times_out_when_broker_never_replies() {
    let name = unique_name();
    let broker = LocalSocketTransport::bind(&name).expect("bind stub broker");

    // A short deadline so the test is fast; the production value is 30s.
    let timeout = Duration::from_millis(200);

    // Stub broker: accept, complete the hello handshake, read the brain's
    // open-stream request, then NEVER reply — hold the conn OPEN for well past
    // the deadline window (so the client times out on the MISSING reply, not on
    // an EOF), then return and drop the conn. We deliberately do NOT wait on the
    // client closing its handle: the pump carrier's abandoned `pump-ipc-reader`
    // thread keeps the client's `RecvHalf` open (split halves share one OS
    // handle), so an EOF-driven join would hang — the very KH 7.6 leak this fix
    // accepts. The bounded sleep sidesteps it; the stub owns its own exit.
    let server = thread::spawn(move || {
        let mut conn = broker.accept().expect("accept brain");
        recv_hello(&mut conn, Role::Brain).expect("hello handshake");
        // The KIND_NET_STREAM_OPEN request the brain sends before it waits.
        let _req = read_frame(&mut conn).expect("read open request");
        // Keep the half-open alive across the client's whole deadline window,
        // then drop it (unblocking the client's reader thread to exit cleanly).
        thread::sleep(timeout * 3);
    });
    let mut brain =
        Brain::cold_start_pump(&name, 0, timeout).expect("connect pump-mode brain");

    let started = Instant::now();
    let err = brain
        .net_open_stream(1, None)
        .expect_err("a never-answered open must error, never block forever");
    let elapsed = started.elapsed();

    assert_eq!(
        err.kind(),
        io::ErrorKind::TimedOut,
        "the bounded read surfaces TimedOut (the pump's supervised-restart signal)"
    );
    assert!(
        elapsed >= timeout,
        "it waited at least the deadline ({elapsed:?} >= {timeout:?})"
    );
    assert!(
        elapsed < timeout * 10,
        "and not wildly past it ({elapsed:?}) — the deadline actually fired"
    );

    // Dropping the brain closes the client half, unblocking the stub's parked
    // read so its thread exits cleanly.
    drop(brain);
    server.join().expect("stub broker thread");
}


A-half model is the brain-read deadline. My B-half int needs a broker-side QUIC hang + the bound configurable for hermetic tests (the int lives in `tests/`, external crate → needs public injection). Read the existing int harness to see what's available.


Bash

echo "=== netbroker.rs lines ==="; wc -l crates/spt-daemon/tests/netbroker.rs crates/spt-daemon/tests/pump.rs; echo "=== heartbeat/restart refs in pump.rs ==="; grep -n "heartbeat\|PEER_PUMP_RESTART\|black\|never_repl\|unreachable\|NetHost\|fn test\|#\[test\]\|run_peer_pump\|supervise" crates/spt-daemon/tests/pump.rs | head -50

Survey int test harness


=== netbroker.rs lines ===
  234 crates/spt-daemon/tests/netbroker.rs
  423 crates/spt-daemon/tests/pump.rs
  657 total
=== heartbeat/restart refs in pump.rs ===
19:use spt_daemon::nethost::{NetConfig, NetHost};
21:    run_peer_pump, PeerResolver, PumpCadence, PumpConfig, PumpHooks, PumpPaths,
67:    let host = NetHost::start(hermetic()).expect("net host start");
106:#[test]
217:        heartbeat: a_root.join("pump-heartbeat.json"),
347:            heartbeat: a_paths.heartbeat.clone(),
364:            let _ = run_peer_pump(


1	//! Peer-pump E2E (D9-1-2): the subnet self-drives. One in-process "node A"
2	//! runs the production pump loop against "node B" running the production
3	//! dispatcher — and registry rows, notif rows, and a mind A never held all
4	//! converge with NO test code driving a single protocol exchange.
5	//!
6	//! Topology: B owns the process's canonical SPT_HOME (its dispatcher's gates
7	//! load from it — the production read path); A's pump-side stores live under
8	//! explicit test roots ([`PumpPaths`]). The resolver is injected (hermetic
9	//! loopback endpoints run no discovery); production resolves by id.
10
11	use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
12	use std::sync::{Arc, Mutex, OnceLock};
13	use std::thread;
14	use std::time::Duration;
15
16	use spt_daemon::brain::Brain;
17	use spt_daemon::broker::Broker;
18	use spt_daemon::dispatch::{run_dispatch_loop, DispatchPaths};
19	use spt_daemon::nethost::{NetConfig, NetHost};
20	use spt_daemon::pump::{
21	    run_peer_pump, PeerResolver, PumpCadence, PumpConfig, PumpHooks, PumpPaths,
22	};
23	use spt_daemon::registryhost::{RegistryGatePolicy, RegistryHost};
24	use spt_daemon::relcache::ReleaseCache;
25	use spt_daemon::release::{sha256_hex, ReleaseMetadata, SignedRelease};
26	use spt_daemon::update::BROKER_RESOURCE_ABI;
27	use spt_net::net::endpoint::{BindScope, LocalDiscovery, RelayPolicy};
28	use spt_net::net::registry::{Instance, Status};
29	use spt_net::net::replicate::RegistryUpdate;
30	use spt_proto::identity::Identity;
31	use spt_store::contextstore::ContextStore;
32	use spt_store::epoch::EpochSource;
33	use spt_store::info::{self, InfoJson};
34	use spt_store::notif::{NotifRow, NotifStore};
35	use spt_store::perch::{self, ParentHint};
36	use spt_store::subnet::SubnetStore;
37	use spt_store::visibility::VisibilityStore;
38	use tempfile::TempDir;
39
40	static HOME: OnceLock<TempDir> = OnceLock::new();
41	static LOCK: Mutex<()> = Mutex::new(());
42	fn init_home() {
43	    HOME.get_or_init(|| {
44	        let dir = TempDir::new().expect("temp dir");
45	        std::env::set_var("SPT_HOME", dir.path());
46	        dir
47	    });
48	}
49
50	static SEQ: AtomicU32 = AtomicU32::new(0);
51	fn unique_name() -> String {
52	    let n = SEQ.fetch_add(1, Ordering::Relaxed);
53	    format!("spt-daemon-d912-{}-{}.sock", std::process::id(), n)
54	}
55
56	fn hermetic() -> NetConfig {
57	    NetConfig {
58	        identity: Identity::generate(),
59	        relay: RelayPolicy::Disabled,
60	        discovery: LocalDiscovery::Off,
61	        scope: BindScope::Loopback,
62	        membership: None,
63	    }
64	}
65
66	fn net_broker(name: &str, dir: &std::path::Path) -> Arc<Broker> {
67	    let host = NetHost::start(hermetic()).expect("net host start");
68	    let broker =
69	        Broker::bind_in_with_net(name, dir.join("effects.log"), Some(host)).expect("bind broker");
70	    let serve = Arc::clone(&broker);
71	    thread::spawn(move || {
72	        let _ = serve.serve();
73	    });
74	    broker
75	}
76
77	fn connect_retry(name: &str) -> Brain {
78	    for _ in 0..300 {
79	        if let Ok(b) = Brain::cold_start(name, 1) {
80	            return b;
81	        }
82	        thread::sleep(Duration::from_millis(10));
83	    }
84	    panic!("brain could not connect");
85	}
86
87	fn stamped(body: &str) -> String {
88	    format!("<!-- spt:source=llm spt:routed_at_ms=1000 spt:node=t spt:vector=t:1 -->\n{body}")
89	}
90
91	/// Poll `probe` until it returns true, panicking with `what` on timeout.
92	fn converge(what: &str, mut probe: impl FnMut() -> bool) {
93	    for _ in 0..600 {
94	        if probe() {
95	            return;
96	        }
97	        thread::sleep(Duration::from_millis(25));
98	    }
99	    panic!("never converged: {what}");
100	}
101
102	// [int->REQ-INST-5] [int->REQ-INST-7] [int->REQ-NOTIF-1] the full self-drive:
103	// A's pump pushes its registry advertisement and notif spool to B and pulls
104	// from B a mind A NEVER HELD (registry-derived bootstrap ref) — all three
105	// converge undriven, A's pump against B's dispatcher, both production loops.
106	#[test]
107	fn pump_and_dispatch_self_drive_the_subnet() {
108	    init_home();
109	    let _g = LOCK.lock().unwrap_or_else(|p| p.into_inner());
110	    let dir = TempDir::new().unwrap();
111	    let subnet = "d912";
112	    let (a_name, b_name) = (unique_name(), unique_name());
113	    let _a_broker = net_broker(&a_name, dir.path());
114	    let _b_broker = net_broker(&b_name, dir.path());
115
116	    let mut a_probe = connect_retry(&a_name);
117	    let mut b_probe = connect_retry(&b_name);
118	    let a_hex = a_probe
119	        .net_status()
120	        .expect("status")
121	        .node_id_hex
122	        .expect("net");
123	    let b_hex = b_probe
124	        .net_status()
125	        .expect("status")
126	        .node_id_hex
127	        .expect("net");
128	    let b_addr = b_probe.net_status().expect("status").addr;
129
130	    // ── B (canonical home): gates + the mind A will bootstrap-pull. ──────
131	    let mut subnets = SubnetStore::load();
132	    subnets.create_subnet(subnet).expect("create subnet");
133	    subnets.save().expect("save subnets");
134	    // Mesh-D6: membership is the roster — B rosters A (both the inbound gate and
135	    // the pump's push target read it). Seed the canonical roster.
136	    let mut roster = spt_store::roster::RosterStore::load();
137	    roster.merge_entry(spt_store::roster::RosterEntry {
138	        pubkey_hex: a_hex.clone(),
139	        subnet: subnet.to_string(),
140	        label: String::new(),
141	        machine_id: String::new(),
142	        address: None,
143	        last_seen: String::new(),
144	        lease_epoch: 1,
145	    });
146	    roster.save().expect("save roster");
147	    let mut vis = VisibilityStore::load();
148	    vis.set_sync_subnets("ling-pumped", vec![subnet.to_string()]);
149	    vis.save().expect("save visibility");
150
151	    let cs_b = ContextStore::open_or_init().expect("B context store");
152	    let p = cs_b.live_context_path("ling-pumped").unwrap();
153	    std::fs::write(&p, stamped("a mind worth following")).unwrap();
154	    cs_b.commit_live("ling-pumped", "seed").unwrap();
155
156	    // B stages a v6 release (the canonical releases dir B's dispatcher serves
157	    // from); A's keys file will trust the signer.
158	    use ed25519_dalek::{Signer, SigningKey};
159	    let sk = SigningKey::from_bytes(&[5u8; 32]);
160	    let artifact = b"pump-propagated binary".to_vec();
161	    let meta = ReleaseMetadata {
162	        version: 6,
163	        channel: "stable".to_string(),
164	        expires_at_ms: u64::MAX,
165	        key_id: "k".to_string(),
166	        artifact_sha256: sha256_hex(&artifact),
167	        brain_ipc_version: spt_daemon::IPC_PROTOCOL_VERSION,
168	        broker_resource_abi: BROKER_RESOURCE_ABI,
169	        product_version: String::new(),
170	    };
171	    let metadata_json = serde_json::to_string(&meta).unwrap();
172	    let sig = sk.sign(metadata_json.as_bytes());
173	    let signed = SignedRelease {
174	        metadata_json,
175	        signature_hex: sig.to_bytes().iter().map(|b| format!("{b:02x}")).collect(),
176	    };
177	    ReleaseCache::open(&perch::spt_home().join("releases"))
178	        .stage(&signed, &artifact)
179	        .expect("stage at B");
180
181	    let b_registry = Arc::new(RegistryHost::new_at(
182	        &b_hex,
183	        EpochSource::load_from(&dir.path().join("b-epoch.json")),
184	        dir.path().join("b-registry"),
185	    ));
186	    let stop = Arc::new(AtomicBool::new(false));
187	    {
188	        let name = b_name.clone();
189	        let reg = Arc::clone(&b_registry);
190	        let flag = Arc::clone(&stop);
191	        let paths = DispatchPaths {
192	            owlery: perch::owlery_dir(),
193	            releases: perch::spt_home().join("releases"),
194	            scratch: dir.path().join("b-scratch"),
195	        };
196	        thread::spawn(move || {
197	            let _ = run_dispatch_loop(&name, &paths, reg, &flag, Duration::from_millis(25));
198	        });
199	    }
200
201	    // ── A (explicit roots): a perch to advertise, a notif row to push, an
202	    //    empty context store that will receive the bootstrap pull. ────────
203	    let a_root = dir.path().join("a");
204	    let a_paths = PumpPaths {
205	        owlery: a_root.join("owlery"),
206	        ctx_root: a_root.join("context"),
207	        notif_db: a_root.join("notifs.db"),
208	        scratch: a_root.join("scratch"),
209	        subnets: a_root.join("subnet.json"),
210	        vis: a_root.join("visibility.json"),
211	        ops: a_root.join("pump-ops.json"),
212	        releases: a_root.join("releases"),
213	        release_keys: a_root.join("release-keys.json"),
214	        pull_marker: a_root.join("sync-pull-now"),
215	        advertise_marker: a_root.join("advertise-now"),
216	        peer_addrs: a_root.join("peer-addrs.json"),
217	        heartbeat: a_root.join("pump-heartbeat.json"),
218	        rotations: a_root.join("rotation-pending.json"),
219	    };
220	    // A trusts B's release signer — the update leg's production gate.
221	    std::fs::create_dir_all(&a_root).unwrap();
222	    let keys_json = serde_json::json!({
223	        "keys": { "k": sk.verifying_key().as_bytes().iter()
224	            .map(|b| format!("{b:02x}")).collect::<String>() },
225	        "channel": "stable",
226	    });
227	    std::fs::write(
228	        &a_paths.release_keys,
229	        serde_json::to_string(&keys_json).unwrap(),
230	    )
231	    .unwrap();
232	    let perch_path = perch::resolve_perch_path_in(&a_paths.owlery, "pump-ag", ParentHint::Infer);
233	    std::fs::create_dir_all(&perch_path).unwrap();
234	    let rec = InfoJson::new("pump-ag", "now", std::process::id(), "s", "ready_agent");
235	    info::write_info(&perch_path, &rec).unwrap();
236
237	    let mut a_subnets = SubnetStore::default();
238	    a_subnets.create_subnet(subnet).expect("A subnet");
239	    a_subnets.save_to(&a_paths.subnets).expect("save A subnets");
240	    // Mesh-D6: A's membership IS the roster — both its inbound gates and its
241	    // pump push target read it. The pump derives the roster path as a sibling of
242	    // `subnets` (subnet.json → roster.json), so write it there.
243	    let mut a_roster = spt_store::roster::RosterStore::default();
244	    a_roster.merge_entry(spt_store::roster::RosterEntry {
245	        pubkey_hex: b_hex.clone(),
246	        subnet: subnet.to_string(),
247	        label: String::new(),
248	        machine_id: String::new(),
249	        address: None,
250	        last_seen: String::new(),
251	        lease_epoch: 1,
252	    });
253	    a_roster
254	        .save_to(&a_root.join("roster.json"))
255	        .expect("save A roster");
256	    let mut a_vis = VisibilityStore::default();
257	    a_vis.set_sync_subnets("ling-pumped", vec![subnet.to_string()]);
258	    a_vis.save_to(&a_paths.vis).expect("save A visibility");
259
260	    let a_store = NotifStore::open_at(&a_paths.notif_db).expect("A notif store");
261	    a_store
262	        .merge_row(&NotifRow {
263	            notif_id: format!("{a_hex}:1"),
264	            subnet: subnet.to_string(),
265	            kind: "agent".to_string(),
266	            from_id: format!("pump-ag@{a_hex}"),
267	            body: "pumped across".to_string(),
268	            created_ms: 1_000,
269	            dismissed: false,
270	            seen: Default::default(),
271	            last_surfaced_ms: 0,
272	        })
273	        .expect("seed A notif row");
274
275	    let a_registry = Arc::new(RegistryHost::new_at(
276	        &a_hex,
277	        EpochSource::load_from(&a_root.join("epoch.json")),
278	        a_root.join("registry"),
279	    ));
280	    // A learned of B's endpoint via an earlier inbound feed (seeded directly
281	    // here — B runs no pump in this test): the bootstrap-ref source.
282	    {
283	        let mut roster = spt_store::roster::RosterStore::default();
284	        roster.merge_entry(spt_store::roster::RosterEntry {
285	            pubkey_hex: b_hex.clone(),
286	            subnet: subnet.to_string(),
287	            label: String::new(),
288	            machine_id: String::new(),
289	            address: None,
290	            last_seen: String::new(),
291	            lease_epoch: 1,
292	        });
293	        let pol = RegistryGatePolicy {
294	            subnets: vec![subnet.to_string()],
295	            roster,
296	        };
297	        a_registry.apply_feed(
298	            &b_hex,
299	            &[RegistryUpdate {
300	                subnet: subnet.to_string(),
301	                endpoint_id: "ling-pumped".to_string(),
302	                instance: Instance {
303	                    node: b_hex.clone(),
304	                    status: Status::Active,
305	                    epoch: 1,
306	                    resources: None,
307	                    last_active_ms: None,
308	                    shell_adapters: Vec::new(),
309	                    node_label: None,
310	                    machine_id: None,
311	                    endpoint_type: None,
312	                },
313	            }],
314	            &pol,
315	        );
316	    }
317
318	    // ── The pump: production loop, injected resolver, fast cadences. ─────
319	    let resolver: PeerResolver = {
320	        let b_hex = b_hex.clone();
321	        let b_addr = b_addr.clone();
322	        Box::new(move |hex| {
323	            if hex == b_hex {
324	                Some(b_addr.clone())
325	            } else {
326	                None
327	            }
328	        })
329	    };
330	    {
331	        let name = a_name.clone();
332	        let reg = Arc::clone(&a_registry);
333	        let flag = Arc::clone(&stop);
334	        let paths = PumpPaths {
335	            owlery: a_paths.owlery.clone(),
336	            ctx_root: a_paths.ctx_root.clone(),
337	            notif_db: a_paths.notif_db.clone(),
338	            scratch: a_paths.scratch.clone(),
339	            subnets: a_paths.subnets.clone(),
340	            vis: a_paths.vis.clone(),
341	            ops: a_paths.ops.clone(),
342	            releases: a_paths.releases.clone(),
343	            release_keys: a_paths.release_keys.clone(),
344	            pull_marker: a_paths.pull_marker.clone(),
345	            advertise_marker: a_paths.advertise_marker.clone(),
346	            peer_addrs: a_paths.peer_addrs.clone(),
347	            heartbeat: a_paths.heartbeat.clone(),
348	            rotations: a_paths.rotations.clone(),
349	        };
350	        thread::spawn(move || {
351	            let config = PumpConfig {
352	                cadence: PumpCadence {
353	                    registry: Duration::from_millis(100),
354	                    notif: Duration::from_millis(100),
355	                    sync: Duration::from_millis(100),
356	                    update: Duration::from_millis(100),
357	                    // Inert here: both loopback nodes feed each other every
358	                    // tick, so nothing is ever silent for 300s.
359	                    registry_evict_after: Duration::from_secs(300),
360	                },
361	                // Gated default (REQ-UPD-4) — the consent notif is the assert.
362	                full_auto_update: false,
363	            };
364	            let _ = run_peer_pump(
365	                &name,
366	                reg,
367	                &paths,
368	                resolver,
369	                config,
370	                PumpHooks::default(),
371	                &flag,
372	            );
373	        });
374	    }
375
376	    // 1. Registry converged at B: A's perch row arrived through the
377	    //    dispatcher's gated registry arm.
378	    converge("B registry holds pump-ag@A", || {
379	        b_registry
380	            .rows(subnet, "pump-ag")
381	            .iter()
382	            .any(|i| i.node == a_hex)
383	    });
384
385	    // 2. Notif converged at B's canonical store (the semilattice join).
386	    let store_b = NotifStore::open().expect("B notif store");
387	    converge("B notif store holds A's row", || {
388	        store_b
389	            .list(subnet)
390	            .map(|rows| rows.iter().any(|r| r.body == "pumped across"))
391	            .unwrap_or(false)
392	    });
393
394	    // 3. The bootstrap pull: A now holds a mind it NEVER had locally — the
395	    //    want-ref came from A's registry, the serve gate ran at B.
396	    let cs_a = ContextStore::open_or_init_in(&a_paths.ctx_root).expect("A context store");
397	    converge("A pulled the registry-derived mind", || {
398	        cs_a.branch_store()
399	            .tip("a-ling-pumped")
400	            .ok()
401	            .flatten()
402	            .is_some()
403	    });
404
405	    // 4. The update leg: A pulled, gated, and staged B's v6 under the
406	    //    production keys file — and surfaced it as a CONSENT notif (REQ-UPD-4
407	    //    gated default), never a silent auto-apply.
408	    let cache_a = ReleaseCache::open(&a_paths.releases);
409	    converge("A staged the propagated release", || {
410	        cache_a
411	            .load_artifact()
412	            .map(|b| b == artifact)
413	            .unwrap_or(false)
414	    });
415	    converge("A holds the consent notif", || {
416	        a_store
417	            .list(subnet)
418	            .map(|rows| rows.iter().any(|r| r.kind == "consent"))
419	            .unwrap_or(false)
420	    });
421
422	    stop.store(true, Ordering::Relaxed);
423	}
424

//! Peer-pump E2E (D9-1-2): the subnet self-drives. One in-process "node A"
//! runs the production pump loop against "node B" running the production
//! dispatcher — and registry rows, notif rows, and a mind A never held all
//! converge with NO test code driving a single protocol exchange.
//!
//! Topology: B owns the process's canonical SPT_HOME (its dispatcher's gates
//! load from it — the production read path); A's pump-side stores live under
//! explicit test roots ([`PumpPaths`]). The resolver is injected (hermetic
//! loopback endpoints run no discovery); production resolves by id.

use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::{Arc, Mutex, OnceLock};
use std::thread;
use std::time::Duration;

use spt_daemon::brain::Brain;
use spt_daemon::broker::Broker;
use spt_daemon::dispatch::{run_dispatch_loop, DispatchPaths};
use spt_daemon::nethost::{NetConfig, NetHost};
use spt_daemon::pump::{
    run_peer_pump, PeerResolver, PumpCadence, PumpConfig, PumpHooks, PumpPaths,
};
use spt_daemon::registryhost::{RegistryGatePolicy, RegistryHost};
use spt_daemon::relcache::ReleaseCache;
use spt_daemon::release::{sha256_hex, ReleaseMetadata, SignedRelease};
use spt_daemon::update::BROKER_RESOURCE_ABI;
use spt_net::net::endpoint::{BindScope, LocalDiscovery, RelayPolicy};
use spt_net::net::registry::{Instance, Status};
use spt_net::net::replicate::RegistryUpdate;
use spt_proto::identity::Identity;
use spt_store::contextstore::ContextStore;
use spt_store::epoch::EpochSource;
use spt_store::info::{self, InfoJson};
use spt_store::notif::{NotifRow, NotifStore};
use spt_store::perch::{self, ParentHint};
use spt_store::subnet::SubnetStore;
use spt_store::visibility::VisibilityStore;
use tempfile::TempDir;

static HOME: OnceLock<TempDir> = OnceLock::new();
static LOCK: Mutex<()> = Mutex::new(());
fn init_home() {
    HOME.get_or_init(|| {
        let dir = TempDir::new().expect("temp dir");
        std::env::set_var("SPT_HOME", dir.path());
        dir
    });
}

static SEQ: AtomicU32 = AtomicU32::new(0);
fn unique_name() -> String {
    let n = SEQ.fetch_add(1, Ordering::Relaxed);
    format!("spt-daemon-d912-{}-{}.sock", std::process::id(), n)
}

fn hermetic() -> NetConfig {
    NetConfig {
        identity: Identity::generate(),
        relay: RelayPolicy::Disabled,
        discovery: LocalDiscovery::Off,
        scope: BindScope::Loopback,
        membership: None,
    }
}

fn net_broker(name: &str, dir: &std::path::Path) -> Arc<Broker> {
    let host = NetHost::start(hermetic()).expect("net host start");
    let broker =
        Broker::bind_in_with_net(name, dir.join("effects.log"), Some(host)).expect("bind broker");
    let serve = Arc::clone(&broker);
    thread::spawn(move || {
        let _ = serve.serve();
    });
    broker
}

fn connect_retry(name: &str) -> Brain {
    for _ in 0..300 {
        if let Ok(b) = Brain::cold_start(name, 1) {
            return b;
        }
        thread::sleep(Duration::from_millis(10));
    }
    panic!("brain could not connect");
}

fn stamped(body: &str) -> String {
    format!("<!-- spt:source=llm spt:routed_at_ms=1000 spt:node=t spt:vector=t:1 -->\n{body}")
}

/// Poll `probe` until it returns true, panicking with `what` on timeout.
fn converge(what: &str, mut probe: impl FnMut() -> bool) {
    for _ in 0..600 {
        if probe() {
            return;
        }
        thread::sleep(Duration::from_millis(25));
    }
    panic!("never converged: {what}");
}

// [int->REQ-INST-5] [int->REQ-INST-7] [int->REQ-NOTIF-1] the full self-drive:
// A's pump pushes its registry advertisement and notif spool to B and pulls
// from B a mind A NEVER HELD (registry-derived bootstrap ref) — all three
// converge undriven, A's pump against B's dispatcher, both production loops.
#[test]
fn pump_and_dispatch_self_drive_the_subnet() {
    init_home();
    let _g = LOCK.lock().unwrap_or_else(|p| p.into_inner());
    let dir = TempDir::new().unwrap();
    let subnet = "d912";
    let (a_name, b_name) = (unique_name(), unique_name());
    let _a_broker = net_broker(&a_name, dir.path());
    let _b_broker = net_broker(&b_name, dir.path());

    let mut a_probe = connect_retry(&a_name);
    let mut b_probe = connect_retry(&b_name);
    let a_hex = a_probe
        .net_status()
        .expect("status")
        .node_id_hex
        .expect("net");
    let b_hex = b_probe
        .net_status()
        .expect("status")
        .node_id_hex
        .expect("net");
    let b_addr = b_probe.net_status().expect("status").addr;

    // ── B (canonical home): gates + the mind A will bootstrap-pull. ──────
    let mut subnets = SubnetStore::load();
    subnets.create_subnet(subnet).expect("create subnet");
    subnets.save().expect("save subnets");
    // Mesh-D6: membership is the roster — B rosters A (both the inbound gate and
    // the pump's push target read it). Seed the canonical roster.
    let mut roster = spt_store::roster::RosterStore::load();
    roster.merge_entry(spt_store::roster::RosterEntry {
        pubkey_hex: a_hex.clone(),
        subnet: subnet.to_string(),
        label: String::new(),
        machine_id: String::new(),
        address: None,
        last_seen: String::new(),
        lease_epoch: 1,
    });
    roster.save().expect("save roster");
    let mut vis = VisibilityStore::load();
    vis.set_sync_subnets("ling-pumped", vec![subnet.to_string()]);
    vis.save().expect("save visibility");

    let cs_b = ContextStore::open_or_init().expect("B context store");
    let p = cs_b.live_context_path("ling-pumped").unwrap();
    std::fs::write(&p, stamped("a mind worth following")).unwrap();
    cs_b.commit_live("ling-pumped", "seed").unwrap();

    // B stages a v6 release (the canonical releases dir B's dispatcher serves
    // from); A's keys file will trust the signer.
    use ed25519_dalek::{Signer, SigningKey};
    let sk = SigningKey::from_bytes(&[5u8; 32]);
    let artifact = b"pump-propagated binary".to_vec();
    let meta = ReleaseMetadata {
        version: 6,
        channel: "stable".to_string(),
        expires_at_ms: u64::MAX,
        key_id: "k".to_string(),
        artifact_sha256: sha256_hex(&artifact),
        brain_ipc_version: spt_daemon::IPC_PROTOCOL_VERSION,
        broker_resource_abi: BROKER_RESOURCE_ABI,
        product_version: String::new(),
    };
    let metadata_json = serde_json::to_string(&meta).unwrap();
    let sig = sk.sign(metadata_json.as_bytes());
    let signed = SignedRelease {
        metadata_json,
        signature_hex: sig.to_bytes().iter().map(|b| format!("{b:02x}")).collect(),
    };
    ReleaseCache::open(&perch::spt_home().join("releases"))
        .stage(&signed, &artifact)
        .expect("stage at B");

    let b_registry = Arc::new(RegistryHost::new_at(
        &b_hex,
        EpochSource::load_from(&dir.path().join("b-epoch.json")),
        dir.path().join("b-registry"),
    ));
    let stop = Arc::new(AtomicBool::new(false));
    {
        let name = b_name.clone();
        let reg = Arc::clone(&b_registry);
        let flag = Arc::clone(&stop);
        let paths = DispatchPaths {
            owlery: perch::owlery_dir(),
            releases: perch::spt_home().join("releases"),
            scratch: dir.path().join("b-scratch"),
        };
        thread::spawn(move || {
            let _ = run_dispatch_loop(&name, &paths, reg, &flag, Duration::from_millis(25));
        });
    }

    // ── A (explicit roots): a perch to advertise, a notif row to push, an
    //    empty context store that will receive the bootstrap pull. ────────
    let a_root = dir.path().join("a");
    let a_paths = PumpPaths {
        owlery: a_root.join("owlery"),
        ctx_root: a_root.join("context"),
        notif_db: a_root.join("notifs.db"),
        scratch: a_root.join("scratch"),
        subnets: a_root.join("subnet.json"),
        vis: a_root.join("visibility.json"),
        ops: a_root.join("pump-ops.json"),
        releases: a_root.join("releases"),
        release_keys: a_root.join("release-keys.json"),
        pull_marker: a_root.join("sync-pull-now"),
        advertise_marker: a_root.join("advertise-now"),
        peer_addrs: a_root.join("peer-addrs.json"),
        heartbeat: a_root.join("pump-heartbeat.json"),
        rotations: a_root.join("rotation-pending.json"),
    };
    // A trusts B's release signer — the update leg's production gate.
    std::fs::create_dir_all(&a_root).unwrap();
    let keys_json = serde_json::json!({
        "keys": { "k": sk.verifying_key().as_bytes().iter()
            .map(|b| format!("{b:02x}")).collect::<String>() },
        "channel": "stable",
    });
    std::fs::write(
        &a_paths.release_keys,
        serde_json::to_string(&keys_json).unwrap(),
    )
    .unwrap();
    let perch_path = perch::resolve_perch_path_in(&a_paths.owlery, "pump-ag", ParentHint::Infer);
    std::fs::create_dir_all(&perch_path).unwrap();
    let rec = InfoJson::new("pump-ag", "now", std::process::id(), "s", "ready_agent");
    info::write_info(&perch_path, &rec).unwrap();

    let mut a_subnets = SubnetStore::default();
    a_subnets.create_subnet(subnet).expect("A subnet");
    a_subnets.save_to(&a_paths.subnets).expect("save A subnets");
    // Mesh-D6: A's membership IS the roster — both its inbound gates and its
    // pump push target read it. The pump derives the roster path as a sibling of
    // `subnets` (subnet.json → roster.json), so write it there.
    let mut a_roster = spt_store::roster::RosterStore::default();
    a_roster.merge_entry(spt_store::roster::RosterEntry {
        pubkey_hex: b_hex.clone(),
        subnet: subnet.to_string(),
        label: String::new(),
        machine_id: String::new(),
        address: None,
        last_seen: String::new(),
        lease_epoch: 1,
    });
    a_roster
        .save_to(&a_root.join("roster.json"))
        .expect("save A roster");
    let mut a_vis = VisibilityStore::default();
    a_vis.set_sync_subnets("ling-pumped", vec![subnet.to_string()]);
    a_vis.save_to(&a_paths.vis).expect("save A visibility");

    let a_store = NotifStore::open_at(&a_paths.notif_db).expect("A notif store");
    a_store
        .merge_row(&NotifRow {
            notif_id: format!("{a_hex}:1"),
            subnet: subnet.to_string(),
            kind: "agent".to_string(),
            from_id: format!("pump-ag@{a_hex}"),
            body: "pumped across".to_string(),
            created_ms: 1_000,
            dismissed: false,
            seen: Default::default(),
            last_surfaced_ms: 0,
        })
        .expect("seed A notif row");

    let a_registry = Arc::new(RegistryHost::new_at(
        &a_hex,
        EpochSource::load_from(&a_root.join("epoch.json")),
        a_root.join("registry"),
    ));
    // A learned of B's endpoint via an earlier inbound feed (seeded directly
    // here — B runs no pump in this test): the bootstrap-ref source.
    {
        let mut roster = spt_store::roster::RosterStore::default();
        roster.merge_entry(spt_store::roster::RosterEntry {
            pubkey_hex: b_hex.clone(),
            subnet: subnet.to_string(),
            label: String::new(),
            machine_id: String::new(),
            address: None,
            last_seen: String::new(),
            lease_epoch: 1,
        });
        let pol = RegistryGatePolicy {
            subnets: vec![subnet.to_string()],
            roster,
        };
        a_registry.apply_feed(
            &b_hex,
            &[RegistryUpdate {
                subnet: subnet.to_string(),
                endpoint_id: "ling-pumped".to_string(),
                instance: Instance {
                    node: b_hex.clone(),
                    status: Status::Active,
                    epoch: 1,
                    resources: None,
                    last_active_ms: None,
                    shell_adapters: Vec::new(),
                    node_label: None,
                    machine_id: None,
                    endpoint_type: None,
                },
            }],
            &pol,
        );
    }

    // ── The pump: production loop, injected resolver, fast cadences. ─────
    let resolver: PeerResolver = {
        let b_hex = b_hex.clone();
        let b_addr = b_addr.clone();
        Box::new(move |hex| {
            if hex == b_hex {
                Some(b_addr.clone())
            } else {
                None
            }
        })
    };
    {
        let name = a_name.clone();
        let reg = Arc::clone(&a_registry);
        let flag = Arc::clone(&stop);
        let paths = PumpPaths {
            owlery: a_paths.owlery.clone(),
            ctx_root: a_paths.ctx_root.clone(),
            notif_db: a_paths.notif_db.clone(),
            scratch: a_paths.scratch.clone(),
            subnets: a_paths.subnets.clone(),
            vis: a_paths.vis.clone(),
            ops: a_paths.ops.clone(),
            releases: a_paths.releases.clone(),
            release_keys: a_paths.release_keys.clone(),
            pull_marker: a_paths.pull_marker.clone(),
            advertise_marker: a_paths.advertise_marker.clone(),
            peer_addrs: a_paths.peer_addrs.clone(),
            heartbeat: a_paths.heartbeat.clone(),
            rotations: a_paths.rotations.clone(),
        };
        thread::spawn(move || {
            let config = PumpConfig {
                cadence: PumpCadence {
                    registry: Duration::from_millis(100),
                    notif: Duration::from_millis(100),
                    sync: Duration::from_millis(100),
                    update: Duration::from_millis(100),
                    // Inert here: both loopback nodes feed each other every
                    // tick, so nothing is ever silent for 300s.
                    registry_evict_after: Duration::from_secs(300),
                },
                // Gated default (REQ-UPD-4) — the consent notif is the assert.
                full_auto_update: false,
            };
            let _ = run_peer_pump(
                &name,
                reg,
                &paths,
                resolver,
                config,
                PumpHooks::default(),
                &flag,
            );
        });
    }

    // 1. Registry converged at B: A's perch row arrived through the
    //    dispatcher's gated registry arm.
    converge("B registry holds pump-ag@A", || {
        b_registry
            .rows(subnet, "pump-ag")
            .iter()
            .any(|i| i.node == a_hex)
    });

    // 2. Notif converged at B's canonical store (the semilattice join).
    let store_b = NotifStore::open().expect("B notif store");
    converge("B notif store holds A's row", || {
        store_b
            .list(subnet)
            .map(|rows| rows.iter().any(|r| r.body == "pumped across"))
            .unwrap_or(false)
    });

    // 3. The bootstrap pull: A now holds a mind it NEVER had locally — the
    //    want-ref came from A's registry, the serve gate ran at B.
    let cs_a = ContextStore::open_or_init_in(&a_paths.ctx_root).expect("A context store");
    converge("A pulled the registry-derived mind", || {
        cs_a.branch_store()
            .tip("a-ling-pumped")
            .ok()
            .flatten()
            .is_some()
    });

    // 4. The update leg: A pulled, gated, and staged B's v6 under the
    //    production keys file — and surfaced it as a CONSENT notif (REQ-UPD-4
    //    gated default), never a silent auto-apply.
    let cache_a = ReleaseCache::open(&a_paths.releases);
    converge("A staged the propagated release", || {
        cache_a
            .load_artifact()
            .map(|b| b == artifact)
            .unwrap_or(false)
    });
    converge("A holds the consent notif", || {
        a_store
            .list(subnet)
            .map(|rows| rows.iter().any(|r| r.kind == "consent"))
            .unwrap_or(false)
    });

    stop.store(true, Ordering::Relaxed);
}


Pump E2E uses real `NetHost` + `Broker` + `run_peer_pump`. For the regression I need (a) a short bound injectable from an external-crate test and (b) a deterministic black-hole peer. I'll mirror the existing `set_roster_exchange` seam: a public `set_quic_op_timeout` backed by an `AtomicU64` (default 10s), read per-op. Read netbroker.rs to place the broker-level error test.