1	//! The broker-owned network host (D4a) — ADR-0004 §B made real for the wire.
2	//!
3	//! [`NetHost`] is the broker's ownership of the node's live network presence:
4	//! the iroh endpoint ([`spt_net::net::endpoint::NetEndpoint`]) and every
5	//! accepted/dialed QUIC connection. A live cross-node connection is exactly the
6	//! kind of resource ADR-0004 §B assigns to the broker — a consumer would lose
7	//! continuity on brain restart — so the endpoint is constructed *here*, in the
8	//! stable kernel, never in the brain. The brain drives net operations over the
9	//! versioned IPC ([`crate::msg::KIND_NET_STATUS`] / [`crate::msg::KIND_NET_DIAL`])
10	//! and may detach/reattach freely; connections persist in the broker's table.
11	//!
12	//! ## Runtime placement (M4-D4-PLAN locked decision)
13	//!
14	//! The broker is otherwise sync/thread-based; iroh requires tokio. So the host
15	//! owns a **dedicated multi-thread tokio runtime** on named worker threads
16	//! (`spt-broker-net`): QUIC objects live in structures only the runtime's tasks
17	//! and the host's own `block_on` bridges touch, and the runtime drops with the
18	//! host (broker teardown stops all net tasks). Spike #3's broker binary proved
19	//! exactly this shape — a broker-owned endpoint surviving brain kill/restart.
20	//!
21	//! ## Ownership closes two §B rows by construction
22	//!
23	//! `MdnsAddressLookup` and the relay session are both constructed *inside* the
24	//! iroh endpoint, so broker ownership of the endpoint closes the ADR-0004 §B
25	//! "mDNS socket / relay session" row with no extra machinery: there is no
26	//! separate socket to own.
27	// [impl->REQ-DAEMON-2]
28
29	use std::collections::{HashMap, HashSet, VecDeque};
30	use std::io;
31	use std::sync::atomic::{AtomicU64, Ordering};
32	use std::sync::{Arc, Mutex};
33
34	use spt_net::net::endpoint::{
35	    BindScope, Connection, EndpointAddr, LocalDiscovery, NetEndpoint, RecvStream, RelayPolicy,
36	    SendStream, SPT_NET_ALPN,
37	};
38	use spt_net::net::mesh::seedproof::ProofRole;
39	use spt_proto::identity::{Identity, PublicKey};
40
41	use crate::broker::SharedSend;
42	use crate::codec::write_frame;
43	use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};
44	use crate::msg::{
45	    net_presence_event_envelope, net_stream_data_envelope, net_stream_eof_envelope,
46	    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
47	};
48
49	/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped
50	/// effects (a dial has no PTY session). Broker session ids are minted from 1
51	/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.
52	pub const NET_EFFECT_SESSION: u64 = 0;
53
54	/// Everything needed to stand the broker's endpoint up: the node's identity
55	/// (the endpoint *is* the node's name — EndpointId == pubkey), the relay stance
56	/// (from `daemon.json` in production; [`RelayPolicy::Disabled`] in hermetic
57	/// tests), and whether mDNS LAN discovery rides along.
58	pub struct NetConfig {
59	    /// The node identity the endpoint binds as (seed → same ed25519 keypair).
60	    pub identity: Identity,
61	    /// Relay stance (ADR-0002): n0 default / self-host / disabled.
62	    pub relay: RelayPolicy,
63	    /// mDNS LAN discovery on/off (off in hermetic tests).
64	    pub discovery: LocalDiscovery,
65	    /// Socket bind scope: [`BindScope::All`] in production;
66	    /// [`BindScope::Loopback`] in hermetic tests (no real interface touched,
67	    /// no per-test-binary Windows firewall prompt).
68	    pub scope: BindScope,
69	    /// Live source of this node's subnet credentials for the connect-time
70	    /// seed-proof (Mesh-D2, REQ-MESH-1). `Some` enables the mutual proof at
71	    /// every dial + accept (fail → drop); `None` disables it entirely — both
72	    /// peers must agree, or the side expecting the control stream would hang.
73	    /// Production wires `Some(SubnetStore::load …)`; mechanics-only tests that
74	    /// don't exercise membership pass `None`.
75	    pub membership: Option<MembershipSource>,
76	}
77
78	/// Default bounded stream-read ring depth (chunks) — same sizing rationale as
79	/// the PTY `OutputLog`: large enough that a brain-restart window never evicts
80	/// unreplayed chunks in practice; bounded so an unsubscribed stream can't
81	/// buffer without limit.
82	const DEFAULT_STREAM_RING_CHUNKS: usize = 4096;
83
84	/// Max bytes pulled per stream read (one ring chunk).
85	const STREAM_READ_CHUNK: usize = 64 * 1024;
86
87	/// One live QUIC connection the broker holds.
88	struct ConnEntry {
89	    /// The connection itself — kept alive by ownership here; dropping it closes
90	    /// the QUIC connection. Cloned (cheap handle) for stream opens.
91	    conn: Connection,
92	    /// The remote node's stable name (hex pubkey), proven by the QUIC handshake.
93	    remote_id_hex: String,
94	    /// The subnets the peer proved current-epoch seed-knowledge of at connect
95	    /// (Mesh-D2, REQ-MESH-1). Empty when seed-proof is disabled (`membership:
96	    /// None`). The D5 gate swap reads this in place of `is_trusted(subnet, …)`;
97	    /// at D2 it is cached but not yet consumed (the `is_trusted` gates still run
98	    /// in parallel — belt + suspenders, no regression).
99	    proven_subnets: HashSet<String>,
100	}
101
102	/// A bounded, sequenced read log for one QUIC stream — the PTY `OutputLog`
103	/// discipline applied to the network (D4b): the stream's single read pump is
104	/// the only producer; a brain connection attaches as the subscriber; every
105	/// subscriber send happens under the enclosing `Mutex`, so live chunks and a
106	/// replay can never interleave out of `seq` order. Chunks produced while no
107	/// brain is attached sit in the ring and replay on the next subscribe — a
108	/// brain restart loses nothing (Spike #3 invariant B/D).
109	// [impl->REQ-DAEMON-2]
110	struct StreamLog {
111	    stream_id: u64,
112	    next_seq: u64,
113	    ring: VecDeque<(u64, Vec<u8>)>,
114	    cap_chunks: usize,
115	    subscriber: Option<SharedSend>,
116	    /// Read side finished cleanly (peer called finish / conn ended).
117	    finished: bool,
118	}
119
120	impl StreamLog {
121	    fn new(stream_id: u64, cap_chunks: usize) -> Self {
122	        StreamLog {
123	            stream_id,
124	            next_seq: 0,
125	            ring: VecDeque::new(),
126	            cap_chunks,
127	            subscriber: None,
128	            finished: false,
129	        }
130	    }
131
132	    /// Append a freshly-read chunk: assign the next `seq`, retain it in the
133	    /// bounded ring, live-send to the current subscriber. Read-pump only.
134	    fn append(&mut self, bytes: &[u8]) {
135	        let seq = self.next_seq;
136	        self.next_seq += 1;
137	        self.ring.push_back((seq, bytes.to_vec()));
138	        while self.ring.len() > self.cap_chunks {
139	            self.ring.pop_front();
140	        }
141	        if let Some(sub) = &self.subscriber {
142	            let frame = net_stream_data_envelope(self.stream_id, seq, bytes);
143	            if let Ok(mut s) = sub.lock() {
144	                let _ = write_frame(&mut *s, &frame);
145	            }
146	        }
147	    }
148
149	    /// Mark the read side finished and tell the subscriber (with the final
150	    /// cursor, so it can verify completeness).
151	    fn finish(&mut self) {
152	        self.finished = true;
153	        if let Some(sub) = &self.subscriber {
154	            let frame = net_stream_eof_envelope(self.stream_id, self.next_seq);
155	            if let Ok(mut s) = sub.lock() {
156	                let _ = write_frame(&mut *s, &frame);
157	            }
158	        }
159	    }
160
161	    /// Attach `sub` as the subscriber, replaying every retained chunk with
162	    /// `seq >= from_seq` (then eof, if already finished) before live streaming.
163	    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
164	        self.subscriber = Some(Arc::clone(&sub));
165	        for (seq, bytes) in self.ring.iter().filter(|(s, _)| *s >= from_seq) {
166	            let frame = net_stream_data_envelope(self.stream_id, *seq, bytes);
167	            if let Ok(mut s) = sub.lock() {
168	                let _ = write_frame(&mut *s, &frame);
169	            }
170	        }
171	        if self.finished {
172	            let frame = net_stream_eof_envelope(self.stream_id, self.next_seq);
173	            if let Ok(mut s) = sub.lock() {
174	                let _ = write_frame(&mut *s, &frame);
175	            }
176	        }
177	    }
178
179	    /// Detach `sub` iff it is the current subscriber (a late disconnect must
180	    /// not unsubscribe a newer brain).
181	    fn detach_if(&mut self, sub: &SharedSend) {
182	        if let Some(cur) = &self.subscriber {
183	            if Arc::ptr_eq(cur, sub) {
184	                self.subscriber = None;
185	            }
186	        }
187	    }
188	}
189
190	/// A bounded, sequenced log of connection liveness events — the broker-owned
191	/// PresenceChannel seam (D4c, REQ-EP-4): the StreamLog ring/cursor discipline
192	/// applied to the conn table itself. `register_conn` appends `connected`; each
193	/// connection's closed-watcher appends `disconnected` after removing the row.
194	/// A brain subscribes with a cursor and replays events produced while it was
195	/// down, so presence survives a brain restart exactly like stream reads do.
196	// [impl->REQ-EP-4]
197	struct PresenceLog {
198	    next_seq: u64,
199	    ring: VecDeque<NetPresenceEvent>,
200	    cap_events: usize,
201	    subscriber: Option<SharedSend>,
202	}
203
204	impl PresenceLog {
205	    fn new(cap_events: usize) -> Self {
206	        PresenceLog {
207	            next_seq: 0,
208	            ring: VecDeque::new(),
209	            cap_events,
210	            subscriber: None,
211	        }
212	    }
213
214	    /// Append a liveness event: assign the next `seq`, retain it in the bounded
215	    /// ring, live-send to the current subscriber. Producers: `register_conn`
216	    /// (connected) and the per-conn closed-watchers (disconnected) — every
217	    /// append serializes on the enclosing `Mutex`, so subscriber frames arrive
218	    /// in strict `seq` order.
219	    fn append(&mut self, kind: &str, conn_id: u64, remote_id_hex: &str) {
220	        let ev = NetPresenceEvent {
221	            seq: self.next_seq,
222	            kind: kind.to_string(),
223	            conn_id,
224	            remote_id_hex: remote_id_hex.to_string(),
225	        };
226	        self.next_seq += 1;
227	        self.ring.push_back(ev.clone());
228	        while self.ring.len() > self.cap_events {
229	            self.ring.pop_front();
230	        }
231	        if let Some(sub) = &self.subscriber {
232	            let frame = net_presence_event_envelope(&ev);
233	            if let Ok(mut s) = sub.lock() {
234	                let _ = write_frame(&mut *s, &frame);
235	            }
236	        }
237	    }
238
239	    /// Attach `sub` as the subscriber, replaying every retained event with
240	    /// `seq >= from_seq` before live streaming (the gapless-resume attach).
241	    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
242	        self.subscriber = Some(Arc::clone(&sub));
243	        for ev in self.ring.iter().filter(|e| e.seq >= from_seq) {
244	            let frame = net_presence_event_envelope(ev);
245	            if let Ok(mut s) = sub.lock() {
246	                let _ = write_frame(&mut *s, &frame);
247	            }
248	        }
249	    }
250
251	    /// Detach `sub` iff it is the current subscriber (a late disconnect must
252	    /// not unsubscribe a newer brain).
253	    fn detach_if(&mut self, sub: &SharedSend) {
254	        if let Some(cur) = &self.subscriber {
255	            if Arc::ptr_eq(cur, sub) {
256	                self.subscriber = None;
257	            }
258	        }
259	    }
260	}
261
262	/// One live QUIC stream the broker holds: the send half (brain writes ride it
263	/// over IPC) + the sequenced read log.
264	struct StreamEntry {
265	    conn_id: u64,
266	    remote_id_hex: String,
267	    /// Whether THIS node opened the stream (`open_stream`) vs the peer
268	    /// (accept loop). The inbound dispatcher (D9-1) claims only
269	    /// peer-initiated streams; without this fact the table is ambiguous —
270	    /// it holds locally-opened outbound streams too (gotcha: a dispatcher
271	    /// that served its own requester stream would deadlock on itself).
272	    initiated_locally: bool,
273	    /// The send half, serialized behind an async mutex (writes happen on the
274	    /// host's runtime via `block_on`).
275	    send: tokio::sync::Mutex<SendStream>,
276	    log: Arc<Mutex<StreamLog>>,
277	}
278
279	/// Tables shared between the host's sync surface and its runtime tasks
280	/// (accept loop, per-conn stream acceptors, per-stream read pumps).
281	struct NetShared {
282	    next_conn_id: AtomicU64,
283	    conns: Mutex<HashMap<u64, ConnEntry>>,
284	    next_stream_id: AtomicU64,
285	    streams: Mutex<HashMap<u64, Arc<StreamEntry>>>,
286	    ring_cap: usize,
287	    /// The conn-liveness event log (D4c) — one per host, all conns interleaved.
288	    presence: Mutex<PresenceLog>,
289	}
290
291	impl NetShared {
292	    fn new() -> Self {
293	        NetShared {
294	            next_conn_id: AtomicU64::new(1),
295	            conns: Mutex::new(HashMap::new()),
296	            next_stream_id: AtomicU64::new(1),
297	            streams: Mutex::new(HashMap::new()),
298	            ring_cap: DEFAULT_STREAM_RING_CHUNKS,
299	            presence: Mutex::new(PresenceLog::new(DEFAULT_STREAM_RING_CHUNKS)),
300	        }
301	    }
302	}
303
304	/// Register a live connection: mint its broker id and spawn its inbound-stream
305	/// acceptor. Must run inside the host's runtime context (the accept loop, or a
306	/// `block_on` future) — it spawns tasks.
307	fn register_conn(
308	    shared: &Arc<NetShared>,
309	    conn: Connection,
310	    proven_subnets: HashSet<String>,
311	) -> (u64, String) {
312	    let remote_id_hex = PublicKey::from_bytes(conn.remote_id().as_bytes())
313	        .expect("an iroh EndpointId is always a valid ed25519 public key")
314	        .to_hex();
315	    let id = shared.next_conn_id.fetch_add(1, Ordering::Relaxed);
316	    shared.conns.lock().unwrap().insert(
317	        id,
318	        ConnEntry {
319	            conn: conn.clone(),
320	            remote_id_hex: remote_id_hex.clone(),
321	            proven_subnets,
322	        },
323	    );
324
325	    // Presence: announce the row BEFORE spawning the closed-watcher, so the
326	    // `connected` event can never race behind its own `disconnected` (D4c).
327	    shared
328	        .presence
329	        .lock()
330	        .unwrap()
331	        .append(PRESENCE_CONNECTED, id, &remote_id_hex);
332
333	    // Closed-watcher: the row's exit path. Awaits the connection's end (clean
334	    // close, conn loss, or our own endpoint close), removes the row — conns
335	    // finally LEAVE the table — then appends the `disconnected` event (row
336	    // gone before the event is observable).
337	    // [impl->REQ-EP-4]
338	    let sh_close = Arc::clone(shared);
339	    let conn_close = conn.clone();
340	    let hex_close = remote_id_hex.clone();
341	    tokio::spawn(async move {
342	        conn_close.closed().await;
343	        sh_close.conns.lock().unwrap().remove(&id);
344	        sh_close
345	            .presence
346	            .lock()
347	            .unwrap()
348	            .append(PRESENCE_DISCONNECTED, id, &hex_close);
349	    });
350
351	    // Inbound-stream acceptor: every peer-opened bidi stream joins the table
352	    // and gets a read pump. (QUIC streams are lazy — the peer's open becomes
353	    // visible here on its first write.) Ends when the connection does.
354	    let sh = Arc::clone(shared);
355	    let hex = remote_id_hex.clone();
356	    tokio::spawn(async move {
357	        while let Ok((send, recv)) = conn.accept_bi().await {
358	            register_stream(&sh, id, &hex, send, recv, false);
359	        }
360	    });
361
362	    (id, remote_id_hex)
363	}
364
365	/// Register a live stream (opened or accepted): mint its broker id and spawn
366	/// its read pump. Must run inside the host's runtime context.
367	/// `initiated_locally` records the direction fact for the table (true only
368	/// from [`NetHost::open_stream`]).
369	fn register_stream(
370	    shared: &Arc<NetShared>,
371	    conn_id: u64,
372	    remote_id_hex: &str,
373	    send: SendStream,
374	    recv: RecvStream,
375	    initiated_locally: bool,
376	) -> u64 {
377	    let id = shared.next_stream_id.fetch_add(1, Ordering::Relaxed);
378	    let log = Arc::new(Mutex::new(StreamLog::new(id, shared.ring_cap)));
379	    shared.streams.lock().unwrap().insert(
380	        id,
381	        Arc::new(StreamEntry {
382	            conn_id,
383	            remote_id_hex: remote_id_hex.to_string(),
384	            initiated_locally,
385	            send: tokio::sync::Mutex::new(send),
386	            log: Arc::clone(&log),
387	        }),
388	    );
389
390	    // The read pump — the stream's single producer. A clean end (peer finish)
391	    // and a torn end (conn lost) both finish the log; the buffered chunks stay
392	    // replayable either way.
393	    let mut recv = recv;
394	    tokio::spawn(async move {
395	        loop {
396	            match recv.read_chunk(STREAM_READ_CHUNK).await {
397	                Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),
398	                Ok(None) | Err(_) => {
399	                    log.lock().unwrap().finish();
400	                    break;
401	                }
402	            }
403	        }
404	    });
405
406	    id
407	}
408
409	/// The broker-owned net endpoint + live connection/stream tables (see module
410	/// docs).
411	pub struct NetHost {
412	    /// The dedicated tokio runtime hosting the endpoint and its accept loop.
413	    /// Dropping the host drops the runtime, stopping every net task with it.
414	    runtime: tokio::runtime::Runtime,
415	    endpoint: Arc<NetEndpoint>,
416	    shared: Arc<NetShared>,
417	    /// `op_id → conn_id` for exactly-once dial replies: a deduped replay of a
418	    /// journaled dial is answered with the connection the first apply opened.
419	    dial_ops: Mutex<HashMap<u64, u64>>,
420	    /// `op_id → stream_id` for exactly-once stream-open replies (D4b), same
421	    /// discipline as `dial_ops`. Shares the one net op-id namespace.
422	    stream_ops: Mutex<HashMap<u64, u64>>,
423	    /// This node's pubkey (the seed-proof's local channel-binding identity).
424	    local_id: PublicKey,
425	    /// The connect-time seed-proof credential source (Mesh-D2). `None` disables
426	    /// the proof (see [`NetConfig::membership`]).
427	    membership: Option<MembershipSource>,
428	    /// The on-connect roster-exchange seam (Mesh-D4, REQ-MESH-2), set on the live
429	    /// host after `start` via [`Self::set_roster_exchange`] (kept off `NetConfig`
430	    /// so the many hermetic test configs need no change — a roster exchange is a
431	    /// layer atop the proof, opt-in per host). `None` ⇒ no roster leg. Read +
432	    /// cloned per connect on the net runtime, so the accept loop (spawned in
433	    /// `start`, before the host exists) shares it through this `Arc`.
434	    roster: Arc<Mutex<Option<RosterExchange>>>,
435	}
436
437	impl NetHost {
438	    /// Bind the broker's endpoint on a fresh dedicated runtime and start the
439	    /// accept loop. The endpoint speaks TWO ALPNs (M7 D3, REQ-SUBNET-2): the
440	    /// trusted [`SPT_NET_ALPN`] (connections join the table as before) and
441	    /// the pre-trust pairing ALPN — routed to the always-on responder
442	    /// ([`crate::pairhost::respond`]), never registered in the stream table.
443	    /// One endpoint for both is deliberate: the pairing transcript binds and
444	    /// pins the REAL node pubkey, and the relay can only route one endpoint
445	    /// per key — the ALPN is the pre-trust/trusted boundary (ADR-0005).
446	    /// The meet-rotation listener loop rides the same runtime.
447	    // [impl->REQ-DAEMON-2]
448	    // [impl->REQ-SUBNET-2]
449	    pub fn start(cfg: NetConfig) -> io::Result<NetHost> {
450	        // Hermeticity guard (REQ-PAIR-8): a loopback-scoped endpoint is the
451	        // "no network reached" promise — the ceremony clock's lazy NTP query
452	        // must honor it (tests must never probe a real NTP server). Real
453	        // scopes keep the decision-18 offset correction.
454	        spt_net::net::pairing::ntp::set_enabled(!matches!(cfg.scope, BindScope::Loopback));
455	        let runtime = tokio::runtime::Builder::new_multi_thread()
456	            .worker_threads(2)
457	            .thread_name("spt-broker-net")
458	            .enable_all()
459	            .build()?;
460	        let endpoint = runtime
461	            .block_on(NetEndpoint::bind_scoped(
462	                &cfg.identity,
463	                cfg.relay.clone(),
464	                cfg.discovery,
465	                vec![
466	                    SPT_NET_ALPN.to_vec(),
467	                    spt_net::net::pairing::wire::SPT_PAIR_ALPN.to_vec(),
468	                ],
469	                cfg.scope,
470	            ))
471	            .map_err(|e| io::Error::other(e.to_string()))?;
472	        let endpoint = Arc::new(endpoint);
473	        let shared = Arc::new(NetShared::new());
474	        let local_pub = cfg.identity.public_key();
475	        let pair_rate = Arc::new(tokio::sync::Mutex::new(
476	            spt_net::net::pairing::ratelimit::PairingRateLimiter::new(),
477	        ));
478
479	        // Accept loop: lives for the endpoint's lifetime on the host's runtime.
480	        // Each handshake completes on its own task so a slow peer can't stall
481	        // the loop; a failed handshake is simply not registered. The pairing
482	        // ALPN routes to the responder; everything else is the trusted table.
483	        let roster = Arc::new(Mutex::new(None::<RosterExchange>));
484	        let ep = Arc::clone(&endpoint);
485	        let sh = Arc::clone(&shared);
486	        let rate = Arc::clone(&pair_rate);
487	        let accept_membership = cfg.membership.clone();
488	        let accept_roster = Arc::clone(&roster);
489	        let accept_ep = Arc::clone(&endpoint);
490	        runtime.spawn(async move {
491	            while let Some(incoming) = ep.accept().await {
492	                let sh = Arc::clone(&sh);
493	                let rate = Arc::clone(&rate);
494	                let membership = accept_membership.clone();
495	                // Snapshot the (opt-in) roster exchange + our live address for
496	                // this connect; the leg runs after the proof verifies.
497	                let roster_rx = accept_roster.lock().unwrap().clone();
498	                let self_addr =
499	                    serde_json::to_value(accept_ep.addr()).unwrap_or(serde_json::Value::Null);
500	                tokio::spawn(async move {
501	                    let Ok(connecting) = incoming.accept() else {
502	                        return;
503	                    };
504	                    let Ok(conn) = connecting.await else { return };
505	                    if conn.alpn() == spt_net::net::pairing::wire::SPT_PAIR_ALPN {
506	                        crate::pairhost::respond(conn, local_pub, rate).await;
507	                    } else {
508	                        // Mesh-D2: prove membership in-band BEFORE the connection
509	                        // joins the table. A non-prover (no shared subnet / bad
510	                        // seed) is dropped — the conn falls out of scope and QUIC
511	                        // closes it. With membership disabled, register as before.
512	                        // Mesh-D4: on success, a roster leg rides the same stream.
513	                        let proven = match membership {
514	                            Some(src) => {
515	                                let creds = src();
516	                                match prove_membership(
517	                                    &conn,
518	                                    local_pub,
519	                                    ProofRole::Acceptor,
520	                                    &creds,
521	                                    roster_rx.as_ref(),
522	                                    &self_addr,
523	                                )
524	                                .await
525	                                {
526	                                    Some(set) => set,
527	                                    None => return, // drop the connection
528	                                }
529	                            }
530	                            None => HashSet::new(),
531	                        };
532	                        register_conn(&sh, conn, proven);
533	                    }
534	                });
535	            }
536	        });
537
538	        // The meet-rotation loop (M7 D3): how a joiner that knows only
539	        // (subnet name, code) finds this responder. Idles with no subnets.
540	        crate::pairhost::spawn_meet_rotation(
541	            runtime.handle(),
542	            Arc::clone(&endpoint),
543	            cfg.relay,
544	            cfg.discovery,
545	            cfg.scope,
546	        );
547
548	        // The limiter Arc lives on in the accept loop; the host keeps no
549	        // handle (per-ceremony state is the responder's, not the table's).
550	        drop(pair_rate);
551
552	        Ok(NetHost {
553	            runtime,
554	            endpoint,
555	            shared,
556	            dial_ops: Mutex::new(HashMap::new()),
557	            stream_ops: Mutex::new(HashMap::new()),
558	            local_id: local_pub,
559	            membership: cfg.membership,
560	            roster,
561	        })
562	    }
563
564	    /// Wire the on-connect roster-exchange seam (Mesh-D4, REQ-MESH-2) onto this
565	    /// live host. The daemon calls this once after `start` with
566	    /// [`crate::seedproofx::production_roster_exchange`]; tests inject their own.
567	    /// Off `NetConfig` by design — see the `roster` field. Takes effect on the
568	    /// next dial/accept (each connect snapshots it).
569	    // [impl->REQ-MESH-2]
570	    pub fn set_roster_exchange(&self, exchange: RosterExchange) {
571	        *self.roster.lock().unwrap() = Some(exchange);
572	    }
573
574	    /// Run the joiner leg of the pairing ceremony (M7 D3, REQ-SUBNET-2) on
575	    /// the host's runtime — the broker IPC `pair-join` arm. The ceremony
576	    /// must run HERE because it binds and pins this endpoint's real node
577	    /// pubkey. Returns the rendered trust outcome, or the classified failure.
578	    // [impl->REQ-SUBNET-2]
579	    pub fn pair_join(
580	        &self,
581	        subnet: &str,
582	        code: &str,
583	        responder_addr: Option<EndpointAddr>,
584	    ) -> Result<String, crate::pairhost::JoinFail> {
585	        let endpoint = Arc::clone(&self.endpoint);
586	        self.runtime.handle().block_on(async move {
587	            crate::pairhost::join_ceremony(&endpoint, subnet, code, responder_addr).await
588	        })
589	    }
590
591	    /// The node's stable name (lowercase-hex pubkey == iroh EndpointId).
592	    pub fn node_id_hex(&self) -> String {
593	        self.endpoint.node_id_hex()
594	    }
595
596	    /// The endpoint's dialable address (id + currently-known direct/relay paths).
597	    pub fn addr(&self) -> EndpointAddr {
598	        self.endpoint.addr()
599	    }
600
601	    /// [`Self::addr`] as the opaque JSON the IPC status reply carries.
602	    pub fn addr_json(&self) -> serde_json::Value {
603	        serde_json::to_value(self.endpoint.addr()).unwrap_or(serde_json::Value::Null)
604	    }
605
606	    /// Live connections (accepted + dialed) currently held.
607	    pub fn conn_count(&self) -> u64 {
608	        self.shared.conns.lock().unwrap().len() as u64
609	    }
610
611	    /// Dial a peer over `SPT_NET_ALPN`, registering the connection in the table.
612	    /// Bridges sync→async via the host's own runtime (the broker's dispatch
613	    /// threads block here; the future runs on the net worker threads).
614	    // [impl->REQ-DAEMON-2]
615	    pub fn dial(&self, addr: EndpointAddr) -> io::Result<(u64, String)> {
616	        let endpoint = Arc::clone(&self.endpoint);
617	        let shared = Arc::clone(&self.shared);
618	        let local_id = self.local_id;
619	        let membership = self.membership.clone();
620	        // Snapshot the (opt-in) roster exchange + our live address for this dial.
621	        let roster_rx = self.roster.lock().unwrap().clone();
622	        let self_addr = serde_json::to_value(self.endpoint.addr()).unwrap_or(serde_json::Value::Null);
623	        self.runtime.handle().block_on(async move {
624	            let conn = endpoint
625	                .connect(addr)
626	                .await
627	                .map_err(|e| io::Error::other(e.to_string()))?;
628	            // Mesh-D2: prove membership in-band before the connection is usable.
629	            // A failed/empty proof drops the dial (the conn never registers).
630	            // Mesh-D4: on success, a roster leg rides the same control stream.
631	            let proven = match membership {
632	                Some(src) => {
633	                    let creds = src();
634	                    prove_membership(
635	                        &conn,
636	                        local_id,
637	                        ProofRole::Dialer,
638	                        &creds,
639	                        roster_rx.as_ref(),
640	                        &self_addr,
641	                    )
642	                    .await
643	                    .ok_or_else(|| {
644	                        io::Error::other("seed-proof failed: peer is not a subnet member")
645	                    })?
646	                }
647	                None => HashSet::new(),
648	            };
649	            Ok(register_conn(&shared, conn, proven))
650	        })
651	    }
652
653	    /// Record that journaled dial `op_id` opened `conn_id` (called inside the
654	    /// `apply_once` closure, so a concurrent deduped replay always finds it).
655	    pub fn record_dial_op(&self, op_id: u64, conn_id: u64) {
656	        self.dial_ops.lock().unwrap().insert(op_id, conn_id);
657	    }
658
659	    /// The connection a journaled dial `op_id` opened, if this broker process
660	    /// still holds it (a broker restart drops connections; the journal alone
661	    /// survives — see `Broker::dispatch_net_dial`).
662	    pub fn dial_op_conn(&self, op_id: u64) -> Option<u64> {
663	        self.dial_ops.lock().unwrap().get(&op_id).copied()
664	    }
665
666	    /// The remote node id (hex) of a held connection.
667	    pub fn conn_remote_hex(&self, conn_id: u64) -> Option<String> {
668	        self.shared
669	            .conns
670	            .lock()
671	            .unwrap()
672	            .get(&conn_id)
673	            .map(|e| e.remote_id_hex.clone())
674	    }
675
676	    /// The subnets the peer of `conn_id` proved current-epoch seed-knowledge of
677	    /// at connect (Mesh-D2, REQ-MESH-1). Empty when seed-proof was disabled or
678	    /// the conn is gone. The seam the D5 gate swap reads in place of
679	    /// `is_trusted(subnet, origin)`; unused by any gate at D2.
680	    // [impl->REQ-MESH-1]
681	    pub fn conn_proven_subnets(&self, conn_id: u64) -> HashSet<String> {
682	        self.shared
683	            .conns
684	            .lock()
685	            .unwrap()
686	            .get(&conn_id)
687	            .map(|e| e.proven_subnets.clone())
688	            .unwrap_or_default()
689	    }
690
691	    /// The OBSERVED peer address of a held connection (remote id + the
692	    /// connection's current path addresses) as opaque JSON — the address-
693	    /// seeding write-back the `net-dialed` reply carries (M8 decision 20,
694	    /// REQ-CONV-1). `null` when the conn is gone, path-less, or fails to
695	    /// serialize: the seed is an optimization, never an error path.
696	    // [impl->REQ-CONV-1]
697	    pub fn conn_remote_addr_json(&self, conn_id: u64) -> serde_json::Value {
698	        let conn = {
699	            let conns = self.shared.conns.lock().unwrap();
700	            match conns.get(&conn_id) {
701	                Some(e) => e.conn.clone(),
702	                None => return serde_json::Value::Null,
703	            }
704	        };
705	        let addr = spt_net::net::endpoint::conn_remote_addr(&conn);
706	        if addr.is_empty() {
707	            return serde_json::Value::Null; // id-only adds nothing over discovery
708	        }
709	        serde_json::to_value(addr).unwrap_or(serde_json::Value::Null)
710	    }
711
712	    /// Open a bidi stream on a held connection, registering it (with its read
713	    /// pump) in the stream table (D4b). Note QUIC streams are lazy: the peer
714	    /// sees the stream on its first write.
715	    // [impl->REQ-DAEMON-2]
716	    pub fn open_stream(&self, conn_id: u64) -> io::Result<u64> {
717	        let (conn, remote_id_hex) = {
718	            let conns = self.shared.conns.lock().unwrap();
719	            let e = conns.get(&conn_id).ok_or_else(|| {
720	                io::Error::new(io::ErrorKind::NotFound, format!("no such conn {conn_id}"))
721	            })?;
722	            (e.conn.clone(), e.remote_id_hex.clone())
723	        };
724	        let shared = Arc::clone(&self.shared);
725	        self.runtime.handle().block_on(async move {
726	            let (send, recv) = conn
727	                .open_bi()
728	                .await
729	                .map_err(|e| io::Error::other(e.to_string()))?;
730	            Ok(register_stream(
731	                &shared,
732	                conn_id,
733	                &remote_id_hex,
734	                send,
735	                recv,
736	                true,
737	            ))
738	        })
739	    }
740
741	    /// Record that journaled open `op_id` minted `stream_id` (inside the
742	    /// `apply_once` closure — same discipline as [`Self::record_dial_op`]).
743	    pub fn record_stream_op(&self, op_id: u64, stream_id: u64) {
744	        self.stream_ops.lock().unwrap().insert(op_id, stream_id);
745	    }
746
747	    /// The stream a journaled open `op_id` minted, if still held.
748	    pub fn stream_op_id(&self, op_id: u64) -> Option<u64> {
749	        self.stream_ops.lock().unwrap().get(&op_id).copied()
750	    }
751
752	    /// A held stream's `(conn_id, remote_id_hex)`.
753	    pub fn stream_conn(&self, stream_id: u64) -> Option<(u64, String)> {
754	        self.shared
755	            .streams
756	            .lock()
757	            .unwrap()
758	            .get(&stream_id)
759	            .map(|e| (e.conn_id, e.remote_id_hex.clone()))
760	    }
761
762	    /// Write `bytes` to a held stream's send side; `finish` closes the send
763	    /// side after the write (the peer's read pump then sees a clean end).
764	    // [impl->REQ-DAEMON-2]
765	    pub fn send_stream(&self, stream_id: u64, bytes: &[u8], finish: bool) -> io::Result<()> {
766	        let entry = {
767	            let streams = self.shared.streams.lock().unwrap();
768	            Arc::clone(streams.get(&stream_id).ok_or_else(|| {
769	                io::Error::new(
770	                    io::ErrorKind::NotFound,
771	                    format!("no such stream {stream_id}"),
772	                )
773	            })?)
774	        };
775	        let bytes = bytes.to_vec();
776	        self.runtime.handle().block_on(async move {
777	            let mut send = entry.send.lock().await;
778	            if !bytes.is_empty() {
779	                send.write_all(&bytes)
780	                    .await
781	                    .map_err(|e| io::Error::other(e.to_string()))?;
782	            }
783	            if finish {
784	                send.finish().map_err(|e| io::Error::other(e.to_string()))?;
785	            }
786	            Ok(())
787	        })
788	    }
789
790	    /// Every held stream's info — the query a re-attaching brain uses to find
791	    /// streams (including peer-initiated ones that arrived while it was down).
792	    pub fn stream_infos(&self) -> Vec<NetStreamInfo> {
793	        let streams = self.shared.streams.lock().unwrap();
794	        streams
795	            .iter()
796	            .map(|(id, e)| {
797	                let log = e.log.lock().unwrap();
798	                NetStreamInfo {
799	                    stream_id: *id,
800	                    conn_id: e.conn_id,
801	                    remote_id_hex: e.remote_id_hex.clone(),
802	                    next_seq: log.next_seq,
803	                    finished: log.finished,
804	                    initiated_locally: e.initiated_locally,
805	                }
806	            })
807	            .collect()
808	    }
809
810	    /// Attach `sub` as a stream's read subscriber, replaying from `from_seq`
811	    /// (gapless resume — the PTY `subscribe` discipline on a QUIC stream).
812	    // [impl->REQ-DAEMON-2]
813	    pub(crate) fn subscribe_stream(
814	        &self,
815	        stream_id: u64,
816	        sub: SharedSend,
817	        from_seq: u64,
818	    ) -> io::Result<()> {
819	        let log = {
820	            let streams = self.shared.streams.lock().unwrap();
821	            let e = streams.get(&stream_id).ok_or_else(|| {
822	                io::Error::new(
823	                    io::ErrorKind::NotFound,
824	                    format!("no such stream {stream_id}"),
825	                )
826	            })?;
827	            Arc::clone(&e.log)
828	        };
829	        log.lock().unwrap().attach(sub, from_seq);
830	        Ok(())
831	    }
832
833	    /// Detach `sub` from a stream iff it is the current subscriber (brain
834	    /// disconnect cleanup; the stream + its ring persist).
835	    pub(crate) fn detach_stream_sub(&self, stream_id: u64, sub: &SharedSend) {
836	        let log = {
837	            let streams = self.shared.streams.lock().unwrap();
838	            streams.get(&stream_id).map(|e| Arc::clone(&e.log))
839	        };
840	        if let Some(log) = log {
841	            log.lock().unwrap().detach_if(sub);
842	        }
843	    }
844
845	    /// Attach `sub` as the presence subscriber, replaying liveness events from
846	    /// `from_seq` (D4c) — the stream `subscribe` discipline on the conn table.
847	    // [impl->REQ-EP-4]
848	    pub(crate) fn subscribe_presence(&self, sub: SharedSend, from_seq: u64) {
849	        self.shared.presence.lock().unwrap().attach(sub, from_seq);
850	    }
851
852	    /// Detach `sub` from presence iff it is the current subscriber (brain
853	    /// disconnect cleanup; the log + its ring persist).
854	    pub(crate) fn detach_presence_sub(&self, sub: &SharedSend) {
855	        self.shared.presence.lock().unwrap().detach_if(sub);
856	    }
857
858	    /// Gracefully close the endpoint: every held connection flushes its QUIC
859	    /// CLOSE so peers' closed-watchers fire promptly (a bare host drop kills
860	    /// the runtime first and leaves peers to an idle timeout).
861	    pub fn close(&self) {
862	        let endpoint = Arc::clone(&self.endpoint);
863	        self.runtime.handle().block_on(async move {
864	            endpoint.close().await;
865	        });
866	    }
867	}
868
869	#[cfg(test)]
870	mod tests {
871	    use super::*;
872
873	    fn hermetic(identity: &Identity) -> NetConfig {
874	        NetConfig {
875	            identity: Identity::from_seed(&identity.seed()),
876	            relay: RelayPolicy::Disabled,
877	            discovery: LocalDiscovery::Off,
878	            scope: BindScope::Loopback,
879	            membership: None,
880	        }
881	    }
882
883	    // [unit->REQ-DAEMON-2] the endpoint is constructed broker-side (inside the
884	    // host's dedicated runtime) with relays disabled, bound to the node's own
885	    // identity, with an empty conn table.
886	    #[test]
887	    fn host_binds_endpoint_to_node_identity() {
888	        let id = Identity::generate();
889	        let host = NetHost::start(hermetic(&id)).expect("host start");
890	        assert_eq!(host.node_id_hex(), id.public_key().to_hex());
891	        assert_eq!(host.conn_count(), 0);
892	        assert!(!host.addr_json().is_null(), "dialable address serializes");
893	    }
894
895	    // [unit->REQ-DAEMON-2] a broker-owned host dials a second in-process host
896	    // over hermetic loopback QUIC; both sides register the connection — the
897	    // dialer in its table, the acceptor via its accept loop.
898	    #[test]
899	    fn two_hosts_dial_over_loopback() {
900	        let a = NetHost::start(hermetic(&Identity::generate())).expect("host a");
901	        let b_id = Identity::generate();
902	        let b = NetHost::start(hermetic(&b_id)).expect("host b");
903
904	        let (conn_id, remote) = a.dial(b.addr()).expect("dial");
905	        assert_eq!(conn_id, 1, "first broker-minted conn id");
906	        assert_eq!(
907	            remote,
908	            b_id.public_key().to_hex(),
909	            "handshake proves the peer"
910	        );
911	        assert_eq!(a.conn_count(), 1);
912
913	        // The acceptor's loop registers the inbound connection (async — poll).
914	        for _ in 0..400 {
915	            if b.conn_count() == 1 {
916	                break;
917	            }
918	            std::thread::sleep(std::time::Duration::from_millis(5));
919	        }
920	        assert_eq!(b.conn_count(), 1, "accept loop registered the inbound conn");
921	        assert_eq!(
922	            b.conn_remote_hex(1).as_deref(),
923	            Some(a.node_id_hex().as_str()),
924	            "acceptor sees the dialer's identity"
925	        );
926	    }
927	}
928

//! The broker-owned network host (D4a) — ADR-0004 §B made real for the wire.
//!
//! [`NetHost`] is the broker's ownership of the node's live network presence:
//! the iroh endpoint ([`spt_net::net::endpoint::NetEndpoint`]) and every
//! accepted/dialed QUIC connection. A live cross-node connection is exactly the
//! kind of resource ADR-0004 §B assigns to the broker — a consumer would lose
//! continuity on brain restart — so the endpoint is constructed *here*, in the
//! stable kernel, never in the brain. The brain drives net operations over the
//! versioned IPC ([`crate::msg::KIND_NET_STATUS`] / [`crate::msg::KIND_NET_DIAL`])
//! and may detach/reattach freely; connections persist in the broker's table.
//!
//! ## Runtime placement (M4-D4-PLAN locked decision)
//!
//! The broker is otherwise sync/thread-based; iroh requires tokio. So the host
//! owns a **dedicated multi-thread tokio runtime** on named worker threads
//! (`spt-broker-net`): QUIC objects live in structures only the runtime's tasks
//! and the host's own `block_on` bridges touch, and the runtime drops with the
//! host (broker teardown stops all net tasks). Spike #3's broker binary proved
//! exactly this shape — a broker-owned endpoint surviving brain kill/restart.
//!
//! ## Ownership closes two §B rows by construction
//!
//! `MdnsAddressLookup` and the relay session are both constructed *inside* the
//! iroh endpoint, so broker ownership of the endpoint closes the ADR-0004 §B
//! "mDNS socket / relay session" row with no extra machinery: there is no
//! separate socket to own.
// [impl->REQ-DAEMON-2]

use std::collections::{HashMap, HashSet, VecDeque};
use std::io;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};

use spt_net::net::endpoint::{
    BindScope, Connection, EndpointAddr, LocalDiscovery, NetEndpoint, RecvStream, RelayPolicy,
    SendStream, SPT_NET_ALPN,
};
use spt_net::net::mesh::seedproof::ProofRole;
use spt_proto::identity::{Identity, PublicKey};

use crate::broker::SharedSend;
use crate::codec::write_frame;
use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};
use crate::msg::{
    net_presence_event_envelope, net_stream_data_envelope, net_stream_eof_envelope,
    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
};

/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped
/// effects (a dial has no PTY session). Broker session ids are minted from 1
/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.
pub const NET_EFFECT_SESSION: u64 = 0;

/// Everything needed to stand the broker's endpoint up: the node's identity
/// (the endpoint *is* the node's name — EndpointId == pubkey), the relay stance
/// (from `daemon.json` in production; [`RelayPolicy::Disabled`] in hermetic
/// tests), and whether mDNS LAN discovery rides along.
pub struct NetConfig {
    /// The node identity the endpoint binds as (seed → same ed25519 keypair).
    pub identity: Identity,
    /// Relay stance (ADR-0002): n0 default / self-host / disabled.
    pub relay: RelayPolicy,
    /// mDNS LAN discovery on/off (off in hermetic tests).
    pub discovery: LocalDiscovery,
    /// Socket bind scope: [`BindScope::All`] in production;
    /// [`BindScope::Loopback`] in hermetic tests (no real interface touched,
    /// no per-test-binary Windows firewall prompt).
    pub scope: BindScope,
    /// Live source of this node's subnet credentials for the connect-time
    /// seed-proof (Mesh-D2, REQ-MESH-1). `Some` enables the mutual proof at
    /// every dial + accept (fail → drop); `None` disables it entirely — both
    /// peers must agree, or the side expecting the control stream would hang.
    /// Production wires `Some(SubnetStore::load …)`; mechanics-only tests that
    /// don't exercise membership pass `None`.
    pub membership: Option<MembershipSource>,
}

/// Default bounded stream-read ring depth (chunks) — same sizing rationale as
/// the PTY `OutputLog`: large enough that a brain-restart window never evicts
/// unreplayed chunks in practice; bounded so an unsubscribed stream can't
/// buffer without limit.
const DEFAULT_STREAM_RING_CHUNKS: usize = 4096;

/// Max bytes pulled per stream read (one ring chunk).
const STREAM_READ_CHUNK: usize = 64 * 1024;

/// One live QUIC connection the broker holds.
struct ConnEntry {
    /// The connection itself — kept alive by ownership here; dropping it closes
    /// the QUIC connection. Cloned (cheap handle) for stream opens.
    conn: Connection,
    /// The remote node's stable name (hex pubkey), proven by the QUIC handshake.
    remote_id_hex: String,
    /// The subnets the peer proved current-epoch seed-knowledge of at connect
    /// (Mesh-D2, REQ-MESH-1). Empty when seed-proof is disabled (`membership:
    /// None`). The D5 gate swap reads this in place of `is_trusted(subnet, …)`;
    /// at D2 it is cached but not yet consumed (the `is_trusted` gates still run
    /// in parallel — belt + suspenders, no regression).
    proven_subnets: HashSet<String>,
}

/// A bounded, sequenced read log for one QUIC stream — the PTY `OutputLog`
/// discipline applied to the network (D4b): the stream's single read pump is
/// the only producer; a brain connection attaches as the subscriber; every
/// subscriber send happens under the enclosing `Mutex`, so live chunks and a
/// replay can never interleave out of `seq` order. Chunks produced while no
/// brain is attached sit in the ring and replay on the next subscribe — a
/// brain restart loses nothing (Spike #3 invariant B/D).
// [impl->REQ-DAEMON-2]
struct StreamLog {
    stream_id: u64,
    next_seq: u64,
    ring: VecDeque<(u64, Vec<u8>)>,
    cap_chunks: usize,
    subscriber: Option<SharedSend>,
    /// Read side finished cleanly (peer called finish / conn ended).
    finished: bool,
}

impl StreamLog {
    fn new(stream_id: u64, cap_chunks: usize) -> Self {
        StreamLog {
            stream_id,
            next_seq: 0,
            ring: VecDeque::new(),
            cap_chunks,
            subscriber: None,
            finished: false,
        }
    }

    /// Append a freshly-read chunk: assign the next `seq`, retain it in the
    /// bounded ring, live-send to the current subscriber. Read-pump only.
    fn append(&mut self, bytes: &[u8]) {
        let seq = self.next_seq;
        self.next_seq += 1;
        self.ring.push_back((seq, bytes.to_vec()));
        while self.ring.len() > self.cap_chunks {
            self.ring.pop_front();
        }
        if let Some(sub) = &self.subscriber {
            let frame = net_stream_data_envelope(self.stream_id, seq, bytes);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
    }

    /// Mark the read side finished and tell the subscriber (with the final
    /// cursor, so it can verify completeness).
    fn finish(&mut self) {
        self.finished = true;
        if let Some(sub) = &self.subscriber {
            let frame = net_stream_eof_envelope(self.stream_id, self.next_seq);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
    }

    /// Attach `sub` as the subscriber, replaying every retained chunk with
    /// `seq >= from_seq` (then eof, if already finished) before live streaming.
    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
        self.subscriber = Some(Arc::clone(&sub));
        for (seq, bytes) in self.ring.iter().filter(|(s, _)| *s >= from_seq) {
            let frame = net_stream_data_envelope(self.stream_id, *seq, bytes);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
        if self.finished {
            let frame = net_stream_eof_envelope(self.stream_id, self.next_seq);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
    }

    /// Detach `sub` iff it is the current subscriber (a late disconnect must
    /// not unsubscribe a newer brain).
    fn detach_if(&mut self, sub: &SharedSend) {
        if let Some(cur) = &self.subscriber {
            if Arc::ptr_eq(cur, sub) {
                self.subscriber = None;
            }
        }
    }
}

/// A bounded, sequenced log of connection liveness events — the broker-owned
/// PresenceChannel seam (D4c, REQ-EP-4): the StreamLog ring/cursor discipline
/// applied to the conn table itself. `register_conn` appends `connected`; each
/// connection's closed-watcher appends `disconnected` after removing the row.
/// A brain subscribes with a cursor and replays events produced while it was
/// down, so presence survives a brain restart exactly like stream reads do.
// [impl->REQ-EP-4]
struct PresenceLog {
    next_seq: u64,
    ring: VecDeque<NetPresenceEvent>,
    cap_events: usize,
    subscriber: Option<SharedSend>,
}

impl PresenceLog {
    fn new(cap_events: usize) -> Self {
        PresenceLog {
            next_seq: 0,
            ring: VecDeque::new(),
            cap_events,
            subscriber: None,
        }
    }

    /// Append a liveness event: assign the next `seq`, retain it in the bounded
    /// ring, live-send to the current subscriber. Producers: `register_conn`
    /// (connected) and the per-conn closed-watchers (disconnected) — every
    /// append serializes on the enclosing `Mutex`, so subscriber frames arrive
    /// in strict `seq` order.
    fn append(&mut self, kind: &str, conn_id: u64, remote_id_hex: &str) {
        let ev = NetPresenceEvent {
            seq: self.next_seq,
            kind: kind.to_string(),
            conn_id,
            remote_id_hex: remote_id_hex.to_string(),
        };
        self.next_seq += 1;
        self.ring.push_back(ev.clone());
        while self.ring.len() > self.cap_events {
            self.ring.pop_front();
        }
        if let Some(sub) = &self.subscriber {
            let frame = net_presence_event_envelope(&ev);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
    }

    /// Attach `sub` as the subscriber, replaying every retained event with
    /// `seq >= from_seq` before live streaming (the gapless-resume attach).
    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
        self.subscriber = Some(Arc::clone(&sub));
        for ev in self.ring.iter().filter(|e| e.seq >= from_seq) {
            let frame = net_presence_event_envelope(ev);
            if let Ok(mut s) = sub.lock() {
                let _ = write_frame(&mut *s, &frame);
            }
        }
    }

    /// Detach `sub` iff it is the current subscriber (a late disconnect must
    /// not unsubscribe a newer brain).
    fn detach_if(&mut self, sub: &SharedSend) {
        if let Some(cur) = &self.subscriber {
            if Arc::ptr_eq(cur, sub) {
                self.subscriber = None;
            }
        }
    }
}

/// One live QUIC stream the broker holds: the send half (brain writes ride it
/// over IPC) + the sequenced read log.
struct StreamEntry {
    conn_id: u64,
    remote_id_hex: String,
    /// Whether THIS node opened the stream (`open_stream`) vs the peer
    /// (accept loop). The inbound dispatcher (D9-1) claims only
    /// peer-initiated streams; without this fact the table is ambiguous —
    /// it holds locally-opened outbound streams too (gotcha: a dispatcher
    /// that served its own requester stream would deadlock on itself).
    initiated_locally: bool,
    /// The send half, serialized behind an async mutex (writes happen on the
    /// host's runtime via `block_on`).
    send: tokio::sync::Mutex<SendStream>,
    log: Arc<Mutex<StreamLog>>,
}

/// Tables shared between the host's sync surface and its runtime tasks
/// (accept loop, per-conn stream acceptors, per-stream read pumps).
struct NetShared {
    next_conn_id: AtomicU64,
    conns: Mutex<HashMap<u64, ConnEntry>>,
    next_stream_id: AtomicU64,
    streams: Mutex<HashMap<u64, Arc<StreamEntry>>>,
    ring_cap: usize,
    /// The conn-liveness event log (D4c) — one per host, all conns interleaved.
    presence: Mutex<PresenceLog>,
}

impl NetShared {
    fn new() -> Self {
        NetShared {
            next_conn_id: AtomicU64::new(1),
            conns: Mutex::new(HashMap::new()),
            next_stream_id: AtomicU64::new(1),
            streams: Mutex::new(HashMap::new()),
            ring_cap: DEFAULT_STREAM_RING_CHUNKS,
            presence: Mutex::new(PresenceLog::new(DEFAULT_STREAM_RING_CHUNKS)),
        }
    }
}

/// Register a live connection: mint its broker id and spawn its inbound-stream
/// acceptor. Must run inside the host's runtime context (the accept loop, or a
/// `block_on` future) — it spawns tasks.
fn register_conn(
    shared: &Arc<NetShared>,
    conn: Connection,
    proven_subnets: HashSet<String>,
) -> (u64, String) {
    let remote_id_hex = PublicKey::from_bytes(conn.remote_id().as_bytes())
        .expect("an iroh EndpointId is always a valid ed25519 public key")
        .to_hex();
    let id = shared.next_conn_id.fetch_add(1, Ordering::Relaxed);
    shared.conns.lock().unwrap().insert(
        id,
        ConnEntry {
            conn: conn.clone(),
            remote_id_hex: remote_id_hex.clone(),
            proven_subnets,
        },
    );

    // Presence: announce the row BEFORE spawning the closed-watcher, so the
    // `connected` event can never race behind its own `disconnected` (D4c).
    shared
        .presence
        .lock()
        .unwrap()
        .append(PRESENCE_CONNECTED, id, &remote_id_hex);

    // Closed-watcher: the row's exit path. Awaits the connection's end (clean
    // close, conn loss, or our own endpoint close), removes the row — conns
    // finally LEAVE the table — then appends the `disconnected` event (row
    // gone before the event is observable).
    // [impl->REQ-EP-4]
    let sh_close = Arc::clone(shared);
    let conn_close = conn.clone();
    let hex_close = remote_id_hex.clone();
    tokio::spawn(async move {
        conn_close.closed().await;
        sh_close.conns.lock().unwrap().remove(&id);
        sh_close
            .presence
            .lock()
            .unwrap()
            .append(PRESENCE_DISCONNECTED, id, &hex_close);
    });

    // Inbound-stream acceptor: every peer-opened bidi stream joins the table
    // and gets a read pump. (QUIC streams are lazy — the peer's open becomes
    // visible here on its first write.) Ends when the connection does.
    let sh = Arc::clone(shared);
    let hex = remote_id_hex.clone();
    tokio::spawn(async move {
        while let Ok((send, recv)) = conn.accept_bi().await {
            register_stream(&sh, id, &hex, send, recv, false);
        }
    });

    (id, remote_id_hex)
}

/// Register a live stream (opened or accepted): mint its broker id and spawn
/// its read pump. Must run inside the host's runtime context.
/// `initiated_locally` records the direction fact for the table (true only
/// from [`NetHost::open_stream`]).
fn register_stream(
    shared: &Arc<NetShared>,
    conn_id: u64,
    remote_id_hex: &str,
    send: SendStream,
    recv: RecvStream,
    initiated_locally: bool,
) -> u64 {
    let id = shared.next_stream_id.fetch_add(1, Ordering::Relaxed);
    let log = Arc::new(Mutex::new(StreamLog::new(id, shared.ring_cap)));
    shared.streams.lock().unwrap().insert(
        id,
        Arc::new(StreamEntry {
            conn_id,
            remote_id_hex: remote_id_hex.to_string(),
            initiated_locally,
            send: tokio::sync::Mutex::new(send),
            log: Arc::clone(&log),
        }),
    );

    // The read pump — the stream's single producer. A clean end (peer finish)
    // and a torn end (conn lost) both finish the log; the buffered chunks stay
    // replayable either way.
    let mut recv = recv;
    tokio::spawn(async move {
        loop {
            match recv.read_chunk(STREAM_READ_CHUNK).await {
                Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),
                Ok(None) | Err(_) => {
                    log.lock().unwrap().finish();
                    break;
                }
            }
        }
    });

    id
}

/// The broker-owned net endpoint + live connection/stream tables (see module
/// docs).
pub struct NetHost {
    /// The dedicated tokio runtime hosting the endpoint and its accept loop.
    /// Dropping the host drops the runtime, stopping every net task with it.
    runtime: tokio::runtime::Runtime,
    endpoint: Arc<NetEndpoint>,
    shared: Arc<NetShared>,
    /// `op_id → conn_id` for exactly-once dial replies: a deduped replay of a
    /// journaled dial is answered with the connection the first apply opened.
    dial_ops: Mutex<HashMap<u64, u64>>,
    /// `op_id → stream_id` for exactly-once stream-open replies (D4b), same
    /// discipline as `dial_ops`. Shares the one net op-id namespace.
    stream_ops: Mutex<HashMap<u64, u64>>,
    /// This node's pubkey (the seed-proof's local channel-binding identity).
    local_id: PublicKey,
    /// The connect-time seed-proof credential source (Mesh-D2). `None` disables
    /// the proof (see [`NetConfig::membership`]).
    membership: Option<MembershipSource>,
    /// The on-connect roster-exchange seam (Mesh-D4, REQ-MESH-2), set on the live
    /// host after `start` via [`Self::set_roster_exchange`] (kept off `NetConfig`
    /// so the many hermetic test configs need no change — a roster exchange is a
    /// layer atop the proof, opt-in per host). `None` ⇒ no roster leg. Read +
    /// cloned per connect on the net runtime, so the accept loop (spawned in
    /// `start`, before the host exists) shares it through this `Arc`.
    roster: Arc<Mutex<Option<RosterExchange>>>,
}

impl NetHost {
    /// Bind the broker's endpoint on a fresh dedicated runtime and start the
    /// accept loop. The endpoint speaks TWO ALPNs (M7 D3, REQ-SUBNET-2): the
    /// trusted [`SPT_NET_ALPN`] (connections join the table as before) and
    /// the pre-trust pairing ALPN — routed to the always-on responder
    /// ([`crate::pairhost::respond`]), never registered in the stream table.
    /// One endpoint for both is deliberate: the pairing transcript binds and
    /// pins the REAL node pubkey, and the relay can only route one endpoint
    /// per key — the ALPN is the pre-trust/trusted boundary (ADR-0005).
    /// The meet-rotation listener loop rides the same runtime.
    // [impl->REQ-DAEMON-2]
    // [impl->REQ-SUBNET-2]
    pub fn start(cfg: NetConfig) -> io::Result<NetHost> {
        // Hermeticity guard (REQ-PAIR-8): a loopback-scoped endpoint is the
        // "no network reached" promise — the ceremony clock's lazy NTP query
        // must honor it (tests must never probe a real NTP server). Real
        // scopes keep the decision-18 offset correction.
        spt_net::net::pairing::ntp::set_enabled(!matches!(cfg.scope, BindScope::Loopback));
        let runtime = tokio::runtime::Builder::new_multi_thread()
            .worker_threads(2)
            .thread_name("spt-broker-net")
            .enable_all()
            .build()?;
        let endpoint = runtime
            .block_on(NetEndpoint::bind_scoped(
                &cfg.identity,
                cfg.relay.clone(),
                cfg.discovery,
                vec![
                    SPT_NET_ALPN.to_vec(),
                    spt_net::net::pairing::wire::SPT_PAIR_ALPN.to_vec(),
                ],
                cfg.scope,
            ))
            .map_err(|e| io::Error::other(e.to_string()))?;
        let endpoint = Arc::new(endpoint);
        let shared = Arc::new(NetShared::new());
        let local_pub = cfg.identity.public_key();
        let pair_rate = Arc::new(tokio::sync::Mutex::new(
            spt_net::net::pairing::ratelimit::PairingRateLimiter::new(),
        ));

        // Accept loop: lives for the endpoint's lifetime on the host's runtime.
        // Each handshake completes on its own task so a slow peer can't stall
        // the loop; a failed handshake is simply not registered. The pairing
        // ALPN routes to the responder; everything else is the trusted table.
        let roster = Arc::new(Mutex::new(None::<RosterExchange>));
        let ep = Arc::clone(&endpoint);
        let sh = Arc::clone(&shared);
        let rate = Arc::clone(&pair_rate);
        let accept_membership = cfg.membership.clone();
        let accept_roster = Arc::clone(&roster);
        let accept_ep = Arc::clone(&endpoint);
        runtime.spawn(async move {
            while let Some(incoming) = ep.accept().await {
                let sh = Arc::clone(&sh);
                let rate = Arc::clone(&rate);
                let membership = accept_membership.clone();
                // Snapshot the (opt-in) roster exchange + our live address for
                // this connect; the leg runs after the proof verifies.
                let roster_rx = accept_roster.lock().unwrap().clone();
                let self_addr =
                    serde_json::to_value(accept_ep.addr()).unwrap_or(serde_json::Value::Null);
                tokio::spawn(async move {
                    let Ok(connecting) = incoming.accept() else {
                        return;
                    };
                    let Ok(conn) = connecting.await else { return };
                    if conn.alpn() == spt_net::net::pairing::wire::SPT_PAIR_ALPN {
                        crate::pairhost::respond(conn, local_pub, rate).await;
                    } else {
                        // Mesh-D2: prove membership in-band BEFORE the connection
                        // joins the table. A non-prover (no shared subnet / bad
                        // seed) is dropped — the conn falls out of scope and QUIC
                        // closes it. With membership disabled, register as before.
                        // Mesh-D4: on success, a roster leg rides the same stream.
                        let proven = match membership {
                            Some(src) => {
                                let creds = src();
                                match prove_membership(
                                    &conn,
                                    local_pub,
                                    ProofRole::Acceptor,
                                    &creds,
                                    roster_rx.as_ref(),
                                    &self_addr,
                                )
                                .await
                                {
                                    Some(set) => set,
                                    None => return, // drop the connection
                                }
                            }
                            None => HashSet::new(),
                        };
                        register_conn(&sh, conn, proven);
                    }
                });
            }
        });

        // The meet-rotation loop (M7 D3): how a joiner that knows only
        // (subnet name, code) finds this responder. Idles with no subnets.
        crate::pairhost::spawn_meet_rotation(
            runtime.handle(),
            Arc::clone(&endpoint),
            cfg.relay,
            cfg.discovery,
            cfg.scope,
        );

        // The limiter Arc lives on in the accept loop; the host keeps no
        // handle (per-ceremony state is the responder's, not the table's).
        drop(pair_rate);

        Ok(NetHost {
            runtime,
            endpoint,
            shared,
            dial_ops: Mutex::new(HashMap::new()),
            stream_ops: Mutex::new(HashMap::new()),
            local_id: local_pub,
            membership: cfg.membership,
            roster,
        })
    }

    /// Wire the on-connect roster-exchange seam (Mesh-D4, REQ-MESH-2) onto this
    /// live host. The daemon calls this once after `start` with
    /// [`crate::seedproofx::production_roster_exchange`]; tests inject their own.
    /// Off `NetConfig` by design — see the `roster` field. Takes effect on the
    /// next dial/accept (each connect snapshots it).
    // [impl->REQ-MESH-2]
    pub fn set_roster_exchange(&self, exchange: RosterExchange) {
        *self.roster.lock().unwrap() = Some(exchange);
    }

    /// Run the joiner leg of the pairing ceremony (M7 D3, REQ-SUBNET-2) on
    /// the host's runtime — the broker IPC `pair-join` arm. The ceremony
    /// must run HERE because it binds and pins this endpoint's real node
    /// pubkey. Returns the rendered trust outcome, or the classified failure.
    // [impl->REQ-SUBNET-2]
    pub fn pair_join(
        &self,
        subnet: &str,
        code: &str,
        responder_addr: Option<EndpointAddr>,
    ) -> Result<String, crate::pairhost::JoinFail> {
        let endpoint = Arc::clone(&self.endpoint);
        self.runtime.handle().block_on(async move {
            crate::pairhost::join_ceremony(&endpoint, subnet, code, responder_addr).await
        })
    }

    /// The node's stable name (lowercase-hex pubkey == iroh EndpointId).
    pub fn node_id_hex(&self) -> String {
        self.endpoint.node_id_hex()
    }

    /// The endpoint's dialable address (id + currently-known direct/relay paths).
    pub fn addr(&self) -> EndpointAddr {
        self.endpoint.addr()
    }

    /// [`Self::addr`] as the opaque JSON the IPC status reply carries.
    pub fn addr_json(&self) -> serde_json::Value {
        serde_json::to_value(self.endpoint.addr()).unwrap_or(serde_json::Value::Null)
    }

    /// Live connections (accepted + dialed) currently held.
    pub fn conn_count(&self) -> u64 {
        self.shared.conns.lock().unwrap().len() as u64
    }

    /// Dial a peer over `SPT_NET_ALPN`, registering the connection in the table.
    /// Bridges sync→async via the host's own runtime (the broker's dispatch
    /// threads block here; the future runs on the net worker threads).
    // [impl->REQ-DAEMON-2]
    pub fn dial(&self, addr: EndpointAddr) -> io::Result<(u64, String)> {
        let endpoint = Arc::clone(&self.endpoint);
        let shared = Arc::clone(&self.shared);
        let local_id = self.local_id;
        let membership = self.membership.clone();
        // Snapshot the (opt-in) roster exchange + our live address for this dial.
        let roster_rx = self.roster.lock().unwrap().clone();
        let self_addr = serde_json::to_value(self.endpoint.addr()).unwrap_or(serde_json::Value::Null);
        self.runtime.handle().block_on(async move {
            let conn = endpoint
                .connect(addr)
                .await
                .map_err(|e| io::Error::other(e.to_string()))?;
            // Mesh-D2: prove membership in-band before the connection is usable.
            // A failed/empty proof drops the dial (the conn never registers).
            // Mesh-D4: on success, a roster leg rides the same control stream.
            let proven = match membership {
                Some(src) => {
                    let creds = src();
                    prove_membership(
                        &conn,
                        local_id,
                        ProofRole::Dialer,
                        &creds,
                        roster_rx.as_ref(),
                        &self_addr,
                    )
                    .await
                    .ok_or_else(|| {
                        io::Error::other("seed-proof failed: peer is not a subnet member")
                    })?
                }
                None => HashSet::new(),
            };
            Ok(register_conn(&shared, conn, proven))
        })
    }

    /// Record that journaled dial `op_id` opened `conn_id` (called inside the
    /// `apply_once` closure, so a concurrent deduped replay always finds it).
    pub fn record_dial_op(&self, op_id: u64, conn_id: u64) {
        self.dial_ops.lock().unwrap().insert(op_id, conn_id);
    }

    /// The connection a journaled dial `op_id` opened, if this broker process
    /// still holds it (a broker restart drops connections; the journal alone
    /// survives — see `Broker::dispatch_net_dial`).
    pub fn dial_op_conn(&self, op_id: u64) -> Option<u64> {
        self.dial_ops.lock().unwrap().get(&op_id).copied()
    }

    /// The remote node id (hex) of a held connection.
    pub fn conn_remote_hex(&self, conn_id: u64) -> Option<String> {
        self.shared
            .conns
            .lock()
            .unwrap()
            .get(&conn_id)
            .map(|e| e.remote_id_hex.clone())
    }

    /// The subnets the peer of `conn_id` proved current-epoch seed-knowledge of
    /// at connect (Mesh-D2, REQ-MESH-1). Empty when seed-proof was disabled or
    /// the conn is gone. The seam the D5 gate swap reads in place of
    /// `is_trusted(subnet, origin)`; unused by any gate at D2.
    // [impl->REQ-MESH-1]
    pub fn conn_proven_subnets(&self, conn_id: u64) -> HashSet<String> {
        self.shared
            .conns
            .lock()
            .unwrap()
            .get(&conn_id)
            .map(|e| e.proven_subnets.clone())
            .unwrap_or_default()
    }

    /// The OBSERVED peer address of a held connection (remote id + the
    /// connection's current path addresses) as opaque JSON — the address-
    /// seeding write-back the `net-dialed` reply carries (M8 decision 20,
    /// REQ-CONV-1). `null` when the conn is gone, path-less, or fails to
    /// serialize: the seed is an optimization, never an error path.
    // [impl->REQ-CONV-1]
    pub fn conn_remote_addr_json(&self, conn_id: u64) -> serde_json::Value {
        let conn = {
            let conns = self.shared.conns.lock().unwrap();
            match conns.get(&conn_id) {
                Some(e) => e.conn.clone(),
                None => return serde_json::Value::Null,
            }
        };
        let addr = spt_net::net::endpoint::conn_remote_addr(&conn);
        if addr.is_empty() {
            return serde_json::Value::Null; // id-only adds nothing over discovery
        }
        serde_json::to_value(addr).unwrap_or(serde_json::Value::Null)
    }

    /// Open a bidi stream on a held connection, registering it (with its read
    /// pump) in the stream table (D4b). Note QUIC streams are lazy: the peer
    /// sees the stream on its first write.
    // [impl->REQ-DAEMON-2]
    pub fn open_stream(&self, conn_id: u64) -> io::Result<u64> {
        let (conn, remote_id_hex) = {
            let conns = self.shared.conns.lock().unwrap();
            let e = conns.get(&conn_id).ok_or_else(|| {
                io::Error::new(io::ErrorKind::NotFound, format!("no such conn {conn_id}"))
            })?;
            (e.conn.clone(), e.remote_id_hex.clone())
        };
        let shared = Arc::clone(&self.shared);
        self.runtime.handle().block_on(async move {
            let (send, recv) = conn
                .open_bi()
                .await
                .map_err(|e| io::Error::other(e.to_string()))?;
            Ok(register_stream(
                &shared,
                conn_id,
                &remote_id_hex,
                send,
                recv,
                true,
            ))
        })
    }

    /// Record that journaled open `op_id` minted `stream_id` (inside the
    /// `apply_once` closure — same discipline as [`Self::record_dial_op`]).
    pub fn record_stream_op(&self, op_id: u64, stream_id: u64) {
        self.stream_ops.lock().unwrap().insert(op_id, stream_id);
    }

    /// The stream a journaled open `op_id` minted, if still held.
    pub fn stream_op_id(&self, op_id: u64) -> Option<u64> {
        self.stream_ops.lock().unwrap().get(&op_id).copied()
    }

    /// A held stream's `(conn_id, remote_id_hex)`.
    pub fn stream_conn(&self, stream_id: u64) -> Option<(u64, String)> {
        self.shared
            .streams
            .lock()
            .unwrap()
            .get(&stream_id)
            .map(|e| (e.conn_id, e.remote_id_hex.clone()))
    }

    /// Write `bytes` to a held stream's send side; `finish` closes the send
    /// side after the write (the peer's read pump then sees a clean end).
    // [impl->REQ-DAEMON-2]
    pub fn send_stream(&self, stream_id: u64, bytes: &[u8], finish: bool) -> io::Result<()> {
        let entry = {
            let streams = self.shared.streams.lock().unwrap();
            Arc::clone(streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?)
        };
        let bytes = bytes.to_vec();
        self.runtime.handle().block_on(async move {
            let mut send = entry.send.lock().await;
            if !bytes.is_empty() {
                send.write_all(&bytes)
                    .await
                    .map_err(|e| io::Error::other(e.to_string()))?;
            }
            if finish {
                send.finish().map_err(|e| io::Error::other(e.to_string()))?;
            }
            Ok(())
        })
    }

    /// Every held stream's info — the query a re-attaching brain uses to find
    /// streams (including peer-initiated ones that arrived while it was down).
    pub fn stream_infos(&self) -> Vec<NetStreamInfo> {
        let streams = self.shared.streams.lock().unwrap();
        streams
            .iter()
            .map(|(id, e)| {
                let log = e.log.lock().unwrap();
                NetStreamInfo {
                    stream_id: *id,
                    conn_id: e.conn_id,
                    remote_id_hex: e.remote_id_hex.clone(),
                    next_seq: log.next_seq,
                    finished: log.finished,
                    initiated_locally: e.initiated_locally,
                }
            })
            .collect()
    }

    /// Attach `sub` as a stream's read subscriber, replaying from `from_seq`
    /// (gapless resume — the PTY `subscribe` discipline on a QUIC stream).
    // [impl->REQ-DAEMON-2]
    pub(crate) fn subscribe_stream(
        &self,
        stream_id: u64,
        sub: SharedSend,
        from_seq: u64,
    ) -> io::Result<()> {
        let log = {
            let streams = self.shared.streams.lock().unwrap();
            let e = streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?;
            Arc::clone(&e.log)
        };
        log.lock().unwrap().attach(sub, from_seq);
        Ok(())
    }

    /// Detach `sub` from a stream iff it is the current subscriber (brain
    /// disconnect cleanup; the stream + its ring persist).
    pub(crate) fn detach_stream_sub(&self, stream_id: u64, sub: &SharedSend) {
        let log = {
            let streams = self.shared.streams.lock().unwrap();
            streams.get(&stream_id).map(|e| Arc::clone(&e.log))
        };
        if let Some(log) = log {
            log.lock().unwrap().detach_if(sub);
        }
    }

    /// Attach `sub` as the presence subscriber, replaying liveness events from
    /// `from_seq` (D4c) — the stream `subscribe` discipline on the conn table.
    // [impl->REQ-EP-4]
    pub(crate) fn subscribe_presence(&self, sub: SharedSend, from_seq: u64) {
        self.shared.presence.lock().unwrap().attach(sub, from_seq);
    }

    /// Detach `sub` from presence iff it is the current subscriber (brain
    /// disconnect cleanup; the log + its ring persist).
    pub(crate) fn detach_presence_sub(&self, sub: &SharedSend) {
        self.shared.presence.lock().unwrap().detach_if(sub);
    }

    /// Gracefully close the endpoint: every held connection flushes its QUIC
    /// CLOSE so peers' closed-watchers fire promptly (a bare host drop kills
    /// the runtime first and leaves peers to an idle timeout).
    pub fn close(&self) {
        let endpoint = Arc::clone(&self.endpoint);
        self.runtime.handle().block_on(async move {
            endpoint.close().await;
        });
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn hermetic(identity: &Identity) -> NetConfig {
        NetConfig {
            identity: Identity::from_seed(&identity.seed()),
            relay: RelayPolicy::Disabled,
            discovery: LocalDiscovery::Off,
            scope: BindScope::Loopback,
            membership: None,
        }
    }

    // [unit->REQ-DAEMON-2] the endpoint is constructed broker-side (inside the
    // host's dedicated runtime) with relays disabled, bound to the node's own
    // identity, with an empty conn table.
    #[test]
    fn host_binds_endpoint_to_node_identity() {
        let id = Identity::generate();
        let host = NetHost::start(hermetic(&id)).expect("host start");
        assert_eq!(host.node_id_hex(), id.public_key().to_hex());
        assert_eq!(host.conn_count(), 0);
        assert!(!host.addr_json().is_null(), "dialable address serializes");
    }

    // [unit->REQ-DAEMON-2] a broker-owned host dials a second in-process host
    // over hermetic loopback QUIC; both sides register the connection — the
    // dialer in its table, the acceptor via its accept loop.
    #[test]
    fn two_hosts_dial_over_loopback() {
        let a = NetHost::start(hermetic(&Identity::generate())).expect("host a");
        let b_id = Identity::generate();
        let b = NetHost::start(hermetic(&b_id)).expect("host b");

        let (conn_id, remote) = a.dial(b.addr()).expect("dial");
        assert_eq!(conn_id, 1, "first broker-minted conn id");
        assert_eq!(
            remote,
            b_id.public_key().to_hex(),
            "handshake proves the peer"
        );
        assert_eq!(a.conn_count(), 1);

        // The acceptor's loop registers the inbound connection (async — poll).
        for _ in 0..400 {
            if b.conn_count() == 1 {
                break;
            }
            std::thread::sleep(std::time::Duration::from_millis(5));
        }
        assert_eq!(b.conn_count(), 1, "accept loop registered the inbound conn");
        assert_eq!(
            b.conn_remote_hex(1).as_deref(),
            Some(a.node_id_hex().as_str()),
            "acceptor sees the dialer's identity"
        );
    }
}


Now find the dispatcher that consumes `conn_proven_subnets` (the D5 serve gate) and the pump.


M4-D9-PLAN.md:138:  `request_update`/`serve_update`, D8-2 notif emit/apply, `reconcile_after_sync`) —
M4-D7-PLAN.md:48:  = the bytes; atomic temp→rename, same posture as D5c commit). `serve_update`
M4-D7-PLAN.md:82:     `serve_update(brain, stream_id, origin_node, net_from_seq, cache, trust)`
crates\spt-daemon\tests\dispatch.rs:209:// serve_update. The serve gate ran from B's canonical roster.
crates\spt-daemon\tests\seedproofx.rs:77:        a.conn_proven_subnets(conn_id),
crates\spt-daemon\tests\seedproofx.rs:86:        b.conn_proven_subnets(1),
crates\spt-daemon\tests\seedproofx.rs:108:        a.conn_proven_subnets(conn_id),
crates\spt-daemon\tests\seedproofx.rs:113:    assert_eq!(b.conn_proven_subnets(1), set(&["home", "work"]));
crates\spt-daemon\tests\rosterprop.rs:169:fn roster_exchange_never_leaks_unproven_subnets() {
crates\spt-daemon\tests\propagate.rs:20:    classify_status, request_update, request_update_status, serve_update, ConvergeState,
crates\spt-daemon\tests\propagate.rs:263:    let outcome = serve_update(
crates\spt-daemon\tests\propagate.rs:451:    // (an honest serve_update never offers a rollback — the gate under test
crates\spt-daemon\tests\propagate.rs:555:    let outcome = serve_update(
crates\spt-store\src\roster.rs:194:    pub fn is_member_any(&self, pubkey_hex: &str) -> bool {
crates\spt-store\src\roster.rs:462:    // [unit->REQ-MESH-5] is_member_any: a node present in ≥1 subnet passes the
crates\spt-store\src\roster.rs:466:    fn is_member_any_spans_subnets_and_honors_tombstones() {
crates\spt-store\src\roster.rs:471:        assert!(r.is_member_any("aa"));
crates\spt-store\src\roster.rs:472:        assert!(!r.is_member_any("zz"), "unknown key is no member anywhere");
crates\spt-store\src\roster.rs:476:        assert!(!r.is_member_any("bb"), "tombstoned in its only subnet");
crates\spt-store\src\roster.rs:480:        assert!(r.is_member_any("aa"), "still a member in another subnet");
crates\spt-daemon\src\dispatch.rs:5://! `serve_sync` / `serve_update` / `apply_notif_feed` / `receive_wan` against
crates\spt-daemon\src\dispatch.rs:33://! | `kind=query` | update pull ([`crate::propagate::serve_update`]) |
crates\spt-daemon\src\dispatch.rs:78:use crate::propagate::serve_update;
crates\spt-daemon\src\dispatch.rs:327:            match serve_update(&mut brain, stream_id, origin, 0, &cache, &roster) {
crates\spt-daemon\src\lib.rs:199:pub use propagate::{request_update, serve_update, UpdatePullOutcome, UpdateServeOutcome};
crates\spt-daemon\src\nethost.rs:99:    proven_subnets: HashSet<String>,
crates\spt-daemon\src\nethost.rs:310:    proven_subnets: HashSet<String>,
crates\spt-daemon\src\nethost.rs:321:            proven_subnets,
crates\spt-daemon\src\nethost.rs:681:    pub fn conn_proven_subnets(&self, conn_id: u64) -> HashSet<String> {
crates\spt-daemon\src\nethost.rs:687:            .map(|e| e.proven_subnets.clone())
crates\spt-daemon\src\relcache.rs:6://! Staging is what makes a node a *server* for its peers — `serve_update`
crates\spt-daemon\src\propagate.rs:7://! responder's [`serve_update`] answers from its **staged, already-verified**
crates\spt-daemon\src\propagate.rs:106:pub fn serve_update(
crates\spt-daemon\src\propagate.rs:134:                            let trusted = roster.is_member_any(origin_node);
crates\spt-daemon\src\propagate.rs:173:                            let trusted = roster.is_member_any(origin_node);
docs\DEBUG-CONVERGE-PLAN.md:7:served by `propagate::serve_update` under the same trust gate as a pull),
MESH-D2-PLAN.md:14:**Decision:** at connect, the two nodes prove the **full set of subnets they share**; `ConnEntry` caches `proven_subnets: HashSet<String>` (current-epoch). D5's gate becomes `conn.proven_subnets.contains(item.subnet)`.
MESH-D2-PLAN.md:54:    proven_subnets: HashSet<String>,   // NEW — current-epoch membership proven at connect
MESH-D2-PLAN.md:58:- `register_conn(shared, conn)` → `register_conn(shared, conn, proven_subnets)` — stores the set; everything else (presence append, closed-watcher, app `accept_bi` loop) unchanged. The app `accept_bi` loop now starts on the conn **after** the ctrl stream is consumed, so stream #1 (ctrl) is never registered as an app stream.
MESH-D2-PLAN.md:59:- New accessor `conn_proven_subnets(conn_id) -> HashSet<String>` (the seam D5 reads; unused by any gate at D2 — belt + suspenders).
MESH-D2-PLAN.md:90:1. **both members → conn + proof** — two in-process `NetHost`s sharing a subnet (same seed+epoch) dial; assert the conn registers on both sides AND `conn_proven_subnets` contains the shared subnet on both.
MESH-D2-PLAN.md:91:2. **multi-subnet set** — two hosts sharing {`home`,`work`} but not `solo`; assert `proven_subnets == {home, work}` on both (the set, not a bool; `solo` absent).
MESH-D2-PLAN.md:99:- Seed-proof runs on every trusted-ALPN dial + accept before the conn is usable; `ConnEntry.proven_subnets` populated; failure/empty drops the conn.
MESH-D7-PLAN.md:3:> JIT plan for `SUBNET-MESH-PLAN.md` §Build phases · phase 7. Activates **REQ-MESH-4** (impl, unit, int). The mesh is already ON (D6); D7 adds the **membership-removal lifecycle**: a real revoke that propagates, a seed rotation that locks the revokee out, and the **re-seed grace** that auto-heals benign offliners across the rotation. Also closes the one deferred D2 seam: the **per-conn `proven_subnets` epoch cross-check** (REQ-MESH-1's "exact-epoch, N-1 the sole exception").
MESH-D7-PLAN.md:51:- **Verify outcome becomes three-valued** per subnet: `Proven{full}` (exact-epoch ∧ on-roster), `Proven{reseed_only}` (epoch == local−1 ∧ prover on-roster ∧ ¬tombstoned), `Denied` (everything else — N-2+, off-roster, tombstoned, forged). Today it is two-valued (exact-epoch pass/fail); D2 cached `proven_subnets: HashSet<String>` — D7 makes it carry the grade.
MESH-D7-PLAN.md:52:- **The deferred per-conn epoch cross-check** (psyche's "deferred per-conn `proven_subnets` epoch cross-check"): the proof already binds `seed_epoch` in the transcript; D7 consults it to grade exact vs prior vs stale, instead of the current binary exact-only verify. This is the same code site, generalized.
MESH-D6-PLAN.md:30:2. **`propagate` serve gate** (`propagate.rs:105–112,133,172`): signature `trust: &TrustStore` → `roster: &RosterStore`; the two `trust.peers.iter().any(|p| p.pubkey_hex == origin_node)` checks → `roster.is_member`-in-any-subnet (iterate the node's subnets, or add a `roster.is_member_any(origin)` helper). Update the caller `dispatch.rs:326–327` (`TrustStore::load()` → `RosterStore::load()`).
MESH-D6-PLAN.md:53:- **D7** — `spt subnet revoke` (propagating tombstone + timeboxed seed rotation), confidential seed push, re-seed grace, the per-conn `proven_subnets` epoch cross-check, cross-node un-tombstone.
MESH-D5-PLAN.md:13:**Why `is_member`, not the `ConnEntry.proven_subnets` set.** Plan decision 2 caches the proven set on the connection "for the D5 gate to read." But the five gates run in the **dispatcher**, which has `origin_node` (handshake-proven hex, KH 7.5) and **no conn/`NetHost` handle** — it cannot read `conn_proven_subnets` without new broker-IPC plumbing. It does not need to: at D5 the two checks are **equivalent**.
MESH-D5-PLAN.md:16:- The only case where `is_member` (directory) and `proven_subnets` (this-epoch crypto) diverge is **seed-epoch skew on a long-lived connection after a rotation** — and rotation is **D7**, which owns the re-proof / force-drop sweep. At D5 every member shares one epoch, so `is_member ⟺ proved-this-epoch`.
MESH-D5-PLAN.md:18:So: **gate on `RosterStore::is_member(subnet, origin)`, loaded fresh per gate call** — the exact shape the gates already use for `TrustStore::is_trusted` (each `*Policy::load()` loads the store fresh). `proven_subnets` stays the **connect-time** enforcement (D2); a per-conn cryptographic cross-check at the gates is deferred to D7 where epoch-skew makes it matter. Zero new plumbing, correct for D6's widen.
MESH-D5-PLAN.md:75:- **D7** — revoke CLI (propagating tombstone + timeboxed rotation), confidential seed push, re-seed grace, the per-conn `proven_subnets` epoch cross-check, cross-node un-tombstone.
MESH-D4-PLAN.md:65:- **D5** — swap the five `is_trusted` gates to `roster.is_member` / the `ConnEntry.proven_subnets` set; hard cutover (delete `peers.json` + the `TrustStore` auth path). **Pre-D5 open item #3** (confirm self-update is node-local per ADR-0016, not riding member conns) is checked *before* D5, not here.
traceable-reqs.toml:771:[Omitted long matching line]


Now read the CLI subnet command pieces + the pump loop. Let me read the relevant cli.rs sections and peerloop.rs.