Write

//! Two-host cross-node E2E — the M4-PLAN §D9b ladder on a REAL rig (D9-6).
//!
//! **Env-gated: runs only with `SPT_TWO_HOST=1` + a role** — CI never sets it,
//! so this is a silent no-op there; the real runs happen on the rig hosts
//! (HFENDULEAM Windows ↔ gravity Linux) per `docs/TWO-HOST-RUNBOOK.md`. Unlike
//! every loopback E2E in this suite, the two sides here are **separate
//! processes on separate machines** talking over a real LAN/WAN path.
//!
//! ## The ladder (one run climbs all of it)
//!
//! pair → register → cross-node message → remote-drive (attach + file
//! transfer) → Psyche sync (both directions) → update self-heal → notif
//! cross-node (insert on A fires on B; dismiss on B replicates to A).
//!
//! ## Roles + rendezvous (no runtime address exchange)
//!
//! Every identity and address is **derived from a shared secret** + static
//! env, so either side can be driven non-interactively (the gravity leg runs
//! under a commit-message-gated CI step; no SSH between the rig hosts):
//!
//! - node identities: `sha256(secret:"node-a"/"node-b")` → [`Identity`] seeds;
//! - the subnet TOTP seed: `sha256(secret:"subnet-totp")[..20]` — role A reads
//!   the 6-digit code off it exactly like a user reading the seed-holder's
//!   `spt pair show-totp` screen (the wire ceremony itself is the REAL
//!   SPAKE2/TOTP exchange — only the operator's eyeballs are simulated);
//! - the release signing key: `sha256(secret:"release")`;
//! - dialable addresses: derived node id + `SPT_TWO_HOST_PEER_IP` + the fixed
//!   ports ([`BindScope::Port`]) — id-only tickets with one static ip:port,
//!   relays and discovery disabled.
//!
//! Role B (seed-holder, server) seeds its stores, answers the pairing
//! ceremony, then runs the production dispatcher + peer pump and *reacts*
//! (dismisses the marker notif, asserts its rungs, waits for A's done-file).
//! Role A (joiner, driver) pairs, then drives every remaining rung and
//! asserts convergence. Cross-host barriers are the protocols themselves —
//! poll-until-deadline on observable store state, never wall-clock guesses.
//!
//! ## Env knobs
//!
//! | var | role | meaning |
//! |-----|------|---------|
//! | `SPT_TWO_HOST` | both | `1` activates (anything else: silent skip) |
//! | `SPT_TWO_HOST_ROLE` | both | `a` (joiner/driver) or `b` (seed-holder/server) |
//! | `SPT_TWO_HOST_SECRET` | both | shared secret (identical on both hosts) |
//! | `SPT_TWO_HOST_PEER_IP` | both | the OTHER host's LAN IP |
//! | `SPT_TWO_HOST_SUBNET` | both | subnet name (default `twohost`) |
//! | `SPT_TWO_HOST_PORT_A` | both | A's broker UDP port (default 7460) |
//! | `SPT_TWO_HOST_PORT_B` | both | B's broker UDP port (default 7461) |
//! | `SPT_TWO_HOST_PORT_PAIR` | both | B's pre-trust pairing port (default 7462) |
//! | `SPT_TWO_HOST_WAIT_SECS` | both | per-rung convergence deadline (default 300) |

use std::net::{IpAddr, SocketAddr};
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::thread;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};

use ed25519_dalek::{Signer, SigningKey};
use sha2::{Digest, Sha256};

use spt_daemon::brain::Brain;
use spt_daemon::broker::Broker;
use spt_daemon::dispatch::{run_dispatch_loop, DispatchPaths};
use spt_daemon::nethost::{NetConfig, NetHost};
use spt_daemon::peerloop::{run_peer_pump, PeerResolver, PumpCadence, PumpConfig, PumpHooks, PumpPaths};
use spt_daemon::registryhost::RegistryHost;
use spt_daemon::relcache::ReleaseCache;
use spt_daemon::release::{sha256_hex, ReleaseMetadata, SignedRelease};
use spt_daemon::update::BROKER_RESOURCE_ABI;
use spt_net::net::endpoint::{
    addr_for_node_hex, BindScope, EndpointAddr, LocalDiscovery, NetEndpoint, RelayPolicy,
};
use spt_net::net::pairing::ratelimit::PairingRateLimiter;
use spt_net::net::pairing::totp::TotpSeed;
use spt_net::net::pairing::wire::{run_initiator, run_responder, SPT_PAIR_ALPN};
use spt_net::net::wanmsg::WanMessage;
use spt_proto::identity::Identity;
use spt_store::contextstore::ContextStore;
use spt_store::epoch::EpochSource;
use spt_store::info::{self, InfoJson};
use spt_store::notif::{NotifRow, NotifStore};
use spt_store::perch::{self, ParentHint};
use spt_store::spool;
use spt_store::subnet::SubnetStore;
use spt_store::trust::TrustStore;
use spt_store::visibility::VisibilityStore;
use tempfile::TempDir;

const ID_A: &str = "ling-twohost-a";
const ID_B: &str = "ling-twohost-b";
const WAN_BODY: &str = "two-host hello";
const NOTIF_BODY: &str = "twohost ping";
const DRIVE_MARKER: &str = "twohost-drive";
const RELEASE_BYTES: &[u8] = b"twohost propagated binary v6";

/// The rig's static configuration, fully determined by env (no runtime
/// exchange between the hosts).
struct Rig {
    subnet: String,
    peer_ip: IpAddr,
    port_a: u16,
    port_b: u16,
    port_pair: u16,
    wait: Duration,
    id_a: Identity,
    id_b: Identity,
    totp: TotpSeed,
    release_sk: SigningKey,
}

impl Rig {
    /// Read the rig from env iff `SPT_TWO_HOST=1` and the role matches;
    /// `None` = silent skip (the CI path).
    fn from_env(role: &str) -> Option<Rig> {
        if std::env::var("SPT_TWO_HOST").as_deref() != Ok("1") {
            return None;
        }
        if std::env::var("SPT_TWO_HOST_ROLE").ok()?.to_ascii_lowercase() != role {
            return None;
        }
        let secret = std::env::var("SPT_TWO_HOST_SECRET")
            .expect("SPT_TWO_HOST_SECRET must be set (same value on both hosts)");
        let peer_ip: IpAddr = std::env::var("SPT_TWO_HOST_PEER_IP")
            .expect("SPT_TWO_HOST_PEER_IP must be set (the other host's IP)")
            .parse()
            .expect("SPT_TWO_HOST_PEER_IP parses as an IP address");
        let mut t20 = [0u8; 20];
        t20.copy_from_slice(&derive32(&secret, "subnet-totp")[..20]);
        Some(Rig {
            subnet: env_or_str("SPT_TWO_HOST_SUBNET", "twohost"),
            peer_ip,
            port_a: env_or("SPT_TWO_HOST_PORT_A", 7460),
            port_b: env_or("SPT_TWO_HOST_PORT_B", 7461),
            port_pair: env_or("SPT_TWO_HOST_PORT_PAIR", 7462),
            wait: Duration::from_secs(env_or("SPT_TWO_HOST_WAIT_SECS", 300)),
            id_a: Identity::from_seed(&derive32(&secret, "node-a")),
            id_b: Identity::from_seed(&derive32(&secret, "node-b")),
            totp: TotpSeed::from_bytes(t20),
            release_sk: SigningKey::from_bytes(&derive32(&secret, "release")),
        })
    }

    fn a_hex(&self) -> String {
        self.id_a.public_key().to_hex()
    }
    fn b_hex(&self) -> String {
        self.id_b.public_key().to_hex()
    }

    /// The peer's broker address as the brain-IPC JSON value: derived node id
    /// + the static ip:port (no relay, no discovery — one direct path).
    fn peer_broker_addr(&self, role: &str) -> serde_json::Value {
        let (hex, port) = match role {
            "a" => (self.b_hex(), self.port_b),
            _ => (self.a_hex(), self.port_a),
        };
        let addr = addr_for_node_hex(&hex)
            .expect("derived node hex is a valid key")
            .with_ip_addr(SocketAddr::new(self.peer_ip, port));
        serde_json::to_value(&addr).expect("EndpointAddr serializes")
    }

    /// B's pre-trust pairing endpoint address (role A dials this).
    fn pair_addr(&self) -> EndpointAddr {
        addr_for_node_hex(&self.b_hex())
            .expect("derived node hex is a valid key")
            .with_ip_addr(SocketAddr::new(self.peer_ip, self.port_pair))
    }
}

fn derive32(secret: &str, tag: &str) -> [u8; 32] {
    let mut h = Sha256::new();
    h.update(secret.as_bytes());
    h.update(b":");
    h.update(tag.as_bytes());
    h.finalize().into()
}

fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
    std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}

fn env_or_str(key: &str, default: &str) -> String {
    std::env::var(key).unwrap_or_else(|_| default.to_string())
}

fn now_secs() -> u64 {
    SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_secs()
}

fn now_ms() -> u64 {
    SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_millis() as u64
}

static OP: AtomicU64 = AtomicU64::new(1);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}

/// Bind this host's broker on its fixed production-scoped port.
fn rig_broker(name: &str, identity: Identity, port: u16, dir: &std::path::Path) -> Arc<Broker> {
    let host = NetHost::start(NetConfig {
        identity,
        relay: RelayPolicy::Disabled,
        discovery: LocalDiscovery::Off,
        scope: BindScope::Port(port),
    })
    .expect("net host start");
    let broker =
        Broker::bind_in_with_net(name, dir.join("effects.log"), Some(host)).expect("bind broker");
    let serve = Arc::clone(&broker);
    thread::spawn(move || {
        let _ = serve.serve();
    });
    broker
}

fn connect_retry(name: &str) -> Brain {
    for _ in 0..300 {
        if let Ok(b) = Brain::cold_start(name, 1) {
            return b;
        }
        thread::sleep(Duration::from_millis(10));
    }
    panic!("brain could not connect");
}

/// Poll `probe` until true; panic with `what` after the rig deadline. The
/// cross-host barrier primitive — every rung waits on observable store state.
fn rig_wait(what: &str, deadline: Duration, mut probe: impl FnMut() -> bool) {
    let end = Instant::now() + deadline;
    while Instant::now() < end {
        if probe() {
            println!("TWOHOST OK: {what}");
            return;
        }
        thread::sleep(Duration::from_millis(250));
    }
    panic!("never converged on the rig: {what}");
}

fn stamped(node: &str, body: &str) -> String {
    format!(
        "<!-- spt:source=llm spt:routed_at_ms={} spt:node={node} spt:vector={node}:1 -->\n{body}",
        now_ms()
    )
}

/// The canonical-store pump paths for this host (the production locations the
/// daemon itself would use — both rig sides run real homes, not test roots).
fn canonical_pump_paths(scratch: PathBuf) -> PumpPaths {
    let home = perch::spt_home();
    PumpPaths {
        owlery: perch::owlery_dir(),
        ctx_root: perch::tracked_dir(),
        notif_db: perch::notifs_file(),
        scratch,
        subnets: perch::subnet_file(),
        trust: perch::trust_file(),
        vis: perch::visibility_file(),
        ops: home.join("pump-ops.json"),
        releases: home.join("releases"),
        release_keys: home.join("release-keys.json"),
        pull_marker: home.join("identity").join("sync-pull-now"),
    }
}

/// Stand up this host's production loops: dispatcher (serve half) + peer pump
/// (drive half), both against the canonical SPT_HOME stores.
fn start_loops(
    broker_name: &str,
    own_hex: &str,
    peer_hex: String,
    peer_addr: serde_json::Value,
    scratch_root: &std::path::Path,
) -> (Arc<RegistryHost>, Arc<AtomicBool>) {
    let home = perch::spt_home();
    let registry = Arc::new(RegistryHost::new_at(
        own_hex,
        EpochSource::load_from(&perch::epoch_file()),
        home.join("registry"),
    ));
    let stop = Arc::new(AtomicBool::new(false));
    {
        let name = broker_name.to_string();
        let reg = Arc::clone(&registry);
        let flag = Arc::clone(&stop);
        let paths = DispatchPaths {
            owlery: perch::owlery_dir(),
            releases: home.join("releases"),
            scratch: scratch_root.join("dispatch"),
        };
        thread::spawn(move || {
            let _ = run_dispatch_loop(&name, &paths, reg, &flag, Duration::from_millis(50));
        });
    }
    {
        let name = broker_name.to_string();
        let reg = Arc::clone(&registry);
        let flag = Arc::clone(&stop);
        let paths = canonical_pump_paths(scratch_root.join("pump"));
        let resolver: PeerResolver =
            Box::new(move |hex| if hex == peer_hex { Some(peer_addr.clone()) } else { None });
        thread::spawn(move || {
            let config = PumpConfig {
                cadence: PumpCadence {
                    registry: Duration::from_millis(500),
                    notif: Duration::from_millis(500),
                    sync: Duration::from_millis(500),
                    update: Duration::from_millis(500),
                },
                // Gated default (REQ-UPD-4): self-heal stages + consent-notifies.
                full_auto_update: false,
            };
            let _ = run_peer_pump(&name, reg, &paths, resolver, config, PumpHooks::default(), &flag);
        });
    }
    (registry, stop)
}

/// Seed this host's visibility (both ladder endpoints sync in the rig subnet)
/// and its trusted release-signer keys file (the update leg's production gate).
fn seed_common_stores(rig: &Rig) {
    let mut vis = VisibilityStore::load();
    vis.set_sync_subnets(ID_A, vec![rig.subnet.clone()]);
    vis.set_sync_subnets(ID_B, vec![rig.subnet.clone()]);
    vis.save().expect("save visibility");
    let keys_json = serde_json::json!({
        "keys": { "twohost": rig.release_sk.verifying_key().as_bytes().iter()
            .map(|b| format!("{b:02x}")).collect::<String>() },
        "channel": "stable",
    });
    std::fs::write(
        perch::spt_home().join("release-keys.json"),
        serde_json::to_string(&keys_json).expect("keys json"),
    )
    .expect("write release-keys.json");
}

/// Create a perch record for `id` on this host (the registry-advertised,
/// WAN-message-addressable unit).
fn seed_perch(id: &str) -> PathBuf {
    let perch_path = perch::resolve_perch_path(id, ParentHint::Infer);
    std::fs::create_dir_all(&perch_path).expect("perch dir");
    let rec = InfoJson::new(id, "now", std::process::id(), "twohost", "ready_agent");
    info::write_info(&perch_path, &rec).expect("info.json");
    perch_path
}

/// Seed this host's own mind (live tier) for `id` — the content the peer's
/// sync pull must replicate byte-for-byte semantics of.
fn seed_mind(id: &str, node: &str, body: &str) {
    let cs = ContextStore::open_or_init().expect("context store");
    let p = cs.live_context_path(id).expect("live path");
    std::fs::write(&p, stamped(node, body)).expect("write mind");
    cs.commit_live(id, "twohost seed").expect("commit mind");
}

/// The deterministically-signed v6 release both sides agree on (B stages it,
/// A's pump pulls + verifies + stages it under the derived trusted key).
fn signed_release(rig: &Rig) -> SignedRelease {
    let meta = ReleaseMetadata {
        version: 6,
        channel: "stable".to_string(),
        expires_at_ms: u64::MAX,
        key_id: "twohost".to_string(),
        artifact_sha256: sha256_hex(RELEASE_BYTES),
        brain_ipc_version: spt_daemon::IPC_PROTOCOL_VERSION,
        broker_resource_abi: BROKER_RESOURCE_ABI,
    };
    let metadata_json = serde_json::to_string(&meta).expect("meta json");
    let sig = rig.release_sk.sign(metadata_json.as_bytes());
    SignedRelease {
        metadata_json,
        signature_hex: sig.to_bytes().iter().map(|b| format!("{b:02x}")).collect(),
    }
}

// ───────────────────────── role B: seed-holder / server ─────────────────────────

// Two-host int evidence (rig-run): the serve half of every rung, plus the
// notif dismiss origination. Tags sit on the rung asserts below.
#[test]
fn two_host_ladder_role_b() {
    let Some(rig) = Rig::from_env("b") else { return };
    let home = TempDir::new().expect("temp home");
    std::env::set_var("SPT_HOME", home.path());
    let scratch = TempDir::new().expect("scratch");
    println!("TWOHOST role B: node {}", rig.b_hex());

    // ── Stores the ceremony + the ladder serve from. B is the seed-holder:
    // the subnet record (derived TOTP seed, epoch 1) exists here FIRST.
    let mut subnets = SubnetStore::load();
    subnets.add_joined(&rig.subnet, *rig.totp.as_bytes(), 1).expect("seed subnet");
    subnets.save().expect("save subnets");
    seed_common_stores(&rig);
    let perch_b = seed_perch(ID_B);
    seed_mind(ID_B, "rig-b", "the mind A pulls across the rig");
    ReleaseCache::open(&perch::spt_home().join("releases"))
        .stage(&signed_release(&rig), RELEASE_BYTES)
        .expect("stage v6 at B");

    // ── Broker on the fixed port + the remote-drive target session. ──
    let broker_name = format!("spt-twohost-b-{}.sock", std::process::id());
    let _broker = rig_broker(&broker_name, rig.id_b.clone(), rig.port_b, scratch.path());
    let mut host_brain = connect_retry(&broker_name);
    #[cfg(unix)]
    let (program, args) = ("cat".to_string(), Vec::<String>::new());
    #[cfg(windows)]
    let (program, args) = ("findstr".to_string(), vec![".".to_string()]);
    let sid = host_brain
        .spawn_session(spt_daemon::msg::SpawnReq {
            program,
            args,
            rows: 24,
            cols: 80,
            endpoint: ID_B.to_string(),
        })
        .expect("spawn drive target");
    // Publish the session id where A's file-fetch rung reads it (the fetch IS
    // the first remote-drive leg).
    let xroot = spt_store::xfer::transfers_dir();
    std::fs::create_dir_all(&xroot).expect("transfer root");
    std::fs::write(xroot.join("sid.txt"), sid.to_string()).expect("write sid");
    println!("TWOHOST role B: drive session {sid} ready");

    // ── Rung 1 — pair (responder). The REAL TOTP-seeded SPAKE2 ceremony over
    // the pre-trust ALPN, cross-host. Trust lands in the canonical store the
    // dispatcher gates read.
    // [int->REQ-PAIR-1]
    let rt = tokio::runtime::Runtime::new().expect("tokio rt");
    rt.block_on(async {
        let ep = NetEndpoint::bind_scoped(
            &rig.id_b,
            RelayPolicy::Disabled,
            LocalDiscovery::Off,
            vec![SPT_PAIR_ALPN.to_vec()],
            BindScope::Port(rig.port_pair),
        )
        .await
        .expect("pairing endpoint bind");
        let end = Instant::now() + rig.wait;
        let mut rate = PairingRateLimiter::new();
        loop {
            assert!(Instant::now() < end, "no pairing ceremony arrived within the rig window");
            let Some(incoming) = ep.accept().await else {
                panic!("pairing endpoint closed before a ceremony arrived")
            };
            let Ok(connecting) = incoming.accept() else { continue };
            let Ok(conn) = connecting.await else { continue };
            let mut trust = TrustStore::load();
            match run_responder(&conn, rig.id_b.public_key(), &subnets, &mut rate, &mut trust, now_secs())
                .await
            {
                Ok(outcome) => {
                    trust.save().expect("save trust");
                    println!("TWOHOST OK: pairing (responder) — {outcome:?}");
                    break;
                }
                Err(e) => eprintln!("pairing attempt failed (joiner may retry): {e}"),
            }
        }
        ep.close().await;
    });
    let trust_now = TrustStore::load();
    assert!(
        trust_now.is_trusted(&rig.subnet, &rig.a_hex()),
        "the ceremony pinned A in B's canonical trust store"
    );

    // ── Production loops: dispatcher (serves A's pulls/feeds) + pump (pushes
    // B's registry/notif/mind, bootstrap-pulls A's). ──
    let (registry, stop) =
        start_loops(&broker_name, &rig.b_hex(), rig.a_hex(), rig.peer_broker_addr("b"), scratch.path());

    // ── Rung 2 — register: A's perch row replicates into B's registry through
    // the gated inbound feed (epoch-leased, trust-gated).
    // [int->REQ-INST-7]
    rig_wait("register: A's perch row replicated to B", rig.wait, || {
        registry.rows(&rig.subnet, ID_A).iter().any(|i| i.node == rig.a_hex())
    });

    // ── Rung 3 — cross-node message: A's WAN send lands in B's perch spool
    // (the atomic offline path) through the dispatcher funnel.
    // [int->REQ-NET-1]
    rig_wait("message: A's WAN message spooled at B", rig.wait, || {
        spool::peek_all_at(&perch_b)
            .map(|rows| rows.iter().any(|(_, _, body, _)| body == WAN_BODY))
            .unwrap_or(false)
    });

    // ── Rung 7a — notif: A's insert fires here, then B dismisses and the
    // dismissal replicates BACK over B's pump (the acceptance leg).
    // [int->REQ-NOTIF-1]
    let store_b = NotifStore::open().expect("B notif store");
    let mut ping_id = String::new();
    rig_wait("notif: A's insert fired on B", rig.wait, || {
        match store_b.list(&rig.subnet) {
            Ok(rows) => match rows.iter().find(|r| r.body == NOTIF_BODY) {
                Some(r) => {
                    ping_id = r.notif_id.clone();
                    true
                }
                None => false,
            },
            Err(_) => false,
        }
    });
    assert!(store_b.dismiss(&ping_id).expect("dismiss"), "B dismisses A's notif (latches once)");
    println!("TWOHOST OK: notif dismissed at B ({ping_id}) — replicating back");

    // ── Rung 5b — Psyche sync, B←A direction: B's pump bootstrap-pulls the
    // mind A seeded (want-ref derived from B's registry, gate served at A).
    // [int->REQ-INST-5]
    let cs_b = ContextStore::open_or_init().expect("B context store");
    rig_wait("sync: B pulled A's mind", rig.wait, || {
        cs_b.branch_store().tip(&format!("a-{ID_A}")).ok().flatten().is_some()
    });

    // ── Hold until A finishes its side (it pushes the done-file as its last
    // act — the file transfer rung doubling as the ladder's completion barrier).
    rig_wait("done-file pushed by A (ladder complete on A)", rig.wait, || {
        spt_store::xfer::transfers_dir().join("done.txt").exists()
    });

    stop.store(true, Ordering::Relaxed);
    println!("TWOHOST role B: ladder complete");
}

// ───────────────────────── role A: joiner / driver ─────────────────────────

// Two-host int evidence (rig-run): the drive half of every rung. Tags sit on
// the rung asserts below.
#[test]
fn two_host_ladder_role_a() {
    let Some(rig) = Rig::from_env("a") else { return };
    let home = TempDir::new().expect("temp home");
    std::env::set_var("SPT_HOME", home.path());
    let scratch = TempDir::new().expect("scratch");
    println!("TWOHOST role A: node {}", rig.a_hex());

    // ── Rung 1 — pair (initiator). A holds NO subnet record yet: the 6-digit
    // code is read off the shared TOTP seed (the operator channel), the ceremony
    // is the real cross-host SPAKE2, and success transfers the subnet seed
    // (A becomes a full member / future seed-holder).
    // [int->REQ-PAIR-1] [int->REQ-PAIR-5]
    let rt = tokio::runtime::Runtime::new().expect("tokio rt");
    rt.block_on(async {
        let ep = NetEndpoint::bind_scoped(
            &rig.id_a,
            RelayPolicy::Disabled,
            LocalDiscovery::Off,
            vec![SPT_PAIR_ALPN.to_vec()],
            BindScope::All,
        )
        .await
        .expect("pairing dialer bind");
        let end = Instant::now() + rig.wait;
        loop {
            assert!(Instant::now() < end, "pairing never completed within the rig window");
            let conn = match ep.connect_with_alpn(rig.pair_addr(), SPT_PAIR_ALPN).await {
                Ok(c) => c,
                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(3)).await;
                    continue;
                }
            };
            let code = rig.totp.code_at(now_secs());
            let mut trust = TrustStore::load();
            let mut subnets = SubnetStore::load();
            match run_initiator(
                &conn,
                rig.id_a.public_key(),
                &rig.subnet,
                &code,
                &mut trust,
                &mut subnets,
                now_secs(),
            )
            .await
            {
                Ok(outcome) => {
                    trust.save().expect("save trust");
                    subnets.save().expect("save subnets");
                    println!("TWOHOST OK: pairing (initiator) — {outcome:?}");
                    break;
                }
                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(5)).await;
                }
            }
        }
        ep.close().await;
    });
    let subnets_now = SubnetStore::load();
    assert_eq!(
        subnets_now.find(&rig.subnet).and_then(|r| r.seed_bytes()),
        Some(*rig.totp.as_bytes()),
        "the ceremony transferred the subnet seed to the joiner (REQ-PAIR-5)"
    );
    assert!(
        TrustStore::load().is_trusted(&rig.subnet, &rig.b_hex()),
        "the ceremony pinned B in A's canonical trust store"
    );

    // ── A's local seeds: perch (registry-advertised), mind (B pulls it),
    // the marker notif (fires on B), the trusted release-keys file. ──
    seed_common_stores(&rig);
    let _perch_a = seed_perch(ID_A);
    seed_mind(ID_A, "rig-a", "the mind B pulls across the rig");
    let store_a = NotifStore::open().expect("A notif store");
    store_a
        .merge_row(&NotifRow {
            notif_id: format!("{}:{}", rig.a_hex(), now_ms()),
            subnet: rig.subnet.clone(),
            kind: "agent".to_string(),
            from_id: format!("{ID_A}@{}", rig.a_hex()),
            body: NOTIF_BODY.to_string(),
            created_ms: now_ms(),
            dismissed: false,
            seen: Default::default(),
            last_surfaced_ms: 0,
        })
        .expect("seed marker notif");

    // ── Broker + production loops. ──
    let broker_name = format!("spt-twohost-a-{}.sock", std::process::id());
    let _broker = rig_broker(&broker_name, rig.id_a.clone(), rig.port_a, scratch.path());
    let (registry, stop) =
        start_loops(&broker_name, &rig.a_hex(), rig.b_hex(), rig.peer_broker_addr("a"), scratch.path());

    // ── Rung 2 — register: B's perch row replicates into A's registry.
    // [int->REQ-INST-7]
    rig_wait("register: B's perch row replicated to A", rig.wait, || {
        registry.rows(&rig.subnet, ID_B).iter().any(|i| i.node == rig.b_hex())
    });

    // ── Rung 3 — cross-node message: the WAN send (B-side spool asserts it).
    // [int->REQ-NET-1]
    let mut a = connect_retry(&broker_name);
    let conn = a.net_dial(rig.peer_broker_addr("a"), Some(op())).expect("dial B");
    let opened = a.net_open_stream(conn.conn_id, Some(op())).expect("open msg stream");
    let msg = WanMessage {
        target: ID_B.to_string(),
        from: ID_A.to_string(),
        body: WAN_BODY.to_string(),
        op_id: "twohost-wan-1".to_string(),
    };
    a.net_stream_send(opened.stream_id, &spt_net::net::ndjson::encode_line(&msg), None, true)
        .expect("send WAN message");
    println!("TWOHOST OK: message sent (B asserts the spool)");

    // ── Rung 4 — remote-drive: fetch the session id off B (file transfer),
    // then attach to B's live PTY, type, and read the echo back.
    // [int->REQ-REACH-1]
    let sid_dest = scratch.path().join("sid.txt");
    let end = Instant::now() + rig.wait;
    let sid: u64 = loop {
        assert!(Instant::now() < end, "sid.txt never became fetchable from B");
        match spt_daemon::xfer::fetch_file(
            &mut a,
            conn.conn_id,
            ID_B,
            "sid.txt",
            &sid_dest,
            &format!("twohost-sid-{}", op()),
            op(),
            &scratch.path().join("xfer-progress"),
        ) {
            Ok(_) => {
                let text = std::fs::read_to_string(&sid_dest).expect("read sid");
                break text.trim().parse().expect("sid parses");
            }
            Err(e) => {
                eprintln!("sid fetch not ready ({e}); retrying");
                thread::sleep(Duration::from_secs(3));
            }
        }
    };
    println!("TWOHOST OK: file fetch (sid {sid})");

    // [int->REQ-INST-8]
    let viewport =
        spt_daemon::attach::request_attach(&mut a, conn.conn_id, sid, 0, op()).expect("attach");
    a.net_stream_subscribe(viewport, 0).expect("subscribe viewport");
    spt_daemon::attach::send_attach_input(&mut a, viewport, format!("{DRIVE_MARKER}\r\n").as_bytes(), op())
        .expect("type into B's session");
    let mut echoed = Vec::new();
    let mut decoder = spt_net::net::attach::AttachDecoder::new();
    let end = Instant::now() + rig.wait;
    while Instant::now() < end {
        match a.read_event().expect("event") {
            spt_daemon::brain::BrokerEvent::NetStreamData { stream_id, bytes, .. }
                if stream_id == viewport =>
            {
                for rec in decoder.push(&bytes) {
                    if let spt_net::net::attach::AttachRecord::Output { data_b64, .. } = rec {
                        echoed.extend(spt_daemon::msg::decode_bytes(&data_b64).expect("b64"));
                    }
                }
                if String::from_utf8_lossy(&echoed).contains(DRIVE_MARKER) {
                    break;
                }
            }
            _ => {}
        }
    }
    assert!(
        String::from_utf8_lossy(&echoed).contains(DRIVE_MARKER),
        "remote-drive echo came back across the rig; got {:?}",
        String::from_utf8_lossy(&echoed)
    );
    println!("TWOHOST OK: remote-drive (attach echo)");

    // ── Rung 5a — Psyche sync, A←B direction: A's pump bootstrap-pulls B's
    // mind (want-ref from A's registry, the gate served at B's dispatcher).
    // [int->REQ-INST-5]
    let cs_a = ContextStore::open_or_init().expect("A context store");
    rig_wait("sync: A pulled B's mind", rig.wait, || {
        cs_a.branch_store().tip(&format!("a-{ID_B}")).ok().flatten().is_some()
    });

    // ── Rung 6 — update self-heal: A's pump pulled B's staged v6, verified it
    // under the derived trusted key, staged it, and surfaced a CONSENT notif
    // (REQ-UPD-4 gated default — never a silent apply).
    // [int->REQ-UPD-1]
    let cache_a = ReleaseCache::open(&perch::spt_home().join("releases"));
    rig_wait("update: B's v6 staged at A through the verify gate", rig.wait, || {
        cache_a.load_artifact().map(|b| b == RELEASE_BYTES).unwrap_or(false)
    });
    rig_wait("update: consent notif surfaced at A", rig.wait, || {
        store_a
            .list(&rig.subnet)
            .map(|rows| rows.iter().any(|r| r.kind == "consent"))
            .unwrap_or(false)
    });

    // ── Rung 7b — notif acceptance leg: B's dismissal replicated back to A.
    // [int->REQ-NOTIF-1]
    rig_wait("notif: B's dismissal replicated to A", rig.wait, || {
        store_a
            .list(&rig.subnet)
            .map(|rows| rows.iter().any(|r| r.body == NOTIF_BODY && r.dismissed))
            .unwrap_or(false)
    });

    // ── Done: push the completion barrier file to B (the push half of the
    // file-transfer rung).
    // [int->REQ-REACH-1]
    let done_src = scratch.path().join("done.txt");
    std::fs::write(&done_src, b"ladder complete on A").expect("write done");
    spt_daemon::xfer::push_file(
        &mut a,
        conn.conn_id,
        ID_B,
        &done_src,
        "done.txt",
        "twohost-done-1",
        op(),
        &scratch.path().join("xfer-progress"),
    )
    .expect("push done-file to B");
    println!("TWOHOST OK: file push (done barrier)");

    stop.store(true, Ordering::Relaxed);
    println!("TWOHOST role A: ladder complete");
}


File created successfully at: C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon\tests\twohost.rs (file state is current in your context — no need to Read it back)

//! Two-host cross-node E2E — the M4-PLAN §D9b ladder on a REAL rig (D9-6).
//!
//! **Env-gated: runs only with `SPT_TWO_HOST=1` + a role** — CI never sets it,
//! so this is a silent no-op there; the real runs happen on the rig hosts
//! (HFENDULEAM Windows ↔ gravity Linux) per `docs/TWO-HOST-RUNBOOK.md`. Unlike
//! every loopback E2E in this suite, the two sides here are **separate
//! processes on separate machines** talking over a real LAN/WAN path.
//!
//! ## The ladder (one run climbs all of it)
//!
//! pair → register → cross-node message → remote-drive (attach + file
//! transfer) → Psyche sync (both directions) → update self-heal → notif
//! cross-node (insert on A fires on B; dismiss on B replicates to A).
//!
//! ## Roles + rendezvous (no runtime address exchange)
//!
//! Every identity and address is **derived from a shared secret** + static
//! env, so either side can be driven non-interactively (the gravity leg runs
//! under a commit-message-gated CI step; no SSH between the rig hosts):
//!
//! - node identities: `sha256(secret:"node-a"/"node-b")` → [`Identity`] seeds;
//! - the subnet TOTP seed: `sha256(secret:"subnet-totp")[..20]` — role A reads
//!   the 6-digit code off it exactly like a user reading the seed-holder's
//!   `spt pair show-totp` screen (the wire ceremony itself is the REAL
//!   SPAKE2/TOTP exchange — only the operator's eyeballs are simulated);
//! - the release signing key: `sha256(secret:"release")`;
//! - dialable addresses: derived node id + `SPT_TWO_HOST_PEER_IP` + the fixed
//!   ports ([`BindScope::Port`]) — id-only tickets with one static ip:port,
//!   relays and discovery disabled.
//!
//! Role B (seed-holder, server) seeds its stores, answers the pairing
//! ceremony, then runs the production dispatcher + peer pump and *reacts*
//! (dismisses the marker notif, asserts its rungs, waits for A's done-file).
//! Role A (joiner, driver) pairs, then drives every remaining rung and
//! asserts convergence. Cross-host barriers are the protocols themselves —
//! poll-until-deadline on observable store state, never wall-clock guesses.
//!
//! ## Env knobs
//!
//! | var | role | meaning |
//! |-----|------|---------|
//! | `SPT_TWO_HOST` | both | `1` activates (anything else: silent skip) |
//! | `SPT_TWO_HOST_ROLE` | both | `a` (joiner/driver) or `b` (seed-holder/server) |
//! | `SPT_TWO_HOST_SECRET` | both | shared secret (identical on both hosts) |
//! | `SPT_TWO_HOST_PEER_IP` | both | the OTHER host's LAN IP |
//! | `SPT_TWO_HOST_SUBNET` | both | subnet name (default `twohost`) |
//! | `SPT_TWO_HOST_PORT_A` | both | A's broker UDP port (default 7460) |
//! | `SPT_TWO_HOST_PORT_B` | both | B's broker UDP port (default 7461) |
//! | `SPT_TWO_HOST_PORT_PAIR` | both | B's pre-trust pairing port (default 7462) |
//! | `SPT_TWO_HOST_WAIT_SECS` | both | per-rung convergence deadline (default 300) |

use std::net::{IpAddr, SocketAddr};
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::thread;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};

use ed25519_dalek::{Signer, SigningKey};
use sha2::{Digest, Sha256};

use spt_daemon::brain::Brain;
use spt_daemon::broker::Broker;
use spt_daemon::dispatch::{run_dispatch_loop, DispatchPaths};
use spt_daemon::nethost::{NetConfig, NetHost};
use spt_daemon::peerloop::{run_peer_pump, PeerResolver, PumpCadence, PumpConfig, PumpHooks, PumpPaths};
use spt_daemon::registryhost::RegistryHost;
use spt_daemon::relcache::ReleaseCache;
use spt_daemon::release::{sha256_hex, ReleaseMetadata, SignedRelease};
use spt_daemon::update::BROKER_RESOURCE_ABI;
use spt_net::net::endpoint::{
    addr_for_node_hex, BindScope, EndpointAddr, LocalDiscovery, NetEndpoint, RelayPolicy,
};
use spt_net::net::pairing::ratelimit::PairingRateLimiter;
use spt_net::net::pairing::totp::TotpSeed;
use spt_net::net::pairing::wire::{run_initiator, run_responder, SPT_PAIR_ALPN};
use spt_net::net::wanmsg::WanMessage;
use spt_proto::identity::Identity;
use spt_store::contextstore::ContextStore;
use spt_store::epoch::EpochSource;
use spt_store::info::{self, InfoJson};
use spt_store::notif::{NotifRow, NotifStore};
use spt_store::perch::{self, ParentHint};
use spt_store::spool;
use spt_store::subnet::SubnetStore;
use spt_store::trust::TrustStore;
use spt_store::visibility::VisibilityStore;
use tempfile::TempDir;

const ID_A: &str = "ling-twohost-a";
const ID_B: &str = "ling-twohost-b";
const WAN_BODY: &str = "two-host hello";
const NOTIF_BODY: &str = "twohost ping";
const DRIVE_MARKER: &str = "twohost-drive";
const RELEASE_BYTES: &[u8] = b"twohost propagated binary v6";

/// The rig's static configuration, fully determined by env (no runtime
/// exchange between the hosts).
struct Rig {
    subnet: String,
    peer_ip: IpAddr,
    port_a: u16,
    port_b: u16,
    port_pair: u16,
    wait: Duration,
    id_a: Identity,
    id_b: Identity,
    totp: TotpSeed,
    release_sk: SigningKey,
}

impl Rig {
    /// Read the rig from env iff `SPT_TWO_HOST=1` and the role matches;
    /// `None` = silent skip (the CI path).
    fn from_env(role: &str) -> Option<Rig> {
        if std::env::var("SPT_TWO_HOST").as_deref() != Ok("1") {
            return None;
        }
        if std::env::var("SPT_TWO_HOST_ROLE").ok()?.to_ascii_lowercase() != role {
            return None;
        }
        let secret = std::env::var("SPT_TWO_HOST_SECRET")
            .expect("SPT_TWO_HOST_SECRET must be set (same value on both hosts)");
        let peer_ip: IpAddr = std::env::var("SPT_TWO_HOST_PEER_IP")
            .expect("SPT_TWO_HOST_PEER_IP must be set (the other host's IP)")
            .parse()
            .expect("SPT_TWO_HOST_PEER_IP parses as an IP address");
        let mut t20 = [0u8; 20];
        t20.copy_from_slice(&derive32(&secret, "subnet-totp")[..20]);
        Some(Rig {
            subnet: env_or_str("SPT_TWO_HOST_SUBNET", "twohost"),
            peer_ip,
            port_a: env_or("SPT_TWO_HOST_PORT_A", 7460),
            port_b: env_or("SPT_TWO_HOST_PORT_B", 7461),
            port_pair: env_or("SPT_TWO_HOST_PORT_PAIR", 7462),
            wait: Duration::from_secs(env_or("SPT_TWO_HOST_WAIT_SECS", 300)),
            id_a: Identity::from_seed(&derive32(&secret, "node-a")),
            id_b: Identity::from_seed(&derive32(&secret, "node-b")),
            totp: TotpSeed::from_bytes(t20),
            release_sk: SigningKey::from_bytes(&derive32(&secret, "release")),
        })
    }

    fn a_hex(&self) -> String {
        self.id_a.public_key().to_hex()
    }
    fn b_hex(&self) -> String {
        self.id_b.public_key().to_hex()
    }

    /// The peer's broker address as the brain-IPC JSON value: derived node id
    /// + the static ip:port (no relay, no discovery — one direct path).
    fn peer_broker_addr(&self, role: &str) -> serde_json::Value {
        let (hex, port) = match role {
            "a" => (self.b_hex(), self.port_b),
            _ => (self.a_hex(), self.port_a),
        };
        let addr = addr_for_node_hex(&hex)
            .expect("derived node hex is a valid key")
            .with_ip_addr(SocketAddr::new(self.peer_ip, port));
        serde_json::to_value(&addr).expect("EndpointAddr serializes")
    }

    /// B's pre-trust pairing endpoint address (role A dials this).
    fn pair_addr(&self) -> EndpointAddr {
        addr_for_node_hex(&self.b_hex())
            .expect("derived node hex is a valid key")
            .with_ip_addr(SocketAddr::new(self.peer_ip, self.port_pair))
    }
}

fn derive32(secret: &str, tag: &str) -> [u8; 32] {
    let mut h = Sha256::new();
    h.update(secret.as_bytes());
    h.update(b":");
    h.update(tag.as_bytes());
    h.finalize().into()
}

fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
    std::env::var(key).ok().and_then(|v| v.parse().ok()).unwrap_or(default)
}

fn env_or_str(key: &str, default: &str) -> String {
    std::env::var(key).unwrap_or_else(|_| default.to_string())
}

fn now_secs() -> u64 {
    SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_secs()
}

fn now_ms() -> u64 {
    SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_millis() as u64
}

static OP: AtomicU64 = AtomicU64::new(1);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}

/// Bind this host's broker on its fixed production-scoped port.
fn rig_broker(name: &str, identity: Identity, port: u16, dir: &std::path::Path) -> Arc<Broker> {
    let host = NetHost::start(NetConfig {
        identity,
        relay: RelayPolicy::Disabled,
        discovery: LocalDiscovery::Off,
        scope: BindScope::Port(port),
    })
    .expect("net host start");
    let broker =
        Broker::bind_in_with_net(name, dir.join("effects.log"), Some(host)).expect("bind broker");
    let serve = Arc::clone(&broker);
    thread::spawn(move || {
        let _ = serve.serve();
    });
    broker
}

fn connect_retry(name: &str) -> Brain {
    for _ in 0..300 {
        if let Ok(b) = Brain::cold_start(name, 1) {
            return b;
        }
        thread::sleep(Duration::from_millis(10));
    }
    panic!("brain could not connect");
}

/// Poll `probe` until true; panic with `what` after the rig deadline. The
/// cross-host barrier primitive — every rung waits on observable store state.
fn rig_wait(what: &str, deadline: Duration, mut probe: impl FnMut() -> bool) {
    let end = Instant::now() + deadline;
    while Instant::now() < end {
        if probe() {
            println!("TWOHOST OK: {what}");
            return;
        }
        thread::sleep(Duration::from_millis(250));
    }
    panic!("never converged on the rig: {what}");
}

fn stamped(node: &str, body: &str) -> String {
    format!(
        "<!-- spt:source=llm spt:routed_at_ms={} spt:node={node} spt:vector={node}:1 -->\n{body}",
        now_ms()
    )
}

/// The canonical-store pump paths for this host (the production locations the
/// daemon itself would use — both rig sides run real homes, not test roots).
fn canonical_pump_paths(scratch: PathBuf) -> PumpPaths {
    let home = perch::spt_home();
    PumpPaths {
        owlery: perch::owlery_dir(),
        ctx_root: perch::tracked_dir(),
        notif_db: perch::notifs_file(),
        scratch,
        subnets: perch::subnet_file(),
        trust: perch::trust_file(),
        vis: perch::visibility_file(),
        ops: home.join("pump-ops.json"),
        releases: home.join("releases"),
        release_keys: home.join("release-keys.json"),
        pull_marker: home.join("identity").join("sync-pull-now"),
    }
}

/// Stand up this host's production loops: dispatcher (serve half) + peer pump
/// (drive half), both against the canonical SPT_HOME stores.
fn start_loops(
    broker_name: &str,
    own_hex: &str,
    peer_hex: String,
    peer_addr: serde_json::Value,
    scratch_root: &std::path::Path,
) -> (Arc<RegistryHost>, Arc<AtomicBool>) {
    let home = perch::spt_home();
    let registry = Arc::new(RegistryHost::new_at(
        own_hex,
        EpochSource::load_from(&perch::epoch_file()),
        home.join("registry"),
    ));
    let stop = Arc::new(AtomicBool::new(false));
    {
        let name = broker_name.to_string();
        let reg = Arc::clone(&registry);
        let flag = Arc::clone(&stop);
        let paths = DispatchPaths {
            owlery: perch::owlery_dir(),
            releases: home.join("releases"),
            scratch: scratch_root.join("dispatch"),
        };
        thread::spawn(move || {
            let _ = run_dispatch_loop(&name, &paths, reg, &flag, Duration::from_millis(50));
        });
    }
    {
        let name = broker_name.to_string();
        let reg = Arc::clone(&registry);
        let flag = Arc::clone(&stop);
        let paths = canonical_pump_paths(scratch_root.join("pump"));
        let resolver: PeerResolver =
            Box::new(move |hex| if hex == peer_hex { Some(peer_addr.clone()) } else { None });
        thread::spawn(move || {
            let config = PumpConfig {
                cadence: PumpCadence {
                    registry: Duration::from_millis(500),
                    notif: Duration::from_millis(500),
                    sync: Duration::from_millis(500),
                    update: Duration::from_millis(500),
                },
                // Gated default (REQ-UPD-4): self-heal stages + consent-notifies.
                full_auto_update: false,
            };
            let _ = run_peer_pump(&name, reg, &paths, resolver, config, PumpHooks::default(), &flag);
        });
    }
    (registry, stop)
}

/// Seed this host's visibility (both ladder endpoints sync in the rig subnet)
/// and its trusted release-signer keys file (the update leg's production gate).
fn seed_common_stores(rig: &Rig) {
    let mut vis = VisibilityStore::load();
    vis.set_sync_subnets(ID_A, vec![rig.subnet.clone()]);
    vis.set_sync_subnets(ID_B, vec![rig.subnet.clone()]);
    vis.save().expect("save visibility");
    let keys_json = serde_json::json!({
        "keys": { "twohost": rig.release_sk.verifying_key().as_bytes().iter()
            .map(|b| format!("{b:02x}")).collect::<String>() },
        "channel": "stable",
    });
    std::fs::write(
        perch::spt_home().join("release-keys.json"),
        serde_json::to_string(&keys_json).expect("keys json"),
    )
    .expect("write release-keys.json");
}

/// Create a perch record for `id` on this host (the registry-advertised,
/// WAN-message-addressable unit).
fn seed_perch(id: &str) -> PathBuf {
    let perch_path = perch::resolve_perch_path(id, ParentHint::Infer);
    std::fs::create_dir_all(&perch_path).expect("perch dir");
    let rec = InfoJson::new(id, "now", std::process::id(), "twohost", "ready_agent");
    info::write_info(&perch_path, &rec).expect("info.json");
    perch_path
}

/// Seed this host's own mind (live tier) for `id` — the content the peer's
/// sync pull must replicate byte-for-byte semantics of.
fn seed_mind(id: &str, node: &str, body: &str) {
    let cs = ContextStore::open_or_init().expect("context store");
    let p = cs.live_context_path(id).expect("live path");
    std::fs::write(&p, stamped(node, body)).expect("write mind");
    cs.commit_live(id, "twohost seed").expect("commit mind");
}

/// The deterministically-signed v6 release both sides agree on (B stages it,
/// A's pump pulls + verifies + stages it under the derived trusted key).
fn signed_release(rig: &Rig) -> SignedRelease {
    let meta = ReleaseMetadata {
        version: 6,
        channel: "stable".to_string(),
        expires_at_ms: u64::MAX,
        key_id: "twohost".to_string(),
        artifact_sha256: sha256_hex(RELEASE_BYTES),
        brain_ipc_version: spt_daemon::IPC_PROTOCOL_VERSION,
        broker_resource_abi: BROKER_RESOURCE_ABI,
    };
    let metadata_json = serde_json::to_string(&meta).expect("meta json");
    let sig = rig.release_sk.sign(metadata_json.as_bytes());
    SignedRelease {
        metadata_json,
        signature_hex: sig.to_bytes().iter().map(|b| format!("{b:02x}")).collect(),
    }
}

// ───────────────────────── role B: seed-holder / server ─────────────────────────

// Two-host int evidence (rig-run): the serve half of every rung, plus the
// notif dismiss origination. Tags sit on the rung asserts below.
#[test]
fn two_host_ladder_role_b() {
    let Some(rig) = Rig::from_env("b") else { return };
    let home = TempDir::new().expect("temp home");
    std::env::set_var("SPT_HOME", home.path());
    let scratch = TempDir::new().expect("scratch");
    println!("TWOHOST role B: node {}", rig.b_hex());

    // ── Stores the ceremony + the ladder serve from. B is the seed-holder:
    // the subnet record (derived TOTP seed, epoch 1) exists here FIRST.
    let mut subnets = SubnetStore::load();
    subnets.add_joined(&rig.subnet, *rig.totp.as_bytes(), 1).expect("seed subnet");
    subnets.save().expect("save subnets");
    seed_common_stores(&rig);
    let perch_b = seed_perch(ID_B);
    seed_mind(ID_B, "rig-b", "the mind A pulls across the rig");
    ReleaseCache::open(&perch::spt_home().join("releases"))
        .stage(&signed_release(&rig), RELEASE_BYTES)
        .expect("stage v6 at B");

    // ── Broker on the fixed port + the remote-drive target session. ──
    let broker_name = format!("spt-twohost-b-{}.sock", std::process::id());
    let _broker = rig_broker(&broker_name, rig.id_b.clone(), rig.port_b, scratch.path());
    let mut host_brain = connect_retry(&broker_name);
    #[cfg(unix)]
    let (program, args) = ("cat".to_string(), Vec::<String>::new());
    #[cfg(windows)]
    let (program, args) = ("findstr".to_string(), vec![".".to_string()]);
    let sid = host_brain
        .spawn_session(spt_daemon::msg::SpawnReq {
            program,
            args,
            rows: 24,
            cols: 80,
            endpoint: ID_B.to_string(),
        })
        .expect("spawn drive target");
    // Publish the session id where A's file-fetch rung reads it (the fetch IS
    // the first remote-drive leg).
    let xroot = spt_store::xfer::transfers_dir();
    std::fs::create_dir_all(&xroot).expect("transfer root");
    std::fs::write(xroot.join("sid.txt"), sid.to_string()).expect("write sid");
    println!("TWOHOST role B: drive session {sid} ready");

    // ── Rung 1 — pair (responder). The REAL TOTP-seeded SPAKE2 ceremony over
    // the pre-trust ALPN, cross-host. Trust lands in the canonical store the
    // dispatcher gates read.
    // [int->REQ-PAIR-1]
    let rt = tokio::runtime::Runtime::new().expect("tokio rt");
    rt.block_on(async {
        let ep = NetEndpoint::bind_scoped(
            &rig.id_b,
            RelayPolicy::Disabled,
            LocalDiscovery::Off,
            vec![SPT_PAIR_ALPN.to_vec()],
            BindScope::Port(rig.port_pair),
        )
        .await
        .expect("pairing endpoint bind");
        let end = Instant::now() + rig.wait;
        let mut rate = PairingRateLimiter::new();
        loop {
            assert!(Instant::now() < end, "no pairing ceremony arrived within the rig window");
            let Some(incoming) = ep.accept().await else {
                panic!("pairing endpoint closed before a ceremony arrived")
            };
            let Ok(connecting) = incoming.accept() else { continue };
            let Ok(conn) = connecting.await else { continue };
            let mut trust = TrustStore::load();
            match run_responder(&conn, rig.id_b.public_key(), &subnets, &mut rate, &mut trust, now_secs())
                .await
            {
                Ok(outcome) => {
                    trust.save().expect("save trust");
                    println!("TWOHOST OK: pairing (responder) — {outcome:?}");
                    break;
                }
                Err(e) => eprintln!("pairing attempt failed (joiner may retry): {e}"),
            }
        }
        ep.close().await;
    });
    let trust_now = TrustStore::load();
    assert!(
        trust_now.is_trusted(&rig.subnet, &rig.a_hex()),
        "the ceremony pinned A in B's canonical trust store"
    );

    // ── Production loops: dispatcher (serves A's pulls/feeds) + pump (pushes
    // B's registry/notif/mind, bootstrap-pulls A's). ──
    let (registry, stop) =
        start_loops(&broker_name, &rig.b_hex(), rig.a_hex(), rig.peer_broker_addr("b"), scratch.path());

    // ── Rung 2 — register: A's perch row replicates into B's registry through
    // the gated inbound feed (epoch-leased, trust-gated).
    // [int->REQ-INST-7]
    rig_wait("register: A's perch row replicated to B", rig.wait, || {
        registry.rows(&rig.subnet, ID_A).iter().any(|i| i.node == rig.a_hex())
    });

    // ── Rung 3 — cross-node message: A's WAN send lands in B's perch spool
    // (the atomic offline path) through the dispatcher funnel.
    // [int->REQ-NET-1]
    rig_wait("message: A's WAN message spooled at B", rig.wait, || {
        spool::peek_all_at(&perch_b)
            .map(|rows| rows.iter().any(|(_, _, body, _)| body == WAN_BODY))
            .unwrap_or(false)
    });

    // ── Rung 7a — notif: A's insert fires here, then B dismisses and the
    // dismissal replicates BACK over B's pump (the acceptance leg).
    // [int->REQ-NOTIF-1]
    let store_b = NotifStore::open().expect("B notif store");
    let mut ping_id = String::new();
    rig_wait("notif: A's insert fired on B", rig.wait, || {
        match store_b.list(&rig.subnet) {
            Ok(rows) => match rows.iter().find(|r| r.body == NOTIF_BODY) {
                Some(r) => {
                    ping_id = r.notif_id.clone();
                    true
                }
                None => false,
            },
            Err(_) => false,
        }
    });
    assert!(store_b.dismiss(&ping_id).expect("dismiss"), "B dismisses A's notif (latches once)");
    println!("TWOHOST OK: notif dismissed at B ({ping_id}) — replicating back");

    // ── Rung 5b — Psyche sync, B←A direction: B's pump bootstrap-pulls the
    // mind A seeded (want-ref derived from B's registry, gate served at A).
    // [int->REQ-INST-5]
    let cs_b = ContextStore::open_or_init().expect("B context store");
    rig_wait("sync: B pulled A's mind", rig.wait, || {
        cs_b.branch_store().tip(&format!("a-{ID_A}")).ok().flatten().is_some()
    });

    // ── Hold until A finishes its side (it pushes the done-file as its last
    // act — the file transfer rung doubling as the ladder's completion barrier).
    rig_wait("done-file pushed by A (ladder complete on A)", rig.wait, || {
        spt_store::xfer::transfers_dir().join("done.txt").exists()
    });

    stop.store(true, Ordering::Relaxed);
    println!("TWOHOST role B: ladder complete");
}

// ───────────────────────── role A: joiner / driver ─────────────────────────

// Two-host int evidence (rig-run): the drive half of every rung. Tags sit on
// the rung asserts below.
#[test]
fn two_host_ladder_role_a() {
    let Some(rig) = Rig::from_env("a") else { return };
    let home = TempDir::new().expect("temp home");
    std::env::set_var("SPT_HOME", home.path());
    let scratch = TempDir::new().expect("scratch");
    println!("TWOHOST role A: node {}", rig.a_hex());

    // ── Rung 1 — pair (initiator). A holds NO subnet record yet: the 6-digit
    // code is read off the shared TOTP seed (the operator channel), the ceremony
    // is the real cross-host SPAKE2, and success transfers the subnet seed
    // (A becomes a full member / future seed-holder).
    // [int->REQ-PAIR-1] [int->REQ-PAIR-5]
    let rt = tokio::runtime::Runtime::new().expect("tokio rt");
    rt.block_on(async {
        let ep = NetEndpoint::bind_scoped(
            &rig.id_a,
            RelayPolicy::Disabled,
            LocalDiscovery::Off,
            vec![SPT_PAIR_ALPN.to_vec()],
            BindScope::All,
        )
        .await
        .expect("pairing dialer bind");
        let end = Instant::now() + rig.wait;
        loop {
            assert!(Instant::now() < end, "pairing never completed within the rig window");
            let conn = match ep.connect_with_alpn(rig.pair_addr(), SPT_PAIR_ALPN).await {
                Ok(c) => c,
                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(3)).await;
                    continue;
                }
            };
            let code = rig.totp.code_at(now_secs());
            let mut trust = TrustStore::load();
            let mut subnets = SubnetStore::load();
            match run_initiator(
                &conn,
                rig.id_a.public_key(),
                &rig.subnet,
                &code,
                &mut trust,
                &mut subnets,
                now_secs(),
            )
            .await
            {
                Ok(outcome) => {
                    trust.save().expect("save trust");
                    subnets.save().expect("save subnets");
                    println!("TWOHOST OK: pairing (initiator) — {outcome:?}");
                    break;
                }
                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(5)).await;
                }
            }
        }
        ep.close().await;
    });
    let subnets_now = SubnetStore::load();
    assert_eq!(
        subnets_now.find(&rig.subnet).and_then(|r| r.seed_bytes()),
        Some(*rig.totp.as_bytes()),
        "the ceremony transferred the subnet seed to the joiner (REQ-PAIR-5)"
    );
    assert!(
        TrustStore::load().is_trusted(&rig.subnet, &rig.b_hex()),
        "the ceremony pinned B in A's canonical trust store"
    );

    // ── A's local seeds: perch (registry-advertised), mind (B pulls it),
    // the marker notif (fires on B), the trusted release-keys file. ──
    seed_common_stores(&rig);
    let _perch_a = seed_perch(ID_A);
    seed_mind(ID_A, "rig-a", "the mind B pulls across the rig");
    let store_a = NotifStore::open().expect("A notif store");
    store_a
        .merge_row(&NotifRow {
            notif_id: format!("{}:{}", rig.a_hex(), now_ms()),
            subnet: rig.subnet.clone(),
            kind: "agent".to_string(),
            from_id: format!("{ID_A}@{}", rig.a_hex()),
            body: NOTIF_BODY.to_string(),
            created_ms: now_ms(),
            dismissed: false,
            seen: Default::default(),
            last_surfaced_ms: 0,
        })
        .expect("seed marker notif");

    // ── Broker + production loops. ──
    let broker_name = format!("spt-twohost-a-{}.sock", std::process::id());
    let _broker = rig_broker(&broker_name, rig.id_a.clone(), rig.port_a, scratch.path());
    let (registry, stop) =
        start_loops(&broker_name, &rig.a_hex(), rig.b_hex(), rig.peer_broker_addr("a"), scratch.path());

    // ── Rung 2 — register: B's perch row replicates into A's registry.
    // [int->REQ-INST-7]
    rig_wait("register: B's perch row replicated to A", rig.wait, || {
        registry.rows(&rig.subnet, ID_B).iter().any(|i| i.node == rig.b_hex())
    });

    // ── Rung 3 — cross-node message: the WAN send (B-side spool asserts it).
    // [int->REQ-NET-1]
    let mut a = connect_retry(&broker_name);
    let conn = a.net_dial(rig.peer_broker_addr("a"), Some(op())).expect("dial B");
    let opened = a.net_open_stream(conn.conn_id, Some(op())).expect("open msg stream");
    let msg = WanMessage {
        target: ID_B.to_string(),
        from: ID_A.to_string(),
        body: WAN_BODY.to_string(),
        op_id: "twohost-wan-1".to_string(),
    };
    a.net_stream_send(opened.stream_id, &spt_net::net::ndjson::encode_line(&msg), None, true)
        .expect("send WAN message");
    println!("TWOHOST OK: message sent (B asserts the spool)");

    // ── Rung 4 — remote-drive: fetch the session id off B (file transfer),
    // then attach to B's live PTY, type, and read the echo back.
    // [int->REQ-REACH-1]
    let sid_dest = scratch.path().join("sid.txt");
    let end = Instant::now() + rig.wait;
    let sid: u64 = loop {
        assert!(Instant::now() < end, "sid.txt never became fetchable from B");
        match spt_daemon::xfer::fetch_file(
            &mut a,
            conn.conn_id,
            ID_B,
            "sid.txt",
            &sid_dest,
            &format!("twohost-sid-{}", op()),
            op(),
            &scratch.path().join("xfer-progress"),
        ) {
            Ok(_) => {
                let text = std::fs::read_to_string(&sid_dest).expect("read sid");
                break text.trim().parse().expect("sid parses");
            }
            Err(e) => {
                eprintln!("sid fetch not ready ({e}); retrying");
                thread::sleep(Duration::from_secs(3));
            }
        }
    };
    println!("TWOHOST OK: file fetch (sid {sid})");

    // [int->REQ-INST-8]
    let viewport =
        spt_daemon::attach::request_attach(&mut a, conn.conn_id, sid, 0, op()).expect("attach");
    a.net_stream_subscribe(viewport, 0).expect("subscribe viewport");
    spt_daemon::attach::send_attach_input(&mut a, viewport, format!("{DRIVE_MARKER}\r\n").as_bytes(), op())
        .expect("type into B's session");
    let mut echoed = Vec::new();
    let mut decoder = spt_net::net::attach::AttachDecoder::new();
    let end = Instant::now() + rig.wait;
    while Instant::now() < end {
        match a.read_event().expect("event") {
            spt_daemon::brain::BrokerEvent::NetStreamData { stream_id, bytes, .. }
                if stream_id == viewport =>
            {
                for rec in decoder.push(&bytes) {
                    if let spt_net::net::attach::AttachRecord::Output { data_b64, .. } = rec {
                        echoed.extend(spt_daemon::msg::decode_bytes(&data_b64).expect("b64"));
                    }
                }
                if String::from_utf8_lossy(&echoed).contains(DRIVE_MARKER) {
                    break;
                }
            }
            _ => {}
        }
    }
    assert!(
        String::from_utf8_lossy(&echoed).contains(DRIVE_MARKER),
        "remote-drive echo came back across the rig; got {:?}",
        String::from_utf8_lossy(&echoed)
    );
    println!("TWOHOST OK: remote-drive (attach echo)");

    // ── Rung 5a — Psyche sync, A←B direction: A's pump bootstrap-pulls B's
    // mind (want-ref from A's registry, the gate served at B's dispatcher).
    // [int->REQ-INST-5]
    let cs_a = ContextStore::open_or_init().expect("A context store");
    rig_wait("sync: A pulled B's mind", rig.wait, || {
        cs_a.branch_store().tip(&format!("a-{ID_B}")).ok().flatten().is_some()
    });

    // ── Rung 6 — update self-heal: A's pump pulled B's staged v6, verified it
    // under the derived trusted key, staged it, and surfaced a CONSENT notif
    // (REQ-UPD-4 gated default — never a silent apply).
    // [int->REQ-UPD-1]
    let cache_a = ReleaseCache::open(&perch::spt_home().join("releases"));
    rig_wait("update: B's v6 staged at A through the verify gate", rig.wait, || {
        cache_a.load_artifact().map(|b| b == RELEASE_BYTES).unwrap_or(false)
    });
    rig_wait("update: consent notif surfaced at A", rig.wait, || {
        store_a
            .list(&rig.subnet)
            .map(|rows| rows.iter().any(|r| r.kind == "consent"))
            .unwrap_or(false)
    });

    // ── Rung 7b — notif acceptance leg: B's dismissal replicated back to A.
    // [int->REQ-NOTIF-1]
    rig_wait("notif: B's dismissal replicated to A", rig.wait, || {
        store_a
            .list(&rig.subnet)
            .map(|rows| rows.iter().any(|r| r.body == NOTIF_BODY && r.dismissed))
            .unwrap_or(false)
    });

    // ── Done: push the completion barrier file to B (the push half of the
    // file-transfer rung).
    // [int->REQ-REACH-1]
    let done_src = scratch.path().join("done.txt");
    std::fs::write(&done_src, b"ladder complete on A").expect("write done");
    spt_daemon::xfer::push_file(
        &mut a,
        conn.conn_id,
        ID_B,
        &done_src,
        "done.txt",
        "twohost-done-1",
        op(),
        &scratch.path().join("xfer-progress"),
    )
    .expect("push done-file to B");
    println!("TWOHOST OK: file push (done barrier)");

    stop.store(true, Ordering::Relaxed);
    println!("TWOHOST role A: ladder complete");
}


Grep

pub fn is_trusted|derive\(.*Clone.*\)]\s*\npub struct Identity|pub struct Identity


No matches found


crates\spt-daemon\tests\twohost.rs:468:        trust_now.is_trusted(&rig.subnet, &rig.a_hex()),
crates\spt-daemon\tests\twohost.rs:605:        TrustStore::load().is_trusted(&rig.subnet, &rig.b_hex()),
crates\spt-net\src\net\pairing\wire.rs:747:        assert!(init_trust.is_trusted("home", &resp_pub.to_hex()));
crates\spt-net\src\net\pairing\wire.rs:748:        assert!(resp_trust.is_trusted("home", &init_pub.to_hex()));
crates\spt-net\src\net\pairing\wire.rs:749:        assert!(!init_trust.is_trusted("home", &init_pub.to_hex()), "self not pinned");
crates\spt-daemon\src\notifsync.rs:57:        self.subnets.iter().any(|s| s == subnet) && self.trust.is_trusted(subnet, origin_node)
crates\spt-daemon\src\notifsync.rs:103:            if !policy.trust.is_trusted(&row.subnet, origin_node) {
crates\spt-daemon\src\registryhost.rs:73:        self.subnets.iter().any(|s| s == subnet) && self.trust.is_trusted(subnet, origin_node)

[Showing results with pagination = limit: 8]


51-/// material; never serialize the seed onto the wire — only [`PublicKey`] is
52-/// public-facing.
53-#[derive(Clone)]
54:pub struct Identity {


Edit

                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(3)).await;
                    continue;
                }

                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    // Blocking sleep is fine: this runtime runs nothing else.
                    thread::sleep(Duration::from_secs(3));
                    continue;
                }


The file C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon\tests\twohost.rs has been updated successfully. (file state is current in your context — no need to Read it back)

                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(3)).await;
                    continue;
                }

                Err(e) => {
                    eprintln!("pairing dial not up yet ({e}); retrying");
                    // Blocking sleep is fine: this runtime runs nothing else.
                    thread::sleep(Duration::from_secs(3));
                    continue;
                }

-                    tokio::time::sleep(Duration::from_secs(3)).await;

+                    // Blocking sleep is fine: this runtime runs nothing else.

+                    thread::sleep(Duration::from_secs(3));


                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(5)).await;
                }

                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    thread::sleep(Duration::from_secs(5));
                }


                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    tokio::time::sleep(Duration::from_secs(5)).await;
                }

                Err(e) => {
                    eprintln!("pairing ceremony failed ({e}); retrying");
                    thread::sleep(Duration::from_secs(5));
                }

-                    tokio::time::sleep(Duration::from_secs(5)).await;

+                    thread::sleep(Duration::from_secs(5));


Bash

cargo build -p spt-net -p spt-daemon --all-targets 2>&1 | tail -40

Build net + daemon with new test


   Compiling spt-msg v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-msg)
   Compiling spt-runtime v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-runtime)
   Compiling spt-term v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-term)
   Compiling spt-net v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-net)
   Compiling spt-live v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-live)
   Compiling spt-daemon v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 26.65s


Write SPT_TWO_HOST=1 gated int tests (D9b ladder)

tests gated on SPT_TWO_HOST=1 + role/peer-addr env, silent skip otherwise. Ladder: pair → register → cross-node message → remote-drive → Psyche sync → update self-heal + notif cross-node leg (insert A→fires B; dismiss B→replicates A). Int tags [int->REQ-NET-1], [int->REQ-INST-5], [int->REQ-UPD-1], [int->REQ-NOTIF-1] + pairing/registry as touched. Stages NOT activated until rig run green.

Write docs/TWO-HOST-RUNBOOK.md

Exact rig procedure HFENDULEAM ↔ gravity-linux: env setup, run order, what green looks like. Evidence appendix filled after real run. Consider [twohost] commit-message CI gate for gravity leg (no SSH from HFENDULEAM).

Run two-host proof on rig, capture evidence

Execute runbook on HFENDULEAM + gravity (likely via [twohost] CI gate for gravity leg). Capture run output into runbook appendix. Then activate int stages on touched reqs in traceable-reqs.toml. Flake-ledger discipline for any rig flake.

Retire gh-repo interim sync (CONTEXT.md note)

After proof green: note in CONTEXT.md — sister gh-repo-sync skill stands down; psyche-sync-setup remains for hub-mode seam but no longer THE sync path.


Compiles first try. Clippy + gate check (untagged run must silent-skip):


cargo clippy -p spt-net -p spt-daemon --all-targets -- -D warnings 2>&1 | tail -25

Clippy deny warnings on touched crates


    Checking spt-msg v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-msg)
    Checking spt-runtime v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-runtime)
    Checking spt-term v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-term)
    Checking spt-net v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-net)
    Checking spt-live v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-live)
    Checking spt-daemon v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.94s


cargo test -p spt-daemon --test twohost 2>&1 | tail -8

Verify ungated run silent-skips


     Running tests\twohost.rs (target\debug\deps\twohost-9affa457e2dcf612.exe)

running 2 tests
test two_host_ladder_role_a ... ok
test two_host_ladder_role_b ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=b SPT_TWO_HOST_SECRET=smoke-local-1 SPT_TWO_HOST_PEER_IP=127.0.0.1 SPT_TWO_HOST_WAIT_SECS=180 cargo test -p spt-daemon --test twohost -- --nocapture 2>&1 | tail -30

Role B local smoke (background server)


Command running in background with ID: bq6rkqx7w. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bq6rkqx7w.output. You will be notified when it completes. To check interim output, use Read on that file path.


sleep 20 && SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=a SPT_TWO_HOST_SECRET=smoke-local-1 SPT_TWO_HOST_PEER_IP=127.0.0.1 SPT_TWO_HOST_WAIT_SECS=180 cargo test -p spt-daemon --test twohost -- --nocapture 2>&1 | tail -40

Role A local smoke (driver)


Command running in background with ID: b1h6wy7d5. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\b1h6wy7d5.output. You will be notified when it completes. To check interim output, use Read on that file path.


<task-notification>
<task-id>b1h6wy7d5</task-id>
<tool-use-id>toolu_01Q4Snmm8PVt2iP63zC5BwWP</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\b1h6wy7d5.output</output-file>
<status>completed</status>
<summary>Background command "Role A local smoke (driver)" completed (exit code 0)</summary>
</task-notification>


Updated task #2 status


Prompt:
<task-notification>
<task-id>b1h6wy7d5</task-id>
<tool-use-id>toolu_01Q4Snmm8PVt2iP63zC5BwWP</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\b1h6wy7d5.output</output-file>
<status>completed</status>
<summary>Background command "Role A local smoke (driver)" completed (exit code 0)</summary>
</task-notification>


Read role A smoke output


    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.38s
     Running tests\twohost.rs (target\debug\deps\twohost-9affa457e2dcf612.exe)

running 2 tests
test two_host_ladder_role_b ... ok
TWOHOST role A: node d4fa77280cdec41f69288612f179a58cec39af298ba5fcbb664ca38ba52094e9
TWOHOST OK: pairing (initiator) — Pinned
UPDATE_STAGED:6:BrainOnly (from 58a9de3ad9a0a81e3ee91b20c2a9a20925834f556cb1f0eb3e040cc3d43dd02b)
TWOHOST OK: register: B's perch row replicated to A

thread 'two_host_ladder_role_a' (30388) panicked at crates\spt-daemon\tests\twohost.rs:645:66:
dial B: Custom { kind: Other, error: "net-dial op already applied but its connection is no longer held (broker restarted); retry with a fresh op_id" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test two_host_ladder_role_a ... FAILED

failures:

failures:
    two_host_ladder_role_a

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.38s

error: test failed, to rerun pass `-p spt-daemon --test twohost`


Smoke caught real bug: test op-ids collide with pump's journaled ops on the shared broker ("op already applied"). Fix: test ops start at high offset.


Edit

static OP: AtomicU64 = AtomicU64::new(1);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}

// High offset: the production pump journals its own ops on the SAME broker —
// test-driven ops must never collide with an already-applied pump op id.
static OP: AtomicU64 = AtomicU64::new(1_000_000_000);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}


static OP: AtomicU64 = AtomicU64::new(1);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}

// High offset: the production pump journals its own ops on the SAME broker —
// test-driven ops must never collide with an already-applied pump op id.
static OP: AtomicU64 = AtomicU64::new(1_000_000_000);
fn op() -> u64 {
    OP.fetch_add(1, Ordering::Relaxed)
}

-static OP: AtomicU64 = AtomicU64::new(1);

+// High offset: the production pump journals its own ops on the SAME broker —

+// test-driven ops must never collide with an already-applied pump op id.

+static OP: AtomicU64 = AtomicU64::new(1_000_000_000);


Bash

Check role B background state


(Bash completed with no output)


<task-notification>
<task-id>bq6rkqx7w</task-id>
<tool-use-id>toolu_01UxH64gJ8LCAkP5FkqY23Tq</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bq6rkqx7w.output</output-file>
<status>completed</status>
<summary>Background command "Role B local smoke (background server)" completed (exit code 0)</summary>
</task-notification>


taskkill //F //IM twohost-9affa457e2dcf612.exe 2>&1; sleep 2; cargo build -p spt-daemon --test twohost 2>&1 | tail -3

Kill stale smoke procs, rebuild test


SUCCESS: The process "twohost-9affa457e2dcf612.exe" with PID 31444 has been terminated.
   Compiling spt-daemon v0.0.0 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.55s


Prompt:
<task-notification>
<task-id>bq6rkqx7w</task-id>
<tool-use-id>toolu_01UxH64gJ8LCAkP5FkqY23Tq</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bq6rkqx7w.output</output-file>
<status>completed</status>
<summary>Background command "Role B local smoke (background server)" completed (exit code 0)</summary>
</task-notification>


Read role B smoke output


    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.40s
     Running tests\twohost.rs (target\debug\deps\twohost-9affa457e2dcf612.exe)

running 2 tests
test two_host_ladder_role_a ... ok
TWOHOST role B: node 58a9de3ad9a0a81e3ee91b20c2a9a20925834f556cb1f0eb3e040cc3d43dd02b
TWOHOST role B: drive session 1 ready
TWOHOST OK: pairing (responder) — Pinned
TWOHOST OK: register: A's perch row replicated to B
test two_host_ladder_role_b has been running for over 60 seconds
PUMP_PEER_FAIL:d4fa77280cdec41f69288612f179a58cec39af298ba5fcbb664ca38ba52094e9:stream ended before Done
error: test failed, to rerun pass `-p spt-daemon --test twohost`

Caused by:
  process didn't exit successfully: `C:\Users\decid\Documents\projects\spt-core\target\debug\deps\twohost-9affa457e2dcf612.exe --nocapture` (exit code: 1)


Pairing + register green both sides before A's op-collision crash. Rerun smoke:


SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=b SPT_TWO_HOST_SECRET=smoke-local-2 SPT_TWO_HOST_PEER_IP=127.0.0.1 SPT_TWO_HOST_WAIT_SECS=180 cargo test -p spt-daemon --test twohost -- --nocapture 2>&1 | tail -35

Role B local smoke take 2


Command running in background with ID: bzx12oonw. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bzx12oonw.output. You will be notified when it completes. To check interim output, use Read on that file path.


sleep 10 && SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=a SPT_TWO_HOST_SECRET=smoke-local-2 SPT_TWO_HOST_PEER_IP=127.0.0.1 SPT_TWO_HOST_WAIT_SECS=180 cargo test -p spt-daemon --test twohost -- --nocapture 2>&1 | tail -45

Role A local smoke take 2


Command running in background with ID: bdoqrzvlq. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bdoqrzvlq.output. You will be notified when it completes. To check interim output, use Read on that file path.


<task-notification>
<task-id>bdoqrzvlq</task-id>
<tool-use-id>toolu_012jsU267fvLbGqvPgCskquH</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bdoqrzvlq.output</output-file>
<status>completed</status>
<summary>Background command "Role A local smoke take 2" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>bzx12oonw</task-id>
<tool-use-id>toolu_01PZFPykpBRrQH3Qn2QdDYnW</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bzx12oonw.output</output-file>
<status>completed</status>
<summary>Background command "Role B local smoke take 2" completed (exit code 0)</summary>
</task-notification>


PowerShell

(Get-NetIPAddress -AddressFamily IPv4 | Where-Object {$_.InterfaceAlias -notmatch 'Loopback|vEthernet|WSL' -and $_.IPAddress -notmatch '^169\.'} | Select-Object InterfaceAlias, IPAddress | Format-Table -AutoSize | Out-String).Trim(); ping -n 1 -w 1000 gravity.local 2>&1 | Select-Object -First 3

Get local LAN IP and try gravity.local mDNS


Error: Exit code 1
InterfaceAlias           IPAddress
--------------           ---------
Local Area Connection* 2 192.168.137.1
Ethernet                 192.168.1.81
Wi-Fi                    192.168.0.237
Tailscale                100.68.35.65
Ping request could not find host gravity.local. Please check the name and try again.


Prompt:
<task-notification>
<task-id>bdoqrzvlq</task-id>
<tool-use-id>toolu_012jsU267fvLbGqvPgCskquH</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bdoqrzvlq.output</output-file>
<status>completed</status>
<summary>Background command "Role A local smoke take 2" completed (exit code 0)</summary>
</task-notification>


Prompt:
<task-notification>
<task-id>bzx12oonw</task-id>
<tool-use-id>toolu_01PZFPykpBRrQH3Qn2QdDYnW</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bzx12oonw.output</output-file>
<status>completed</status>
<summary>Background command "Role B local smoke take 2" completed (exit code 0)</summary>
</task-notification>


Write docs/TWO-HOST-RUNBOOK.md


Both smoke runs finished — checking:


Bash

echo "=== ROLE A ==="; tail -40 "C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bdoqrzvlq.output"; echo "=== ROLE B ==="; tail -30 "C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\648159b6-16b2-4568-b643-7b0c98c1871d\tasks\bzx12oonw.output"

Read both smoke outputs


=== ROLE A ===
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.40s
     Running tests\twohost.rs (target\debug\deps\twohost-9affa457e2dcf612.exe)

running 2 tests
test two_host_ladder_role_b ... ok
TWOHOST role A: node 41c55a8d0d457578fe8e5ce0e8cd52bc0610f7af785b0cc84d1be65901950320
TWOHOST OK: pairing (initiator) — Pinned
UPDATE_STAGED:6:BrainOnly (from 634119261c551f314d9888bdd62397ef9ed7bf6ef398689f615d7de93096a8fe)
TWOHOST OK: register: B's perch row replicated to A
TWOHOST OK: message sent (B asserts the spool)
TWOHOST OK: file fetch (sid 1)
TWOHOST OK: remote-drive (attach echo)
TWOHOST OK: sync: A pulled B's mind
TWOHOST OK: update: B's v6 staged at A through the verify gate
TWOHOST OK: update: consent notif surfaced at A
TWOHOST OK: notif: B's dismissal replicated to A
TWOHOST OK: file push (done barrier)
TWOHOST role A: ladder complete
test two_host_ladder_role_a ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.61s

=== ROLE B ===
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.45s
     Running tests\twohost.rs (target\debug\deps\twohost-9affa457e2dcf612.exe)

running 2 tests
test two_host_ladder_role_a ... ok
TWOHOST role B: node 634119261c551f314d9888bdd62397ef9ed7bf6ef398689f615d7de93096a8fe
TWOHOST role B: drive session 1 ready
TWOHOST OK: pairing (responder) — Pinned
TWOHOST OK: register: A's perch row replicated to B
TWOHOST OK: message: A's WAN message spooled at B
TWOHOST OK: notif: A's insert fired on B
TWOHOST OK: notif dismissed at B (41c55a8d0d457578fe8e5ce0e8cd52bc0610f7af785b0cc84d1be65901950320:1780582028075) — replicating back
TWOHOST OK: sync: B pulled A's mind
TWOHOST OK: done-file pushed by A (ladder complete on A)
TWOHOST role B: ladder complete
test two_host_ladder_role_b ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 18.76s


PowerShell

tailscale status 2>&1 | Select-Object -First 15

List tailscale peers


100.68.35.65    hfenduleam                    brandon@  windows  -
100.125.242.88  enlyzeam                      brandon@  windows  -
100.72.210.9    flaerdine                     brandon@  android  -
100.100.62.43   gravity-linux                 brandon@  linux    -
100.65.194.82   gravity-runner                brandon@  windows  -
100.98.197.12   kitsubito                     brandon@  linux    idle; offers exit node
100.112.85.82   lepicep                       brandon@  linux    offline, last seen 13d ago
100.98.213.33   philip-vibe                   philip@   linux    offline, last seen 14d ago
100.122.242.42  philips-desktop-threadripper  philip@   windows  offline, last seen 1h ago
100.98.214.87   philips-desktop               philip@   windows  -
100.94.119.17   philips-iphone-12             philip@   iOS      offline, last seen 18d ago
100.78.177.97   philips-macbook-pro           philip@   macOS    offline, last seen 6d ago


gravity-linux on tailscale: `100.100.62.43`. Stable IPs both sides — rig addressing solved. Verify reachability:


Ping gravity over tailscale


    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 13ms, Maximum = 33ms, Average = 23ms


Bash

ssh -o ConnectTimeout=8 -o BatchMode=yes -o StrictHostKeyChecking=accept-new brandon@100.100.62.43 "hostname && echo SSH-OK" 2>&1 | tail -3

Try SSH to gravity via tailscale IP


Warning: Permanently added '100.100.62.43' (ED25519) to the list of known hosts.
brandon@100.100.62.43: Permission denied (publickey,password).


for u in decide decidel reavo gravity ubuntu; do timeout 8 ssh -o ConnectTimeout=5 -o BatchMode=yes $u@100.100.62.43 "echo SSH-OK as \$(whoami)" 2>/dev/null && break; done; echo "done"

Try common usernames for gravity SSH


done


timeout 12 tailscale ssh gravity-linux "echo TS-SSH-OK as \$(whoami)" 2>&1 | tail -2

Try Tailscale SSH to gravity


No ED25519 host key is known for gravity-linux.gazella-geological.ts.net. and you have requested strict checking.
Host key verification failed.


timeout 15 tailscale ssh -o StrictHostKeyChecking=accept-new gravity-linux "echo TS-SSH-OK as \$(whoami)" 2>&1 | tail -2

Tailscale SSH accepting host key


* It automatically checks the destination server's SSH host key against the
  node's SSH host key as advertised via the Tailscale coordination server.


timeout 15 tailscale ssh brandon@gravity-linux "echo TS-SSH-OK as \$(whoami)" 2>&1 | tail -3

Tailscale SSH with explicit user


ssh-keyscan -T 8 -t ed25519 gravity-linux.gazella-geological.ts.net >> ~/.ssh/known_hosts 2>/dev/null; timeout 15 tailscale ssh brandon@gravity-linux "echo TS-SSH-OK as \$(whoami)" 2>&1 | tail -2

Add gravity host key, retry tailscale SSH


ssh-keyscan -T 8 -t ed25519 100.100.62.43 2>/dev/null | head -2; echo "---"; tailscale ssh 2>&1 | head -5

Keyscan IP directly; tailscale ssh usage


100.100.62.43 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOddjgrprkl6b8PiWGSm05PmRGQVgsA2NzFXmAgYPa73
---
usage: tailscale ssh [user@]<host>


echo "gravity-linux.gazella-geological.ts.net,100.100.62.43 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOddjgrprkl6b8PiWGSm05PmRGQVgsA2NzFXmAgYPa73" >> ~/.ssh/known_hosts; timeout 15 tailscale ssh brandon@gravity-linux "echo TS-SSH-OK as \$(whoami)" 2>&1 | tail -2

Manually add FQDN host key, retry


Write

# Two-host rig runbook — the D9b ladder on real hardware

How to run the `SPT_TWO_HOST` integration ladder (`crates/spt-daemon/tests/twohost.rs`)
across two real machines, and what green looks like. This is the M4 cross-node proof
(M4-PLAN §D9b): **pair → register → cross-node message → remote-drive → Psyche sync →
update self-heal → notif cross-node** — every rung over a real network path between two
separate processes on two separate hosts, no loopback shortcuts.

The test is **env-gated**: without `SPT_TWO_HOST=1` + a role it silent-skips (CI's
normal sweeps never run it). The `int` evidence tags live on the gated tests; the
requirement `int` stages activate only after a real rig run is green (TRACEABILITY.md
rule: tags on real evidence).

## The rig

| host | OS | role | tailscale address |
|------|----|------|-------------------|
| HFENDULEAM | Windows 11 | **a** — joiner / driver | `100.68.35.65` |
| gravity-linux | Ubuntu 22.04 | **b** — seed-holder / server | `100.100.62.43` |

Tailscale addresses are the rig's stable transport (the hosts sit on different
physical subnets and gravity's hostname does not resolve from HFENDULEAM; the
tailnet IPs are static). Any reachable IP pair works — LAN addresses included.

## How the two sides find each other (no runtime exchange)

Everything either side needs is **derived from a shared secret + static env**, so both
legs run non-interactively (the gravity leg is driven by a commit-message-gated CI
step — there is no SSH between the rig hosts):

- node identities ← `sha256(secret:"node-a"/"node-b")`;
- the subnet TOTP seed ← `sha256(secret:"subnet-totp")[..20]` — role A computes the
  6-digit code locally, standing in for the user reading the seed-holder's
  `spt pair show-totp` screen. **The wire ceremony is the real SPAKE2/TOTP
  exchange** (run_initiator/run_responder over the pre-trust ALPN, cross-host);
  only the operator's eyeballs are simulated;
- the release signing key ← `sha256(secret:"release")`;
- dialable addresses ← derived node id + the peer's IP + fixed UDP ports
  (`BindScope::Port`): relays and discovery disabled, one direct path each way.

## Env contract

| var | role | meaning | rig value |
|-----|------|---------|-----------|
| `SPT_TWO_HOST` | both | `1` activates; anything else silent-skips | `1` |
| `SPT_TWO_HOST_ROLE` | both | `a` (joiner/driver) / `b` (seed-holder/server) | per host |
| `SPT_TWO_HOST_SECRET` | both | shared secret, identical both sides | any string |
| `SPT_TWO_HOST_PEER_IP` | both | the OTHER host's IP | see table above |
| `SPT_TWO_HOST_SUBNET` | both | subnet name | default `twohost` |
| `SPT_TWO_HOST_PORT_A` | both | A's broker UDP port | default `7460` |
| `SPT_TWO_HOST_PORT_B` | both | B's broker UDP port | default `7461` |
| `SPT_TWO_HOST_PORT_PAIR` | both | B's pre-trust pairing UDP port | default `7462` |
| `SPT_TWO_HOST_WAIT_SECS` | both | per-rung convergence deadline | `900` on the rig (absorbs runner start skew); default `300` |

## Prerequisites

- Clocks NTP-synced (the pairing TOTP window is ±30 s).
- UDP reachable both directions on the three ports. The Windows side may need a
  one-time inbound allow (run elevated):
  `netsh advfirewall firewall add rule name="spt twohost rig" dir=in action=allow protocol=UDP localport=7460-7462`
  In practice A's pump dials B first and B's replies ride the same UDP 4-tuple, so
  a stateful default usually passes — add the rule if role B logs
  `PUMP_PEER_FAIL`/dial timeouts against A.
- Both sides on the same commit (the wire formats must match).

## Procedure 1 — CI-driven rig run (the standard path)

Both rig hosts are the repo's self-hosted runners, so one tagged push runs both legs:

1. Land the change you want proven; wait for normal CI green.
2. Push a commit whose message contains **`[twohost]`** (an empty commit is fine):
   `git commit --allow-empty -m "test(rig): two-host ladder run [twohost]"`
3. The `twohost-a` (hfenduleam) and `twohost-b` (gravity) jobs gate on the tag and
   on `needs: test` — they release together after both test legs finish, which keeps
   the two roles inside each other's `WAIT` window.
4. Verify FINAL conclusions sha-pinned (gotchas #7/#13):
   `gh run list --commit <sha> --json name,conclusion`
5. Capture both job logs' `TWOHOST OK:` lines into the evidence appendix below.

## Procedure 2 — manual two-terminal run

On gravity (start first):

```sh
SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=b \
SPT_TWO_HOST_SECRET=rig-<date> SPT_TWO_HOST_PEER_IP=100.68.35.65 \
SPT_TWO_HOST_WAIT_SECS=900 \
cargo test -p spt-daemon --test twohost -- --nocapture
```

On HFENDULEAM (PowerShell):

```powershell
$env:SPT_TWO_HOST="1"; $env:SPT_TWO_HOST_ROLE="a"
$env:SPT_TWO_HOST_SECRET="rig-<date>"; $env:SPT_TWO_HOST_PEER_IP="100.100.62.43"
$env:SPT_TWO_HOST_WAIT_SECS="900"
cargo test -p spt-daemon --test twohost -- --nocapture
```

Role A retries the pairing dial until B's ceremony endpoint answers, so exact start
order only matters within the `WAIT` window.

## What green looks like

Role A prints, in order:

```
TWOHOST OK: pairing (initiator) — Pinned
TWOHOST OK: register: B's perch row replicated to A
TWOHOST OK: message sent (B asserts the spool)
TWOHOST OK: file fetch (sid <n>)
TWOHOST OK: remote-drive (attach echo)
TWOHOST OK: sync: A pulled B's mind
TWOHOST OK: update: B's v6 staged at A through the verify gate
TWOHOST OK: update: consent notif surfaced at A
TWOHOST OK: notif: B's dismissal replicated to A
TWOHOST OK: file push (done barrier)
TWOHOST role A: ladder complete
```

Role B prints, in order:

```
TWOHOST OK: pairing (responder) — Pinned
TWOHOST OK: register: A's perch row replicated to B
TWOHOST OK: message: A's WAN message spooled at B
TWOHOST OK: notif: A's insert fired on B
TWOHOST OK: notif dismissed at B (<id>) — replicating back
TWOHOST OK: sync: B pulled A's mind
TWOHOST OK: done-file pushed by A (ladder complete on A)
TWOHOST role B: ladder complete
```

Plus `test result: ok` on both sides. Any rung that never converges panics with
`never converged on the rig: <rung>` — file a flake-ledger entry, don't shrug.

## What each rung proves

| rung | evidence | requirement |
|------|----------|-------------|
| pairing ceremony, cross-host SPAKE2 + seed transfer | trust pinned both sides; joiner holds the subnet seed | REQ-PAIR-1, REQ-PAIR-5 |
| registry replication, both directions | each side's perch row in the other's gated registry | REQ-INST-7 |
| WAN message | A's send lands in B's perch spool through the dispatcher funnel | REQ-NET-1 |
| remote-drive | A fetches a file off B, types into B's live PTY, reads the echo, pushes a file back | REQ-INST-8, REQ-REACH-1 |
| Psyche sync, both directions | each side bootstrap-pulls a mind it never held (registry-derived want-ref, gate served remotely) | REQ-INST-5 |
| update self-heal | B's staged v6 verified + staged at A under the trusted key; consent notif surfaced (gated default) | REQ-UPD-1, REQ-UPD-4 posture |
| notif cross-node | insert on A fires on B; B's dismissal replicates back to A | REQ-NOTIF-1 |

## Evidence appendix

> Filled from the first green rig run; later reruns replace it.

*(pending — no rig run captured yet)*


File created successfully at: C:\Users\decid\Documents\projects\spt-core\docs\TWO-HOST-RUNBOOK.md (file state is current in your context — no need to Read it back)

# Two-host rig runbook — the D9b ladder on real hardware

How to run the `SPT_TWO_HOST` integration ladder (`crates/spt-daemon/tests/twohost.rs`)
across two real machines, and what green looks like. This is the M4 cross-node proof
(M4-PLAN §D9b): **pair → register → cross-node message → remote-drive → Psyche sync →
update self-heal → notif cross-node** — every rung over a real network path between two
separate processes on two separate hosts, no loopback shortcuts.

The test is **env-gated**: without `SPT_TWO_HOST=1` + a role it silent-skips (CI's
normal sweeps never run it). The `int` evidence tags live on the gated tests; the
requirement `int` stages activate only after a real rig run is green (TRACEABILITY.md
rule: tags on real evidence).

## The rig

| host | OS | role | tailscale address |
|------|----|------|-------------------|
| HFENDULEAM | Windows 11 | **a** — joiner / driver | `100.68.35.65` |
| gravity-linux | Ubuntu 22.04 | **b** — seed-holder / server | `100.100.62.43` |

Tailscale addresses are the rig's stable transport (the hosts sit on different
physical subnets and gravity's hostname does not resolve from HFENDULEAM; the
tailnet IPs are static). Any reachable IP pair works — LAN addresses included.

## How the two sides find each other (no runtime exchange)

Everything either side needs is **derived from a shared secret + static env**, so both
legs run non-interactively (the gravity leg is driven by a commit-message-gated CI
step — there is no SSH between the rig hosts):

- node identities ← `sha256(secret:"node-a"/"node-b")`;
- the subnet TOTP seed ← `sha256(secret:"subnet-totp")[..20]` — role A computes the
  6-digit code locally, standing in for the user reading the seed-holder's
  `spt pair show-totp` screen. **The wire ceremony is the real SPAKE2/TOTP
  exchange** (run_initiator/run_responder over the pre-trust ALPN, cross-host);
  only the operator's eyeballs are simulated;
- the release signing key ← `sha256(secret:"release")`;
- dialable addresses ← derived node id + the peer's IP + fixed UDP ports
  (`BindScope::Port`): relays and discovery disabled, one direct path each way.

## Env contract

| var | role | meaning | rig value |
|-----|------|---------|-----------|
| `SPT_TWO_HOST` | both | `1` activates; anything else silent-skips | `1` |
| `SPT_TWO_HOST_ROLE` | both | `a` (joiner/driver) / `b` (seed-holder/server) | per host |
| `SPT_TWO_HOST_SECRET` | both | shared secret, identical both sides | any string |
| `SPT_TWO_HOST_PEER_IP` | both | the OTHER host's IP | see table above |
| `SPT_TWO_HOST_SUBNET` | both | subnet name | default `twohost` |
| `SPT_TWO_HOST_PORT_A` | both | A's broker UDP port | default `7460` |
| `SPT_TWO_HOST_PORT_B` | both | B's broker UDP port | default `7461` |
| `SPT_TWO_HOST_PORT_PAIR` | both | B's pre-trust pairing UDP port | default `7462` |
| `SPT_TWO_HOST_WAIT_SECS` | both | per-rung convergence deadline | `900` on the rig (absorbs runner start skew); default `300` |

## Prerequisites

- Clocks NTP-synced (the pairing TOTP window is ±30 s).
- UDP reachable both directions on the three ports. The Windows side may need a
  one-time inbound allow (run elevated):
  `netsh advfirewall firewall add rule name="spt twohost rig" dir=in action=allow protocol=UDP localport=7460-7462`
  In practice A's pump dials B first and B's replies ride the same UDP 4-tuple, so
  a stateful default usually passes — add the rule if role B logs
  `PUMP_PEER_FAIL`/dial timeouts against A.
- Both sides on the same commit (the wire formats must match).

## Procedure 1 — CI-driven rig run (the standard path)

Both rig hosts are the repo's self-hosted runners, so one tagged push runs both legs:

1. Land the change you want proven; wait for normal CI green.
2. Push a commit whose message contains **`[twohost]`** (an empty commit is fine):
   `git commit --allow-empty -m "test(rig): two-host ladder run [twohost]"`
3. The `twohost-a` (hfenduleam) and `twohost-b` (gravity) jobs gate on the tag and
   on `needs: test` — they release together after both test legs finish, which keeps
   the two roles inside each other's `WAIT` window.
4. Verify FINAL conclusions sha-pinned (gotchas #7/#13):
   `gh run list --commit <sha> --json name,conclusion`
5. Capture both job logs' `TWOHOST OK:` lines into the evidence appendix below.

## Procedure 2 — manual two-terminal run

On gravity (start first):

```sh
SPT_TWO_HOST=1 SPT_TWO_HOST_ROLE=b \
SPT_TWO_HOST_SECRET=rig-<date> SPT_TWO_HOST_PEER_IP=100.68.35.65 \
SPT_TWO_HOST_WAIT_SECS=900 \
cargo test -p spt-daemon --test twohost -- --nocapture
```

On HFENDULEAM (PowerShell):

```powershell
$env:SPT_TWO_HOST="1"; $env:SPT_TWO_HOST_ROLE="a"
$env:SPT_TWO_HOST_SECRET="rig-<date>"; $env:SPT_TWO_HOST_PEER_IP="100.100.62.43"
$env:SPT_TWO_HOST_WAIT_SECS="900"
cargo test -p spt-daemon --test twohost -- --nocapture
```

Role A retries the pairing dial until B's ceremony endpoint answers, so exact start
order only matters within the `WAIT` window.

## What green looks like

Role A prints, in order:

```
TWOHOST OK: pairing (initiator) — Pinned
TWOHOST OK: register: B's perch row replicated to A
TWOHOST OK: message sent (B asserts the spool)
TWOHOST OK: file fetch (sid <n>)
TWOHOST OK: remote-drive (attach echo)
TWOHOST OK: sync: A pulled B's mind
TWOHOST OK: update: B's v6 staged at A through the verify gate
TWOHOST OK: update: consent notif surfaced at A
TWOHOST OK: notif: B's dismissal replicated to A
TWOHOST OK: file push (done barrier)
TWOHOST role A: ladder complete
```

Role B prints, in order:

```
TWOHOST OK: pairing (responder) — Pinned
TWOHOST OK: register: A's perch row replicated to B
TWOHOST OK: message: A's WAN message spooled at B
TWOHOST OK: notif: A's insert fired on B
TWOHOST OK: notif dismissed at B (<id>) — replicating back
TWOHOST OK: sync: B pulled A's mind
TWOHOST OK: done-file pushed by A (ladder complete on A)
TWOHOST role B: ladder complete
```

Plus `test result: ok` on both sides. Any rung that never converges panics with
`never converged on the rig: <rung>` — file a flake-ledger entry, don't shrug.

## What each rung proves

| rung | evidence | requirement |
|------|----------|-------------|
| pairing ceremony, cross-host SPAKE2 + seed transfer | trust pinned both sides; joiner holds the subnet seed | REQ-PAIR-1, REQ-PAIR-5 |
| registry replication, both directions | each side's perch row in the other's gated registry | REQ-INST-7 |
| WAN message | A's send lands in B's perch spool through the dispatcher funnel | REQ-NET-1 |
| remote-drive | A fetches a file off B, types into B's live PTY, reads the echo, pushes a file back | REQ-INST-8, REQ-REACH-1 |
| Psyche sync, both directions | each side bootstrap-pulls a mind it never held (registry-derived want-ref, gate served remotely) | REQ-INST-5 |
| update self-heal | B's staged v6 verified + staged at A under the trusted key; consent notif surfaced (gated default) | REQ-UPD-1, REQ-UPD-4 posture |
| notif cross-node | insert on A fires on B; B's dismissal replicates back to A | REQ-NOTIF-1 |

## Evidence appendix

> Filled from the first green rig run; later reruns replace it.

*(pending — no rig run captured yet)*


Now CI jobs: