Instrumenting nethost.rs — `subscribe_stream` lock waits + `StreamLog::append` pressure. Checking its imports + the append site.


Grep

^use |use std::sync|AtomicU64|Ordering|fn append|impl StreamLog|fn attach\b


29:use std::collections::{HashMap, HashSet, VecDeque};
30:use std::io;
31:use std::sync::atomic::{AtomicU64, Ordering};
32:use std::sync::{Arc, Mutex};
33:use std::time::Duration;
35:use tokio::io::{AsyncReadExt, AsyncWriteExt, DuplexStream, ReadHalf, WriteHalf};
37:use spt_net::net::endpoint::{
41:use spt_net::net::mesh::seedproof::ProofRole;
42:use spt_proto::identity::{Identity, PublicKey};
44:use crate::broker::SharedSend;
45:use crate::codec::write_frame;
46:use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};
47:use crate::msg::{
170:impl StreamLog {
192:    fn append(&mut self, bytes: &[u8]) {
238:    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
294:    fn append(&mut self, kind: &str, conn_id: u64, remote_id_hex: &str) {
316:    fn attach(&mut self, sub: SharedSend, from_seq: u64) {
403:    next_conn_id: AtomicU64,
405:    next_stream_id: AtomicU64,
415:            next_conn_id: AtomicU64::new(1),
417:            next_stream_id: AtomicU64::new(1),
436:    let id = shared.next_conn_id.fetch_add(1, Ordering::Relaxed);
505:    let id = shared.next_stream_id.fetch_add(1, Ordering::Relaxed);
619:    quic_op_timeout_ms: AtomicU64,
748:            quic_op_timeout_ms: AtomicU64::new(BROKER_QUIC_OP_TIMEOUT_MS),
765:        Duration::from_millis(self.quic_op_timeout_ms.load(Ordering::Relaxed))
776:            .store(dur.as_millis() as u64, Ordering::Relaxed);
974:        let id = self.shared.next_conn_id.fetch_add(1, Ordering::Relaxed);
1427:        use std::sync::atomic::{AtomicUsize, Ordering as AtOrd};


/spt:commune with immediate next steps and broad summary of the project's status + end goal. we'll resume in the next session


nethost.rs has the atomics imported. Adding the broker-helper import + stream counters, then instrumenting the StreamLog lock sites.


44	use crate::broker::SharedSend;
45	use crate::codec::write_frame;
46	use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};
47	use crate::msg::{
48	    net_presence_event_envelope, net_stream_data_envelope, net_stream_eof_envelope,
49	    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
50	};
51
52	/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped
53	/// effects (a dial has no PTY session). Broker session ids are minted from 1
54	/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.
55	pub const NET_EFFECT_SESSION: u64 = 0;

use crate::broker::SharedSend;
use crate::codec::write_frame;
use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};
use crate::msg::{
    net_presence_event_envelope, net_stream_data_envelope, net_stream_eof_envelope,
    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
};

/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped
/// effects (a dial has no PTY session). Broker session ids are minted from 1
/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.
pub const NET_EFFECT_SESSION: u64 = 0;


Edit

use crate::broker::SharedSend;
use crate::codec::write_frame;

use crate::broker::SharedSend;
#[allow(unused_imports)]
use crate::broker::{wedge_trace_on, WT_DRAIN_APPENDS};
use crate::codec::write_frame;


The file C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon\src\nethost.rs has been updated successfully. (file state is current in your context — no need to Read it back)

use crate::broker::SharedSend;
use crate::codec::write_frame;

use crate::broker::SharedSend;
#[allow(unused_imports)]
use crate::broker::{wedge_trace_on, WT_DRAIN_APPENDS};
use crate::codec::write_frame;

use crate::seedproofx::{prove_membership, MembershipSource, RosterExchange};


Adding stream-pressure counters + instrumenting the StreamLog lock sites.


    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
};

    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
};

// DIAGNOSTIC (SPT_WEDGE_TRACE): StreamLog-lock pressure from the serve_attach
// forward — the Theory-1 suspect (the attacher's subscribe attach() starving on the
// StreamLog Mutex behind the per-chunk forward append). Off by default.
static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);


    NetPresenceEvent, NetStreamInfo, PRESENCE_CONNECTED, PRESENCE_DISCONNECTED,
};

// DIAGNOSTIC (SPT_WEDGE_TRACE): StreamLog-lock pressure from the serve_attach
// forward — the Theory-1 suspect (the attacher's subscribe attach() starving on the
// StreamLog Mutex behind the per-chunk forward append). Off by default.
static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

+// DIAGNOSTIC (SPT_WEDGE_TRACE): StreamLog-lock pressure from the serve_attach

+// forward — the Theory-1 suspect (the attacher's subscribe attach() starving on the

+// StreamLog Mutex behind the per-chunk forward append). Off by default.

+static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);

+static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);

+static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped

/// effects (a dial has no PTY session). Broker session ids are minted from 1

/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.


Now instrument `subscribe_stream` — the streams-lock + StreamLog-lock waits (Theory-1 smoking gun):


1224	        sub: SharedSend,
1225	        from_seq: u64,
1226	    ) -> io::Result<()> {
1227	        let log = {
1228	            let streams = self.shared.streams.lock().unwrap();
1229	            let e = streams.get(&stream_id).ok_or_else(|| {
1230	                io::Error::new(
1231	                    io::ErrorKind::NotFound,
1232	                    format!("no such stream {stream_id}"),
1233	                )
1234	            })?;
1235	            Arc::clone(&e.log)
1236	        };
1237	        log.lock().unwrap().attach(sub, from_seq);
1238	        Ok(())
1239	    }
1240
1241	    /// Detach `sub` from a stream iff it is the current subscriber (brain
1242	    /// disconnect cleanup; the stream + its ring persist).
1243	    pub(crate) fn detach_stream_sub(&self, stream_id: u64, sub: &SharedSend) {
1244	        let log = {
1245	            let streams = self.shared.streams.lock().unwrap();

        sub: SharedSend,
        from_seq: u64,
    ) -> io::Result<()> {
        let log = {
            let streams = self.shared.streams.lock().unwrap();
            let e = streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?;
            Arc::clone(&e.log)
        };
        log.lock().unwrap().attach(sub, from_seq);
        Ok(())
    }

    /// Detach `sub` from a stream iff it is the current subscriber (brain
    /// disconnect cleanup; the stream + its ring persist).
    pub(crate) fn detach_stream_sub(&self, stream_id: u64, sub: &SharedSend) {
        let log = {
            let streams = self.shared.streams.lock().unwrap();


        let log = {
            let streams = self.shared.streams.lock().unwrap();
            let e = streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?;
            Arc::clone(&e.log)
        };
        log.lock().unwrap().attach(sub, from_seq);
        Ok(())
    }

    /// Detach `sub` from a stream iff it is the current subscriber (brain

        // DIAGNOSTIC (SPT_WEDGE_TRACE): the attacher's net-stream subscribe. Time the
        // two locks it takes — the brief `streams` map lock, then the per-stream
        // StreamLog lock for attach(). Theory 1 = StreamLog log_lock_wait_us is HUGE
        // (attach starves behind the serve_attach forward's per-chunk append on the
        // same StreamLog mutex). Theory 2 = both waits ~tiny (the lock is not it).
        if wedge_trace_on() {
            let ts = std::time::Instant::now();
            let log = {
                let streams = self.shared.streams.lock().unwrap();
                let streams_lock_wait_us = ts.elapsed().as_micros();
                let e = streams.get(&stream_id).ok_or_else(|| {
                    io::Error::new(io::ErrorKind::NotFound, format!("no such stream {stream_id}"))
                })?;
                let log = Arc::clone(&e.log);
                eprintln!(
                    "WEDGE_TRACE subscribe_stream stream={stream_id} \
                     streams_lock_wait_us={streams_lock_wait_us} drain_appends={} \
                     stream_appends={}",
                    WT_DRAIN_APPENDS.load(Ordering::Relaxed),
                    WT_STREAM_APPENDS.load(Ordering::Relaxed),
                );
                log
            };
            let tl = std::time::Instant::now();
            let mut g = log.lock().unwrap();
            let log_lock_wait_us = tl.elapsed().as_micros();
            let ta = std::time::Instant::now();
            g.attach(sub, from_seq);
            eprintln!(
                "WEDGE_TRACE subscribe_stream stream={stream_id} \
                 streamlog_log_lock_wait_us={log_lock_wait_us} attach_us={} \
                 stream_append_wait_total_us={} stream_append_wait_max_us={}",
                ta.elapsed().as_micros(),
                WT_STREAM_APPEND_WAIT_US.load(Ordering::Relaxed),
                WT_STREAM_APPEND_WAIT_MAX_US.load(Ordering::Relaxed),
            );
            return Ok(());
        }
        let log = {
            let streams = self.shared.streams.lock().unwrap();
            let e = streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?;
            Arc::clone(&e.log)
        };
        log.lock().unwrap().attach(sub, from_seq);
        Ok(())
    }

    /// Detach `sub` from a stream iff it is the current subscriber (brain


        // DIAGNOSTIC (SPT_WEDGE_TRACE): the attacher's net-stream subscribe. Time the
        // two locks it takes — the brief `streams` map lock, then the per-stream
        // StreamLog lock for attach(). Theory 1 = StreamLog log_lock_wait_us is HUGE
        // (attach starves behind the serve_attach forward's per-chunk append on the
        // same StreamLog mutex). Theory 2 = both waits ~tiny (the lock is not it).
        if wedge_trace_on() {
            let ts = std::time::Instant::now();
            let log = {
                let streams = self.shared.streams.lock().unwrap();
                let streams_lock_wait_us = ts.elapsed().as_micros();
                let e = streams.get(&stream_id).ok_or_else(|| {
                    io::Error::new(io::ErrorKind::NotFound, format!("no such stream {stream_id}"))
                })?;
                let log = Arc::clone(&e.log);
                eprintln!(
                    "WEDGE_TRACE subscribe_stream stream={stream_id} \
                     streams_lock_wait_us={streams_lock_wait_us} drain_appends={} \
                     stream_appends={}",
                    WT_DRAIN_APPENDS.load(Ordering::Relaxed),
                    WT_STREAM_APPENDS.load(Ordering::Relaxed),
                );
                log
            };
            let tl = std::time::Instant::now();
            let mut g = log.lock().unwrap();
            let log_lock_wait_us = tl.elapsed().as_micros();
            let ta = std::time::Instant::now();
            g.attach(sub, from_seq);
            eprintln!(
                "WEDGE_TRACE subscribe_stream stream={stream_id} \
                 streamlog_log_lock_wait_us={log_lock_wait_us} attach_us={} \
                 stream_append_wait_total_us={} stream_append_wait_max_us={}",
                ta.elapsed().as_micros(),
                WT_STREAM_APPEND_WAIT_US.load(Ordering::Relaxed),
                WT_STREAM_APPEND_WAIT_MAX_US.load(Ordering::Relaxed),
            );
            return Ok(());
        }
        let log = {
            let streams = self.shared.streams.lock().unwrap();
            let e = streams.get(&stream_id).ok_or_else(|| {
                io::Error::new(
                    io::ErrorKind::NotFound,
                    format!("no such stream {stream_id}"),
                )
            })?;
            Arc::clone(&e.log)
        };
        log.lock().unwrap().attach(sub, from_seq);
        Ok(())
    }

    /// Detach `sub` from a stream iff it is the current subscriber (brain

+        // DIAGNOSTIC (SPT_WEDGE_TRACE): the attacher's net-stream subscribe. Time the

+        // two locks it takes — the brief `streams` map lock, then the per-stream

+        // StreamLog lock for attach(). Theory 1 = StreamLog log_lock_wait_us is HUGE

+        // (attach starves behind the serve_attach forward's per-chunk append on the

+        // same StreamLog mutex). Theory 2 = both waits ~tiny (the lock is not it).

+            let ts = std::time::Instant::now();

+                let streams = self.shared.streams.lock().unwrap();

+                let streams_lock_wait_us = ts.elapsed().as_micros();

+                let e = streams.get(&stream_id).ok_or_else(|| {

+                    io::Error::new(io::ErrorKind::NotFound, format!("no such stream {stream_id}"))

+                let log = Arc::clone(&e.log);

+                    "WEDGE_TRACE subscribe_stream stream={stream_id} \

+                     streams_lock_wait_us={streams_lock_wait_us} drain_appends={} \

+                     stream_appends={}",

+                    WT_DRAIN_APPENDS.load(Ordering::Relaxed),

+                    WT_STREAM_APPENDS.load(Ordering::Relaxed),

+            let tl = std::time::Instant::now();

+            let mut g = log.lock().unwrap();

+            let log_lock_wait_us = tl.elapsed().as_micros();

+            let ta = std::time::Instant::now();

+                 streamlog_log_lock_wait_us={log_lock_wait_us} attach_us={} \

+                 stream_append_wait_total_us={} stream_append_wait_max_us={}",

+                ta.elapsed().as_micros(),

+                WT_STREAM_APPEND_WAIT_US.load(Ordering::Relaxed),

+                WT_STREAM_APPEND_WAIT_MAX_US.load(Ordering::Relaxed),


Adding the forward-pressure count + wait at the StreamLog append call site. Finding it.


546:                    Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),
562:                        Ok(n) => log.lock().unwrap().append(&buf[..n]),


Two append sites (546, 562) — the stream read pumps. Adding a timed-append helper + wiring both. Reading the context for types.


536	    // so before pulling the next chunk the pump WAITS while the undrained ring is
537	    // at its `retentive_cap`. The unread bytes stay in the transport's buffer, so
538	    // QUIC stream flow-control (or the loopback duplex's bounded buffer) lags the
539	    // SENDER — "lag, never loss" (CONTEXT:262). A `drain` wakes `room`. Ordinary
540	    // streams have `retentive_cap == 0`, so this gate is a no-op for them.
541	    tokio::spawn(async move {
542	        match recv {
543	            RecvHalf::Quic(mut recv) => loop {
544	                await_ring_room(&log, &room).await;
545	                match recv.read_chunk(STREAM_READ_CHUNK).await {
546	                    Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),
547	                    Ok(None) | Err(_) => {
548	                        log.lock().unwrap().finish();
549	                        break;
550	                    }
551	                }
552	            },
553	            RecvHalf::Loopback(mut recv) => {
554	                let mut buf = vec![0u8; STREAM_READ_CHUNK];
555	                loop {
556	                    await_ring_room(&log, &room).await;
557	                    match recv.read(&mut buf).await {
558	                        Ok(0) | Err(_) => {
559	                            log.lock().unwrap().finish();
560	                            break;
561	                        }
562	                        Ok(n) => log.lock().unwrap().append(&buf[..n]),
563	                    }
564	                }
565	            }

    // so before pulling the next chunk the pump WAITS while the undrained ring is
    // at its `retentive_cap`. The unread bytes stay in the transport's buffer, so
    // QUIC stream flow-control (or the loopback duplex's bounded buffer) lags the
    // SENDER — "lag, never loss" (CONTEXT:262). A `drain` wakes `room`. Ordinary
    // streams have `retentive_cap == 0`, so this gate is a no-op for them.
    tokio::spawn(async move {
        match recv {
            RecvHalf::Quic(mut recv) => loop {
                await_ring_room(&log, &room).await;
                match recv.read_chunk(STREAM_READ_CHUNK).await {
                    Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),
                    Ok(None) | Err(_) => {
                        log.lock().unwrap().finish();
                        break;
                    }
                }
            },
            RecvHalf::Loopback(mut recv) => {
                let mut buf = vec![0u8; STREAM_READ_CHUNK];
                loop {
                    await_ring_room(&log, &room).await;
                    match recv.read(&mut buf).await {
                        Ok(0) | Err(_) => {
                            log.lock().unwrap().finish();
                            break;
                        }
                        Ok(n) => log.lock().unwrap().append(&buf[..n]),
                    }
                }
            }


Loopback append at 562 is the attacher's path. Adding the timed-append helper + wiring both sites.


static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// DIAGNOSTIC (SPT_WEDGE_TRACE): the stream read-pump's StreamLog append, timed.
/// Measures the lock-acquire wait the forward incurs per chunk — the attacher's
/// `subscribe_stream` attach() contends on the SAME StreamLog mutex (Theory 1).
fn wt_stream_append(log: &Arc<Mutex<StreamLog>>, bytes: &[u8]) {
    if wedge_trace_on() {
        let t0 = std::time::Instant::now();
        let mut g = log.lock().unwrap();
        let w = t0.elapsed().as_micros() as u64;
        WT_STREAM_APPENDS.fetch_add(1, Ordering::Relaxed);
        WT_STREAM_APPEND_WAIT_US.fetch_add(w, Ordering::Relaxed);
        WT_STREAM_APPEND_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);
        g.append(bytes);
    } else {
        log.lock().unwrap().append(bytes);
    }
}


static WT_STREAM_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_STREAM_APPEND_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// DIAGNOSTIC (SPT_WEDGE_TRACE): the stream read-pump's StreamLog append, timed.
/// Measures the lock-acquire wait the forward incurs per chunk — the attacher's
/// `subscribe_stream` attach() contends on the SAME StreamLog mutex (Theory 1).
fn wt_stream_append(log: &Arc<Mutex<StreamLog>>, bytes: &[u8]) {
    if wedge_trace_on() {
        let t0 = std::time::Instant::now();
        let mut g = log.lock().unwrap();
        let w = t0.elapsed().as_micros() as u64;
        WT_STREAM_APPENDS.fetch_add(1, Ordering::Relaxed);
        WT_STREAM_APPEND_WAIT_US.fetch_add(w, Ordering::Relaxed);
        WT_STREAM_APPEND_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);
        g.append(bytes);
    } else {
        log.lock().unwrap().append(bytes);
    }
}

+/// DIAGNOSTIC (SPT_WEDGE_TRACE): the stream read-pump's StreamLog append, timed.

+/// Measures the lock-acquire wait the forward incurs per chunk — the attacher's

+/// `subscribe_stream` attach() contends on the SAME StreamLog mutex (Theory 1).

+fn wt_stream_append(log: &Arc<Mutex<StreamLog>>, bytes: &[u8]) {

+        let t0 = std::time::Instant::now();

+        let mut g = log.lock().unwrap();

+        let w = t0.elapsed().as_micros() as u64;

+        WT_STREAM_APPENDS.fetch_add(1, Ordering::Relaxed);

+        WT_STREAM_APPEND_WAIT_US.fetch_add(w, Ordering::Relaxed);

+        WT_STREAM_APPEND_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);

+        log.lock().unwrap().append(bytes);

/// The reserved [`crate::effect::EffectKey`] session namespace for net-scoped

/// effects (a dial has no PTY session). Broker session ids are minted from 1

/// (`Broker::next_id` starts at 1), so `0` can never collide with a real session.


Ok(Some(chunk)) => wt_stream_append(&log, &chunk.bytes),


-                    Ok(Some(chunk)) => log.lock().unwrap().append(&chunk.bytes),

+                    Ok(Some(chunk)) => wt_stream_append(&log, &chunk.bytes),


Ok(n) => wt_stream_append(&log, &buf[..n]),


-                        Ok(n) => log.lock().unwrap().append(&buf[..n]),

+                        Ok(n) => wt_stream_append(&log, &buf[..n]),


All probes added. Compile-checking v3.


Bash

cd "C:\Users\decid\Documents\projects\spt-core" && cargo check -p spt-daemon 2>&1 | tail -12; echo "[check $?]"

Compile-check v3 net-layer probes


Command running in background with ID: bmokd9mgu. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bmokd9mgu.output. You will be notified when it completes. To check interim output, use Read on that file path.


<task-notification>
<task-id>bmokd9mgu</task-id>
<tool-use-id>toolu_01CJis4DsAz3bkJzZ7ygiCtn</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bmokd9mgu.output</output-file>
<status>completed</status>
<summary>Background command "Compile-check v3 net-layer probes" completed (exit code 0)</summary>
</task-notification>


1	    Checking spt-daemon v0.12.1 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
2	    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.72s
3	[check 0]
4

    Checking spt-daemon v0.12.1 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.72s
[check 0]


Prompt:
<task-notification>
<task-id>bmokd9mgu</task-id>
<tool-use-id>toolu_01CJis4DsAz3bkJzZ7ygiCtn</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bmokd9mgu.output</output-file>
<status>completed</status>
<summary>Background command "Compile-check v3 net-layer probes" completed (exit code 0)</summary>
</task-notification>


Compiles clean. Committing + pushing v3 (doyle validates on kitsubito immediately; I note it's compile-validated, not yet run-validated, since the session is wrapping).


cd "C:\Users\decid\Documents\projects\spt-core" && git add crates/spt-daemon/src/broker.rs crates/spt-daemon/src/nethost.rs && git commit -q -F - <<'EOF'
DIAGNOSTIC (do not merge): wedge-trace-v3 — net-stream subscribe / StreamLog probes to discriminate lock vs throughput

Off v2 (no_ack base). Adds SPT_WEDGE_TRACE probes on the NET-STREAM attach path — the
attacher's REAL subscribe (net_dial_loopback viewer) — to settle Theory 1 (StreamLog
lock) vs Theory 2 (serde/throughput control-vs-data-plane), which the gdb dump left
open (all broker threads CPU-busy in serde, none lock-blocked; net-stream reply is the
uninstrumented gap):
- broker.rs: wedge_trace_on() + WT_DRAIN_APPENDS made pub(crate) so the net layer
  cross-references flood progress. dispatch_net_stream_subscribe(2205): time
  subscribe_stream enter->exit + ok, vs drain_appends.
- nethost.rs subscribe_stream(1227): streams_lock_wait_us + StreamLog log_lock_wait_us
  + attach_us (Theory 1 smoking gun = HUGE streamlog log_lock_wait).
- nethost.rs the stream read-pump appends (546 QUIC, 562 loopback) routed through
  wt_stream_append: WT_STREAM_APPENDS count + acquire-wait totals (the serve_attach
  forward pressure on the SAME StreamLog mutex the subscribe attaches to).

Read: streamlog_log_lock_wait_us HUGE + stream_append_wait big => Theory 1 (fix the
StreamLog lock). log_lock_wait ~tiny but net_stream_subscribe exit late / subscribed
still false => Theory 2 (control-plane delivery starved by serde — fix = prioritize
control frames / batch serde / coalesce output). Compile-validated (Windows); run on
kitsubito. Off by default, never ships.

Co-authored by: todlando
EOF
git push -u origin wedge-trace-v3 2>&1 | tail -2 && git checkout v0.13.0-p1b-ack-deadlock 2>&1 | tail -1

Commit + push v3, return to delivery branch


To https://github.com/SaberMage/spt-core.git
 * [new branch]      wedge-trace-v3 -> wedge-trace-v3
Switched to branch 'v0.13.0-p1b-ack-deadlock'


v3 pushed. Sending doyle the run command, then communing (session ending).


$OWL send doyle <<'EOF'
wedge-trace-v3 PUSHED (off no_ack base; compile-validated on Windows, NOT run-validated — diagnostic eprintlns only, low risk; if it panics on kitsubito you'll see instantly). Returned to delivery HEAD.

RUN: git fetch && git checkout wedge-trace-v3
  SPT_WEDGE_TRACE=1 cargo nextest run -p spt-daemon --test inject_control_wedge -E 'test(a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach)' --no-capture 2>&1 | grep WEDGE_TRACE
  (and p0)

READ (the discriminator):
- `WEDGE_TRACE subscribe_stream ... streamlog_log_lock_wait_us=N attach_us=M stream_append_wait_total_us=...` — N HUGE (seconds) => THEORY 1: the attacher's attach() starves on the StreamLog mutex behind the read-pump forward append (fix = StreamLog lock: drop-lock-across, or replay off-lock like OutputLog's W1).
- `WEDGE_TRACE net_stream_subscribe stream=.. subscribe_stream_us=K ok=true` with K small + streamlog_log_lock_wait≈0 BUT subscribed STILL false => THEORY 2: the subscribe RESOLVES fine, the wedge is the REPLY/output delivery riding the serde-saturated path (fix = prioritize control-plane / batch serde / coalesce output).
- WT_STREAM_APPENDS = the loopback read-pump forward volume (the pressure).

I'm communing to my Psyche + ending this session now (user wrapping). On resume I'll pick up from your v3 numbers + the named theory. Project memory written (v013-rc-attach-flood-wedge) so next-session-me resumes cold with the full root-cause arc. Over to you for the run. -todlando
EOF

Send v3 run command + handoff note to doyle


➜ SENT:doyle


Now the commune. The Psyche snapshot is stale (ended at "wedge-trace-v2, awaiting 15-thread gdb map"). Writing only the delta since then.


Write

<live-context>
SESSION ENDING — resume handoff. The wedge investigation closed many rounds since the last snapshot (which stopped at "wedge-trace-v2, awaiting doyle's 15-thread gdb map").

WHAT HAPPENED (delta): doyle's gdb maps on kitsubito refuted EVERY lock hypothesis in turn — ACK-on-SendHalf (symptom, not root), CPU/HOL-behind-flood, StreamLog-lock, and PtySession.writer (all red herrings). The clean sleep-600 dump was conclusive: NO thread lock-blocked; ALL broker worker threads CPU-saturated in serde_json serializing the PTY echo-output flood (OutputLog::append→output_envelope→serde at seq ~1.96M; controller_writer→write_frame→serde). The PTY line-discipline ECHOes the input flood into an output flood. So the consolidated root: a concurrent rc --view (net-stream loopback) attach is not serviced+DELIVERED within the 30s watchdog because the control-plane subscribe reply rides the serde-saturated data-plane output path. NOT a lock — a real throughput / control-vs-data-plane-priority product bug. The two-phase CI split did its job: it isolated this real signal from CI load noise.

OPERATOR RULING: keep root-causing + fix (greenlit). I pushed wedge-trace-v3 (off the no_ack base) — it discriminates Theory 1 (StreamLog Mutex contention: subscribe attach() starves behind the read-pump forward append) vs Theory 2 (serde throughput: subscribe resolves fine but reply delivery starves). doyle is running it on kitsubito now.

IMMEDIATE NEXT STEPS (on resume):
1. Read doyle's wedge-trace-v3 numbers. `streamlog_log_lock_wait_us` HUGE => Theory 1 (fix = off-lock StreamLog replay / drop-lock-across, like OutputLog's W1). ~tiny but subscribe still false => Theory 2 (fix = prioritize control-plane frames / batch-serde / coalesce output rate).
2. Apply the GATED design fix (doyle gates the root first; product currently UNTOUCHED — don't patch blind; we burned 4 wrong hypotheses, validate-before-fold caught each).
3. g2_no_commit_deadline is a SEPARATE item (translation::InjectFloor commit-deadline/floor-release on forkpty) — still to dig; held.
4. Once all 3 gates green on kitsubito → doyle folds to delivery-control → deployah cuts v0.13.0.

INTERACTIONS: doyle = the gate/gdb partner (has sudo gdb on kitsubito, the only online Linux box; gravity-linux was offline 5d — my stale-memory head-fake, corrected). Division of labor that worked: doyle runs gdb/repro on kitsubito, I author SPT_WEDGE_TRACE instrumentation + read code + name the root. Instrumentation branches: wedge-trace-instrumentation / -v2 / -v3 (env-gated, do-not-merge).

REFLECTION: a long, honest root-cause grind — five hypotheses proposed and refuted by empirical data before any fix. The discipline (gdb + targeted instrumentation + validate-before-fold) is what kept us from shipping wrong patches. Project memory written (v013-rc-attach-flood-wedge.md) so a cold resume has the full arc.
</live-context>
<project-context>
v0.13.0 release status (branch v0.13.0-p1b-ack-deadlock, HEAD 71378d8): BLOCKED on 3 failing `inject_control_wedge` gates, now root-caused to a REAL product throughput bug — a concurrent rc --view attach (net-stream loopback viewer → dispatch_net_stream_subscribe → NetHost::subscribe_stream) is not delivered within 30s while the session emits a sustained PTY output flood; the broker CPU-saturates serializing every output envelope (serde_json per chunk) and the control-plane subscribe reply has no priority over the data-plane flood. The test invariant is VALID (a viewer must attach to a busy session); the bug is product-side throughput / control-vs-data-plane priority, not the test.

SHIPPED this cycle (on the delivery branch): (1) the two-phase nextest CI split — Phase A runs the light pool at full parallelism, then Phase B runs the heavy real-broker/PTY/daemon-tree class serialized on a now-quiet box — which structurally fixed the load-induced flakes AND isolated the 3 genuine product wedges; HEAVY filter = 39 binaries/91 tests, byte-identical across .config/nextest.toml + both ci.yml -E phases; (2) --no-fail-fast on both phases; (3) a robust deadline-poll fix for spawn_env_reaches_child (was a real read-race); (4) a Linux pre-checkout reap of `yes FLOOD` forkpty orphans; (5) job_escape_e2e reap-poll 10s→25s. Commit 71378d8 swapped the wedge drivers to send_effect_no_ack (the real ackless operator seam, REQ-HAZARD-INPUT-ACK-BACKPRESSURE) — correct seam-realism but NOT the fix (the wedge persists ackless).

END GOAL: cut v0.13.0. Path = fix the rc-attach-under-output-flood throughput bug (Theory 1 StreamLog lock vs Theory 2 serde/control-plane — pending wedge-trace-v3 numbers) + resolve g2 (separate InjectFloor commit-deadline item) → all 3 gates green on kitsubito (both runners x3) → doyle folds to delivery-control → deployah cuts the release. Then post-v0.13.0 follow-ons. Product code must stay untouched until doyle gates the chosen root (four prior hypotheses were empirically refuted).
</project-context>


File created successfully at: C:\Users\decid\Documents\projects\spt-core\.claude\todlando-commune.md (file state is current in your context — no need to Read it back)

<live-context>
SESSION ENDING — resume handoff. The wedge investigation closed many rounds since the last snapshot (which stopped at "wedge-trace-v2, awaiting doyle's 15-thread gdb map").

WHAT HAPPENED (delta): doyle's gdb maps on kitsubito refuted EVERY lock hypothesis in turn — ACK-on-SendHalf (symptom, not root), CPU/HOL-behind-flood, StreamLog-lock, and PtySession.writer (all red herrings). The clean sleep-600 dump was conclusive: NO thread lock-blocked; ALL broker worker threads CPU-saturated in serde_json serializing the PTY echo-output flood (OutputLog::append→output_envelope→serde at seq ~1.96M; controller_writer→write_frame→serde). The PTY line-discipline ECHOes the input flood into an output flood. So the consolidated root: a concurrent rc --view (net-stream loopback) attach is not serviced+DELIVERED within the 30s watchdog because the control-plane subscribe reply rides the serde-saturated data-plane output path. NOT a lock — a real throughput / control-vs-data-plane-priority product bug. The two-phase CI split did its job: it isolated this real signal from CI load noise.

OPERATOR RULING: keep root-causing + fix (greenlit). I pushed wedge-trace-v3 (off the no_ack base) — it discriminates Theory 1 (StreamLog Mutex contention: subscribe attach() starves behind the read-pump forward append) vs Theory 2 (serde throughput: subscribe resolves fine but reply delivery starves). doyle is running it on kitsubito now.

IMMEDIATE NEXT STEPS (on resume):
1. Read doyle's wedge-trace-v3 numbers. `streamlog_log_lock_wait_us` HUGE => Theory 1 (fix = off-lock StreamLog replay / drop-lock-across, like OutputLog's W1). ~tiny but subscribe still false => Theory 2 (fix = prioritize control-plane frames / batch-serde / coalesce output rate).
2. Apply the GATED design fix (doyle gates the root first; product currently UNTOUCHED — don't patch blind; we burned 4 wrong hypotheses, validate-before-fold caught each).
3. g2_no_commit_deadline is a SEPARATE item (translation::InjectFloor commit-deadline/floor-release on forkpty) — still to dig; held.
4. Once all 3 gates green on kitsubito → doyle folds to delivery-control → deployah cuts v0.13.0.

INTERACTIONS: doyle = the gate/gdb partner (has sudo gdb on kitsubito, the only online Linux box; gravity-linux was offline 5d — my stale-memory head-fake, corrected). Division of labor that worked: doyle runs gdb/repro on kitsubito, I author SPT_WEDGE_TRACE instrumentation + read code + name the root. Instrumentation branches: wedge-trace-instrumentation / -v2 / -v3 (env-gated, do-not-merge).

REFLECTION: a long, honest root-cause grind — five hypotheses proposed and refuted by empirical data before any fix. The discipline (gdb + targeted instrumentation + validate-before-fold) is what kept us from shipping wrong patches. Project memory written (v013-rc-attach-flood-wedge.md) so a cold resume has the full arc.
</live-context>
<project-context>
v0.13.0 release status (branch v0.13.0-p1b-ack-deadlock, HEAD 71378d8): BLOCKED on 3 failing `inject_control_wedge` gates, now root-caused to a REAL product throughput bug — a concurrent rc --view attach (net-stream loopback viewer → dispatch_net_stream_subscribe → NetHost::subscribe_stream) is not delivered within 30s while the session emits a sustained PTY output flood; the broker CPU-saturates serializing every output envelope (serde_json per chunk) and the control-plane subscribe reply has no priority over the data-plane flood. The test invariant is VALID (a viewer must attach to a busy session); the bug is product-side throughput / control-vs-data-plane priority, not the test.

SHIPPED this cycle (on the delivery branch): (1) the two-phase nextest CI split — Phase A runs the light pool at full parallelism, then Phase B runs the heavy real-broker/PTY/daemon-tree class serialized on a now-quiet box — which structurally fixed the load-induced flakes AND isolated the 3 genuine product wedges; HEAVY filter = 39 binaries/91 tests, byte-identical across .config/nextest.toml + both ci.yml -E phases; (2) --no-fail-fast on both phases; (3) a robust deadline-poll fix for spawn_env_reaches_child (was a real read-race); (4) a Linux pre-checkout reap of `yes FLOOD` forkpty orphans; (5) job_escape_e2e reap-poll 10s→25s. Commit 71378d8 swapped the wedge drivers to send_effect_no_ack (the real ackless operator seam, REQ-HAZARD-INPUT-ACK-BACKPRESSURE) — correct seam-realism but NOT the fix (the wedge persists ackless).

END GOAL: cut v0.13.0. Path = fix the rc-attach-under-output-flood throughput bug (Theory 1 StreamLog lock vs Theory 2 serde/control-plane — pending wedge-trace-v3 numbers) + resolve g2 (separate InjectFloor commit-deadline item) → all 3 gates green on kitsubito (both runners x3) → doyle folds to delivery-control → deployah cuts the release. Then post-v0.13.0 follow-ons. Product code must stay untouched until doyle gates the chosen root (four prior hypotheses were empirically refuted).
</project-context>


Commune written to `.claude/todlando-commune.md` (delta only — Self listener will ingest + delete it). Session wrapped.

**Done this turn:**
- Operator greenlit keep-fixing → pushed **wedge-trace-v3** (discriminates Theory 1 StreamLog-lock vs Theory 2 serde-throughput); compile-clean. doyle running it on kitsubito now.
- Sent doyle the run command + read-key + handoff.
- Communed the delta to Psyche; project memory (`v013-rc-attach-flood-wedge.md`) already written.

**Status for next session:**
- **Project:** spt-core, cutting **v0.13.0** (branch `v0.13.0-p1b-ack-deadlock`, HEAD 71378d8).
- **Shipped:** two-phase nextest CI split (Phase A light-parallel → Phase B heavy-serial-on-quiet-box) — fixed load flakes AND isolated 3 genuine product wedges.
- **Open blocker:** the 3 `inject_control_wedge` gates fail on a **real throughput bug** — a concurrent rc --view attach starves under sustained PTY output flood (broker CPU-saturates serializing output envelopes; control-plane reply has no priority over data-plane). Five lock hypotheses refuted by gdb; product untouched.
- **Resume:** read doyle's v3 numbers → name Theory 1 vs 2 → apply gated design fix → dig g2 (separate InjectFloor item) → all 3 green → fold → **deployah cuts v0.13.0** (end goal).

Caveman out. Resume next session from doyle's v3 verdict.