Bash

$OWL send doyle <<'EOF'
ACK — 3 confirmed REAL PRODUCT BUGS, isolated zero-load on kitsubito. Evidence accepted. Writing the SPT_WEDGE_TRACE instrumentation NOW; pushing to a throwaway branch off the test HEAD so you check it out + run on kitsubito (don't need my own access yet — agreed).

CODE-GROUNDED LEAD for your parallel gdb cut (so you know what to look for): suspect b is mechanically supported by the source —
- DRAIN holder: broker.rs:1451 `log_drain.lock().unwrap().append(chunk)` runs for EVERY flood-output chunk from `yes FLOOD` on the session's single drain thread.
- SUBSCRIBE waiter: broker.rs:1584 `log.lock().unwrap().resolve_subscribe(...)` — the concurrent attach must take the SAME OutputLog mutex to register+replay before the reply (1590).
std Mutex is UNFAIR on Linux; a quiet box floods MORE output -> the drain reacquires the lock thousands of times/sec -> the one-shot subscribe acquire starves = subscribed=false. Output-volume-driven => worse-on-quiet, exactly the observed inversion.
So in your `thread apply all bt`: the dispatch/subscribe thread parked in a FUTEX wait whose lock addr == the drain thread's held mutex == CONFIRMS suspect b at broker.rs:1584. A write()/poll() syscall instead => different root.

Instrumentation will print: subscribe lock_wait_us (the smoking gun), drain append-count + acquire-wait totals (the pressure), and the sessions-lock wait (suspect c control). Pushing shortly. -todlando
EOF

Ack doyle + send code-grounded gdb lead


➜ SENT:doyle


Writing the instrumentation. Finding a clean insertion anchor for the helper after the imports.


66	};
67	use crate::nethost::{NetHost, NET_EFFECT_SESSION};
68	use crate::translation::{key_to_bytes, InjectFloor, KeyCmd, ToBinary, TranslationChild};
69	use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};
70
71	/// The shared, serialized send side of one brain connection. Output frames,
72	/// replay frames, command acks, and exit/error events all write through this.
73	/// `pub(crate)`: the net host's stream logs (D4b) live-send through the same
74	/// handle under the same lock discipline.
75	pub(crate) type SharedSend = Arc<Mutex<SendHalf>>;
76
77	/// Default bounded output-log depth (chunks). Large enough that a brain restart

};
use crate::nethost::{NetHost, NET_EFFECT_SESSION};
use crate::translation::{key_to_bytes, InjectFloor, KeyCmd, ToBinary, TranslationChild};
use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};

/// The shared, serialized send side of one brain connection. Output frames,
/// replay frames, command acks, and exit/error events all write through this.
/// `pub(crate)`: the net host's stream logs (D4b) live-send through the same
/// handle under the same lock discipline.
pub(crate) type SharedSend = Arc<Mutex<SendHalf>>;

/// Default bounded output-log depth (chunks). Large enough that a brain restart


Edit

use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};

/// The shared, serialized send side of one brain connection. Output frames,

use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};

// ── DIAGNOSTIC-ONLY: sustained-input-wedge instrumentation (SPT_WEDGE_TRACE=1) ──
// Off by default; behaviour-neutral; never ships a logic change. Measures whether
// a concurrent attach's `subscribe` starves on the OutputLog mutex behind the
// drain thread's flood-output `append` (REQ-HAZARD-EFFECT-JOURNAL-PTY-WEDGE /
// REQ-HAZARD-PTY-INPUT-WRITER-WEDGE root-cause hunt). Run the inject_control_wedge
// gates on a forkpty box with the env set and read the WEDGE_TRACE lines.
fn wedge_trace_on() -> bool {
    use std::sync::OnceLock;
    static ON: OnceLock<bool> = OnceLock::new();
    *ON.get_or_init(|| std::env::var_os("SPT_WEDGE_TRACE").is_some())
}
/// Drain-thread OutputLog-lock pressure counters (only touched when tracing).
static WT_DRAIN_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_DRAIN_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_DRAIN_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// The shared, serialized send side of one brain connection. Output frames,


The file C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon\src\broker.rs has been updated successfully. (file state is current in your context — no need to Read it back)

use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};

/// The shared, serialized send side of one brain connection. Output frames,

use crate::transport::{recv_hello, DaemonTransport, LocalSocketTransport};

// ── DIAGNOSTIC-ONLY: sustained-input-wedge instrumentation (SPT_WEDGE_TRACE=1) ──
// Off by default; behaviour-neutral; never ships a logic change. Measures whether
// a concurrent attach's `subscribe` starves on the OutputLog mutex behind the
// drain thread's flood-output `append` (REQ-HAZARD-EFFECT-JOURNAL-PTY-WEDGE /
// REQ-HAZARD-PTY-INPUT-WRITER-WEDGE root-cause hunt). Run the inject_control_wedge
// gates on a forkpty box with the env set and read the WEDGE_TRACE lines.
fn wedge_trace_on() -> bool {
    use std::sync::OnceLock;
    static ON: OnceLock<bool> = OnceLock::new();
    *ON.get_or_init(|| std::env::var_os("SPT_WEDGE_TRACE").is_some())
}
/// Drain-thread OutputLog-lock pressure counters (only touched when tracing).
static WT_DRAIN_APPENDS: AtomicU64 = AtomicU64::new(0);
static WT_DRAIN_WAIT_US: AtomicU64 = AtomicU64::new(0);
static WT_DRAIN_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// The shared, serialized send side of one brain connection. Output frames,

use crate::translation::{key_to_bytes, InjectFloor, KeyCmd, ToBinary, TranslationChild};

+// ── DIAGNOSTIC-ONLY: sustained-input-wedge instrumentation (SPT_WEDGE_TRACE=1) ──

+// Off by default; behaviour-neutral; never ships a logic change. Measures whether

+// a concurrent attach's `subscribe` starves on the OutputLog mutex behind the

+// drain thread's flood-output `append` (REQ-HAZARD-EFFECT-JOURNAL-PTY-WEDGE /

+// REQ-HAZARD-PTY-INPUT-WRITER-WEDGE root-cause hunt). Run the inject_control_wedge

+// gates on a forkpty box with the env set and read the WEDGE_TRACE lines.

+    static ON: OnceLock<bool> = OnceLock::new();

+    *ON.get_or_init(|| std::env::var_os("SPT_WEDGE_TRACE").is_some())

+/// Drain-thread OutputLog-lock pressure counters (only touched when tracing).

+static WT_DRAIN_APPENDS: AtomicU64 = AtomicU64::new(0);

+static WT_DRAIN_WAIT_US: AtomicU64 = AtomicU64::new(0);

+static WT_DRAIN_WAIT_MAX_US: AtomicU64 = AtomicU64::new(0);

/// replay frames, command acks, and exit/error events all write through this.

/// `pub(crate)`: the net host's stream logs (D4b) live-send through the same


Now the drain probe (holder side, @1451):


        let log_drain = Arc::clone(&log);
        let drain = session
            .drain(move |chunk| {
                let job = log_drain.lock().unwrap().append(chunk);
                if let Some(job) = job {
                    if let Err(epoch) = job.deliver() {
                        log_drain.lock().unwrap().mark_controller_gone(epoch);
                    }
                }
            })

        let log_drain = Arc::clone(&log);
        let drain = session
            .drain(move |chunk| {
                // DIAGNOSTIC (SPT_WEDGE_TRACE): time how long THIS drain thread waits
                // to (re)acquire the OutputLog mutex per flood-output chunk, and how
                // often — the pressure side of the subscribe-starvation hypothesis.
                let job = if wedge_trace_on() {
                    let t0 = std::time::Instant::now();
                    let mut g = log_drain.lock().unwrap();
                    let w = t0.elapsed().as_micros() as u64;
                    let n = WT_DRAIN_APPENDS.fetch_add(1, Ordering::Relaxed) + 1;
                    WT_DRAIN_WAIT_US.fetch_add(w, Ordering::Relaxed);
                    WT_DRAIN_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);
                    if n % 2000 == 0 {
                        eprintln!(
                            "WEDGE_TRACE drain appends={n} acquire_wait_total_us={} acquire_wait_max_us={}",
                            WT_DRAIN_WAIT_US.load(Ordering::Relaxed),
                            WT_DRAIN_WAIT_MAX_US.load(Ordering::Relaxed),
                        );
                    }
                    g.append(chunk)
                } else {
                    log_drain.lock().unwrap().append(chunk)
                };
                if let Some(job) = job {
                    if let Err(epoch) = job.deliver() {
                        log_drain.lock().unwrap().mark_controller_gone(epoch);
                    }
                }
            })


        let log_drain = Arc::clone(&log);
        let drain = session
            .drain(move |chunk| {
                let job = log_drain.lock().unwrap().append(chunk);
                if let Some(job) = job {
                    if let Err(epoch) = job.deliver() {
                        log_drain.lock().unwrap().mark_controller_gone(epoch);
                    }
                }
            })

        let log_drain = Arc::clone(&log);
        let drain = session
            .drain(move |chunk| {
                // DIAGNOSTIC (SPT_WEDGE_TRACE): time how long THIS drain thread waits
                // to (re)acquire the OutputLog mutex per flood-output chunk, and how
                // often — the pressure side of the subscribe-starvation hypothesis.
                let job = if wedge_trace_on() {
                    let t0 = std::time::Instant::now();
                    let mut g = log_drain.lock().unwrap();
                    let w = t0.elapsed().as_micros() as u64;
                    let n = WT_DRAIN_APPENDS.fetch_add(1, Ordering::Relaxed) + 1;
                    WT_DRAIN_WAIT_US.fetch_add(w, Ordering::Relaxed);
                    WT_DRAIN_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);
                    if n % 2000 == 0 {
                        eprintln!(
                            "WEDGE_TRACE drain appends={n} acquire_wait_total_us={} acquire_wait_max_us={}",
                            WT_DRAIN_WAIT_US.load(Ordering::Relaxed),
                            WT_DRAIN_WAIT_MAX_US.load(Ordering::Relaxed),
                        );
                    }
                    g.append(chunk)
                } else {
                    log_drain.lock().unwrap().append(chunk)
                };
                if let Some(job) = job {
                    if let Err(epoch) = job.deliver() {
                        log_drain.lock().unwrap().mark_controller_gone(epoch);
                    }
                }
            })

-                let job = log_drain.lock().unwrap().append(chunk);

+                // DIAGNOSTIC (SPT_WEDGE_TRACE): time how long THIS drain thread waits

+                // to (re)acquire the OutputLog mutex per flood-output chunk, and how

+                // often — the pressure side of the subscribe-starvation hypothesis.

+                let job = if wedge_trace_on() {

+                    let t0 = std::time::Instant::now();

+                    let mut g = log_drain.lock().unwrap();

+                    let w = t0.elapsed().as_micros() as u64;

+                    let n = WT_DRAIN_APPENDS.fetch_add(1, Ordering::Relaxed) + 1;

+                    WT_DRAIN_WAIT_US.fetch_add(w, Ordering::Relaxed);

+                    WT_DRAIN_WAIT_MAX_US.fetch_max(w, Ordering::Relaxed);

+                            "WEDGE_TRACE drain appends={n} acquire_wait_total_us={} acquire_wait_max_us={}",

+                            WT_DRAIN_WAIT_US.load(Ordering::Relaxed),

+                            WT_DRAIN_WAIT_MAX_US.load(Ordering::Relaxed),

+                    log_drain.lock().unwrap().append(chunk)


Now the subscribe waiter probe (@1577-1591) — sessions-lock wait (suspect c) + the log-lock wait smoking gun + resolve duration + reply:


        let log = {
            let sessions = self.sessions.lock().unwrap();
            let h = sessions
                .get(&req.session_id)
                .ok_or_else(|| format!("no such session {}", req.session_id))?;
            Arc::clone(&h.log)
        };
        let outcome = log.lock().unwrap().resolve_subscribe(
            Arc::clone(send),
            req.from_seq,
            req.intent,
            req.by,
        );
        send_frame(send, &subscribed_envelope(req.session_id, outcome));
        Ok(req.session_id)

        // DIAGNOSTIC (SPT_WEDGE_TRACE): this is the concurrent attach's subscribe.
        // Time the two locks it must take — the brief `sessions` map lock (suspect c,
        // shared with input/drain) and then the OutputLog mutex (suspect b, held by
        // the drain per flood-output chunk) — plus the in-lock resolve+replay. If
        // `log_lock_wait_us` blows up while `drain_appends_so_far` is large, the
        // subscribe is starving on the OutputLog mutex behind the flood drain.
        let traced = wedge_trace_on();
        let sid = req.session_id;
        let log = {
            let ts = std::time::Instant::now();
            let sessions = self.sessions.lock().unwrap();
            if traced {
                eprintln!(
                    "WEDGE_TRACE subscribe sid={sid} sessions_lock_wait_us={}",
                    ts.elapsed().as_micros()
                );
            }
            let h = sessions
                .get(&req.session_id)
                .ok_or_else(|| format!("no such session {}", req.session_id))?;
            Arc::clone(&h.log)
        };
        let outcome = if traced {
            let t0 = std::time::Instant::now();
            let mut g = log.lock().unwrap();
            let log_lock_wait_us = t0.elapsed().as_micros();
            let t1 = std::time::Instant::now();
            let o = g.resolve_subscribe(Arc::clone(send), req.from_seq, req.intent, req.by);
            drop(g);
            eprintln!(
                "WEDGE_TRACE subscribe sid={sid} log_lock_wait_us={log_lock_wait_us} \
                 resolve_us={} drain_appends_so_far={}",
                t1.elapsed().as_micros(),
                WT_DRAIN_APPENDS.load(Ordering::Relaxed),
            );
            o
        } else {
            log.lock().unwrap().resolve_subscribe(
                Arc::clone(send),
                req.from_seq,
                req.intent,
                req.by,
            )
        };
        let t_reply = std::time::Instant::now();
        send_frame(send, &subscribed_envelope(req.session_id, outcome));
        if traced {
            eprintln!(
                "WEDGE_TRACE subscribe sid={sid} reply_send_us={}",
                t_reply.elapsed().as_micros()
            );
        }
        Ok(req.session_id)


        let log = {
            let sessions = self.sessions.lock().unwrap();
            let h = sessions
                .get(&req.session_id)
                .ok_or_else(|| format!("no such session {}", req.session_id))?;
            Arc::clone(&h.log)
        };
        let outcome = log.lock().unwrap().resolve_subscribe(
            Arc::clone(send),
            req.from_seq,
            req.intent,
            req.by,
        );
        send_frame(send, &subscribed_envelope(req.session_id, outcome));
        Ok(req.session_id)

        // DIAGNOSTIC (SPT_WEDGE_TRACE): this is the concurrent attach's subscribe.
        // Time the two locks it must take — the brief `sessions` map lock (suspect c,
        // shared with input/drain) and then the OutputLog mutex (suspect b, held by
        // the drain per flood-output chunk) — plus the in-lock resolve+replay. If
        // `log_lock_wait_us` blows up while `drain_appends_so_far` is large, the
        // subscribe is starving on the OutputLog mutex behind the flood drain.
        let traced = wedge_trace_on();
        let sid = req.session_id;
        let log = {
            let ts = std::time::Instant::now();
            let sessions = self.sessions.lock().unwrap();
            if traced {
                eprintln!(
                    "WEDGE_TRACE subscribe sid={sid} sessions_lock_wait_us={}",
                    ts.elapsed().as_micros()
                );
            }
            let h = sessions
                .get(&req.session_id)
                .ok_or_else(|| format!("no such session {}", req.session_id))?;
            Arc::clone(&h.log)
        };
        let outcome = if traced {
            let t0 = std::time::Instant::now();
            let mut g = log.lock().unwrap();
            let log_lock_wait_us = t0.elapsed().as_micros();
            let t1 = std::time::Instant::now();
            let o = g.resolve_subscribe(Arc::clone(send), req.from_seq, req.intent, req.by);
            drop(g);
            eprintln!(
                "WEDGE_TRACE subscribe sid={sid} log_lock_wait_us={log_lock_wait_us} \
                 resolve_us={} drain_appends_so_far={}",
                t1.elapsed().as_micros(),
                WT_DRAIN_APPENDS.load(Ordering::Relaxed),
            );
            o
        } else {
            log.lock().unwrap().resolve_subscribe(
                Arc::clone(send),
                req.from_seq,
                req.intent,
                req.by,
            )
        };
        let t_reply = std::time::Instant::now();
        send_frame(send, &subscribed_envelope(req.session_id, outcome));
        if traced {
            eprintln!(
                "WEDGE_TRACE subscribe sid={sid} reply_send_us={}",
                t_reply.elapsed().as_micros()
            );
        }
        Ok(req.session_id)

// post-removal subscribe gets a prompt "no such session" error — both

// non-silent. The rc-side status-gate (a) + first-event backstop (b) cover

+        // DIAGNOSTIC (SPT_WEDGE_TRACE): this is the concurrent attach's subscribe.

+        // Time the two locks it must take — the brief `sessions` map lock (suspect c,

+        // shared with input/drain) and then the OutputLog mutex (suspect b, held by

+        // the drain per flood-output chunk) — plus the in-lock resolve+replay. If

+        // `log_lock_wait_us` blows up while `drain_appends_so_far` is large, the

+        // subscribe is starving on the OutputLog mutex behind the flood drain.

+            let ts = std::time::Instant::now();

+                    "WEDGE_TRACE subscribe sid={sid} sessions_lock_wait_us={}",

+                    ts.elapsed().as_micros()

-        let outcome = log.lock().unwrap().resolve_subscribe(

+            let t0 = std::time::Instant::now();

+            let mut g = log.lock().unwrap();

+            let log_lock_wait_us = t0.elapsed().as_micros();

+            let t1 = std::time::Instant::now();

+            let o = g.resolve_subscribe(Arc::clone(send), req.from_seq, req.intent, req.by);

+                "WEDGE_TRACE subscribe sid={sid} log_lock_wait_us={log_lock_wait_us} \

+                 resolve_us={} drain_appends_so_far={}",

+                t1.elapsed().as_micros(),

+                WT_DRAIN_APPENDS.load(Ordering::Relaxed),

+            log.lock().unwrap().resolve_subscribe(

+        let t_reply = std::time::Instant::now();

+                "WEDGE_TRACE subscribe sid={sid} reply_send_us={}",

+                t_reply.elapsed().as_micros()


Targets 1/3/4 instrumented (subscribe recv→reply window, OutputLog mutex both sides, sessions-lock wait). Target 2 (HOL) is covered by elimination — the subscribe is on its own conn/thread, and the `drain_appends_so_far` counter quantifies the flood; I'll note that to doyle rather than bloat the hot input path. Compile-checking the instrumentation.


Bash

cd "C:\Users\decid\Documents\projects\spt-core" && cargo check -p spt-daemon 2>&1 | tail -8; echo "[check $?]"

Compile-check broker instrumentation


<owl_messages>
<owl_message from="doyle" priority="high">ACK — suspect-b code grounding is excellent (drain 1451 append under log_drain.lock vs subscribe 1584 resolve_subscribe under same OutputLog mutex; std Mutex unfair on Linux; output-volume-driven = worse-on-quiet = matches the inversion). passwordless sudo confirmed on kitsubito so I can sudo gdb -p past ptrace_scope=1 — running the live thread-dump NOW to confirm the subscribe thread is futex-parked on the drain's held mutex. Keep writing SPT_WEDGE_TRACE as the precise numeric confirm regardless; my gdb cut is the fast localize. Results incoming. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


Command running in background with ID: bo3yff3jj. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bo3yff3jj.output. You will be notified when it completes. To check interim output, use Read on that file path.


<task-notification>
<task-id>bo3yff3jj</task-id>
<tool-use-id>toolu_013z9mQcvPv5FEm6WvF4cXCe</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bo3yff3jj.output</output-file>
<status>completed</status>
<summary>Background command "Compile-check broker instrumentation" completed (exit code 0)</summary>
</task-notification>


doyle (high-priority) acked: passwordless sudo on kitsubito → running live `sudo gdb -p` thread-dump now to confirm the subscribe thread is futex-parked on the drain's held mutex. Keep SPT_WEDGE_TRACE as the precise numeric confirm. His gdb localize incoming.

My compile-check is running. Quick ack to doyle, then read it.


$OWL send doyle <<'EOF'
ACK — gdb live thread-dump is the right fast localize; SPT_WEDGE_TRACE coming as the numeric confirm. WHAT TO LOOK FOR in your `thread apply all bt`:
- The SUBSCRIBE/dispatch thread: parked in `__lll_lock_wait` / futex under `std::sync::Mutex::lock` inside `Broker::dispatch_subscribe` -> `OutputLog::resolve_subscribe` (broker.rs:1584). That = suspect b CONFIRMED.
- The DRAIN thread: should be the lock HOLDER — runnable/looping in `OutputLog::append` (broker.rs:1451) or just-released, cycling fast. Cross-check the futex addr: subscribe's wait-addr == the Mutex the drain thread holds.
- If instead the subscribe thread is in a `write()`/`sendto()`/`poll()` syscall (not futex), it's NOT lock starvation -> different root (reply-send backpressure or the attacher-side never sending the subscribe).
- BONUS: g2 (op_flushed=false) — if its parked thread differs, it may be the commit-deadline/floor path, not the subscribe mutex; note where ITS dispatch thread sits.

My instrumentation compile-check is running; pushing the SPT_WEDGE_TRACE branch next so you can run the numeric pass after the gdb cut. Standing by for your stacks. -todlando
EOF

Ack doyle + gdb read guidance


Prompt:
<task-notification>
<task-id>bo3yff3jj</task-id>
<tool-use-id>toolu_013z9mQcvPv5FEm6WvF4cXCe</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\bo3yff3jj.output</output-file>
<status>completed</status>
<summary>Background command "Compile-check broker instrumentation" completed (exit code 0)</summary>
</task-notification>


Read compile-check result


    Checking spt-daemon v0.12.1 (C:\Users\decid\Documents\projects\spt-core\crates\spt-daemon)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.85s
[check 0]


Instrumentation compiles clean. Validating the trace format on Windows (won't wedge, but should print WEDGE_TRACE lines + no panic), then push to a throwaway branch for doyle.


cd "C:\Users\decid\Documents\projects\spt-core" && SPT_WEDGE_TRACE=1 cargo nextest run -p spt-daemon --test inject_control_wedge -E 'test(a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach)' --no-capture 2>&1 | grep -E "WEDGE_TRACE|PASS|FAIL" | head -20

Traced a_journaled run on Windows, check format


Command running in background with ID: b043ot8jh. Output is being written to: C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\b043ot8jh.output. You will be notified when it completes. To check interim output, use Read on that file path.


<system-reminder>Warning: the file exists but is shorter than the provided offset (1). The file has 1 lines.</system-reminder>


Recompiling the test with the instrumentation. Awaiting the trace-format validation.


<task-notification>
<task-id>b043ot8jh</task-id>
<tool-use-id>toolu_01U1CHySjs6FxxBUwp2ZAboy</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\b043ot8jh.output</output-file>
<status>completed</status>
<summary>Background command "Traced a_journaled run on Windows, check format" completed (exit code 0)</summary>
</task-notification>


Compiling. While waiting — plan: create throwaway branch `wedge-trace-instrumentation` off the delivery HEAD, commit the diagnostic there (keeping the delivery branch clean of throwaway instrumentation), push, give doyle the run command, then return to the delivery branch.


1	WEDGE_TRACE subscribe sid=1 sessions_lock_wait_us=0
2	WEDGE_TRACE subscribe sid=1 log_lock_wait_us=0 resolve_us=332 drain_appends_so_far=0
3	WEDGE_TRACE subscribe sid=1 reply_send_us=35
4	WEDGE_TRACE subscribe sid=1 sessions_lock_wait_us=0
5	WEDGE_TRACE subscribe sid=1 log_lock_wait_us=0 resolve_us=768 drain_appends_so_far=213
6	WEDGE_TRACE subscribe sid=1 reply_send_us=32
7	        PASS [   3.084s] (1/1) spt-daemon::inject_control_wedge a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach
8

WEDGE_TRACE subscribe sid=1 sessions_lock_wait_us=0
WEDGE_TRACE subscribe sid=1 log_lock_wait_us=0 resolve_us=332 drain_appends_so_far=0
WEDGE_TRACE subscribe sid=1 reply_send_us=35
WEDGE_TRACE subscribe sid=1 sessions_lock_wait_us=0
WEDGE_TRACE subscribe sid=1 log_lock_wait_us=0 resolve_us=768 drain_appends_so_far=213
WEDGE_TRACE subscribe sid=1 reply_send_us=32
        PASS [   3.084s] (1/1) spt-daemon::inject_control_wedge a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach


Prompt:
<task-notification>
<task-id>b043ot8jh</task-id>
<tool-use-id>toolu_01U1CHySjs6FxxBUwp2ZAboy</tool-use-id>
<output-file>C:\Users\decid\AppData\Local\Temp\claude\C--Users-decid-Documents-projects-spt-core\97787e5e-64b2-4afd-b3fe-becaa2ea8030\tasks\b043ot8jh.output</output-file>
<status>completed</status>
<summary>Background command "Traced a_journaled run on Windows, check format" completed (exit code 0)</summary>
</task-notification>


Trace format validated on Windows: prints cleanly, no panic, test still PASSES (behavior-neutral). As expected for Windows — `log_lock_wait_us=0`, `drain_appends_so_far=213` (ConPTY absorbs → low flood, no starvation). On Linux forkpty the `yes FLOOD` will drive thousands of appends; if `log_lock_wait_us` then blows up → suspect b confirmed. Two subscribes traced (initial + re-subscribe).

Pushing to throwaway branch for doyle.


cd "C:\Users\decid\Documents\projects\spt-core" && git stash list >/dev/null 2>&1; echo "dirty:"; git status -s; git checkout -b wedge-trace-instrumentation 2>&1 | tail -2 && git add crates/spt-daemon/src/broker.rs && git commit -q -F - <<'EOF'
DIAGNOSTIC (do not merge): SPT_WEDGE_TRACE instrumentation for the sustained-input-wedge root-cause

Env-gated (SPT_WEDGE_TRACE=1), off by default, behaviour-neutral — measures whether
the concurrent attach's `subscribe` starves on the OutputLog mutex behind the drain
thread's flood-output `append`. Throwaway branch off the test HEAD for doyle to run
the 3 confirmed-failing gates on kitsubito; not for v0.13.0.

Probes (broker.rs):
- dispatch_subscribe (1584): sessions_lock_wait_us (suspect c), log_lock_wait_us
  (suspect b — the smoking gun), resolve_us, reply_send_us, and drain_appends_so_far
  (flood volume at the moment of subscribe).
- drain append (1451): per-chunk OutputLog acquire-wait, summarized every 2000
  appends (acquire_wait_total_us / _max_us) — the holder-side pressure.

Run: SPT_WEDGE_TRACE=1 cargo nextest run -p spt-daemon --test inject_control_wedge \
  -E 'test(a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach)' --no-capture
(then p0_paste_wedge_..., g2_no_commit_deadline_...). Read the WEDGE_TRACE lines.

Co-authored by: todlando
EOF
git push -u origin wedge-trace-instrumentation 2>&1 | tail -3

Commit + push instrumentation to throwaway branch


dirty:
 M crates/spt-daemon/src/broker.rs
Switched to a new branch 'wedge-trace-instrumentation'
branch 'wedge-trace-instrumentation' set up to track 'origin/wedge-trace-instrumentation'.
To https://github.com/SaberMage/spt-core.git
 * [new branch]      wedge-trace-instrumentation -> wedge-trace-instrumentation


Pushed `wedge-trace-instrumentation`. Returning to the delivery branch (keep it clean), then giving doyle the run commands + interpretation key.


cd "C:\Users\decid\Documents\projects\spt-core" && git checkout v0.13.0-p1b-ack-deadlock 2>&1 | tail -2; git status -sb | head -1

Return to delivery branch, confirm clean


Your branch is up to date with 'origin/v0.13.0-p1b-ack-deadlock'.
Switched to branch 'v0.13.0-p1b-ack-deadlock'
## v0.13.0-p1b-ack-deadlock...origin/v0.13.0-p1b-ack-deadlock


$OWL send doyle <<'EOF'
INSTRUMENTATION PUSHED -> branch `wedge-trace-instrumentation` (off the test HEAD 18416fd; throwaway, do-not-merge, diagnostic-only). Compiles clean; validated on Windows (prints + no panic + test still PASS = behaviour-neutral).

RUN ON KITSUBITO (warm; --no-capture to see the eprintln stderr):
  git fetch && git checkout wedge-trace-instrumentation
  SPT_WEDGE_TRACE=1 cargo nextest run -p spt-daemon --test inject_control_wedge -E 'test(a_journaled_input_wedge_does_not_starve_a_concurrent_rc_attach)' --no-capture
  (repeat for test(p0_paste_wedge_parked_write_does_not_starve_attach_or_wedge_broker) and test(g2_no_commit_deadline_faults_binary_and_does_not_wedge_controller_input))

READ THESE LINES:
- `WEDGE_TRACE subscribe ... log_lock_wait_us=N drain_appends_so_far=M` — THE smoking gun. SUSPECT B CONFIRMED if log_lock_wait_us is huge (>>1e6 = seconds) while drain_appends_so_far is large (thousands). If log_lock_wait_us≈0 -> NOT lock starvation, look elsewhere (resolve_us, reply_send_us, or the subscribe line never prints = frame never reached dispatch_subscribe).
- `WEDGE_TRACE drain appends=K acquire_wait_total_us=.. acquire_wait_max_us=..` (every 2000 appends) — holder-side pressure / flood rate.
- sessions_lock_wait_us — suspect c control (expect ~0).

WINDOWS BASELINE (for contrast, from my box just now, test PASSES): log_lock_wait_us=0, drain_appends_so_far=213 (ConPTY absorbs the paste -> tiny flood -> no starvation). On Linux forkpty `yes FLOOD` should drive appends into the thousands; if log_lock_wait_us then explodes, that IS the inversion + suspect b.

Cross-check against your gdb stacks: if the subscribe thread is futex-parked AND log_lock_wait_us is seconds, that's airtight. Send me the trace numbers + your stacks and I'll name the exact root + line. Delivery branch untouched (still 18416fd). -todlando
EOF

Send run commands + interpretation to doyle


➜ SENT:doyle


Instrumentation delivered to doyle (branch `wedge-trace-instrumentation`, run commands + interpretation key + Windows baseline for contrast). Delivery branch untouched at 18416fd. doyle runs the numeric pass on kitsubito alongside his live gdb thread-dump; I analyze both to name the exact root + line, then we scope the fix (no product patch until he gates the root).

Standing by for his trace numbers + stacks.


ScheduleWakeup

Instrumentation pushed; awaiting doyle's kitsubito WEDGE_TRACE numbers + gdb stacks to name the root cause, arriving via the armed Monitor — fallback heartbeat only.

Prompt: <<autonomous-loop-dynamic>>


<owl_messages>
<owl_message from="doyle" priority="high">GDB SMOKING GUN — suspect b (OutputLog mutex) REFUTED. Real root is the SendHalf mutex in send_frame. The broker conn-dispatch thread is parked here:
  #2 Mutex::lock_contended
  #3 Mutex::lock (self=0x70c934000c50)  [Mutex<interprocess SendHalf>]
  #4 Mutex<SendHalf>::lock
  #5 broker::send_frame (broker.rs:2150)
  #6 broker::Broker::dispatch_input (broker.rs:1690)
  #7 broker::Broker::handle_conn (broker.rs:1272)
  #8 broker::serve::closure (broker.rs:1180)
NOT resolve_subscribe / NOT OutputLog. The drain thread (Thread 9) is RUNNABLE inside OutputLog::append->output_envelope->serde_json (msg.rs:734) i.e. busy serializing flood output, holding OutputLog but that's not what blocks dispatch. The dispatch thread blocks acquiring a conn's SendHalf in send_frame while ACKing journaled input.

WORKING THEORY: the driver attached (w.attach), so the broker pushes massive flood OUTPUT to the driver's conn SendHalf; dispatch_input's ACK send_frame needs the SAME SendHalf -> starves on it under unfair std Mutex. send_frame at 2150 locks Mutex<SendHalf> per frame. Pulling the full 15-thread map now to find (a) the ATTACHER's conn-handler thread + what IT's blocked on (subscribe reply path), and (b) who HOLDS 0x70c934000c50. Will confirm whether the subscribe-reply send is also starving on a SendHalf. Hold SPT_WEDGE_TRACE tweaks until I send the full map — the probe targets may shift from OutputLog to SendHalf/send_frame. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


Next wakeup scheduled for 22:56:00 (in 1239s). Nothing more to do this turn — the harness re-invokes you when the wakeup fires or a task-notification arrives.


doyle (high-priority) acked: **gdb refuted suspect b.** Real root = `Mutex<SendHalf>` contention in `send_frame` (broker.rs:2150), hit by `dispatch_input`'s per-input ACK (1690) on the **driver's conn SendHalf** — which the broker is simultaneously hammering with flood OUTPUT delivery (driver attached as controller). Drain thread is runnable (serializing flood JSON), holding OutputLog but not the blocker. Good thing we gdb'd instead of patching my wrong hypothesis. HOLDING all SPT_WEDGE_TRACE tweaks until his full 15-thread map.

Respecting the hold — no instrumentation changes. Prepping by reading the cited SendHalf/send_frame/dispatch_input/controller-delivery code so I can name the root + fix-shape fast when his map lands.


Bash

$OWL send doyle <<'EOF'
ACK — suspect b REFUTED by gdb, real root = Mutex<SendHalf> contention in send_frame (broker.rs:2150), not OutputLog. Exactly why we gdb'd instead of patching my hypothesis. HOLDING all SPT_WEDGE_TRACE tweaks until your full thread map — agreed the probe target shifts to SendHalf/send_frame.

NOT idle while you map — reading the SendHalf sharing path (send_frame:2150, dispatch_input:1690, the controller-delivery writer thread, and how ONE conn's SendHalf is shared between (a) flood-output delivery and (b) the per-input ACK) so I can name the root + fix-shape the instant your map lands. No code changes.

EARLY READ of your theory (will verify in source): the driver is the CONTROLLER (test does w.attach), so the broker fans flood OUTPUT to the driver's SendHalf via the controller writer thread WHILE dispatch_input ACKs each of the ~5000 journaled inputs through the SAME SendHalf -> unfair std Mutex -> the ACK (and anything else on that conn) starves. KEY question for your map that decides the fix: is the ATTACHER's subscribe-reply (send_frame at 1590) blocked on the ATTACHER's OWN SendHalf, or is the attacher's whole conn-handler thread blocked upstream (e.g. waiting on a shared lock the wedged driver dispatch holds)? That distinguishes "per-conn SendHalf self-contention" (fix: don't serialize output-delivery and acks on one mutex) from "cross-conn lock coupling". Send the map + who holds 0x70c934000c50 and I'll pin it. -todlando
EOF

Ack doyle + offer code-prep analysis


Reading the SendHalf contention path.


2138	            .ok_or_else(|| format!("no such session {}", req.session_id))?;
2139	        h.session.kill().map_err(|e| format!("kill failed: {e}"))
2140	    }
2141	}
2142
2143	/// The default broker-owned effect-journal path: `<spt_home>/daemon-effects.log`.
2144	pub(crate) fn default_journal_path() -> std::path::PathBuf {
2145	    spt_store::perch::spt_home().join("daemon-effects.log")
2146	}
2147
2148	/// Write a frame to the brain, ignoring a broken-pipe failure (brain gone).
2149	fn send_frame(send: &SharedSend, frame: &Envelope) {
2150	    if let Ok(mut s) = send.lock() {
2151	        let _ = write_frame(&mut *s, frame);
2152	    }
2153	}
2154
2155	/// Send an `error` event to the brain (best-effort).
2156	fn send_error(send: &SharedSend, message: &str) {
2157	    let frame = Envelope::new(
2158	        KIND_ERROR,
2159	        serde_json::to_value(ErrorEvent {
2160	            message: message.to_string(),
2161	        })
2162	        .expect("ErrorEvent serializes"),
2163	    );
2164	    send_frame(send, &frame);
2165	}
2166
2167	#[cfg(test)]

            .ok_or_else(|| format!("no such session {}", req.session_id))?;
        h.session.kill().map_err(|e| format!("kill failed: {e}"))
    }
}

/// The default broker-owned effect-journal path: `<spt_home>/daemon-effects.log`.
pub(crate) fn default_journal_path() -> std::path::PathBuf {
    spt_store::perch::spt_home().join("daemon-effects.log")
}

/// Write a frame to the brain, ignoring a broken-pipe failure (brain gone).
fn send_frame(send: &SharedSend, frame: &Envelope) {
    if let Ok(mut s) = send.lock() {
        let _ = write_frame(&mut *s, frame);
    }
}

/// Send an `error` event to the brain (best-effort).
fn send_error(send: &SharedSend, message: &str) {
    let frame = Envelope::new(
        KIND_ERROR,
        serde_json::to_value(ErrorEvent {
            message: message.to_string(),
        })
        .expect("ErrorEvent serializes"),
    );
    send_frame(send, &frame);
}

#[cfg(test)]


1660	        match req.op_id {
1661	            Some(op) => {
1662	                let key = (req.session_id, op);
1663	                let outcome = self
1664	                    .journal
1665	                    .apply_once(key, EffectKind::PtyWrite, || {
1666	                        // Buffer-or-ENQUEUE INSIDE the journal effect so the op is
1667	                        // still recorded applied + acked (dedup-safe) whether the byte
1668	                        // buffers (inject floor held), enqueues, or is dropped under
1669	                        // backpressure. The effect is now the non-blocking enqueue —
1670	                        // PtyWrite is ephemeral, so Applied-at-enqueue is sound
1671	                        // (V0.13.0-P0 design); the ack means "accepted + ordered + will
1672	                        // land," and rc does not gate on landing.
1673	                        if !buffered(&bytes) {
1674	                            input.enqueue(bytes.clone());
1675	                        }
1676	                        Ok(())
1677	                    })
1678	                    .map_err(|e| format!("input enqueue failed: {e}"))?;
1679	                // Ack exactly-once — but ONLY when the caller requested it
1680	                // (REQ-HAZARD-INPUT-ACK-BACKPRESSURE). The fire-and-forward
1681	                // operator/rc path sends `ack=false`: writing the ack back onto the
1682	                // same brain↔broker conn it is flooding with input fills the return
1683	                // direction and deadlocks the per-conn handler (the wedge). With no
1684	                // ack the handler never writes back while servicing the flood, so it
1685	                // always drains. The acked path (`shellchan`, one-at-a-time) still
1686	                // gets its `KIND_APPLIED`. Dedup is unaffected either way (the
1687	                // applied-set is the exactly-once anchor, not the ack frame).
1688	                // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
1689	                if req.ack {
1690	                    send_frame(
1691	                        send,
1692	                        &applied_envelope(req.session_id, op, outcome == Outcome::Applied),
1693	                    );
1694	                }
1695	                Ok(())
1696	            }
1697	            None => {
1698	                if !buffered(&bytes) {
1699	                    input.enqueue(bytes);
1700	                }
1701	                Ok(())
1702	            }
1703	        }
1704	    }

        match req.op_id {
            Some(op) => {
                let key = (req.session_id, op);
                let outcome = self
                    .journal
                    .apply_once(key, EffectKind::PtyWrite, || {
                        // Buffer-or-ENQUEUE INSIDE the journal effect so the op is
                        // still recorded applied + acked (dedup-safe) whether the byte
                        // buffers (inject floor held), enqueues, or is dropped under
                        // backpressure. The effect is now the non-blocking enqueue —
                        // PtyWrite is ephemeral, so Applied-at-enqueue is sound
                        // (V0.13.0-P0 design); the ack means "accepted + ordered + will
                        // land," and rc does not gate on landing.
                        if !buffered(&bytes) {
                            input.enqueue(bytes.clone());
                        }
                        Ok(())
                    })
                    .map_err(|e| format!("input enqueue failed: {e}"))?;
                // Ack exactly-once — but ONLY when the caller requested it
                // (REQ-HAZARD-INPUT-ACK-BACKPRESSURE). The fire-and-forward
                // operator/rc path sends `ack=false`: writing the ack back onto the
                // same brain↔broker conn it is flooding with input fills the return
                // direction and deadlocks the per-conn handler (the wedge). With no
                // ack the handler never writes back while servicing the flood, so it
                // always drains. The acked path (`shellchan`, one-at-a-time) still
                // gets its `KIND_APPLIED`. Dedup is unaffected either way (the
                // applied-set is the exactly-once anchor, not the ack frame).
                // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
                if req.ack {
                    send_frame(
                        send,
                        &applied_envelope(req.session_id, op, outcome == Outcome::Applied),
                    );
                }
                Ok(())
            }
            None => {
                if !buffered(&bytes) {
                    input.enqueue(bytes);
                }
                Ok(())
            }
        }
    }


**Major lead in the code.** dispatch_input (1689-1693): the ACK `send_frame` fires only when `req.ack` is true — and the comment documents **REQ-HAZARD-INPUT-ACK-BACKPRESSURE**: "writing the ack back onto the same brain↔broker conn it is flooding with input fills the return direction and deadlocks the per-conn handler (the wedge). The fire-and-forward operator/rc path sends ack=false." So the production rc path AVOIDS this exact wedge via ack=false. The question that decides everything: does the **test driver's `send_effect` send ack=true** (re-triggering the hazard the real path dodges)? Checking.


Grep

fn send_effect|fn attach\b|ack:|InputReq\s*\{|\.ack\b|pub fn send_input


<owl_messages>
<owl_message from="doyle" priority="high">FULL 15-THREAD MAP done. SUSPECT B (OutputLog) looks REFUTED by gdb; real wedge is the SendHalf send path:
- Thread 6 (driver-conn dispatch handler): BLOCKED futex on Mutex<SendHalf>(0x..0c50) in send_frame(2150) <- dispatch_input(1690) <- handle_conn(1272). This conn handler can't finish ACKing input.
- Thread 9 (DRAIN): RUNNABLE, busy in OutputLog::append->output_envelope->serde_json (msg.rs:734) — holds OutputLog but is NOT what blocks dispatch; no thread is parked waiting on OutputLog.
- Thread 10 (input_writer): parked in write_input on the PTY (by design, off dispatch). Good.
- No thread in resolve_subscribe/OutputLog-wait at all.
So the contention is the per-conn SendHalf mutex in send_frame, NOT the OutputLog mutex. Your reply_send_us probe is the exact cross-check — if reply_send_us (or the ACK send) is the seconds-scale wait and log_lock_wait_us≈0, that CONFIRMS SendHalf-send starvation, not OutputLog. Running your instrumented branch on kitsubito now for the numbers. If log_lock_wait_us≈0, pivot the probe to time the send_frame SendHalf acquire in dispatch_input + the output-writer's hold on that same SendHalf. Numbers incoming. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


448:    pub fn send_input(&mut self, bytes: &[u8]) -> io::Result<()> {
452:            serde_json::to_value(InputReq {
456:                ack: true,
475:    pub fn send_effect(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
488:    pub fn send_effect_no_ack(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
494:    fn send_effect_inner(&mut self, op_id: u64, bytes: &[u8], ack: bool) -> io::Result<()> {
498:            serde_json::to_value(InputReq {
796:    pub fn attach(&mut self, session_id: u64, from_seq: u64) -> io::Result<()> {
1070:            serde_json::to_value(EndpointInputReq {
1265:fn contains(haystack: &[u8], needle: &[u8]) -> bool {


The grep already exposes the likely real story before doyle's map resolves: `send_input` uses `ack: true` (456), and a separate **`send_effect_no_ack`** exists (488) — so `send_effect` (the test driver's call) is the **ACKED** variant. The comment at dispatch_input says the production rc/operator flood path uses **ack=false** precisely to avoid this SendHalf wedge (REQ-HAZARD-INPUT-ACK-BACKPRESSURE). So the test driver may be flooding via the ACKED path that production never uses → a non-production self-wedge, not a product bug. Acking doyle + surfacing this as the lead to verify.


Bash

$OWL send doyle <<'EOF'
ACK — full map: Thread6 driver-dispatch futex-parked on Mutex<SendHalf> in send_frame(2150)<-dispatch_input(1690); drain runnable (not the blocker); nobody waiting on OutputLog. SendHalf-send starvation, suspect b dead. Agreed my reply_send_us / ACK-send probe is the cross-check; running it is right.

BUT I think I found WHY, and it may move the fix from product to TEST. Reading brain.rs:
- dispatch_input only sends the ACK send_frame(1690) when req.ack==true. The comment there IS REQ-HAZARD-INPUT-ACK-BACKPRESSURE: "the fire-and-forward operator/rc path sends ack=false: writing the ack back onto the same conn it is flooding ... deadlocks the per-conn handler (the wedge). The acked path (shellchan, one-at-a-time) still gets KIND_APPLIED." i.e. PRODUCTION rc/operator FLOOD is ack=FALSE by design to dodge exactly this SendHalf wedge.
- brain.rs has BOTH send_effect (475) and send_effect_no_ack (488); send_input is ack:true (456). The test driver pumps `w.send_effect(op,&chunk)` = the ACKED variant.

HYPOTHESIS (verifying in source right now): the wedge gates drive the sustained flood with send_effect = ack=TRUE, so the broker ACKs all ~5000 ops back onto the SAME driver conn that's also receiving controller flood-output -> SendHalf self-contention = the documented hazard. But the REAL rc/operator drive floods ack=FALSE (send_effect_no_ack), which never writes back on the flooded conn and so never wedges. If so, these 3 are the TEST mis-modeling the driver (acked flood) rather than a product regression — and the product's real ackless path is fine.

CONFIRMING NOW: (1) send_effect passes ack=true, (2) what the real rc attach/drive path emits (ack true|false). If real=false, the fix is the test driver -> send_effect_no_ack, NOT a product SendHalf change. Don't sink time into a product SendHalf fix until I confirm. Numbers from your run will agree: ACK-send wait huge, log_lock_wait≈0. Verifying + back in a min. -todlando
EOF

Ack + surface ack-mode hypothesis to doyle


Verifying: read send_effect's ack value + find which variant the real rc/drive path uses in non-test src.


475	    pub fn send_effect(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
476	        self.send_effect_inner(op_id, bytes, true)
477	    }
478
479	    /// Inject input idempotently like [`Brain::send_effect`] but request NO
480	    /// [`KIND_APPLIED`] ack (REQ-HAZARD-INPUT-ACK-BACKPRESSURE). The operator/rc
481	    /// attach path is fire-and-forward — `op_id` is for dedup only and the ack is
482	    /// discarded by `serve_attach` — and it can flood N input frames on ONE
483	    /// brain↔broker conn without returning to read. With no ack the broker's
484	    /// per-conn handler never writes back while servicing that flood, so the return
485	    /// direction can't fill and deadlock the handler. Exactly-once is preserved: the
486	    /// broker still dedups by `(session, op_id)` at its applied-set.
487	    // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
488	    pub fn send_effect_no_ack(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
489	        self.send_effect_inner(op_id, bytes, false)
490	    }
491
492	    /// Shared body for [`Brain::send_effect`] / [`Brain::send_effect_no_ack`]: the
493	    /// only difference is whether the broker replies a `KIND_APPLIED` ack.
494	    fn send_effect_inner(&mut self, op_id: u64, bytes: &[u8], ack: bool) -> io::Result<()> {
495	        let sid = self.require_session()?;
496	        self.send(
497	            KIND_INPUT,
498	            serde_json::to_value(InputReq {
499	                session_id: sid,
500	                data_b64: encode_bytes(bytes),
501	                op_id: Some(op_id),
502	                ack,
503	            })
504	            .expect("InputReq serializes"),
505	        )
506	    }
507
508	    /// Resize the current session's PTY.
509	    pub fn resize(&mut self, rows: u16, cols: u16) -> io::Result<()> {

    pub fn send_effect(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
        self.send_effect_inner(op_id, bytes, true)
    }

    /// Inject input idempotently like [`Brain::send_effect`] but request NO
    /// [`KIND_APPLIED`] ack (REQ-HAZARD-INPUT-ACK-BACKPRESSURE). The operator/rc
    /// attach path is fire-and-forward — `op_id` is for dedup only and the ack is
    /// discarded by `serve_attach` — and it can flood N input frames on ONE
    /// brain↔broker conn without returning to read. With no ack the broker's
    /// per-conn handler never writes back while servicing that flood, so the return
    /// direction can't fill and deadlock the handler. Exactly-once is preserved: the
    /// broker still dedups by `(session, op_id)` at its applied-set.
    // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
    pub fn send_effect_no_ack(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
        self.send_effect_inner(op_id, bytes, false)
    }

    /// Shared body for [`Brain::send_effect`] / [`Brain::send_effect_no_ack`]: the
    /// only difference is whether the broker replies a `KIND_APPLIED` ack.
    fn send_effect_inner(&mut self, op_id: u64, bytes: &[u8], ack: bool) -> io::Result<()> {
        let sid = self.require_session()?;
        self.send(
            KIND_INPUT,
            serde_json::to_value(InputReq {
                session_id: sid,
                data_b64: encode_bytes(bytes),
                op_id: Some(op_id),
                ack,
            })
            .expect("InputReq serializes"),
        )
    }

    /// Resize the current session's PTY.
    pub fn resize(&mut self, rows: u16, cols: u16) -> io::Result<()> {


crates\spt-daemon\src\brain.rs:123:    /// distinguishes). The exactly-once ack for a [`Brain::send_effect`] (B5).
crates\spt-daemon\src\brain.rs:447:    /// sequenced injection), use [`Brain::send_effect`].
crates\spt-daemon\src\brain.rs:448:    pub fn send_input(&mut self, bytes: &[u8]) -> io::Result<()> {
crates\spt-daemon\src\brain.rs:472:    /// conn — use [`Brain::send_effect_no_ack`] to avoid the ack-backpressure
crates\spt-daemon\src\brain.rs:475:    pub fn send_effect(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
crates\spt-daemon\src\brain.rs:479:    /// Inject input idempotently like [`Brain::send_effect`] but request NO
crates\spt-daemon\src\brain.rs:488:    pub fn send_effect_no_ack(&mut self, op_id: u64, bytes: &[u8]) -> io::Result<()> {
crates\spt-daemon\src\brain.rs:492:    /// Shared body for [`Brain::send_effect`] / [`Brain::send_effect_no_ack`]: the
crates\spt-daemon\src\attach.rs:10://! - inbound [`AttachRecord::Input`] → [`Brain::send_effect`] with the
crates\spt-daemon\src\attach.rs:205:                                    brain.send_effect_no_ack(op_id, &bytes)?
crates\spt-daemon\src\attach.rs:240:                            brain.send_effect(op, &b)?;
crates\spt-daemon\src\shellchan.rs:11://!   `send_effect` input seam, exactly-once per spool row id). `http` refuses
crates\spt-daemon\src\shellchan.rs:171:/// each row rides [`Brain::send_effect`] with the row id as the durable
crates\spt-daemon\src\shellchan.rs:205:            .send_effect(row_id as u64, &bytes)
crates\spt-daemon\src\shellhost.rs:146:/// frames ride the broker's `send_effect` input seam); **http** refuses at
crates\spt-daemon\src\shellhost.rs:207:/// stdin and `spt shell cmd` can drive it through the `send_effect` input


**CONFIRMED.** `send_effect` (475) → `ack=true`. `send_effect_no_ack` (488) → `ack=false`, tagged `[impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]` with the doc: "The operator/rc attach path is fire-and-forward ... can flood N input frames on ONE conn ... With no ack the per-conn handler never writes back ... so the return direction can't fill and deadlock." And the **real rc attach inbound path uses it**: `attach.rs:205 brain.send_effect_no_ack(op_id, &bytes)`. But `attach.rs:240` uses acked `send_effect` — need to characterize 205 vs 240 before I claim "test mis-models." Reading.


185	                            // clear-race), so this loop no longer stamps them.
186	                            brain.attach_as(session_id, from_seq, req_intent, Some(origin_node))?;
187	                            attached = true;
188	                        }
189	                        AttachRecord::Input { data_b64, op_id }
190	                            if attached && intent != AttachIntent::Viewer =>
191	                        {
192	                            let bytes = decode_bytes(&data_b64)
193	                                .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
194	                            match role {
195	                                // Confirmed controller — exactly-once at the PTY
196	                                // (the operator-minted id rides the B5 journal). NO-ACK
197	                                // (REQ-HAZARD-INPUT-ACK-BACKPRESSURE): this path is
198	                                // fire-and-forward (the ack is discarded here) and can
199	                                // flood N input frames on this one conn; an acked write
200	                                // back would fill the return direction and deadlock the
201	                                // broker's per-conn handler. Dedup still holds at the
202	                                // broker (op_id applied-set).
203	                                // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
204	                                Some(ServeRole::Controller) => {
205	                                    brain.send_effect_no_ack(op_id, &bytes)?
206	                                }
207	                                // Outcome not yet known — buffer; flushed on a
208	                                // controller confirmation, discarded on busy/viewer.
209	                                None => pending_input.push((op_id, bytes)),
210	                                // A viewer never reaches here (intent gate).
211	                                Some(ServeRole::Viewer) => {}
212	                            }
213	                        }
214	                        AttachRecord::Resize { rows, cols }
215	                            if attached && intent != AttachIntent::Viewer =>
216	                        {
217	                            // Controller → PTY resize (controller-EXCLUSIVE; the
218	                            // broker rejects it on a viewer's stream). A viewer's
219	                            // Resize is dropped here by the intent gate too.
220	                            let _ = brain.resize(rows, cols);
221	                        }
222	                        // Input/Resize before Request, a viewer's input/resize,
223	                        // or target-direction records echoed back: noise — ignore.
224	                        _ => {}
225	                    }
226	                }
227	            }
228
229	            // ── Broker → loop: the resolved controller/viewer role. ─────────
230	            BrokerEvent::Subscribed { outcome, .. } => {
231	                use crate::msg::SubscribeOutcome as O;
232	                match outcome {
233	                    O::Controller | O::TookControl => {
234	                        role = Some(ServeRole::Controller);
235	                        // A driver attached — the wake edge (controller-only;
236	                        // idempotent on an already-active seat).
237	                        feed_rest(endpoint, crate::resting::RestEvent::Wake);
238	                        // Flush any input that raced ahead of the reply.
239	                        for (op, b) in pending_input.drain(..) {
240	                            brain.send_effect(op, &b)?;
241	                        }
242	                    }
243	                    O::Viewer => {
244	                        role = Some(ServeRole::Viewer);
245	                        // Wake-neutral: a viewer never wakes the endpoint, never
246	                        // keeps it awake (Gap B). Discard any racing input.

                            // clear-race), so this loop no longer stamps them.
                            brain.attach_as(session_id, from_seq, req_intent, Some(origin_node))?;
                            attached = true;
                        }
                        AttachRecord::Input { data_b64, op_id }
                            if attached && intent != AttachIntent::Viewer =>
                        {
                            let bytes = decode_bytes(&data_b64)
                                .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
                            match role {
                                // Confirmed controller — exactly-once at the PTY
                                // (the operator-minted id rides the B5 journal). NO-ACK
                                // (REQ-HAZARD-INPUT-ACK-BACKPRESSURE): this path is
                                // fire-and-forward (the ack is discarded here) and can
                                // flood N input frames on this one conn; an acked write
                                // back would fill the return direction and deadlock the
                                // broker's per-conn handler. Dedup still holds at the
                                // broker (op_id applied-set).
                                // [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE]
                                Some(ServeRole::Controller) => {
                                    brain.send_effect_no_ack(op_id, &bytes)?
                                }
                                // Outcome not yet known — buffer; flushed on a
                                // controller confirmation, discarded on busy/viewer.
                                None => pending_input.push((op_id, bytes)),
                                // A viewer never reaches here (intent gate).
                                Some(ServeRole::Viewer) => {}
                            }
                        }
                        AttachRecord::Resize { rows, cols }
                            if attached && intent != AttachIntent::Viewer =>
                        {
                            // Controller → PTY resize (controller-EXCLUSIVE; the
                            // broker rejects it on a viewer's stream). A viewer's
                            // Resize is dropped here by the intent gate too.
                            let _ = brain.resize(rows, cols);
                        }
                        // Input/Resize before Request, a viewer's input/resize,
                        // or target-direction records echoed back: noise — ignore.
                        _ => {}
                    }
                }
            }

            // ── Broker → loop: the resolved controller/viewer role. ─────────
            BrokerEvent::Subscribed { outcome, .. } => {
                use crate::msg::SubscribeOutcome as O;
                match outcome {
                    O::Controller | O::TookControl => {
                        role = Some(ServeRole::Controller);
                        // A driver attached — the wake edge (controller-only;
                        // idempotent on an already-active seat).
                        feed_rest(endpoint, crate::resting::RestEvent::Wake);
                        // Flush any input that raced ahead of the reply.
                        for (op, b) in pending_input.drain(..) {
                            brain.send_effect(op, &b)?;
                        }
                    }
                    O::Viewer => {
                        role = Some(ServeRole::Viewer);
                        // Wake-neutral: a viewer never wakes the endpoint, never
                        // keeps it awake (Gap B). Discard any racing input.


**Root cause found — evidence-complete.** attach.rs:205 (real rc/operator steady controller input) = `send_effect_no_ack` (ackless, the sustained flood path); attach.rs:240 (acked `send_effect`) is only a **one-shot bounded flush** of input that raced ahead of the subscribe reply — never a sustained flood. So the real sustained rc drive is ackless by design. The **test driver floods ~5000 ops with `send_effect` (ack=true)** — re-triggering the exact hazard (REQ-HAZARD-INPUT-ACK-BACKPRESSURE) that production avoids. It's a test mis-model, not a product regression. Sending doyle the definitive case.


$OWL send doyle <<'EOF'
ROOT CAUSE — CONFIRMED IN SOURCE. It's the TEST DRIVER's ack mode, not a product regression. Your gdb + this code = airtight:

CHAIN:
- brain.rs:475 `send_effect` -> send_effect_inner(.., ack=TRUE). brain.rs:488 `send_effect_no_ack` -> ack=FALSE, tagged [impl->REQ-HAZARD-INPUT-ACK-BACKPRESSURE].
- dispatch_input (broker.rs:1689) sends the ACK send_frame(2150) ONLY when req.ack==true -> locks Mutex<SendHalf> on the DRIVER's conn per op. That's your Thread-6 futex park.
- The driver is the controller, so the broker fans flood OUTPUT to that SAME SendHalf via the controller writer -> the ~5000 ACKs starve on it (unfair std Mutex) -> dispatch handler can't drain -> the concurrent attach never gets serviced. subscribed=false.

WHY IT'S A TEST BUG:
- REAL rc/operator sustained input = ACKLESS. attach.rs:205 (ServeRole::Controller inbound Input) = `brain.send_effect_no_ack(op_id,&bytes)` — fire-and-forward, EXACTLY to dodge this wedge (doc 196-203).
- attach.rs:240's acked `send_effect` is ONLY the one-shot flush of `pending_input` that raced ahead of the subscribe reply — bounded, never a flood.
- So NO real path floods sustained input with ack=true. But the wedge gates' driver loops pump `w.send_effect(op,&chunk)` = ACKED, thousands of times. They model the operator with the wrong seam.

FIX (test, not product): switch the sustained-flood drivers in inject_control_wedge.rs to `send_effect_no_ack` (the real rc seam). Ackless flood -> broker never writes ACKs back on the flooded conn -> dispatch drains -> concurrent attach serviced. Facet-B stays non-vacuous: still a sustained journaled-input flood via the REAL operator seam, still asserts the attach is serviced. The product's ackless path is already correct (REQ-HAZARD-INPUT-ACK-BACKPRESSURE) — no SendHalf change needed.

YOUR NUMBERS WILL AGREE: log_lock_wait_us≈0 (suspect b dead) and the wedge is the ACK send_frame, which only fires because ack=true.

CAVEAT TO GATE: g2 asserts op_flushed=false (commit-deadline/floor), a different assertion — I need to check whether g2's driver is also acked-flood (same root) or a genuinely separate stall. p0 + a_journaled are the ack-mode root for sure.

WANT ME TO: (1) apply the test-driver -> send_effect_no_ack fix on the 3 gates + push, you validate on kitsubito? (2) first check g2 specifically? Your gate. I'll hold the product untouched. -todlando
EOF

Send definitive root-cause to doyle


➜ SENT:doyle


Definitive root-cause sent. While doyle validates, resolving my own caveat — reading g2 to see if it's the same acked-flood root or a separate stall.


<owl_messages>
<owl_message from="doyle" priority="high">HUGE insight — ack=true vs ack=false is very likely it, and it MATCHES my gdb exactly: Thread 6 is blocked in send_frame INSIDE dispatch_input = the ACK write, which only fires when req.ack==true. So an acked sustained flood self-contends on the conn's SendHalf vs the controller flood-output. REQ-HAZARD-INPUT-ACK-BACKPRESSURE is literally the documented dodge. Strong.

CRITICAL per-test distinction before we call it "just a test bug" — confirm in source for EACH:
1. p0_paste: real rc/operator PASTE flood = ack=FALSE in production? If yes -> test mis-models with send_effect(ack=true) -> TEST FIX (driver -> send_effect_no_ack). Clean.
2. a_journaled: the JOURNALED-input path is exactly-once (op_id) -> does it LEGITIMATELY use ack=true in production? AND does production ever SUSTAIN-flood journaled input, or is it one-at-a-time like shellchan? If journaled is always one-at-a-time/acked-serially in prod, a 5000-op acked journaled flood is ALSO a test mis-model (prod never floods it). If prod CAN sustain acked journaled flood concurrent with an attach -> REAL product bug. This one decides product-vs-test for the headline gate.
3. g2: commit-deadline/floor path — different; tell me its ack model + whether op_flushed=false is the same SendHalf-ACK self-contention or the floor-release logic.

I'm applying your SPT_WEDGE_TRACE via PATCH (kitsubito has no git creds — CI injects a token; I'm git-apply'ing the diff on a clean 18416fd base) + running for the numbers to corroborate (expect log_lock_wait_us≈0, the wait on the ACK/SendHalf send). Numbers + per-test source verdict together = airtight. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


89:/// be parked forever inside a blocking socket op — e.g. the driver's `send_effect`
503:    //    (op_id-carrying send_effect → broker dispatch_input → journal.apply_once
519:        // deadlock send_effect on a plain Whole conn (observed: it stalled after a
526:        // Bind the driver brain to the same session so send_effect has a target.
533:        // the journal lock) until teardown. Each send_effect writes a frame; the
536:            if w.send_effect(op, &chunk).is_err() {
693:    // `send_effect` and the attacher in its subscribe — neither observes `stop`, so
1133:fn g2_no_commit_deadline_faults_binary_and_does_not_wedge_controller_input() {
1164:    let op_flushed =
1185:        "=== W2 G2 NO-COMMIT FAULT GATE: delivered={delivered} op_flushed={op_flushed} \
1188:         (op_flushed=true → controller input never wedged; raw_fallback → binary faulted) ==="
1194:        op_flushed,
1766:    //    (op_id-carrying send_effect → dispatch_input → apply_once(PtyWrite) =
1789:            if w.send_effect(op, &chunk).is_err() {


doyle (high-priority) acked: ack-mode matches his gdb (Thread 6 = the ACK write in dispatch_input, fires only on req.ack==true). He wants a **per-test** product-vs-test verdict, especially #2 (a_journaled headline): does production ever SUSTAIN-flood *acked* journaled input concurrent with an attach? He's git-apply'ing my trace branch on kitsubito for corroborating numbers.

From my earlier src grep, the acked `send_effect` production callers are attach.rs:240 (one-shot pending flush) and shellchan.rs:205 ("one-at-a-time"). Confirming shellchan is one-at-a-time + reading g2 to answer #3, then sending the consolidated verdict.


$OWL send doyle <<'EOF'
ACK — per-test verdict incoming, this is the right gate. STRONG PRELIM on your critical #2 (a_journaled): in PRODUCTION nobody SUSTAIN-floods ACKED journaled input. The only acked `send_effect` callers in src are:
- attach.rs:240 — ONE-SHOT flush of pending_input that raced the subscribe reply (bounded, not a flood).
- shellchan.rs:205 — spool rows, your own dispatch_input comment calls it "the acked path (shellchan, ONE-AT-A-TIME)".
The SUSTAINED rc/operator drive is ackless (attach.rs:205 send_effect_no_ack). So a 5000-op sustained ACKED journaled flood = NO real path -> a_journaled is ALSO a test mis-model, same as p0. Confirming shellchan is truly one-at-a-time (not fire-many-acked) + reading g2 now to finish #3. Full per-test table in my next msg. -todlando
EOF

Ack + prelim #2 verdict to doyle


Reading g2 + confirming shellchan's one-at-a-time pattern to finish the per-test table.


1133	fn g2_no_commit_deadline_faults_binary_and_does_not_wedge_controller_input() {
1134	    init_wedge_home();
1135	    let dir = tempfile::tempdir().expect("tempdir");
1136	    // The binary emits the payload + a key but NEVER `{commit}` → the broker's
1137	    // commit deadline must fire. Shrink the deadline so the structural outcome
1138	    // (floor releases → buffered input flushes → binary faults) is observed fast.
1139	    std::env::set_var("XLATE_FIXTURE_MODE", "nocommit");
1140	    std::env::set_var("SPT_INJECT_COMMIT_DEADLINE_MS", "400");
1141	    std::env::remove_var("XLATE_FIXTURE_STDIN_LOG");
1142
1143	    let name = unique_name();
1144	    let endpoint = "xlate-g2-ep";
1145	    let (broker, sid, mut controller) =
1146	        spawn_xlate_session(&name, &dir.path().join("g2"), endpoint);
1147	    // Bind the controller brain to the session so its keystrokes have a target.
1148	    controller.attach(sid, 0).expect("controller attach");
1149
1150	    // Deliver an event whose binary never commits → the floor OPENS, buffers
1151	    // operator input, and the worker hits the 400ms deadline.
1152	    let envelope = br#"<EVENT type="message" from="peer">no-commit event</EVENT>"#;
1153	    let delivered = deliver_endpoint_event(&name, endpoint, envelope, Duration::from_secs(5));
1154
1155	    // While the floor is (briefly) held, the operator types — Enter-terminated so the
1156	    // echo child flushes the line. After the 400ms deadline the floor must RELEASE and
1157	    // the buffered keystrokes flush to the PTY (then echo back over the ring). Send a
1158	    // few times (some before, some after the deadline) so the flush is exercised; then
1159	    // observe the marker echo — proof the controller input did NOT wedge.
1160	    for _ in 0..4 {
1161	        let _ = controller.send_input(b"OPERATOR_KEYS_FLUSHED\r");
1162	        let _ = controller.read_event_until(Some(Instant::now() + Duration::from_millis(150)));
1163	    }
1164	    let op_flushed =
1165	        output_contains_within(&name, sid, b"OPERATOR_KEYS_FLUSHED", Duration::from_secs(8));
1166
1167	    // The session stays serviceable after the FAULT.
1168	    let sessions_answered =
1169	        probe(&name, KIND_SESSIONS, KIND_SESSIONS_REPLY, Duration::from_secs(5));
1170
1171	    // After the binary faulted, a SUBSEQUENT inbound delivery falls back to RAW
1172	    // inject: the whole envelope bytes land on the PTY directly (the binary no longer
1173	    // drives the choreography). The raw envelope ends in `\r` so the echo child
1174	    // flushes the echoed line; assert the marker reaches the PTY via the ring.
1175	    let raw_envelope = b"<EVENT>RAW_FALLBACK_MARKER</EVENT>\r";
1176	    let delivered2 = deliver_endpoint_event(&name, endpoint, raw_envelope, Duration::from_secs(5));
1177	    let raw_fallback_reached =
1178	        output_contains_within(&name, sid, b"RAW_FALLBACK_MARKER", Duration::from_secs(8));
1179
1180	    if let Some(pid) = broker.session_pid(sid) {
1181	        kill_pid(pid);
1182	    }
1183
1184	    eprintln!(
1185	        "=== W2 G2 NO-COMMIT FAULT GATE: delivered={delivered} op_flushed={op_flushed} \
1186	         sessions_answered={sessions_answered} delivered2={delivered2} \
1187	         raw_fallback_reached={raw_fallback_reached} \
1188	         (op_flushed=true → controller input never wedged; raw_fallback → binary faulted) ==="
1189	    );
1190
1191	    // [int->REQ-MSG-IDLE-TRANSLATION-BINARY]
1192	    assert!(delivered, "the no-commit event must still be accepted by the broker");
1193	    assert!(
1194	        op_flushed,
1195	        "after the commit deadline the floor must RELEASE and buffered operator \
1196	         keystrokes must flush to the PTY — the controller's input must NOT wedge \
1197	         (the mandatory bounded-buffer guard; doyle non-negotiable = never park forever)"

fn g2_no_commit_deadline_faults_binary_and_does_not_wedge_controller_input() {
    init_wedge_home();
    let dir = tempfile::tempdir().expect("tempdir");
    // The binary emits the payload + a key but NEVER `{commit}` → the broker's
    // commit deadline must fire. Shrink the deadline so the structural outcome
    // (floor releases → buffered input flushes → binary faults) is observed fast.
    std::env::set_var("XLATE_FIXTURE_MODE", "nocommit");
    std::env::set_var("SPT_INJECT_COMMIT_DEADLINE_MS", "400");
    std::env::remove_var("XLATE_FIXTURE_STDIN_LOG");

    let name = unique_name();
    let endpoint = "xlate-g2-ep";
    let (broker, sid, mut controller) =
        spawn_xlate_session(&name, &dir.path().join("g2"), endpoint);
    // Bind the controller brain to the session so its keystrokes have a target.
    controller.attach(sid, 0).expect("controller attach");

    // Deliver an event whose binary never commits → the floor OPENS, buffers
    // operator input, and the worker hits the 400ms deadline.
    let envelope = br#"<EVENT type="message" from="peer">no-commit event</EVENT>"#;
    let delivered = deliver_endpoint_event(&name, endpoint, envelope, Duration::from_secs(5));

    // While the floor is (briefly) held, the operator types — Enter-terminated so the
    // echo child flushes the line. After the 400ms deadline the floor must RELEASE and
    // the buffered keystrokes flush to the PTY (then echo back over the ring). Send a
    // few times (some before, some after the deadline) so the flush is exercised; then
    // observe the marker echo — proof the controller input did NOT wedge.
    for _ in 0..4 {
        let _ = controller.send_input(b"OPERATOR_KEYS_FLUSHED\r");
        let _ = controller.read_event_until(Some(Instant::now() + Duration::from_millis(150)));
    }
    let op_flushed =
        output_contains_within(&name, sid, b"OPERATOR_KEYS_FLUSHED", Duration::from_secs(8));

    // The session stays serviceable after the FAULT.
    let sessions_answered =
        probe(&name, KIND_SESSIONS, KIND_SESSIONS_REPLY, Duration::from_secs(5));

    // After the binary faulted, a SUBSEQUENT inbound delivery falls back to RAW
    // inject: the whole envelope bytes land on the PTY directly (the binary no longer
    // drives the choreography). The raw envelope ends in `\r` so the echo child
    // flushes the echoed line; assert the marker reaches the PTY via the ring.
    let raw_envelope = b"<EVENT>RAW_FALLBACK_MARKER</EVENT>\r";
    let delivered2 = deliver_endpoint_event(&name, endpoint, raw_envelope, Duration::from_secs(5));
    let raw_fallback_reached =
        output_contains_within(&name, sid, b"RAW_FALLBACK_MARKER", Duration::from_secs(8));

    if let Some(pid) = broker.session_pid(sid) {
        kill_pid(pid);
    }

    eprintln!(
        "=== W2 G2 NO-COMMIT FAULT GATE: delivered={delivered} op_flushed={op_flushed} \
         sessions_answered={sessions_answered} delivered2={delivered2} \
         raw_fallback_reached={raw_fallback_reached} \
         (op_flushed=true → controller input never wedged; raw_fallback → binary faulted) ==="
    );

    // [int->REQ-MSG-IDLE-TRANSLATION-BINARY]
    assert!(delivered, "the no-commit event must still be accepted by the broker");
    assert!(
        op_flushed,
        "after the commit deadline the floor must RELEASE and buffered operator \
         keystrokes must flush to the PTY — the controller's input must NOT wedge \
         (the mandatory bounded-buffer guard; doyle non-negotiable = never park forever)"


190	        Brain::cold_start(broker_name, now_ms()).map_err(|e| format!("broker connect: {e}"))?;
191	    let sessions = brain.sessions().map_err(|e| format!("sessions: {e}"))?;
192	    let sid = sessions
193	        .sessions
194	        .iter()
195	        .find(|s| s.endpoint == label)
196	        .map(|s| s.session_id)
197	        .ok_or_else(|| format!("no broker session for {label} (shell offline?)"))?;
198	    brain.attach(sid, 0).map_err(|e| format!("attach: {e}"))?;
199
200	    let mut delivered = 0usize;
201	    for (row_id, _from, body, _at) in rows {
202	        let mut bytes = body.into_bytes();
203	        bytes.push(b'\n');
204	        brain
205	            .send_effect(row_id as u64, &bytes)
206	            .map_err(|e| format!("send row {row_id}: {e}"))?;
207	        // Drain interleaved output until this row's exactly-once ack.
208	        loop {
209	            match brain.read_event().map_err(|e| format!("read: {e}"))? {
210	                BrokerEvent::Applied { op_id, .. } if op_id == row_id as u64 => break,
211	                BrokerEvent::Exit { .. } => return Err("shell binary exited mid-delivery".into()),
212	                BrokerEvent::Error { message } => return Err(format!("broker error: {message}")),
213	                _ => {}
214	            }

        Brain::cold_start(broker_name, now_ms()).map_err(|e| format!("broker connect: {e}"))?;
    let sessions = brain.sessions().map_err(|e| format!("sessions: {e}"))?;
    let sid = sessions
        .sessions
        .iter()
        .find(|s| s.endpoint == label)
        .map(|s| s.session_id)
        .ok_or_else(|| format!("no broker session for {label} (shell offline?)"))?;
    brain.attach(sid, 0).map_err(|e| format!("attach: {e}"))?;

    let mut delivered = 0usize;
    for (row_id, _from, body, _at) in rows {
        let mut bytes = body.into_bytes();
        bytes.push(b'\n');
        brain
            .send_effect(row_id as u64, &bytes)
            .map_err(|e| format!("send row {row_id}: {e}"))?;
        // Drain interleaved output until this row's exactly-once ack.
        loop {
            match brain.read_event().map_err(|e| format!("read: {e}"))? {
                BrokerEvent::Applied { op_id, .. } if op_id == row_id as u64 => break,
                BrokerEvent::Exit { .. } => return Err("shell binary exited mid-delivery".into()),
                BrokerEvent::Error { message } => return Err(format!("broker error: {message}")),
                _ => {}
            }


All 3 resolved:
- **shellchan.rs:208-210** confirms strictly **one-at-a-time** (sends one acked `send_effect`, blocks reading until that row's `Applied` ack before the next). No concurrent acked flood anywhere in production.
- **g2 is different** — it does NOT flood: 4 `controller.send_input` keystrokes (1161), and `op_flushed=false` is the **InjectFloor commit-deadline/floor-release/flush** path (no-commit fault), not SendHalf-ACK contention. Separate investigation.

Sending doyle the per-test verdict table.


$OWL send doyle <<'EOF'
PER-TEST VERDICT (source-confirmed):

1. p0_paste_wedge -> TEST MIS-MODEL. Real rc/operator PASTE flood is ackless (attach.rs:205 send_effect_no_ack). Test driver floods w.send_effect (ack=true). FIX: driver -> send_effect_no_ack.

2. a_journaled (HEADLINE, your critical #2) -> TEST MIS-MODEL. Production has NO sustained-acked-journaled-flood path. Confirmed all acked send_effect callers:
   - attach.rs:240 = one-shot pending_input flush (bounded).
   - shellchan.rs:201-214 = strictly ONE-AT-A-TIME: sends one send_effect(row_id), then BLOCKS in a loop reading until that row's Applied ack before the next row. Never concurrent.
   The sustained drive in prod is ackless (attach.rs:205). Journaling (op_id exactly-once) is INDEPENDENT of ack — send_effect_no_ack is still journaled+deduped, just no ack frame. So a 5000-op acked journaled flood models no real path. And it's BY DESIGN per REQ-HAZARD-INPUT-ACK-BACKPRESSURE (acked = one-at-a-time/bounded; flood = ackless), not a product gap. FIX: driver -> send_effect_no_ack. Facet-B stays non-vacuous (still a real sustained journaled flood via the real operator seam, still asserts the attach is serviced).

3. g2_no_commit_deadline -> NOT the ack root, SEPARATE. It does not flood — 4x controller.send_input + reads. op_flushed=false is the InjectFloor commit-deadline path: binary never commits -> floor opens+buffers operator keys -> after the 400ms deadline the floor must RELEASE + flush buffered keys to the PTY. The fail = buffered keys didn't flush/echo within 8s on the quiet Linux box. Could be (a) a real floor-release bug on forkpty, or (b) the 400ms deadline / 8s echo-wait timing. NEEDS its own trace — different subsystem (translation::InjectFloor), not SendHalf. I'll dig g2 separately.

RECOMMENDATION: apply the driver -> send_effect_no_ack fix to p0 + a_journaled NOW (clean test fix, product untouched, matches REQ-HAZARD-INPUT-ACK-BACKPRESSURE); investigate g2 as a distinct item. Your numbers (log_lock_wait≈0 + the wait on the dispatch_input ACK send) will corroborate 1+2.

WANT ME TO apply 1+2 (drivers -> send_effect_no_ack) + push, then dig g2? Your gate. Product stays untouched. -todlando
EOF

Send per-test verdict + recommendation to doyle


Per-test verdict delivered: p0 + a_journaled are clean test mis-models (driver → `send_effect_no_ack`, product untouched, matches REQ-HAZARD-INPUT-ACK-BACKPRESSURE); g2 is a separate InjectFloor commit-deadline/floor-release item. Recommended applying 1+2 + digging g2 separately — awaiting doyle's gate + his corroborating kitsubito numbers (log_lock_wait≈0 + the ACK-send wait) before I push any fix.

Holding deep g2 work until doyle confirms it's mine to chase (avoid speculative effort while the gate question is open). Ready to apply 1+2 or dig g2 on his word.


ScheduleWakeup

Delivered per-test root-cause verdict; awaiting doyle's gate (apply driver→no_ack fix / assign g2) + his corroborating kitsubito trace numbers via the armed Monitor — fallback heartbeat.

Prompt: <<autonomous-loop-dynamic>>


<owl_messages>
<owl_message from="doyle" priority="high">ROOT CAUSE ACCEPTED — AIRTIGHT. gdb (Thread6 parked in ACK send_frame on SendHalf) + your source chain (ack=true only-path, real operator = send_effect_no_ack per attach.rs:205 / REQ-HAZARD-INPUT-ACK-BACKPRESSURE) + my trace numbers (drain appends=566000, OutputLog acquire_wait_total=0.43s total / 41ms max = NOT the bottleneck -> suspect b dead). The wedge is the per-op ACK self-contending vs flood-output on the driver-conn SendHalf, and ONLY fires because the test floods with the acked seam. PRODUCT IS SOUND — ackless real path already correct. This is a TEST mis-model the two-phase split exposed by removing load noise.

GATED DIRECTION — do in THIS order, push when done, I validate each on kitsubito:
1. g2 FIRST (don't lump): confirm whether g2's driver is the SAME acked-flood root or a genuinely separate stall (op_flushed=false is a floor-release/commit-deadline assertion, different shape). Report which. If same root -> same fix; if separate -> separate analysis before any change.
2. NON-VACUITY GATE for a_journaled (load-bearing): confirm send_effect_no_ack STILL carries the op_id and goes through the journal apply_once (the journaled-input-in-flight condition). a_journaled exists to prove facet B on the JOURNALED path; if no_ack bypasses the journal, switching makes it vacuous -> then we need the ackless-but-journaled seam, not a bare no_ack. Show me the no_ack path still journals.
3. Apply driver -> send_effect_no_ack on the gates whose root+non-vacuity you've confirmed (p0 for sure; a_journaled iff #2 holds; g2 iff #1 same-root). Keep each assertion intact + non-vacuous (still a sustained flood via the REAL operator seam). Tag/comment the seam choice referencing REQ-HAZARD-INPUT-ACK-BACKPRESSURE so the next reader knows acked-flood is the wrong model.
4. Product UNTOUCHED. traceable-reqs green. Push to v0.13.0-p1b-ack-deadlock; I run all 3 isolated on kitsubito + then fold + --no-fail-fast CI.

Superb root-cause work — gdb+trace+source triangulation is exactly right. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


Next wakeup scheduled for 22:58:00 (in 1001s). Nothing more to do this turn — the harness re-invokes you when the wakeup fires or a task-notification arrives.


doyle (high-priority) acked: **ROOT CAUSE ACCEPTED — AIRTIGHT.** His numbers nail it: drain appends=566000 but OutputLog acquire_wait=0.43s total/41ms max = not the bottleneck → suspect b dead; the wedge is the per-op ACK self-contending with flood-output on the driver SendHalf, fires only because the test floods the acked seam. Product sound. Gated 4-step order: g2 first (same-root or separate?), a_journaled non-vacuity gate (no_ack must still journal), then apply where confirmed, product untouched, push.

Both prerequisite checks already resolve from code I've read:
- **g2 = SEPARATE** (4 `send_input` keystrokes + InjectFloor commit-deadline; not the acked-flood) → no change, separate analysis.
- **a_journaled non-vacuity HOLDS**: dispatch_input (1660-1694) runs `journal.apply_once` on `op_id == Some(_)` **regardless of ack**; `req.ack` only gates the ACK reply. `send_effect_no_ack` sends `op_id: Some, ack: false` → still journaled exactly-once.

Acking + executing the gated plan.


Bash

$OWL send doyle <<'EOF'
ACK — root accepted, numbers airtight (566k appends but OutputLog wait 0.43s/41ms = not it; ACK-on-SendHalf is). Executing your gated order:

#1 g2 = SEPARATE (confirmed in source, NO change). g2 does NOT flood the acked seam — it sends 4x controller.send_input + reads; op_flushed=false is the translation::InjectFloor commit-deadline/floor-release path (binary never commits -> floor opens+buffers operator keys -> after 400ms deadline floor must RELEASE+flush to PTY). Different subsystem, different shape. Per your rule (separate -> separate analysis before any change) g2 gets NO seam change now; I'll dig it as its own item after this push.

#2 NON-VACUITY HOLDS for a_journaled (source-confirmed). dispatch_input (broker.rs:1660-1694): `match req.op_id { Some(op) => { journal.apply_once((sid,op), PtyWrite, ||enqueue); if req.ack { send ACK } } }`. The journal apply_once runs on op_id==Some REGARDLESS of ack; `req.ack` ONLY gates the ACK frame. send_effect_no_ack sends op_id:Some + ack:false -> STILL journaled exactly-once, in-flight. So switching keeps the journaled-input-in-flight condition -> facet B stays non-vacuous.

#3 APPLYING NOW: a_journaled + p0 drivers `w.send_effect` -> `w.send_effect_no_ack` (the REAL operator seam, attach.rs:205). Removing the now-vestigial cfg(unix) pacing I added under the refuted CPU-storm hypothesis (the real fix is no_ack; doyle's 566k-append/0.43s data shows fast flood is fine ackless) + adding a comment citing REQ-HAZARD-INPUT-ACK-BACKPRESSURE so the next reader knows acked-flood is the wrong model. Assertions UNTOUCHED. Product UNTOUCHED.

Build + traceable-reqs green, then push to v0.13.0-p1b-ack-deadlock for your kitsubito validation. g2 dig follows. -todlando
EOF

Ack + confirm g2-separate + non-vacuity, start applying


<owl_messages>
<owl_message from="doyle" priority="high">GATE RULING:

p0 + a_journaled: APPROVED — apply driver -> send_effect_no_ack + push. Your non-vacuity proof for a_journaled is the key unlock: journaling (op_id/exactly-once) is INDEPENDENT of ack, so no_ack stays journaled+deduped -> facet-B remains a real sustained JOURNALED flood via the REAL operator seam, still asserts the attach is serviced. All acked callers confirmed bounded/one-at-a-time (attach.rs:240 one-shot, shellchan one-at-a-time) = no sustained-acked path in prod. Clean test fix, product untouched. Tag the seam choice w/ a comment citing REQ-HAZARD-INPUT-ACK-BACKPRESSURE so the next reader knows acked-flood is the wrong model.

g2: HOLD — do NOT apply any fix or timing-bump yet. It's a SEPARATE root (InjectFloor commit-deadline/floor-release, not SendHalf). op_flushed=false could be a REAL floor-release bug on forkpty — that would be the ONE genuine product issue in this cluster, and I won't let it be masked by a timing-bump. Root-cause it FIRST (same gdb+trace rigor): when the 400ms commit deadline fires with no {commit}, does the InjectFloor actually RELEASE + flush the buffered operator keys to the PTY on forkpty? Instrument the floor-release path; show whether the keys flush-but-slow (timing) or never-flush (real bug). Report before any change.

PLAN: push p0+a_journaled now; I validate both isolated on kitsubito (expect PASS with no_ack). In parallel dig g2. Don't fold to delivery-control yet — I'll fold once all 3 are green (p0+a_journaled fixed, g2 root-caused+resolved). traceable green, product untouched. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.


doyle (high-priority) acked: **p0 + a_journaled APPROVED** (apply no_ack + push, tag w/ REQ-HAZARD-INPUT-ACK-BACKPRESSURE). **g2 HOLD** — do NOT touch; it could be the ONE real product issue (InjectFloor floor-release on forkpty); root-cause first with gdb+trace rigor, report before any change. Don't fold until all 3 green.

Applying p0 + a_journaled. Reading the current driver regions for precise edits.


496	    // ── A REAL flooding child that IGNORES stdin (its PTY input buffer fills). ──
497	    let mut spawner = connect_retry(&name);
498	    let sid = spawner
499	        .spawn_session(flood_spawn_req("wedge-jrnl-ep"))
500	        .expect("spawn flood child");
501
502	    // ── The WEDGE driver: on its OWN connection, pump journaled input
503	    //    (op_id-carrying send_effect → broker dispatch_input → journal.apply_once
504	    //    runs write_input INSIDE the global journal lock). On Unix the full input
505	    //    buffer parks write_input, holding the lock; on Windows ConPTY absorbs it.
506	    //    The driver fires a BOUNDED burst of MODEST-sized ops (so a backed-up
507	    //    broker IPC never blocks the driver's own socket write — the burst always
508	    //    completes and the thread exits cleanly, never hanging the test process)
509	    //    and then idles, checking the stop flag. ──
510	    let stop = Arc::new(AtomicBool::new(false));
511	    let pumped = Arc::new(AtomicU64::new(0));
512	    let driver_name = name.clone();
513	    let driver_stop = Arc::clone(&stop);
514	    let driver_pumped = Arc::clone(&pumped);
515	    let driver = thread::spawn(move || {
516	        // PUMP carrier (cold_start_pump): the reader thread drains the driver's
517	        // OWN incoming frames (KIND_APPLIED acks + session output) so the broker's
518	        // reply side never backs up and blocks reading our input — which would
519	        // deadlock send_effect on a plain Whole conn (observed: it stalled after a
520	        // few ops). The pump keeps the driver's socket healthy so it can keep a
521	        // journaled write in-flight without ever wedging the CLIENT.
522	        let mut w = match Brain::cold_start_pump(&driver_name, 1, Duration::from_secs(2)) {
523	            Ok(b) => b,
524	            Err(_) => return,
525	        };
526	        // Bind the driver brain to the same session so send_effect has a target.
527	        if w.attach(sid, 0).is_err() {
528	            return;
529	        }
530	        let chunk = vec![b'W'; 16 * 1024];
531	        let mut op = 1u64;
532	        // Sustain journaled input: keep ops in flight (and, on Unix, parked under
533	        // the journal lock) until teardown. Each send_effect writes a frame; the
534	        // pump reader drains replies concurrently so the write never deadlocks.
535	        while !driver_stop.load(std::sync::atomic::Ordering::Relaxed) {
536	            if w.send_effect(op, &chunk).is_err() {
537	                break;
538	            }
539	            driver_pumped.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
540	            op += 1;
541	            // Drain queued replies (acks/output) so the carrier stays healthy, then
542	            // PACE with an explicit floor sleep. The doc above says "sustain a write
543	            // in-flight", but the bare `while !stop` pumped ~6094 ops and pegged the
544	            // core, out-competing the SIBLING attacher thread for scheduling under
545	            // contention so its subscribe was never serviced in 30s (the kitsubito
546	            // reliability victim, subscribed=false). A small sleep keeps a journaled
547	            // write continuously in-flight (facet-B's load is intact) while yielding
548	            // the CPU the attacher needs — sustained, not storming.
549	            let _ = w.read_event_until(Some(std::time::Instant::now() + Duration::from_millis(20)));
550	            // Unix-only: the CPU storm is a forkpty problem. On Windows ConPTY

    // ── A REAL flooding child that IGNORES stdin (its PTY input buffer fills). ──
    let mut spawner = connect_retry(&name);
    let sid = spawner
        .spawn_session(flood_spawn_req("wedge-jrnl-ep"))
        .expect("spawn flood child");

    // ── The WEDGE driver: on its OWN connection, pump journaled input
    //    (op_id-carrying send_effect → broker dispatch_input → journal.apply_once
    //    runs write_input INSIDE the global journal lock). On Unix the full input
    //    buffer parks write_input, holding the lock; on Windows ConPTY absorbs it.
    //    The driver fires a BOUNDED burst of MODEST-sized ops (so a backed-up
    //    broker IPC never blocks the driver's own socket write — the burst always
    //    completes and the thread exits cleanly, never hanging the test process)
    //    and then idles, checking the stop flag. ──
    let stop = Arc::new(AtomicBool::new(false));
    let pumped = Arc::new(AtomicU64::new(0));
    let driver_name = name.clone();
    let driver_stop = Arc::clone(&stop);
    let driver_pumped = Arc::clone(&pumped);
    let driver = thread::spawn(move || {
        // PUMP carrier (cold_start_pump): the reader thread drains the driver's
        // OWN incoming frames (KIND_APPLIED acks + session output) so the broker's
        // reply side never backs up and blocks reading our input — which would
        // deadlock send_effect on a plain Whole conn (observed: it stalled after a
        // few ops). The pump keeps the driver's socket healthy so it can keep a
        // journaled write in-flight without ever wedging the CLIENT.
        let mut w = match Brain::cold_start_pump(&driver_name, 1, Duration::from_secs(2)) {
            Ok(b) => b,
            Err(_) => return,
        };
        // Bind the driver brain to the same session so send_effect has a target.
        if w.attach(sid, 0).is_err() {
            return;
        }
        let chunk = vec![b'W'; 16 * 1024];
        let mut op = 1u64;
        // Sustain journaled input: keep ops in flight (and, on Unix, parked under
        // the journal lock) until teardown. Each send_effect writes a frame; the
        // pump reader drains replies concurrently so the write never deadlocks.
        while !driver_stop.load(std::sync::atomic::Ordering::Relaxed) {
            if w.send_effect(op, &chunk).is_err() {
                break;
            }
            driver_pumped.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
            op += 1;
            // Drain queued replies (acks/output) so the carrier stays healthy, then
            // PACE with an explicit floor sleep. The doc above says "sustain a write
            // in-flight", but the bare `while !stop` pumped ~6094 ops and pegged the
            // core, out-competing the SIBLING attacher thread for scheduling under
            // contention so its subscribe was never serviced in 30s (the kitsubito
            // reliability victim, subscribed=false). A small sleep keeps a journaled
            // write continuously in-flight (facet-B's load is intact) while yielding
            // the CPU the attacher needs — sustained, not storming.
            let _ = w.read_event_until(Some(std::time::Instant::now() + Duration::from_millis(20)));
            // Unix-only: the CPU storm is a forkpty problem. On Windows ConPTY


<owl_messages>
<owl_message from="doyle" priority="high">ACK plan — all correct. Pacing removal APPROVED (it was the refuted CPU-storm fix; no_ack is the real one, and my 566k-append/0.43s data proves fast ackless flood is fine). Non-vacuity proof solid (apply_once gated on op_id==Some, ack only gates the ACK frame). g2 held as separate = right.

Push p0+a_journaled when green; I'll validate BOTH isolated on kitsubito immediately (expect PASS: subscribed=true, got_output=true), then run the FULL inject_control_wedge binary to confirm no sibling regression from the seam/pacing change. g2 dig in parallel — send findings when you have the floor-release trace. I fold to delivery-control only when all 3 are green. Standing by for your push. -doyle</owl_message>
</owl_messages>

[OWL SYSTEM - HIGHEST PRIORITY] You have received high-priority owl messages above. STOP your current task and acknowledge each high-priority message BEFORE continuing with any tool use. If you are about to spawn subagent(s), run them in the background so you remain available for incoming messages.