---
name: v013-forkpty-groundtruth
description: "v0.13.0 wedge gates are RED on Linux forkpty — the real unsolved core is a live-attach forward-gap reject on ring-roll-behind; b4+skip-to-live don't cover it. Code-gate PASSes refuted by real integration."
metadata: 
  node_type: memory
  type: project
  originSessionId: 05672504-0818-43ea-8de3-4797dc3039b6
---

2026-06-21 (doyle gen-19). Combined forkpty CI run (wedge-trace-v4 @7074ba0 = b4 + skip-to-live + w5_a2-CutB + pump-carrier, Linux kitsubito, PR#27 run 27903471062) REFUTES the "near-fold" board. 5 RED:

**HEADLINE — a_journaled (b4's OWN gate) is RED on Linux.** `ctrl event: "output gap: got seq 5968 want 4504"`. A LIVE controller attach falls behind the hard forkpty flood → ring rolls 4504..5967 out → broker sends 5968 → the serving brain's LEGACY reject-gap (read_event) FATALS → attach_received_pty_output=false. b4 DID decouple the drain (drain_appends→98993, c1-EVICT fires) but "decoupled" ≠ "attach receives output" — the attach dies EARLY (~seq 4504), before any eviction. **Windows passed run31 only because ConPTY floods slower → never falls far enough to roll the ring. Platform-masked false-green; Linux was never actually green.** Prior run c240643 (no skip-to-live) Linux test ALSO failed → pre-existing, not introduced.

**THE REAL CORE (keystone, bigger than "g2 reconcile"):** a live attach (controller AND viewer) that falls behind the ring under flood gets a FORWARD SEQ GAP and read_event fatals. b4 (drain-decouple) + skip-to-live (snap-above is EVICTION-triggered, KIND_VIEWER_EVICTED only) do NOT cover the ring-roll-behind forward-gap on the LIVE stream. Same root kills a_journaled / p0_paste / attach.rs:1071 wedged_viewer / g2-output. Candidate (NOT ruled): generalize snap-above resume-from-floor to the live forward-gap, not just the eviction marker. Viewer = B2-safe; CONTROLLER = B2 question (advances delivered_through). Needs harness read (controller-vs-viewer + does it run serve_attach) + design with todlando.

**w5_a2 Cut B REFUTED:** cby_after_abandon=Some(None) with the QUIET child too → clear is NOT output-path (my ruling premise WRONG; todlando's Disconnected-output mechanism incomplete). Real: drop(operator) over loopback closes the socket → broker detects CONN CLOSE (FIN) → clears, regardless of output. In-process loopback drop CANNOT stage a true half-open (no-FIN) abandon. w5_a2 asserts an unstageable premise on loopback.

**g2 CONFIRMED no-flood:** delivered=TRUE+delivered2=TRUE (acceptance non-blocking, my no-flood finding right). Failure = op_flushed=false + raw_fallback_reached=false = injected bytes' ECHO never reaches the ring = same attach-receives-no-output family, NOT dispatch starvation.

**LESSON (recurring [[v0121-realharness-reopen]]):** I issued code-gate PASS on skip-to-live + w5_a2 from source-read + GREEN UNITS; real forkpty integration refuted both. Green-units ≠ green-integration. Gate wedge fixes on the forkpty run BEFORE declaring PASS, not after.

**UPDATE 2026-06-21 (run 27919479506, keystone d79f02e cherry-picked):** the ring-roll-snap keystone WORKS. a_journaled ✅ (attach_received_pty_output=TRUE) + p0_paste ✅ (attach_received_output=TRUE) — both headline RED gates GREEN. Phase B 87 pass / 3 fail (was 5). The VIEWER trio is done (b4 viewer-starve + skip-to-live + ring-roll-snap). 3 remain, all NON-viewer:
- **wedged_viewer_does_not_stall_controller (attach.rs:1071)** = a b4 SIDE-EFFECT on the CONTROLLER. b4 made the controller a non-blocking try_send that DROPS frames on a full channel; pre-b4 the inline sleep-poll blocked the drain to the controller rate (no drops, no gaps). Now a controller behind its own echo drops → forward gap → controller brain strict reject-gap (B2) FATALS (`got 6134 want 4643`). Controller CAN'T snap (B2). Fix direction (gate-pending): controller resume-from-floor (re-subscribe from delivered_through = frozen contiguous cursor → RE-FETCH dropped frames from ring, exactly-once IF ring retains). OPEN B2 question: can ring roll past delivered_through? Repro-first dispatched to todlando. This is the real remaining viewer-drain-MILESTONE blocker (b4 caused it).
- **w5_a2** = REQ-HAZARD-DRIVEN-BY-SELFHEAL (controller self-heal). Ruling CONFIRMED: flip to assert conn-close-evict-correct; defer no-FIN half-open to REQ-HAZARD-DRIVEN-BY-IDLE-REMOTE-EVICT. Separate subsystem.
- **g2** = REQ-MSG-IDLE-TRANSLATION-BINARY (W2 commit-deadline floor). delivered=TRUE; op_flushed/raw_fallback FALSE = floor-flush under forkpty echo OR output_contains_within's raw legacy-reject-gap brain.attach. Separate subsystem, does NOT gate the viewer-drain fold.

**UPDATE 2 (run 27920375097, controller-gap-resume f0c2ee0 cherry-picked):** the fix FIRES correctly — wedged_viewer now surfaces `ControllerIrrecoverablyBehind{floor:4500}` (clean, not a hang). a_journaled + p0_paste stay GREEN. ROOT (CONTEXT.md:343 grounding): the doc STILL says "ONE authoritative BLOCKING controller ... STRICT exactly-once" but b4 made it NON-BLOCKING-drop (CONTROLLER_CHANNEL_DEPTH=4096 try_send+drop) → **stale doc, design contradiction**. The wedged_viewer 200KB burst (>8192 frames) rolls the ring (4096) past the frozen delivered_through (4500) → frames GONE → exactly-once genuinely impossible → ControllerIrrecoverablyBehind is CORRECT. Burst regimes: <4096 no-gap; 4096-8192 recoverable re-fetch; >8192 irrecoverable. RULING dispatched: (A) wedged_viewer (REQ-HAZARD-VIEWER-ISOLATION) asserts controller NOT-STALLED = prompt clean outcome (ISOLATED OR clean-irrecoverable), don't `.expect`-panic on the error — robust, no fragile regime-2 sizing; recoverable re-fetch already proven by the in-process UNIT. (C) CONTEXT.md amendment: controller non-blocking-drop, exactly-once = best-effort (re-fetch while ring retains; ring-exceeding flood = marked data-loss). (D) DEFER to v0.13.x: irrecoverable handling = snap-to-live-with-loss-marker (better UX than error-and-end; lost frames gone regardless) under REQ-HAZARD-CONTROLLER-IRRECOVERABLE-BEHIND.

**UPDATE 3 (run 223eb8b=1ac64b9, milestone signal): VIEWER-DRAIN MILESTONE COMPLETE on forkpty.** Phase B 88 pass / 2 fail. wedged_viewer PASS ✅ (accepts clean irrecoverable) + a_journaled ✅ + p0_paste ✅. The full viewer-drain set is GREEN on Linux forkpty: b4 viewer-starve + skip-to-live + ring-roll-snap + controller-gap-resume. The 2 remaining fails are NOT viewer-drain: (1) w5_a2 = ARTIFACT (wedge vehicle runs the un-flipped test; Linux value Some(None) → the b4-branch #[cfg(unix)] assert Some(None) passes = proven-by-data); (2) g2 = separate W2 subsystem (REQ-MSG-IDLE-TRANSLATION-BINARY floor). 

NEXT = OPERATOR-GATED FOLD: merge the b4 branch (b4 + skip-to-live d79f02e + ring-roll-snap + controller-gap-resume f0c2ee0 + w5_a2-flip e363b26 + wedged_viewer-rework 1ac64b9 + tag-comment 2d4dae8) → delivery-control (which has pump-carrier-fix) → full matrix ×3 both runners → activate int stages (rule 5) → deployah v0.13.0. SCOPE DECISION pending operator: ship viewer-drain + defer g2 (W2) to v0.13.x, OR fix g2 first.

**UPDATE 4 (operator chose FIX-G2-FIRST; g2 root localized repro-first):** g2 root CONFIRMED via forkpty trace ladder — NOT a W2 floor bug. The input chain works end-to-end on Linux (dispatch_input buffered_in_floor=true → flush_inject_floor enqueued 88 → write_input 88 bytes to PTY — the FAULT-revert DOES flush, root #1 refuted; my reader-reject-gap root #2 also refuted, reader gets acc_len=0 ZERO output not a gap). The missing line = ZERO OutputLog::append → the forkpty child didn't ECHO. echo_program()=`cat` (unix)/`findstr .` (windows). `cat`'s stdout is LINE-BUFFERED on a TTY (flushes on \n); g2 keys are \r-terminated (no \n) + forkpty PTY no CR→NL (ICRNL) → cat never flushes → no echo → empty ring → reader sees nothing. Windows findstr echoes. So g2 = TEST-FIXTURE echo-observability under forkpty, the W2 floor is VINDICATED (repro-first saved "fixing" working code). FIX (todlando, fixture): \r\n terminator OR an unbuffered/per-byte unix echoer (stdbuf -o0 cat / tiny echoer); apply to any sibling cat-echo-of-\r tests. FLAGGED separate (non-blocking): does spt forkpty PTY default termios match a real interactive terminal (ICRNL/ECHO)? possible latent Enter-key gap, shells set own termios so prod likely fine. LESSON: repro-first localization (trace ladder, which-line-absent) beats blind-fixing — refuted TWO wrong root theories (floor-flush, reader-gap) before landing on the fixture.

**UPDATE 5 (g2 REAL root + GREEN):** my cat-line-buffer theory ALSO refuted (todlando: g3 hard-asserts a \r-terminated echo through the SAME cat reader + PASSES on Linux → \r echoes fine). CODE-DEFINITIVE root: xlate_choreo_fixture emits XOFF (ctrl+s 0x13) unconditionally then RETURNS in nocommit mode BEFORE the XON (ctrl+q) the happy path sends → g2 leaves the PTY XOFF-paused → forkpty IXON strands ALL echo → write_input 88 → zero append. Windows ConPTY ignores XOFF → masked. FIX (2f57fea, KEEPABLE test-fixture fix): nocommit branch emits XON before its fault-return. g2 GREEN ✅ (0.4s, was 16.7s). **ALL wedge-file failures resolved: a_journaled + p0_paste + wedged_viewer + g2 GREEN on forkpty; w5_a2 flip proven-by-data (confirmed by the fold matrix).** Phase B 89/1 (the 1 = w5_a2 wedge-vehicle artifact, green on b4 w/ flip). 

**v0.13.0 NOW FOLDING:** todlando reverts diag commits (b8ec32b+9323593) → clean tree → fold b4 branch → delivery-control (has pump-carrier-fix + registry-knob) → I gate full matrix x3 both runners → activate held int stages (REQ-HAZARD-VIEWER-STARVE/SKIP-TO-LIVE/RING-ROLL-SNAP/CONTROLLER-GAP-RESUME/MSG-IDLE-TRANSLATION-BINARY) → deployah v0.13.0 (OPERATOR-CONFIRMED at publish). KEEPABLE fixes: skip-to-live d79f02e, ring-roll-snap, controller-gap-resume f0c2ee0, w5_a2 flip e363b26, wedged_viewer rework 1ac64b9, g2 XOFF fix 2f57fea. Operator chose fix-g2-first (=ship after) — g2 now green → ship authorized. Logged FLAG (non-blocking): forkpty PTY default termios CR→NL — possible latent Enter-key gap, follow-up.

**UPDATE 6 (fold matrix run 1 — w5_a2 flake = MY ruling error):** fold pushed to v0.13.0-delivery-control @8c6b253 (clean merge: pump-carrier-fix 7a1cc68+7c09819 ff'd + b4 merged + 5 int stages activated, PR#26→main). Matrix run 1 (27925083635): ONLY w5_a2 RED on Linux — a_journaled/p0_paste/wedged_viewer/g1-g4 GREEN both runners, Windows job fully green. The w5_a2 fail = MY w5_a2-flip ruling error: this run cby_after_abandon=Some(Some) (conn-close FIN NOT detected in the 2s window) → the #[cfg(unix)] hard-assert Some(None) FAILED. Linux QUIC conn-close detection is TIMING-VARIABLE (not deterministically <2s) — the exact flake class I'd flagged for Windows, now bit unix. CORRECTED RULING: w5_a2 = CHARACTERIZATION-ONLY both platforms (remove the hard cby_after_abandon assert, keep cby_attached precondition + eprintln) — RETURNS it to its documented "characterization not hard-gate" nature; REQ-HAZARD-DRIVEN-BY-SELFHEAL's hard int is gap_b (driven_by_selfheal.rs), w5_a1/w5_a2 are the characterization arm, so no coverage loss. LESSON: never hard-assert a timing-variable QUIC conn-close outcome against a fixed read window. todlando pushing the fix → re-run matrix x3 → expect green → operator-confirmed publish.

**UPDATE 7 (x3 attempt: Windows p0_paste flake → OPERATOR chose HARDEN-FIRST):** w5_a2 char-only fix (bed1878) landed. Matrix x3: run 1/3 GREEN both runners ✅; run 2/3 Linux GREEN, Windows p0_paste FAIL (concurrent_attach_subscribed=FALSE = subscribe not serviced within the attacher's cold_start_pump 2s deadline). Intermittent (passed Windows run1 + Linux both) → NOT a regression = the known Windows-shared-runner load-contention class, likely WORSENED by this session hammering ~10 matrix runs on hfenduleam (the Windows runner = doyle's own box). HYPOTHESIS: the 2s pump deadline is too tight for the loaded box — the subscribe is serviced but >2s → net_stream_subscribe times out → subscribed=false = a MEASUREMENT artifact from the pump deadline (not the old carrier artifact, not a real wedge; the paste-wedge fix works). OPERATOR RULING (AskUserQuestion): HARDEN the Windows path before publish (not ship-around). Dispatched todlando repro-first: confirm serviced-but-late (→ widen the a_journaled+p0_paste attach-gate pump deadline / CI-headroom env-knob like SPT_REGISTRY_BUSY_TIMEOUT_MS) vs never-serviced (→ real dispatch starvation, deeper fix, don't widen=mask). doyle PAUSING CI load on hfenduleam so the repro is clean. v0.13.0 publish held until p0_paste hardened + x3 clean.

**UPDATE 8 (RESOLVED + PUBLISHED v0.13.0 counter 27, 2026-06-21).** The flake root took THREE attribution passes before landing — fix was right each time, mechanism/knob-name wrong twice:
- Pass 1 (todlando): "subscribed=false = the 30s result_rx watchdog defaulting." Shipped SPT_ATTACH_GATE_WATCHDOG_MS (default 30s/CI 120s). REFUTED by the kept source-attribution print: failure showed `source=channel after 2.016s`, NOT WATCHDOG-DEFAULT → watchdog never reached, knob ORTHOGONAL.
- Pass 2 (doyle): "attacher's cold_start_pump 2s CONNECT timed out, no fallback." REFUTED by todlando's empiric proof (DEADLINE_MS=1 fails but the connect-Err print never fired). cold_start_pump's socket connect is UNBOUNDED.
- **REAL root (final):** the `Duration` arg to `cold_start_pump` is `io_timeout` = the per-IPC-CALL READ deadline (brain.rs:311-323, call_deadline=now+io_timeout). `request_attach`→`net_open_stream` reads the journaled NetStreamOpened ack bounded by that deadline (brain.rs:1066-68). Co-tenant scheduler starvation on the shared self-hosted Windows box pushes the SERVICED ack past 2s → read Errs → request_attach Err → attacher sends (false,false) via channel ~2.0s, 2s BEFORE the 30s watchdog. NOT a connect, NOT the watchdog, NOT a product wedge. Failed even SERIALIZED (cancel-in-progress) → trigger is shared-box co-tenant load (live agents share hfenduleam), not my matrix stacking.
- **FIX @45cc479** (test+CI only, zero product diff): env-tunable `SPT_ATTACH_IPC_DEADLINE_MS` (default 2s tight / CI 30000), one helper `attach_ipc_deadline()` wired to all 9 result-gating cold_start_pump sites; attribution print moved to the request_attach Err arm; watchdog knob kept as outer ceiling. x3 serialized GREEN (per-attempt verified). Published v0.13.0 @ d263a94, counter 27, linux 3f196564… win de4ad840…, baseline v0.12.1 (DID ship, counter 26), no-fold, v0.12.1 tag kept. perri notified (adapter surfaces: [message-idle-translation-binary], [session.resume]).
- **LESSONS:** (1) the cheap source-attribution print we INSISTED on keeping is exactly what refuted pass-1 — keep diagnostic attribution in flaky gates. (2) `gh run rerun` + a short-sleep `gh run watch` attaches to the STALE completed prior attempt and false-greens — poll until status leaves "completed" before watching AND verify per-ATTEMPT conclusion, never trust watch exit code alone. (3) release counter comes from PUBLISHED spt-releases metadata, not memory derivation — see [[release-counter-from-published]].