diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ceb42e..99fc071 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable user-facing changes to **omp-spt** (the Spacetime adapter for oh-my- > Each release below is keyed to the **adapter version of truth** (the version `spt adapter list` reports and the GitHub release tag). omp-spt starts its own version line at 0.1.0; the sections from [0.17.3] down are the inherited **claude-spt** lineage this project forked from, retained for history. +## [0.3.20] - 2026-07-23 + +> Requires spt-core **v0.39.1 or newer** and Oh My Pi **v16.3.15 or newer**. Update with `spt adapter update omp-spt`, then restart existing endpoints so they load the corrected extension. + +### Fixed +- **Deaf sessions close instead of staying silently ONLINE.** Daemon-restart autostart replay could spawn sessions whose bring-up looked healthy (bind, token, listener) but whose accepted deliveries never entered an agent turn (#10). The extension now enforces delivery turn liveness on every spawn path: a message stranded on an idle session past the liveness deadline is force-resubmitted (a hung in-flight submission cannot wedge recovery), then the degraded ` · comms recovering...` rail surfaces, and on continued deafness the endpoint fails closed — pending custody is released with failure outcomes, the SPT session ends, and the hosted TUI shuts down so daemon lifecycle or the operator brings up a fresh, working session. +- **Stale endpoint-state cache is invalidated on listener death.** A restarted daemon can hold persisted activity state that no longer matches the extension's cache; the next publication after a listener death now republishes current truth instead of being skipped as already-current. + ## [0.3.19] - 2026-07-23 > Requires spt-core **v0.39.1 or newer** and Oh My Pi **v16.3.15 or newer**. Update with `spt adapter update omp-spt`, then restart existing endpoints so they load the corrected extension. diff --git a/DELIVERY-LIVENESS-PLAN.md b/DELIVERY-LIVENESS-PLAN.md new file mode 100644 index 0000000..acd82f0 --- /dev/null +++ b/DELIVERY-LIVENESS-PLAN.md @@ -0,0 +1,71 @@ +# Delivery liveness plan (issue #10) + +## Scope + +Issue #10: on adapter 0.3.19 the daemon-restart autostart replay path spawns sessions whose +bring-up looks healthy (bind, token, listener) but whose inbound deliveries never start an agent +turn — the #9 defect-B pattern, resurfacing on a spawn path the adapter cannot distinguish from +any other daemon-driven spawn (no marker in the published launch contract). Any daemon death +(OOM, update, crash) funnels through this path, silently re-deafening endpoints. + +The adapter-side root gap: the extension *trusts* that `pi.sendUserMessage` on an idle session +atomically starts a turn and has no verification that a delivered message ever produced one. A +deaf session is indistinguishable from a healthy idle one on every observable the extension +currently tracks — exactly the "dead communication path looks healthy" hazard, one rail deeper. + +## Decisions + +- **Path-independent invariant, not path detection.** The daemon replay spawn is not + distinguishable from the manifest side, so cover it (and every current and future deaf mode) + with one invariant: *an accepted listener delivery on an idle session either enters a turn + within the liveness deadline, or the endpoint visibly self-heals and, failing that, closes* + (ADR-0010 doctrine: native delivery self-heals or closes). +- **Watchdog condition**: `pendingListener` non-empty while OMP reports idle. Any real turn — + peer-triggered or operator-typed — consumes pending envelopes through the existing context + hook, so a session that stays pending+idle across a full deadline has demonstrably lost + receivability. Busy sessions never count toward the deadline. +- **Escalation ladder** on deadline expiry (deadline injectable, default 120s): + 1. First expiry: force-resubmit pending items (per-item submission epoch invalidates a hung + or stale in-flight submission so the guard flag cannot wedge recovery). + 2. Second expiry: resubmit again and surface the existing warning-styled + ` · comms recovering...` health rail (component `delivery`). + 3. Third expiry: fail closed — release pending custody with failure outcomes, end the SPT + session, shut down the hosted TUI. A deaf-but-ONLINE endpoint becomes visibly offline; + daemon-side lifecycle (or the operator) brings up a fresh, working session, which is the + operationally verified recovery (#9/#10: suspend/wake). +- **Turn liveness resets** the ladder: `agent_start` clears the attempt counter and timer. +- Stale `endpointState` cache invalidation on listener death: a restarted daemon may hold + persisted state that no longer matches the extension's cache; after a listener death the next + publication must not be skipped by the cache. +- Daemon-side aspects are findings, not workarounds (public-surface-only constraint): + - F-033: autostart replay yields a session whose live deliveries never kick turns while + `spt send` reports plain `SENT(WAN)` — needs daemon-side root-cause; adapter now closes + deaf sessions instead of presenting them healthy. + - The 1.4G daemon RSS / OOM ask in issue #10 is spt-core territory — recorded in the same + finding for doyle. + +## Non-goals + +- No bring-up probe turn (spends tokens on every healthy spawn; watchdog only acts when real + custody is at stake). +- No daemon-side replay change (not our repo; finding filed). +- No second durable spool, receipts, or acknowledgment above spt-core custody. + +## Tasks + +1. Requirement `REQ-HAZARD-DELIVERY-TURN-LIVENESS` (doc/impl/unit) + KNOWN-HAZARDS entry + + harness-contract failure-behavior wording. +2. Extension: liveness watchdog state, arm/disarm sites (accept, submit, resubmit, agent_start, + context consumption, shutdown), escalation ladder, submission-epoch resubmit hardening, + endpointState cache invalidation on listener death. +3. Focused tests: deadline expiry resubmits; hung submission cannot block recovery; second + expiry shows degraded health; third expiry fails closed with failure outcomes; agent_start + resets the ladder; busy sessions never expire; watchdog disarms when context consumes items. +4. Findings ledger F-033; CHANGELOG; version bump 0.3.20 across manifest/package/Cargo/docs + (version-consistency gate is authoritative). +5. Gates: `node --test tests/omp-extension.mjs`, `sh ci/run-gates.sh`, `traceable-reqs check`. + +## Gate + +Extension tests, deterministic gates, and traceability all pass; the plan's invariant has a +focused failing-then-passing test for each ladder rung. diff --git a/README.md b/README.md index 9069907..edc61aa 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ release with its bundled native extension and OMP plugin skills. ## Install -The v0.3.19 release asset supports **x86_64 Windows and x86_64 Linux only**. It +The v0.3.20 release asset supports **x86_64 Windows and x86_64 Linux only**. It contains Windows MSVC, Linux GNU, and static Linux musl x86_64 helpers, but no macOS or Arm64 payload. The musl helper is a compatibility tier for OMP-capable hosts; it is not a generic Alpine support claim. Pick a supported shell below; diff --git a/adapter/omp-spt.toml b/adapter/omp-spt.toml index 7cc8421..7c172dc 100644 --- a/adapter/omp-spt.toml +++ b/adapter/omp-spt.toml @@ -9,7 +9,7 @@ [adapter] name = "omp-spt" kind = "harness" -version = "0.3.19" +version = "0.3.20" # v0.39.1 preserves broker-hosted controllability across the required # `api bind --set-session-id` -> `api listen --session-id` sequence. min_spt_core_version = "0.39.1" diff --git a/adapter/strings/omp-spt.mjs b/adapter/strings/omp-spt.mjs index 8c3e43b..aab2271 100644 --- a/adapter/strings/omp-spt.mjs +++ b/adapter/strings/omp-spt.mjs @@ -408,6 +408,8 @@ const DEFAULT_ACCEPTED_QUEUE_LIMIT = 128; const DEFAULT_ACCEPTED_BYTES_LIMIT = 1024 * 1024; const DEFAULT_SHUTDOWN_BUDGET_MS = 1_800; const DEFAULT_SHUTDOWN_COMMAND_TIMEOUT_MS = 300; +const DEFAULT_DELIVERY_LIVENESS_MS = 120_000; +const DEFAULT_DELIVERY_LIVENESS_ATTEMPT_LIMIT = 3; async function mapWithConcurrency(items, concurrency, task) { const results = new Array(items.length); @@ -668,6 +670,12 @@ export function createOmpSpt(overrides = {}) { ]; const listenerStableMs = overrides.listenerStableMs === false ? undefined : (overrides.listenerStableMs ?? 30_000); + const deliveryLivenessMs = + overrides.deliveryLivenessMs === false + ? undefined + : (overrides.deliveryLivenessMs ?? DEFAULT_DELIVERY_LIVENESS_MS); + const deliveryLivenessAttemptLimit = + overrides.deliveryLivenessAttemptLimit ?? DEFAULT_DELIVERY_LIVENESS_ATTEMPT_LIMIT; const checkUpdates = overrides.checkUpdates ?? true; const updateProbeTimeoutMs = overrides.updateProbeTimeoutMs ?? 1_500; const fetchLatestAdapterVersion = @@ -782,6 +790,8 @@ export function createOmpSpt(overrides = {}) { let listenerRestartCount = 0; let listenerStableTimer; let restartTimer; + let livenessTimer; + let livenessAttempts = 0; let stateRetryTimer; let stateRetryAttempt = 0; let bindPromise; @@ -1573,6 +1583,10 @@ export function createOmpSpt(overrides = {}) { clearTimer(listenerStableTimer); listenerStableTimer = undefined; } + if (livenessTimer !== undefined) { + clearTimer(livenessTimer); + livenessTimer = undefined; + } const child = listener; listener = undefined; listenerBuffer = ""; @@ -1670,7 +1684,11 @@ export function createOmpSpt(overrides = {}) { listenerSenderCounts.set(sender, ordinal); item.stub = senderStub(sender, ordinal); } + // A liveness resubmission supersedes any in-flight submission; its stale + // callbacks must not settle the item a second time. + const epoch = item.submissionEpoch ?? 0; const failSubmission = (error) => { + if ((item.submissionEpoch ?? 0) !== epoch) return; item.submissionPending = false; listenerAwaitingProvider.delete(item); const index = pendingListener.indexOf(item); @@ -1689,6 +1707,7 @@ export function createOmpSpt(overrides = {}) { return true; } void submission.then(() => { + if ((item.submissionEpoch ?? 0) !== epoch) return; item.submissionPending = false; if ( !stopping && @@ -1707,6 +1726,87 @@ export function createOmpSpt(overrides = {}) { function resubmitUnobservedListenerItems() { for (const item of pendingListener) submitListenerItem(item); + armDeliveryWatchdog(); + } + + function ompIdle() { + return typeof runtimeCtx?.isIdle === "function" + ? runtimeCtx.isIdle() + : !agentActive && desiredState !== "busy"; + } + + // [impl->REQ-HAZARD-DELIVERY-TURN-LIVENESS] + function armDeliveryWatchdog() { + if (deliveryLivenessMs === undefined || stopping) return; + if (livenessTimer !== undefined || pendingListener.length === 0) return; + livenessTimer = setTimer(() => { + livenessTimer = undefined; + void enforceDeliveryLiveness(); + }, deliveryLivenessMs); + livenessTimer?.unref?.(); + } + + function forceResubmitListenerItem(item) { + item.submissionEpoch = (item.submissionEpoch ?? 0) + 1; + item.submissionPending = false; + listenerAwaitingProvider.delete(item); + submitListenerItem(item); + } + + // [impl->REQ-HAZARD-DELIVERY-TURN-LIVENESS] + async function enforceDeliveryLiveness() { + if (stopping) return; + if (pendingListener.length === 0) { + livenessAttempts = 0; + return; + } + if (!ompIdle()) { + livenessAttempts = 0; + armDeliveryWatchdog(); + return; + } + livenessAttempts += 1; + const staleness = new Error( + `accepted deliveries entered no turn across ${livenessAttempts} × ${deliveryLivenessMs}ms on an idle session`, + ); + if (livenessAttempts >= deliveryLivenessAttemptLimit) { + markCommsFailure( + "delivery", + "omp-spt is closing this endpoint: it accepts deliveries but never runs their turns", + staleness, + ); + await failDeafSession(staleness); + return; + } + if (livenessAttempts > 1) { + markCommsFailure( + "delivery", + "omp-spt deliveries are not entering turns; resubmitting", + staleness, + ); + } + for (const item of [...pendingListener]) forceResubmitListenerItem(item); + armDeliveryWatchdog(); + } + + // [impl->REQ-HAZARD-DELIVERY-TURN-LIVENESS] + async function failDeafSession(error) { + if (stopping && shutdownMode) return teardownPromise ?? Promise.resolve(); + if (fatalPromise) return fatalPromise; + fatalPromise = (async () => { + ui?.setStatus("omp-spt", "spt delivery dead"); + logError( + "omp-spt accepted deliveries never entered a turn; closing the deaf endpoint", + error, + ); + try { + await teardownSession("endpoint lost turn receivability"); + } catch (teardownError) { + logError("omp-spt session teardown failed", teardownError); + } + runtimeCtx?.shutdown(); + })(); + return fatalPromise; } function admitOverflowItem() { @@ -1715,11 +1815,17 @@ export function createOmpSpt(overrides = {}) { overflowItem = undefined; pendingListener.push(item); submitListenerItem(item); + armDeliveryWatchdog(); } // [impl->REQ-OMP-COMMS-RECOVERY] function handleListenerDeath(reason) { listenerBuffer = ""; + // [impl->REQ-HAZARD-DELIVERY-TURN-LIVENESS] + // A dead listener often means daemon churn; the restarted daemon's endpoint + // state can differ from this cache, so the next publication must not be + // skipped as already-current. + endpointState = undefined; if (listenerStableTimer !== undefined) { clearTimer(listenerStableTimer); listenerStableTimer = undefined; @@ -1832,6 +1938,7 @@ export function createOmpSpt(overrides = {}) { } pendingListener.push(event); submitListenerItem(event); + armDeliveryWatchdog(); } }); child.stderr.on("data", (chunk) => @@ -2032,6 +2139,9 @@ export function createOmpSpt(overrides = {}) { pi.on("agent_start", async () => { if (stopping) return; + // [impl->REQ-HAZARD-DELIVERY-TURN-LIVENESS] + livenessAttempts = 0; + clearCommsFailure("delivery"); turnCompletionPromise = undefined; turnAssistantBaseline = observedAssistantBaseline; turnContextObserved = false; diff --git a/adapter/strings/package.json b/adapter/strings/package.json index 51547eb..8042c3b 100644 --- a/adapter/strings/package.json +++ b/adapter/strings/package.json @@ -1,6 +1,6 @@ { "name": "omp-spt", - "version": "0.3.19", + "version": "0.3.20", "private": true, "type": "module", "omp": { diff --git a/docs-site/llms-full.txt b/docs-site/llms-full.txt index 62400b4..4680c72 100644 --- a/docs-site/llms-full.txt +++ b/docs-site/llms-full.txt @@ -78,7 +78,7 @@ You need [Oh My Pi 16.3.15 or newer](https://github.com/can1357/oh-my-pi/commit/ and [`spt-core`](https://sabermage.github.io/spt-releases). There is no separate plugin installation step because the adapter release carries its OMP plugin skills. -The v0.3.19 `omp-spt` release supports **x86_64 Windows and x86_64 Linux only**. +The v0.3.20 `omp-spt` release supports **x86_64 Windows and x86_64 Linux only**. Its archive contains Windows MSVC, Linux GNU, and static Linux musl x86_64 helpers, but no macOS or Arm64 payload. The musl helper is a compatibility tier for OMP-capable hosts; it is not a generic Alpine support claim. @@ -251,6 +251,13 @@ its restart budget, the adapter fails closed: it reports the problem in the OMP outcomes for pending messages, ends the Spacetime session, and shuts down the hosted TUI. A half-bound endpoint is never presented as healthy. + +Accepted deliveries must also prove turn liveness. If a delivered message sits on an idle session +past the liveness deadline, the extension resubmits it, then surfaces the degraded-comms rail, and +on continued deafness closes the endpoint the same fail-closed way. A session whose bring-up looks +healthy but whose deliveries never run turns — as the daemon-restart autostart replay produced — +becomes visibly offline instead of silently swallowing custody. + ## Updates and packaging Each release is one fat `adapter.spt` archive containing the supported adapter binaries, manifest, diff --git a/docs-site/src/quickstart.md b/docs-site/src/quickstart.md index 6b59a92..7b95568 100644 --- a/docs-site/src/quickstart.md +++ b/docs-site/src/quickstart.md @@ -9,7 +9,7 @@ You need [Oh My Pi 16.3.15 or newer](https://github.com/can1357/oh-my-pi/commit/ and [`spt-core`](https://sabermage.github.io/spt-releases). There is no separate plugin installation step because the adapter release carries its OMP plugin skills. -The v0.3.19 `omp-spt` release supports **x86_64 Windows and x86_64 Linux only**. +The v0.3.20 `omp-spt` release supports **x86_64 Windows and x86_64 Linux only**. Its archive contains Windows MSVC, Linux GNU, and static Linux musl x86_64 helpers, but no macOS or Arm64 payload. The musl helper is a compatibility tier for OMP-capable hosts; it is not a generic Alpine support claim. diff --git a/docs-site/src/reference/harness-contract.md b/docs-site/src/reference/harness-contract.md index ae9e93a..4b3cd66 100644 --- a/docs-site/src/reference/harness-contract.md +++ b/docs-site/src/reference/harness-contract.md @@ -82,6 +82,13 @@ its restart budget, the adapter fails closed: it reports the problem in the OMP outcomes for pending messages, ends the Spacetime session, and shuts down the hosted TUI. A half-bound endpoint is never presented as healthy. + +Accepted deliveries must also prove turn liveness. If a delivered message sits on an idle session +past the liveness deadline, the extension resubmits it, then surfaces the degraded-comms rail, and +on continued deafness closes the endpoint the same fail-closed way. A session whose bring-up looks +healthy but whose deliveries never run turns — as the daemon-restart autostart replay produced — +becomes visibly offline instead of silently swallowing custody. + ## Updates and packaging Each release is one fat `adapter.spt` archive containing the supported adapter binaries, manifest, diff --git a/docs/KNOWN-HAZARDS.md b/docs/KNOWN-HAZARDS.md index 62ec82b..a6fed67 100644 --- a/docs/KNOWN-HAZARDS.md +++ b/docs/KNOWN-HAZARDS.md @@ -73,3 +73,26 @@ requirement must point at production behavior and a focused test. hosted-harness match key. Fresh and resume launch use the same resolver. - **cite:** ADR-0009 and the retired bridge incident's executable-collision finding. + +## 5. A deaf session presents as a healthy idle endpoint + + + +- **Failure:** Bring-up looks healthy — bind succeeds, the listener child runs, + `spt send` reports live delivery — but submitted messages never enter an + agent turn. The daemon-restart autostart replay produced exactly this + (issue #10): custody drains destructively into a session that will never + consume it, precisely when nobody is watching. +- **Invariant:** An accepted delivery on an idle session enters a turn within + the liveness deadline. On expiry the extension self-heals by force-resubmitting + (a hung in-flight submission cannot wedge recovery), then surfaces the + degraded-comms rail, and finally fails closed: pending custody is released + with failure outcomes, the SPT session ends, and the hosted TUI shuts down. + A deaf endpoint becomes visibly offline instead of silently ONLINE. +- **Mapping / notes:** The watchdog arms only while pending deliveries coexist + with OMP-reported idleness; any real turn (peer or operator) consumes pending + envelopes through the context hook and resets the ladder. Busy sessions never + age toward the deadline. Spawn-path detection is impossible from the + published launch contract, so the invariant covers every spawn path. +- **cite:** ADR-0010 doctrine (delivery self-heals or closes), issues #9/#10, + finding F-033. diff --git a/docs/PARITY.md b/docs/PARITY.md index 142bd67..1c5f5ba 100644 --- a/docs/PARITY.md +++ b/docs/PARITY.md @@ -9,14 +9,14 @@ ## Versioned comparison baseline -Current baseline: `omp-spt v0.3.19` → `BigscreenVR/claude-spt-bs v0.25.1`. +Current baseline: `omp-spt v0.3.20` → `BigscreenVR/claude-spt-bs v0.25.1`. `omp-spt` v0.3.4 was published at 2026-07-16 10:48:22 UTC. The first repository commit after that release, `413d6908d3a514edf93e0e53d4fb1f4b46ff7269`, landed at 2026-07-19 09:49:10 UTC and fixes the start of the work interval that produced v0.3.5. `claude-spt-bs` v0.25.1 was the latest published sister release at that instant, published from `BigscreenVR/claude-spt-bs` at 2026-07-19 04:55:53 UTC. -This baseline identifies the exact sister behavior used to define v0.3.19 parity. It is not a +This baseline identifies the exact sister behavior used to define v0.3.20 parity. It is not a claim that later `claude-spt` capabilities are covered. Any subsequent feature-parity work must first select and record the exact published `claude-spt` release being consulted. Every parity-informed `omp-spt` release must update this baseline to its own version; the release diff --git a/docs/SPT-CORE-FINDINGS.md b/docs/SPT-CORE-FINDINGS.md index fb00182..041b30a 100644 --- a/docs/SPT-CORE-FINDINGS.md +++ b/docs/SPT-CORE-FINDINGS.md @@ -29,6 +29,7 @@ | F-024 | 2026-07-01 | **CLOSED-REFRAMED (doyle, same day) — transport EXONERATED; symptom = F-023 on the remote node.** The `(spooled)` token in `SENT(WAN)` output is RECEIVER-CONFIRMED (only prints on a confirmed round-trip): probe `DIAG-7c1e` reached ENLYZEAM post-restart and sits in ball-b's spool undelivered (`spooled` not `delivered` = no live TCP listener at receive time). Operator workaround: any prompt typed into ball-b UPS-drains the stuck messages. Residues stay real as separate diagnostics in the wave: recv-pull silence (context-sync loop, not messaging) + PENDING-no-DONE ledger semantics | Cross-node delivery HFENDULEAM→ENLYZEAM never lands (reverse direction works instantly): `spt send ball-b` returns `SENT(WAN)… (spooled)` but nothing ever reaches the peer — not even the destination's hook-drain path, so the op never reaches the remote spool. Supporting: 32 never-DONE `net-send` effects in the ledger, recv-pull bundle loop silent since the last daemon restart, daemon holds relay-only connections | | F-031 | 2026-07-08 | **REPORTED to doyle (perri) — docs gap; adapter worked around in v0.17.3 (REQ-HAZARD-SEND-STATUS-STDERR).** `spt send`'s result token (`SENT:` / `QUEUED:` / `DEFERRED:` / `NO_PERCH:`) is written to **STDERR**, not stdout — stdout is empty on a plain `spt send`. This is undocumented in the published harness-contract / CLI reference (they describe the tokens as the send's *result* without naming the stream). It bit the claude-spt tag-messaging hook: the hook classified send outcomes off captured **stdout** (nulling stderr), so every peer `@<…@>` send saw `""` and was reported `NO_PERCH` in the confirm-back even though it DELIVERED — a silent false-negative that made agents re-send by hand. Root-caused offline via an isolated hook-binary rig (peer send returned `raw=Some("")` while stderr carried `SENT:doyle`). Adapter fix: capture stdout+stderr merged for send classification. **Needs (doyle):** either (a) document that `spt send` emits its status token on stderr (so adapters read the right stream), or (b) mirror the token to stdout for scriptable capture (`$(spt send …)`), which would match the least-surprise convention every other queryable verb follows (`whoami --json`, `endpoint list` → stdout). | `spt send`'s success/failure status token is emitted on STDERR (stdout empty), undocumented in the published surface — an adapter that captures stdout (the natural `$(…)` convention) sees nothing and misclassifies every successful send as unreachable | | F-032 | 2026-07-19 | **CONFIRMED spt-core defect (hertz, grounded in spt-core CONTEXT.md); omp-spt v0.3.4 promoted by explicit operator decision on 2026-07-19 with `spt endpoint stop` as acceptance cleanup.** Core must synchronously reap the broker-owned host after the shutdown boundary/cascade, then clear address and liveness while retaining `Suspended`; no new adapter callback is required. | `spt endpoint shutdown ` records Active→Suspended but leaves the broker PTY, native OMP process, `api listen` child, address, and `alive=true` until a separate `endpoint stop` | +| F-033 | 2026-07-23 | **REPORTED to doyle (omp-spt issue #10) — adapter hardened in v0.3.20 (delivery turn-liveness watchdog closes deaf sessions); daemon-side root cause open.** Also carries the reporter's daemon-footprint ask: 1.4G RSS peak on a 3.8G host OOM-killed the daemon (restart counter 3), and every daemon death funnels endpoints through this replay path | Daemon-restart autostart replay spawns sessions that bind and listen healthily but whose live deliveries never enter turns (`spt send` returns plain `SENT(WAN)`, message lands in digest, session db frozen for hours); suspend/wake recovers. The published launch contract gives the adapter no way to distinguish or detect the replay spawn | > **F-021 / F-022 (NOT spt-core findings — claude-spt parity items, tracked as REQs; surfaced during the 2026-06-24 checkpoint-commune grill)** — (F-021) claude-spt polls only on `UserPromptSubmit` (between turns); legacy spt also polls+injects on **PreToolUse** = the mid-turn half of live-agent reachability (the main path by which a live agent receives a message WHILE working). `api poll` already exists on the public surface → adapter wiring, not an spt-core gap → **`REQ-DIST-PRETOOL-POLL`**. (F-022) subagent worker-perch reachability: wiring is present (`subagent-start.sh` → `api worker-start`; `hostable_types` includes `Worker`) but runtime `spt send`-to-a-worker is UNVERIFIED → validation item **`REQ-DIST-WORKER-PERCH-REACH`**. Neither is an spt-core public-surface gap. @@ -1697,3 +1698,32 @@ asserts host/listener exit, no replacement listener, no address, `alive=false`, an unreachable old host before the command succeeds. For v0.3.4, the operator explicitly accepted `spt endpoint stop` as the release-acceptance cleanup and directed stable promotion on 2026-07-19. The core defect remains open and must not be represented as fixed by the adapter release. + +## F-033 — daemon-restart autostart replay spawns deaf-but-healthy sessions; OOM makes the path common + +**Reported:** 2026-07-23, from omp-spt issue #10 (same deployment as issue #9, adapter 0.3.19, +spt-core 0.40.0). +**Status:** reported to doyle; adapter hardened in v0.3.20; daemon-side root cause open. + +After a systemd OOM-kill (`Failed with result 'oom-kill'`, 1.4G daemon RSS peak on a 3.8G host, +restart counter 3), the restarted daemon replayed the saved endpoint entry and spawned a fresh +session. Its bring-up log is indistinguishable from a healthy session (`SID_BIND … no live seed — +bound from --session-id` → `BOUND … token=…`), and `spt send` returns plain `SENT(WAN)` (live +delivery), yet no delivered message ever starts an agent turn: the session db stayed frozen for +hours across multiple sends. Explicit `spt endpoint run --create` and suspend/wake bring-ups on the +same node and versions deliver correctly (verified for #9), and suspend/wake also recovers a +replay-deafened endpoint. + +Two public-surface gaps: + +1. The published launch contract carries no marker distinguishing the autostart replay spawn from + any other daemon-driven spawn, so an adapter cannot special-case or even detect the broken + path at bring-up. Whatever differs must be daemon-side (delivery routing, persisted endpoint + state, or listener registration after restart) and needs daemon-side root-cause. +2. The daemon's memory footprint (1.4G peak on a small host) makes OOM-driven restarts — and + therefore this replay path — routine on small nodes, precisely when nobody is attached. + +Adapter-side hardening shipped in v0.3.20 (ADR-0020, `REQ-HAZARD-DELIVERY-TURN-LIVENESS`): accepted +deliveries stranded on an idle session past a liveness deadline are resubmitted, then the endpoint +surfaces degraded comms, then fails closed so the deafness becomes visible and recoverable instead +of a silent black hole. This bounds the damage; it does not fix the replay path itself. diff --git a/docs/adr/0020-accepted-deliveries-prove-turn-liveness-or-close.md b/docs/adr/0020-accepted-deliveries-prove-turn-liveness-or-close.md new file mode 100644 index 0000000..6be0eb9 --- /dev/null +++ b/docs/adr/0020-accepted-deliveries-prove-turn-liveness-or-close.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Accepted deliveries prove turn liveness or the endpoint closes + +An accepted listener delivery on an idle session must enter an agent turn within the liveness deadline. The extension arms a watchdog whenever accepted custody coexists with OMP-reported idleness; a real turn of any origin consumes pending envelopes through the context hook and resets it, and busy sessions never age toward the deadline. On expiry the extension self-heals by force-resubmitting the stranded deliveries (a per-item submission epoch invalidates hung in-flight submissions so they cannot wedge recovery), then surfaces the degraded-comms rail, and on continued deafness fails closed: it releases pending custody, ends the SPT session, and shuts down the hosted TUI so the endpoint is visibly offline rather than silently deaf while ONLINE. This narrows ADR-0010's never-stop-local-work doctrine deliberately: the arming condition proves no local turn is running, and issue #10 showed the daemon-restart autostart replay can produce sessions whose healthy-looking bring-up (bind, token, listener) hides permanently dead turn delivery — a state the published launch contract gives the adapter no way to detect at spawn, since replay is indistinguishable from any other daemon-driven launch. Detecting the replay path was rejected as impossible from the public surface; a bring-up probe turn was rejected because it spends model tokens on every healthy spawn while the watchdog acts only when real custody is at stake; unbounded resubmission without closure was rejected because it preserves the silent black hole the invariant exists to eliminate. diff --git a/tests/omp-extension.mjs b/tests/omp-extension.mjs index 42bc6fe..5caaeed 100644 --- a/tests/omp-extension.mjs +++ b/tests/omp-extension.mjs @@ -263,6 +263,8 @@ function createHarness(options = {}) { shutdownCommandTimeoutMs: options.shutdownCommandTimeoutMs, shortformCommandTimeoutMs: options.shortformCommandTimeoutMs, listenerStableMs: options.listenerStableMs ?? false, + deliveryLivenessMs: options.deliveryLivenessMs ?? false, + deliveryLivenessAttemptLimit: options.deliveryLivenessAttemptLimit, killForceMs: options.killForceMs ?? 4, killGraceMs: options.killGraceMs ?? 3, listenerBufferLimit: options.listenerBufferLimit, @@ -1727,6 +1729,131 @@ async function testResumedIdleOverridesStaleLifecycleBusyState() { await harness.emit("session_shutdown"); } +// [unit->REQ-HAZARD-DELIVERY-TURN-LIVENESS] +async function testDeliveryLivenessResubmitsThenClosesDeafSession() { + const harness = createHarness({ deliveryLivenessMs: 500 }); + await harness.emit("session_start"); + harness.children[0].stdout.emit( + "data", + 'replayed bring-up looks healthy', + ); + await flush(); + assert.deepEqual(harness.submitted, ['']); + + await harness.clock.runNext(500); + assert.deepEqual( + harness.submitted, + ['', ''], + "the first liveness deadline must resubmit the stranded delivery", + ); + assert.equal(harness.shutdowns, 0); + assert.doesNotMatch(harness.statuses.at(-1).text, /comms recovering/); + + await harness.clock.runNext(500); + assert.equal(harness.submitted.length, 3); + assert.match( + harness.statuses.at(-1).text, + /comms recovering/, + "the second liveness deadline must surface the degraded-comms rail", + ); + + await harness.clock.runNext(500); + assert.equal(harness.shutdowns, 1, "a deaf endpoint must close, not stay ONLINE"); + assert.ok( + harness.calls.some((call) => call.args[3] === "session-end"), + "failing closed must end the SPT session", + ); + assert.equal( + harness.statuses.at(-1).text, + "spt delivery dead", + "the closed endpoint names its failure on the status rail", + ); +} + +// [unit->REQ-HAZARD-DELIVERY-TURN-LIVENESS] +async function testDeliveryLivenessIgnoresBusySessions() { + let idle = false; + const harness = createHarness({ deliveryLivenessMs: 500, isIdle: () => idle }); + await harness.emit("session_start"); + harness.children[0].stdout.emit("data", 'queued'); + await flush(); + assert.equal(harness.submitted.length, 1); + + await harness.clock.runNext(500); + assert.equal(harness.submitted.length, 1, "a busy session must never age toward the deadline"); + assert.equal(harness.shutdowns, 0); + assert.deepEqual(harness.clock.delays(), [500], "the watchdog re-arms while custody is pending"); + + idle = true; + await harness.clock.runNext(500); + await harness.clock.runNext(500); + await harness.clock.runNext(500); + assert.equal(harness.shutdowns, 1, "idleness resumes the escalation ladder"); +} + +// [unit->REQ-HAZARD-DELIVERY-TURN-LIVENESS] +async function testDeliveryLivenessDisarmsWhenContextConsumesCustody() { + const harness = createHarness({ deliveryLivenessMs: 500 }); + await harness.emit("session_start"); + harness.children[0].stdout.emit("data", 'consumed'); + await flush(); + + await harness.emit("context", { + messages: [{ role: "user", content: '' }], + }); + await harness.clock.runNext(500); + assert.deepEqual(harness.clock.delays(), [], "consumed custody must disarm the watchdog"); + assert.equal(harness.shutdowns, 0); + await harness.emit("session_shutdown"); +} + +// [unit->REQ-HAZARD-DELIVERY-TURN-LIVENESS] +async function testHungSubmissionCannotWedgeLivenessRecovery() { + const harness = createHarness({ + deliveryLivenessMs: 500, + onSubmit: () => new Promise(() => {}), + }); + await harness.emit("session_start"); + harness.children[0].stdout.emit("data", 'hung'); + await flush(); + assert.equal(harness.submitted.length, 1); + + await harness.clock.runNext(500); + assert.equal( + harness.submitted.length, + 2, + "a never-settling submission must not block the liveness resubmission", + ); + await harness.emit("session_shutdown"); +} + +// [unit->REQ-HAZARD-DELIVERY-TURN-LIVENESS] +async function testTurnStartResetsLivenessLadder() { + const harness = createHarness({ deliveryLivenessMs: 500 }); + await harness.emit("session_start"); + harness.children[0].stdout.emit("data", 'late turn'); + await flush(); + + await harness.clock.runNext(500); + await harness.clock.runNext(500); + assert.match(harness.statuses.at(-1).text, /comms recovering/); + + await harness.emit("before_agent_start", { prompt: "listener wake", systemPrompt: [] }); + await harness.emit("agent_start"); + assert.doesNotMatch( + harness.statuses.at(-1).text, + /comms recovering/, + "an entered turn must clear the degraded delivery rail", + ); + const boundary = await harness.emit("context", { + messages: [{ role: "user", content: '' }], + }); + await harness.emit("agent_end", { messages: boundary?.messages ?? [] }); + await harness.clock.runNext(500); + assert.equal(harness.shutdowns, 0, "a recovered session must not continue the old ladder"); + await harness.emit("session_shutdown"); +} + // [unit->REQ-HAZARD-ABNORMAL-TURN-RECEIVABILITY] async function testAbnormalTurnsRestoreReceivability() { for (const [label, completionEvent] of [ @@ -2100,6 +2227,11 @@ await testStartupBriefHintsAndUpdateNotices(); await testActiveTurnBoundaryDeliveryAndFallback(); await testAbnormalTurnsRestoreReceivability(); await testResumedIdleOverridesStaleLifecycleBusyState(); +await testDeliveryLivenessResubmitsThenClosesDeafSession(); +await testDeliveryLivenessIgnoresBusySessions(); +await testDeliveryLivenessDisarmsWhenContextConsumesCustody(); +await testHungSubmissionCannotWedgeLivenessRecovery(); +await testTurnStartResetsLivenessLadder(); await testCompletionStopReasonGatesSideEffects(); await testCompactionPreservesLocalAssistantOutput(); await testPeerShortformParsingAndDispatch(); diff --git a/tools/omp-spt/Cargo.lock b/tools/omp-spt/Cargo.lock index 0067ede..1153844 100644 --- a/tools/omp-spt/Cargo.lock +++ b/tools/omp-spt/Cargo.lock @@ -40,7 +40,7 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "omp-spt" -version = "0.3.19" +version = "0.3.20" dependencies = [ "getrandom", "serde_json", diff --git a/tools/omp-spt/Cargo.toml b/tools/omp-spt/Cargo.toml index 21ccc25..70ae5c4 100644 --- a/tools/omp-spt/Cargo.toml +++ b/tools/omp-spt/Cargo.toml @@ -3,7 +3,7 @@ # [impl->REQ-DIST-BINARY-CONSOLIDATE] [package] name = "omp-spt" -version = "0.3.19" +version = "0.3.20" edition = "2021" publish = false diff --git a/traceable-reqs.toml b/traceable-reqs.toml index 37aa14b..7f9501c 100644 --- a/traceable-reqs.toml +++ b/traceable-reqs.toml @@ -169,6 +169,11 @@ id = "REQ-HAZARD-ABNORMAL-TURN-RECEIVABILITY" title = "Cancellation, interruption, and failed OMP turns settle affected custody and restore endpoint receivability" required_stages = ["doc", "impl", "unit"] +[[requirements]] +id = "REQ-HAZARD-DELIVERY-TURN-LIVENESS" +title = "An accepted delivery on an idle session enters a turn within the liveness deadline or the endpoint self-heals and then visibly closes" +required_stages = ["doc", "impl", "unit"] + [[requirements]] id = "REQ-PARITY-STARTUP-BRIEF" title = "Activated OMP sessions receive concise identity, roster, messaging, continuity, lifecycle, subnet, and version guidance"