# M9 Wave 3 (G3) — JIT execution plan: session-digest log projection

> **STATUS: GREENLIT (doyle, 2026-06-13). §2 RULED.** Wave 3 of M9 per
> `M9-PLAN.md §Wave 3` + ADR-0008 amendment (2026-06-12) + CONTEXT.md `session
> digest` entry (lines 334–343). Branch: `m9-digest-logs` (lands as one PR).
> Approach A confirmed, field names + collapse rule approved; the §2 sharpening
> (digest-record contract is a *published* surface; one schema, no mapping
> layer; migrate the stray native fixture) is folded in. Source-grounded against
> the live tree at `58f034f`. Executor: todlando.

---

## 1. What this wave does (recap)

Re-found the live activity buffer on **normalized session logs**, deleting the
PTY-parse engine. Product surface (CONTEXT.md 334–343) is unchanged: N-turn
window (~3), collapsed tool sprints, ~25-char arg truncation, snapshot pull +
structured-delta stream, address-gated, opt-in Option-C persistence. Only the
**source** changes: PTY bytes → the same normalized history records the
echo-commune consumes. New outcome: topology-independent (Claude-Code-hosted
endpoints get digests via Path A JSONL).

**End state at G3:** `spt endpoint digest <id>` renders for a CC-hosted
endpoint; the PTY-parse path is gone; REQ-TERM-4's title + evidence match the
log-source reality.

---

## 2. Record→Turn source-tagging — the digest-record contract (RULED: doyle 2026-06-13)

The digest is a **structured, source-tagged** projection (input / agent /
collapsed-tool turns) that spt-core computes and refreshes incrementally. The
history subsystem treats each `HistoryRecord` as an **opaque line**
(`crates/spt-live/src/history.rs:30-35`, "spt-core only delimits, never
parses"), and the echo-commune — the sibling consumer the ADR points at —
**never parses** records: it concatenates `r.raw` and pipes them to an
adapter-owned summarizer command (`crates/spt-live/src/echo.rs:99-104`). So the
projection cannot inherit echo's opaque-pipe pattern: a structured,
incrementally-tailed digest must **source-tag** each record. CONTEXT.md/M9-PLAN
leave the mechanism to the executor ("executor picks the mechanism with the
history subsystem's grain"; "incremental normalize story is design-phase
concern").

### Approach A — spt-core PUBLISHES a digest-record contract (RULED)

**Correct framing (doyle's sharpening):** A is NOT "adapters already emit this
shape." A is **spt-core publishes a digest-record contract that projectable
adapters must target.** The `{"r":"u"}` line at `history.rs:249` is a
*history-subsystem* test fixture whose discriminator is `r ∈ {u,a}` — a
**different** schema, cited in error in the draft as pre-alignment. It is not
evidence; A is a new published surface, not a free ride.

**The single contract (no dual schema, no mapping layer):**
- **History-the-subsystem stays opaque — UNCHANGED.** It carries `raw` verbatim
  (`history.rs:30-31`), adapter-defined; echo's raw-pipe (`echo.rs:99-104`)
  untouched. No REQ-HAZARD on history opacity is tripped.
- **The digest owns ONE record contract** — an spt-core-published overlay on the
  *content* of `raw`. JSON line: `role ∈ {input, agent, tool}` (enumerated,
  exact), `text: string`, optional `tool: {name, arg}` (present iff
  `role == "tool"`). Tool sprints = **consecutive `role:"tool"` records
  collapsed** into one `ToolSprint`. **APPROVED.**
- **Exactly one schema in the tree.** No `r`/`u → role`/`input` mapping layer,
  ever. **Decision (RULED, my call per doyle): migrate the stray native
  `{"r":"u"}` history-test fixture (`history.rs:249`) to the digest contract
  shape** — cosmetic for that test (it still only exercises *verbatim carry* of
  an opaque line; REQ-SEAM-HISTORY semantics unchanged), but it removes the
  second-schema optics so the codebase shows ONE record JSON shape. Chosen over
  "settle the digest contract AS the native record shape" because native
  history must stay adapter-opaque for echo/resume — the digest schema must not
  become the universal native shape.

**Published-surface treatment (because adapters encode against it):**
- Enumerate `role` values exactly (`input`/`agent`/`tool`) — closed set.
- **Document the contract in CONTEXT.md + `docs/MANIFEST.md`** — the MANIFEST
  `[pty_digest]` section is **replaced** by the digest-record contract doc, not
  merely deleted (it remains the digest's schema home, now log-source-shaped).
- **Forward-compat:** unknown field / malformed line / unknown `role` → **skip**
  the record (never panic, never abort the projection).

**Why A (verified by doyle):** the only candidate with no manifest seam
(M9-PLAN:162 verified), the only clean CC-fixture E2E path (the mock adapter's
Path-A normalize emits the contract → projectable directly), and it keeps the
history subsystem opaque. Rejected alternative: an adapter digest-normalize
command (echo-style) — **adds a manifest seam, contradicts M9-PLAN:162**, and a
subprocess per refresh kills incremental tail.

---

## 3. Retire inventory — surgical, NOT wholesale file-deletion

The Map's "delete entire `digest.rs`" overstates it. The **output + delta
types** must survive (the hub and CLI depend on them); only the **parser** dies.

### KEEP (survives the retire)
- `spt-term`: `Digest`, `Turn`, `DigestEntry`, `ToolUse`, `DigestConfig` — the
  projected-digest **output shape** + presentation knobs (window depth, arg
  truncation). Imported by `digesthub.rs:29`.
- `spt-daemon::digest`: `DigestUpdate` (delta payload), `common_prefix_len`
  (delta diff), the version/`from`/`newly_completed` delta machinery, the
  `ingest_*`→`hub.publish` bridge, `persist_turn` (Option-C). These are
  **source-independent** — they operate on `Digest`/`Turn`, not on bytes.
- `spt-daemon::digesthub` — **entire file unchanged** (hub, control channel,
  `pull_snapshot`/`follow`, render/JSON helpers). It already consumes
  `Digest`+`DigestUpdate`; it never sees the source.

### DELETE (the PTY-parse machinery)
- `spt-term/src/digest.rs`: `DigestParser`, `DigestPatterns`, `DigestError`, the
  regex-over-bytes `feed()`/`digest()` + their unit tests. **Split the file**:
  relocate the surviving output types (`Digest`/`Turn`/`DigestEntry`/`ToolUse`/
  `DigestConfig`) into a `digest_types`-style module (or keep file, gut parser).
- `spt-daemon/src/digest.rs`: `DigestEngine::from_manifest` (PtyDigest seam),
  `with_parts(patterns,…)`, `feed(bytes:&[u8])`, the `parser` field, the
  byte-feed unit tests. The struct **becomes** the projection engine (§4).
- `spt-runtime`: `PtyDigest` struct + `[pty_digest]` serde/validate
  (`manifest.rs:699+`, tests `902/909/924`).
- `spt-daemon/src/brain.rs:760` `run_digest_feed` (the PTY byte-loop) → replaced
  by a record-projection driver (§4).
- `crates/spt-daemon/tests/digest.rs` B8 E2E — re-wired (§5), not just deleted.

### REPLACE / MIGRATE (not a clean delete)
- `docs/MANIFEST.md:113-145` `[pty_digest]` section → **replaced** by the
  digest-record contract doc (§2), not deleted — it stays the digest's schema
  home, now log-source-shaped.
- `crates/spt-live/src/history.rs:249` native fixture `{"r":"u"}`/`{"r":"a"}` →
  **migrated** to the digest-record contract shape (cosmetic; REQ-SEAM-HISTORY
  verbatim-carry semantics unchanged — kills second-schema optics, §2).

---

## Execution status (live)

- **commit 0** ✅ `714f6b8` — this plan (doyle-vetted).
- **T7a commit 1** ✅ `82ead73` — `spt-term::projection`: published digest-record
  contract + `project`/`project_lines` + 8 units. Additive; parser still green.
- **T7a commit 2** ✅ `6033326` — `api digest-entry` push door +
  `spt_store::history::append_digest_entry` (separate `digest.log`) + units.
- **T7b core** ✅ `5a09577` — on-demand projector + hub `project_and_publish` +
  reproject verb + handlers + removed `run_digest_feed` + CC-fixture E2E.
- **T7b retire+re-point** ✅ (this commit) — deleted the spt-term parser +
  `PtyDigest` + `[pty_digest]` (schema re-blessed); retitled REQ-TERM-4 (honest
  log-source title) with its `impl`/`unit`/`int` tags on the projection + E2E;
  migrated the native fixture to the contract; MANIFEST/CONTEXT/ADR-0008/docs-site
  /CHANGELOG swept; deferred autonomous-freshness + Option-C re-home logged to
  `docs/DEFERRED.md`. **G3 ready** — `traceable-reqs check` EXIT=0, clippy -D +
  workspace tests green, `xtask check` OK. doyle pinged for the G3 gate.

### Discovery (affects T7b scope — surfaced to doyle 2026-06-13)

`Brain::run_digest_feed` (the PTY byte-feed driver, `brain.rs:760`) has **no
production caller** — only its definition, a doc cross-ref, and the daemon-half
integration test (`spt-daemon/tests/digest.rs`) drive it. The daemon serves the
digest **control channel** (`daemon.rs:213` → `serve_digest_control`, the
pull/subscribe surface) but **nothing ever fed the hub in production**: B8
shipped the engine + hub + control channel + a manually-driven E2E, never the
live session→engine feed. So in production today `spt endpoint digest <id>`
always resolves "no live digest" (the hub is never published to).

**Consequence:** T7b is **retire + a fresh log-sourced wire**, not "replace a
live path." The byte-feed primitive can be removed cleanly (no live integration
depends on it). The open scope question (doyle): for G3, does the **delta-stream
live-nudge** (file-watch → publish, brand-new work that never existed) belong in
scope, or is a **topology-independent snapshot-pull** — the daemon projecting
from the endpoint's records on demand at the snapshot handler, matching the
pre-M9 wiring level + proven by the CC-fixture E2E — the G3 bar, with live
nudging deferred to the frontend milestone that actually consumes the stream?

## 4. T7a — projection engine (new code) — ✅ DONE

1. **Digest-record schema** (RULED §2): a small parse fn
   `record_to_tagged(raw: &str) -> Option<TaggedRecord>` reading the published
   contract (`role ∈ {input,agent,tool}`, `text`, optional `tool{name,arg}`).
   Malformed / unknown-field / unknown-role line → skipped (forward-compat,
   never panics).
2. **Projection**: `project(records: &[…], config: &DigestConfig) -> Digest` —
   fold tagged records into `Turn`s, applying the unchanged formula (N-turn
   window slide, consecutive-tool collapse into one `ToolSprint`, ~25-char arg
   truncation). Reuses the surviving `Digest`/`Turn`/`DigestEntry` types.
3. **`DigestEngine` re-core**: replace `parser`+`feed(bytes)` with
   `feed_records(&[…]) -> Option<DigestUpdate>` (or `project_into` + the
   existing `common_prefix_len` delta diff — **delta machinery unchanged**, only
   its input changes from a parsed-byte digest to a projected-record digest).
   `snapshot()`, `version()`, `full_update()`, Option-C `newly_completed` all
   stay.
4. **Incremental tail** (design note — executor picks mechanism per the history
   grain): Path B native `history.log` tails cleanly (append-only file →
   file-watch + byte/record cursor). Path A (fetcher / locate+normalize) is a
   bounded full re-pull. **Fallback if Path A grain too coarse** (M9-PLAN:206):
   Path-B-first incremental + Path-A snapshot-only, an accepted degradation
   **flagged at G3, not silently shipped** (surface = a `tracing` warn + a G3
   note). Recommend: cursor-based tail over the normalized record stream,
   re-using `fetch_history`; mechanism final pick is an in-T7a design note.
5. **`api digest-entry`** — the push door for log-less adapters (entries arrive
   pre-formed in the §2 schema). Mirror the `api history-log` →
   `spt::api::reporting::append_history` precedent (a new verb + IPC sink that
   feeds the projection/hub directly).
6. **Units**: projection table from fixture records (input/agent/tool-sprint
   collapse, window slide, arg truncation); `digest-entry` merge; malformed-line
   skip. Tag `[unit->REQ-TERM-4]` **on the new tests** (moved from the deleted
   parser units in the same commit — §6).

---

## 5. T7b — re-wire + retire + CC E2E

1. **Re-wire the feed source**: `brain.rs` driver now reads the endpoint's
   normalized records (via the history subsystem) and calls `feed_records` →
   `hub.publish` (the hub + control channel + CLI are **untouched** — they
   already speak `Digest`/`DigestUpdate`).
2. **Retire** everything in §3 DELETE, with units removed alongside their code.
3. **MANIFEST.md**: **replace** `[pty_digest]` (113–145) with the
   digest-record contract doc (§2 — `role`/`text`/`tool`, enumerated roles,
   malformed→skip); note the digest rides `[history]` (no manifest *seam*) but
   the **record contract** is published here. Mirror the contract enumeration
   into CONTEXT.md (digest entry, 334–343).
4. **Fixture migration**: `history.rs:249` `{"r":"u"}` → digest-contract shape
   (cosmetic; commit message states it's optics-only, semantics unchanged).
5. **CC-fixture E2E** (`int` — the topology-independence proof): a mock adapter
   with `[history]` Path-A `locate_normalize` over a fixture JSONL (CC-style
   per-session log); assert `spt endpoint digest <id>` renders the expected
   N-turn window + collapsed sprint. New test file (proposed
   `crates/spt-daemon/tests/digest.rs` rewritten, or
   `crates/spt/tests/cc_digest_e2e.rs` — executor picks; **flag the filename
   choice at G3**). Tag `[int->REQ-TERM-4]`.

---

## 6. REQ-TERM-4 re-point (restoration discipline — one commit, never un-evidenced)

In the **same commit** the old evidence is deleted:
1. **Retitle** in `traceable-reqs.toml:415`:
   `"Live activity buffer (session digest): projection of normalized session
   logs, spt endpoint digest pull + structured-delta stream, api digest-entry
   push, opt-in Path-B log"` (drops "adapter-supplied patterns over broker
   PTY"). `required_stages = ["impl","unit","int"]` unchanged.
2. **Move tags**: `[impl->REQ-TERM-4]` onto the projection engine + `digest-entry`
   sink; `[unit->REQ-TERM-4]` onto the new projection/merge units;
   `[int->REQ-TERM-4]` onto the CC-fixture E2E — as the old parser/byte-feed/B8
   tags are deleted. **Never a commit where REQ-TERM-4 is un-evidenced**
   (M9-PLAN:204 — the `int` replacement lands with the retire).
3. `traceable-reqs check` EXIT=0 at the commit.

---

## 7. G3 gate checklist (doyle, pre-merge + milestone close)

- [ ] REQ-TERM-4 re-point audit — no orphaned tags, title honest to log-source.
- [ ] Retire completeness — **zero PTY-parse remnants** (`PtyDigest`,
      `DigestParser`, `[pty_digest]`, byte-`feed` all gone; grep clean).
- [ ] CC-fixture E2E green (topology-independence proven).
- [ ] Full suite: workspace tests 0-fail, `clippy -D`, `--no-default-features`,
      xtask check, `traceable-reqs check` EXIT=0.
- [ ] Digest-record contract **published**: enumerated in CONTEXT.md (334–343)
      + `docs/MANIFEST.md` (replacing `[pty_digest]`, not just deleted).
- [ ] Native fixture (`history.rs:249`) migrated to the contract shape (one
      schema in the tree, no mapping layer).
- [ ] Docs sweep: mdBook digest/manifest pages, CHANGELOG `[Unreleased]`,
      ADR-0008 status note (implementation debt discharged), CONTEXT.md 341
      "implementation note" updated (debt cleared).
- [ ] History-grain fallback decision flagged (Path-A coarse? B-first taken?).
- [ ] E2E filename choice flagged.

---

## 8. Commit choreography (atomic, evidence-in-commit)

1. `feat(digest): record→Turn projection engine + schema (T7a)` — new projection,
   schema parse, units; `DigestEngine` re-core keeping delta machinery.
2. `feat(api): digest-entry push door for log-less adapters (T7a)`.
3. `feat(digest): re-wire brain feed to normalized records; retire PTY-parse +
   re-point REQ-TERM-4 (T7b)` — the retire + retitle + tag-move + CC E2E in one
   commit (restoration discipline; tree never un-evidenced).
4. `docs(m9): MANIFEST drop [pty_digest], ADR-0008/CONTEXT debt cleared,
   CHANGELOG (G3)`.

Each commit: `traceable-reqs check` EXIT=0, CI-targeted local sweep green.

---

## 9. Risks / watch-items

- **Opaque-record vs structured-projection tension** (§2) — the one real
  decision; resolved by scoping the parse to the *digest's* contract, leaving
  history opaque. STOP-and-ping if A is rejected.
- **History-grain coarseness** (M9-PLAN:206) — Path-A re-pull may be too coarse
  for near-realtime tail; fallback is B-first + A-snapshot-only, **flagged not
  silent**.
- **Don't expand scope** — no shell (M10), no rc (M11), no frontend pane. Digest
  presentation stays spt-core-owned; extraction stays adapter-owned.
- **Quietest-tree discipline** — Wave 3 deletes code + re-points active evidence;
  lands last, after G2 + gateway-wan gates close (per M9-PLAN sequencing).