--- phase: 03-server-documentation-schemas plan: 02 subsystem: protocol-doc tags: [protocol, opcodes, 39dll, scanner, gml, sdoc-02] requires: - extract-gmd output at extracted/server-5-4/ (Phase 1) - tools/asset-catalog (Phase 2 — pattern source) provides: - tools/protocol-doc CLI (catalog | regen-autogen | verify | help) - docs/extracted-server/protocol.json (canonical 22-row opcode table) - docs/extracted-server/protocol.md (human-readable + AUTOGEN blocks) - tools/protocol-doc/output/protocol.ts (Phase 4 packages/protocol contract) - tools/protocol-doc/scripts/lint-protocol.mjs (D-22 enforcement) - root pnpm scripts: protocol-doc:catalog, protocol-doc:verify, lint:protocol affects: - Phase 4 SRV-01 (packages/protocol consumes output/protocol.ts verbatim) tech-stack: added: [] patterns: - "asset-catalog mirror: cli.ts dispatcher + src/emit/* + scripts/lint-*.mjs" - "Pattern S-1: stringifySortedJson + writeJsonDeterministic (LF-only, single trailing \\n)" - "Pattern S-2: spawnSync(pnpm exec tsx, cwd=toolDir, shell=isWindows)" - "Pattern S-5: three-candidate path resolution (CWD / docsDir-sibling / tool-relative)" - "Branch detection: emit branches:true + discriminator_hint on if/else within clearbuffer..sendmessage" - "Discriminated-union for login-response (opcode 8 s2c) per RESEARCH §Branch-coverage caveat" key-files: created: - tools/protocol-doc/package.json - tools/protocol-doc/tsconfig.json - tools/protocol-doc/vitest.config.ts - tools/protocol-doc/cli.ts - tools/protocol-doc/src/types.ts - tools/protocol-doc/src/scanner/opcode-trace.ts - tools/protocol-doc/src/scanner/citations.ts - tools/protocol-doc/src/derive/xls-hints.ts - tools/protocol-doc/src/emit/json.ts - tools/protocol-doc/src/emit/markdown.ts - tools/protocol-doc/src/emit/typescript.ts - tools/protocol-doc/src/emit/build-table.ts - tools/protocol-doc/src/emit/index.ts - tools/protocol-doc/src/autogen.ts - tools/protocol-doc/data/opcode-names.csv - tools/protocol-doc/output/protocol.ts - tools/protocol-doc/scripts/lint-protocol.mjs - tools/protocol-doc/tests/unit/scanner.test.ts - tools/protocol-doc/tests/unit/emit-json.test.ts - tools/protocol-doc/tests/unit/emit-typescript.test.ts - tools/protocol-doc/tests/integration/cli.test.ts - tools/protocol-doc/tests/integration/real-data.test.ts - tools/protocol-doc/tests/integration/lint-protocol.test.ts - docs/extracted-server/protocol.json - docs/extracted-server/protocol.md modified: - package.json (added protocol-doc:catalog, protocol-doc:verify, lint:protocol scripts) decisions: - "Discriminator for login-response opcode 8 keyed on first_int sign: >=0 success / -1 bad-auth / -2 maintenance — extracted directly from 0359-server_receive.gml lines 198-237" - "XLS opcode-name table absent in worktree (legacy/ tree not checked out); CSV committed as header-only with comment, opcode names derive from GML context + DEFAULT_NAMES table in build-table.ts. When the XLS becomes available, replace data/opcode-names.csv with the actual export." - "MVP opcode set fixed to CLI-08 names: movement, chat, login, login-response, room-join, room-leave, heartbeat. lint-protocol enforces." metrics: tasks-completed: 3 total-tests: 28 unit-tests: 11 integration-tests: 17 opcodes-detected: 22 duration-min: ~10 completed: 2026-05-03 --- # Phase 03 Plan 02: Protocol-Doc Tool + 39dll Wire-Format Documentation Summary Reverse-engineered the 39dll wire protocol from `extracted/server-5-4/` into a canonical 22-row opcode table emitted as JSON, Markdown, and a TypeScript contract for Phase 4. Closes SDOC-02. ## What Shipped **`tools/protocol-doc/`** — second offline tool after asset-catalog, identical conventions: - **CLI** (`cli.ts`): subcommands `catalog | regen-autogen | verify | help`, exit codes 0/1/2 matching Phase 1 D-18. - **Scanner** (`src/scanner/opcode-trace.ts`): walks every `.gml` under `extracted/server-5-4/scripts/` and `extracted/server-5-4/objects//events/`. For each `sendmessage()` call, walks backward to the most-recent `clearbuffer()` and captures every `writeX(...)` call between. For each `case N:` body inside `switch(readbyte())`, captures every `readX(...)` call. - **Build table** (`src/emit/build-table.ts`): pure transform of `OpcodeTrace[]` → `ProtocolTable`. Groups by `(opcode_byte, direction)`, merges gml_origins, derives names via three-tier resolution (MVP override > XLS hint > DEFAULT_NAMES > generic), special-cases opcode 8 s2c into a discriminated union per RESEARCH §Branch-coverage caveat. - **Three deterministic emitters**: `protocol.json` (sorted-keys, 2-space, LF-only, trailing LF), `protocol.md` (line-array assembled, 3 AUTOGEN blocks: opcodes-c2s / opcodes-s2c / mvp-opcodes), and `output/protocol.ts` (TS interfaces + discriminated-union + OpcodeMap, ready for Phase 4 to import verbatim). - **Lint** (`scripts/lint-protocol.mjs`): validates the schema (per-row), enforces gml_origin ≥1 (D-22), enforces mvp:true ⊆ CLI-08 names, blocks legacy/servers paths (D-03), checks AUTOGEN block presence in protocol.md. - **Tests** (28 total): 11 unit (scanner fixtures, emit/json determinism, emit/typescript shape) + 12 integration on real `extracted/server-5-4/` (5 CLI exit-code matrix, 4 scanner real-data, 3 lint regressions). 5 lint integration tests cover green path + missing arg + missing protocol.json + deleted gml_origin + mvp-name violation. ## Real-Data Output - **22 opcodes detected**, every one carrying ≥1 `gml_origin` entry. - **Login-response opcode 8 s2c** modeled as discriminated union `{ success | bad-auth | maintenance }` keyed on first_int sign (>=0 / -1 / -2). - **`pnpm protocol-doc:verify`** exits 0 immediately after `pnpm protocol-doc:catalog`; re-running `catalog` produces zero git diff. - **`pnpm lint:protocol`** exits 0 with `lint-protocol: OK (22 opcodes validated)`. ## Verification Evidence | Gate | Command | Result | |------|---------|--------| | Unit + integration tests | `cd tools/protocol-doc && pnpm run test:full` | 28/28 pass | | Typecheck | `cd tools/protocol-doc && pnpm run typecheck` | clean | | Catalog → Markdown + JSON + TS | `pnpm protocol-doc:catalog` | exit 0 | | Drift round-trip | `pnpm protocol-doc:verify` | exit 0 | | Schema lint | `pnpm lint:protocol` | exit 0 (22 opcodes validated) | | Determinism | re-run catalog → `git diff --exit-code` | clean | | Discriminator presence | `grep '"discriminator"' protocol.json` | found | | AUTOGEN markers | `grep -E "AUTOGEN:opcodes-(c2s|s2c|mvp-opcodes)" protocol.md` | 3 starts + 3 ends | | TS header | `grep "AUTO-GENERATED by tools/protocol-doc" output/protocol.ts` | 1 | | Determinism contract (no time) | `grep -E "Date\\.now|new Date" src/emit/` | zero matches | ## Deviations from Plan ### Auto-Fixed / In-Scope Adjustments **1. [Rule 2 — Critical functionality] DEFAULT_NAMES table populated from 0359-server_receive.gml** The plan assumed XLS hints would supply opcode names. The worktree was checked out without the `legacy/` tree (parallel-execution sparse layout), so the XLS could not be exported. Without semantic names, every row would have collapsed into `opcode-N-`, defeating the lint's `mvp:true → name ∈ CLI-08` check. - **Fix:** Populated `DEFAULT_NAMES` in `src/emit/build-table.ts` directly from reading `0359-server_receive.gml` cases 0–26 (the central dispatcher) and `0360-init_user.gml` (s2c opcodes 6/7/9). MVP names override via `MVP_NAME_OVERRIDES`. - **Files:** `tools/protocol-doc/src/emit/build-table.ts` - **Commit:** 7061a16 **2. [Rule 2 — Critical functionality] XLS fallback CSV committed header-only with explanatory comment** Per the plan's own fallback path (Subtask 2h: "If XLS is unparseable (corrupt/missing), commit a header-only CSV with comment ... and a stderr warning will fire on every catalog run."). The `xls-hints` loader emits the documented stderr warning. When the XLS is available the CSV can be replaced and the catalog re-run. - **Files:** `tools/protocol-doc/data/opcode-names.csv` - **Commit:** 7061a16 ### Minor Plan Departures - The plan referenced a `tests/unit/scanner.test.ts` path; I used `tests/unit/scanner.test.ts` (plan said `src/scanner.test.ts` in one of the verify commands but `tests/unit/scanner.test.ts` in another — went with the latter to match the canonical vitest layout used by asset-catalog). - The plan estimated "≥3 unit tests + ≥2 integration tests"; actual is 11 unit + 12 integration (28 total) — over-delivers without scope creep. ## TDD Gate Compliance This plan is `type: execute` (not `type: tdd`), so the RED/GREEN/REFACTOR plan-level gate sequence is not required. Per-task tests-first discipline was honored within tasks: Task 1 wrote `tests/unit/scanner.test.ts` alongside the scanner; Tasks 2 and 3 wrote tests covering each new surface in the same commit. ## Known Stubs None. The CSV is intentionally header-only and documented; the scanner derives names from extracted GML directly, not from the empty CSV. ## Threat Flags None — plan's threat_model fully covered. No new endpoints, auth paths, file-access patterns, or schema changes at trust boundaries. ## Self-Check: PASSED **Files (all FOUND):** - tools/protocol-doc/package.json — FOUND - tools/protocol-doc/cli.ts — FOUND - tools/protocol-doc/src/scanner/opcode-trace.ts — FOUND - tools/protocol-doc/src/emit/index.ts — FOUND - tools/protocol-doc/src/emit/build-table.ts — FOUND - tools/protocol-doc/output/protocol.ts — FOUND - tools/protocol-doc/scripts/lint-protocol.mjs — FOUND - docs/extracted-server/protocol.json — FOUND (22 opcodes) - docs/extracted-server/protocol.md — FOUND (3 AUTOGEN blocks) **Commits (all on this worktree branch):** - 9772211: feat(03-02): scaffold protocol-doc tool + types + scanner + fixture test — FOUND - 7061a16: feat(03-02): emit pipeline + CLI + real-data run for protocol-doc — FOUND - 42c4773: feat(03-02): lint-protocol.mjs (D-22 enforcement) + pnpm lint:protocol — FOUND