TO: doyle FROM: todlando SUBJECT: DV5SZHGQ — registry lost-update source review BASE: 7890ead39bb7f14ed44aaae44b0951f098ffe9ac TREE: .worktrees/297-enforcement-shape SCOPE: Read-only source investigation. No product edits, builds, tests, reproduction, or analyzer startup. This report is the only new artifact. VERDICT Hertz's synchronization gap is source-proven. There is ONE Rust REGISTRY_WRITE declaration, but broker and brain have independent process-local instances. Their registry read/modify/publish transactions do not exclude one another. The proposed incident interleaving fits the reported missing alpha.md exactly, but remains INFERRED, not a captured writer trace. A green 10x cohort would not refute this gap. Do not weaken or retry ARM 11's identity assertion. (a) ACTUAL LOCK SITES AND PROCESS OWNERS Paths below are relative to the pinned tree. - crates/spt-daemon/src/servehost.rs:18-20 declares static REGISTRY_WRITE: std::sync::Mutex<()>. - servehost.rs:148-153, with_registry_write, takes that mutex. livehost.rs:1287 calls this helper around the reaper's load, expiration mutation, snapshot cleanup, and save. - servehost.rs:171-173, apply_at, independently takes the SAME declaration's mutex in its own process. Its six mutation arms are Add (save :190), Attach (:229), AddScoped (:243), AddInputReference (:261), Remove (:272), and Reconcile (:285). List is also inside this local guard. Docs/LAN dispatches return before it. - Broker control: daemon.rs:478-482 runs seed control; seedmap.rs:182-183 dispatches ServeRequest to servehost::handle. Brain process: brainproc.rs:1260-1297 creates Command and calls spawn; brainproc.rs:358 starts livehost; livehost.rs:1420 invokes the reaper on the 5-second reconciliation pulse. These are not two threads sharing one address-space mutex. - crates/spt-store/src/serving.rs:138-172: load_at has no lock; save_at validates and serializes the WHOLE registry, then calls atomic_write_bytes_durable. No compare-and-swap/version check. atomic.rs:131-160 writes a unique temporary file and renames it over the destination. Atomic publication prevents torn JSON, not lost updates. - Full crates search found the production ServingRegistry mutation owners above; webserve.rs:335-338 is a snapshot reader. CLI attachment registration is broker IPC, sequential per source (crates/spt/src/attach.rs:86-115), not an independent production disk writer. (b) STALE-WRITE PATH Let R0 contain report.md, report~1.md, and the expired ephemeral entry. 1. Brain loads R0 under its own mutex. 2. Broker loads R0, adds alpha, durably saves R1, acknowledges success. 3. Brain removes the expired entry from its OLD R0 and saves that snapshot, erasing alpha. 4. Broker loads that result, adds beta, saves and acknowledges. 5. List observes report.md, report~1.md, beta.md, precisely the reported membership. The test creates the short-lived expired entry immediately before the two-attachment arm (webserve_attachment_e2e.rs:507-546). The two returned URLs prove successful individual registration responses, not preservation against a later stale publication. Default attachment TTL is 30 days; ordinary expiration does not explain seconds-old alpha. The overwrite includes next_suffix and name_owners, not just entries (serving.rs:126-133). Therefore the mechanism also risks orphaned snapshot bytes and erasing historical name ownership, potentially violating ADR-0057's stale-link identity guarantee. Those consequences are source-derived risks, not incident measurements. (c) MINIMAL PRODUCT FIX AND OWNERSHIP Use one per-registry CROSS-PROCESS file-lock guard, not another static mutex. Acquire it BEFORE the fresh load and retain it through mutation, snapshot side effects/rollback, and durable publication. Both broker and brain must use that same storage-owned guard. Runtime ownership: three files: 1. crates/spt-store/src/serving.rs — common lock acquisition/RAII guard for the resolved registry path, stable sibling sentinel such as serve/.registry.lock. Reuse the existing fs2 sentinel pattern in info.rs:871-886. fs2 0.4 already belongs to spt-store; no new dependency is needed. Derive the sentinel in one owner, not independently in each caller. 2. crates/spt-daemon/src/servehost.rs — replace the static mutex and its obsolete wrapper/call sites with the store guard. All six mutation arms retain their existing explicit save/error behavior. 3. crates/spt-daemon/src/livehost.rs — acquire that same guard before loading; hold through reap, snapshot unlink, and save. Correct the same-process synchronization comment. Do NOT lock only save_at: the snapshot can already be stale. Do NOT lock the registry file itself: atomic replacement changes the file being locked. Do NOT unlink the sentinel on unlock. Do NOT unconditionally auto-save from a generic transaction wrapper: missing-input/no-op paths intentionally publish nothing. Preserve Attach's cleanup on failed registration/save and the reaper's existing unlink-before-publication order and attachment-only containment guard. HTTP readers can retain atomic, lock-free snapshot reads. Preserve Docs/LAN bypasses. Moving reaping into the broker, or adding a Reap IPC operation, would also establish a single writer, but expands lifecycle/protocol ownership and is not the minimal repair. (d) TEST AND PRODUCTION BLAST RADIUS; SIZE ESTIMATE Existing regression servehost.rs:703-756 (a_reap_shaped_pass_cannot_clobber_a_concurrent_add) uses two threads and sleep timing. It cannot prove the process boundary. Replace/extend that coverage with a controlled two-process transaction overlap, using the repository's tests/fixtures worker-bin pattern (spt-store/Cargo.toml:8-17, tests/wtlock_two_process_int.rs). Assert the concurrent registration survives, the expired entry retires, and persisted name allocation/history stays coherent; preserve same-process writer coverage. Include process-exit lock release so a crashed holder cannot strand writes. Use explicit barriers and bounded child cleanup, not repeated timing races. Additional live-daemon test writers must join the guard: - crates/spt/tests/webserve_attachment_e2e.rs:676-681 and :701-708 directly modify the registry while the daemon runs. These execute AFTER ARM 11, so cannot explain its earlier alpha failure. Their current reread/retry loops are not transaction protection. - crates/spt/tests/webserve_cross_node_e2e.rs:662-666, :676-680, and :760-762 directly change audience while the node runs. These also need the shared guard, tightly scoped to each mutation rather than across subsequent HTTP/CLI requests. Isolated low-level store/HTTP fixtures and the twohost_web.rs:702-706 pre-broker seed need not be migrated merely for appearance if raw load/save primitives remain available with their serialization contract. Scope estimate, not an implemented patch: 3 runtime files, 2 existing E2E files, plus a dedicated process regression, its worker fixture and Cargo fixture declaration — approximately 8 code/manifest files, plus the applicable traceability/docs/changelog updates. No JSON schema, public CLI, serving URL, TTL policy, or production dependency change is needed. Production risks: real serialization can delay the brain reconciliation pulse behind a large attachment copy/save, and delay broker serving mutations behind reaping; retain lock failure as an error, never proceed unlocked. The lock coordinates only participating binaries. An old broker plus new brain remains unsafe if that old broker writes this registry without the file lock. Brain-only update is therefore not sufficient activation for such a mixed pair; both writer processes must be running the fixed implementation. This is a rollout constraint to account for, not a request to restart production now. After edit GO: deterministic process regression, relevant store/daemon cells, unchanged attachment and cross-node E2E assertions, Windows and Linux locking coverage, traceable-reqs check, then the required fresh-SHA consumer gates. Existing evidence at 7890ead3 does not certify a future changed SHA. STATUS Investigation complete; no lane opened and no product changes made. Waiting for verdict/edit GO. Hertz retains ownership of the 10x cohort.