From https://github.com/BigscreenVR/spt-bs-core * branch test/twohost-inbound-probe -> FETCH_HEAD d882297f test(twohost): ask whether B's datagrams reach A, before the ceremony does .github/workflows/ci.yml | 4 +- .github/workflows/golden.yml | 68 ++++- CHANGELOG.md | 89 +++--- Cargo.lock | 28 +- Cargo.toml | 2 +- crates/spt-daemon/tests/mesh.rs | 30 +- crates/spt-daemon/tests/mesh_recovery.rs | 90 ++---- crates/spt-daemon/tests/pump.rs | 34 +-- crates/spt-daemon/tests/registry_lifecycle.rs | 161 ++-------- crates/spt-daemon/tests/twohost.rs | 71 +---- crates/spt-daemon/tests/twohost_web.rs | 155 +++++++++- crates/spt-test-support/src/lib.rs | 52 ---- crates/spt/tests/common/mod.rs | 93 ------ crates/spt/tests/common/reap.rs | 77 +---- crates/spt/tests/resident_service_e2e.rs | 68 +---- crates/spt/tests/webserve_attachment_e2e.rs | 103 +------ docs-site/src/changelog.md | 85 ++---- docs/FLAKE-LEDGER.md | 7 +- docs/INFRA-REGISTER.md | 411 -------------------------- docs/RELEASE-RUNBOOK.md | 16 - 20 files changed, 388 insertions(+), 1256 deletions(-) --- diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bf9b3008..17c58137 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -90,9 +90,7 @@ jobs: matrix: runner: [[self-hosted, Linux, kitsubito], [self-hosted, Windows, hfenduleam]] runs-on: ${{ matrix.runner }} - # 40 (was 25): main's Windows unit job measured 22m at e4444413 (run - # 34261096301), 3min under its own wall, for the same box-wide slowdown. - timeout-minutes: 40 + timeout-minutes: 25 steps: - uses: actions/checkout@v5 # IR-73: This floor is not protective, so it runs after checkout. diff --git a/.github/workflows/golden.yml b/.github/workflows/golden.yml index 5a56de87..40f06699 100644 --- a/.github/workflows/golden.yml +++ b/.github/workflows/golden.yml @@ -158,18 +158,12 @@ jobs: HEAVY: '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' # A wedged test must not block the single self-hosted runner slot (the # 2026-06-03 handoff.rs ConPTY stall burned 22 hosted minutes unbounded). - # 80 (was 50; the note this replaces narrated kitsubito at 25 -> 50): - # WINDOWS is the long pole now, not the Linux runner. The green Windows - # test job ran 30m53s (08-30, run 33296634901) and 33m21s (09-06, run - # 34017906638); at v0.68.0 it needs ~56min - r3 att2 (job 102352551368) - # was CANCELLED by this wall at 49m59s having reached step 30 with Phase A - # (17m14s) and Phase B (20m25s) GREEN, so docs-drift and every step after - # it were unreachable. 80 = that measured need plus ~40% for the box's - # day-to-day variance. The real fix is the Windows fs-heavy slowdown - # (spt-store monic/contextstore/syncmerge and spt-daemon::sync at 3-4x - # their 09-06 wall) - INFRA-REGISTER entry, number to follow. A cap cannot - # fix a slow box; it can only stop hiding it. - timeout-minutes: 80 + # 50 (was 25): the slow Linux runner (kitsubito) needs more than 25min for a + # full build + the whole suite (the int tests carry fixed wall-clock sleeps); + # a version-bump full rebuild pushed v0.7.0 over. The real fix is the build + # speedup (mold/nextest — ci-kitsubito-speedup-plan); this keeps the wedge + # guard generous, not unbounded. + timeout-minutes: 50 steps: # Pre-checkout exe-lock guard (Windows): a PRIOR run's leaked workspace # daemon (an spt.exe spawned from this checkout's target/) can survive into @@ -1112,6 +1106,31 @@ jobs: # this host OWNS the files, hfenduleam fetches them through its own # loopback listener. Its own bin, its own ports (+20), after the ladder. # [int->REQ-WEB-CROSS-NODE-PROXY] + # THE INBOUND PROBE, before the ceremony (releases#272). A golden run spent + # 900 s on EACH half discovering that B's datagrams never reached A: the + # Windows host is BlockInbound on every profile and no rule covered the + # test binary, so every QUIC dial died on its 10 s bound -- 75 of them, + # cadenced at exactly 12.00 s, never one ADMIT. At B a dropped datagram is + # indistinguishable from a delivered one, so it was triaged as product, + # then as rig, before it was measured as box. + # + # SEPARATE INVOCATION, not a filter inside the ceremony run: within one + # `cargo test` the cells are threads and nothing orders them, so only a + # step boundary puts this FIRST. + # + # SAME BINARY, not a shell probe: the Windows rule is PER-EXECUTABLE, so an + # allow rule naming pwsh or python says nothing about twohost_web-.exe + # -- a shell probe would pass the moment the PROBER was allowed and certify + # the failure it exists to catch. The assertion lives on A, which is the + # only side that can see arrival. + - name: Two-host inbound probe — role B (sender) + shell: bash + run: cargo test -p spt-daemon --test twohost_web two_host_inbound_probe -- --nocapture + env: + SPT_TWO_HOST: "1" + SPT_TWO_HOST_ROLE: "b" + SPT_TWO_HOST_SECRET: "rig-twohost-ci" + SPT_TWO_HOST_PEER_IP: "100.68.35.65" # hfenduleam (tailscale) - name: Two-host web serving — role B (owner) shell: bash run: bash .github/bench/wrap.sh twohost-web-b -- cargo test -p spt-daemon --test twohost_web -- --nocapture @@ -1249,6 +1268,31 @@ jobs: # serving cells — three cells, each its own listener + broker, fetching # kitsubito's files through this host's loopback. After the ladder. # [int->REQ-WEB-CROSS-NODE-PROXY] + # THE INBOUND PROBE, before the ceremony (releases#272). A golden run spent + # 900 s on EACH half discovering that B's datagrams never reached A: the + # Windows host is BlockInbound on every profile and no rule covered the + # test binary, so every QUIC dial died on its 10 s bound -- 75 of them, + # cadenced at exactly 12.00 s, never one ADMIT. At B a dropped datagram is + # indistinguishable from a delivered one, so it was triaged as product, + # then as rig, before it was measured as box. + # + # SEPARATE INVOCATION, not a filter inside the ceremony run: within one + # `cargo test` the cells are threads and nothing orders them, so only a + # step boundary puts this FIRST. + # + # SAME BINARY, not a shell probe: the Windows rule is PER-EXECUTABLE, so an + # allow rule naming pwsh or python says nothing about twohost_web-.exe + # -- a shell probe would pass the moment the PROBER was allowed and certify + # the failure it exists to catch. The assertion lives on A, which is the + # only side that can see arrival. + - name: Two-host inbound probe — role A (receipt asserted here) + shell: pwsh + run: cargo test -p spt-daemon --test twohost_web two_host_inbound_probe -- --nocapture + env: + SPT_TWO_HOST: "1" + SPT_TWO_HOST_ROLE: "a" + SPT_TWO_HOST_SECRET: "rig-twohost-ci" + SPT_TWO_HOST_PEER_IP: "100.98.197.12" # kitsubito (tailscale) - name: Two-host web serving — role A (requester) shell: pwsh run: .github/bench/wrap.ps1 -Step "twohost-web-a" -Command "cargo test -p spt-daemon --test twohost_web -- --nocapture" diff --git a/CHANGELOG.md b/CHANGELOG.md index eafe18d6..943fdc5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,17 +10,44 @@ breaks something, or changes the observable behavior of existing surfaces broadly; **patch** for fixes, and for additive opt-in capability — a new key, flag, or page that no existing user can encounter without opting into it. -## [0.68.0] - 2026-09-08 - -Web serving. Files, directories, adapter documentation and the changelog are -reachable at node-prefixed addresses on the local server, and an address that -names another machine in the subnet is answered by the machine that owns it. -Messages can carry attachments the receiver pulls on demand, every message -gets a short ID to show or reply to, and a machine with no spt on it can be -handed the binary over the local network. The `XFER` access surface is retired. +## [Unreleased] ### Added + +- `spt serve lan --bootstrap` hands the spt binary to a machine that is not + yet a node. It is a SECOND listener on port 5470, bound on all interfaces, + off by default and off again after every daemon restart; `--stop` takes it + down and a bare `spt serve lan` reports its state. It serves only + `/bin//spt[.exe]`, that file's `.release.json` sidecar, and an + `/install` page - everything else answers not-found naming the listener, and + the docs server on 5474 stays loopback-only throughout. While it is up, + anyone who can reach the socket may pull the binary, so the now-signal + carries a standing LAN-EXPOSED line for the whole window. + +- The bootstrap listener serves the artifacts of the applied signed release + set or it refuses to start, naming why + (`LAN_BOOTSTRAP_REFUSED:unsigned-exe`, `:sha-mismatch`, `:set-not-applied`). + Each platform is re-checked as it is served, so a tampered or missing + artifact answers `LAN_BOOTSTRAP_TRIPLE_UNAVAILABLE` for that platform alone + while the others keep serving. Start prints one `sha256 ` line + per platform, and `spt install` gains `--expect-sha256` (refuses anything but + those exact bytes, printing what it downloaded first) and `--release-json` + (re-verifies the signature and digest against its own built-in release keys) + so the two machines' hashes can be compared by eye. + +- The changelog is now a page of the local docs (`spt docs url`, then + **Changelog** under Reference), so what a release changed can be read on + the machine being updated without a repository checkout. The page is + generated from `CHANGELOG.md` and checked against it in CI. + +- An adapter can publish its own documentation: the new optional + `[adapter].docs_dir` manifest key names a directory inside the adapter, + and it is served at `//a//docs/`. Without the key that URL + answers not-found naming the facet; it never serves anything else in its + place. The key is adapter-relative — an absolute path or a path that + leaves the adapter's own directory is refused by name, both when the + manifest is read and again on every request. - `spt serve add`, `spt serve rm`, and `spt serve list --json` manage live file and directory references. Same-name registrations receive stable numbered @@ -43,34 +70,6 @@ handed the binary over the local network. The `XFER` access surface is retired. cached; `HEAD` and `Range` requests are honored by the owner. An owner that refuses answers 403 naming `WEB`; an owner that cannot be reached answers 502 naming the node, within a bounded time. -- `spt send --attachment ` sends a file with a message. The file's bytes - are captured as they are at send time, so later edits or a deletion do not - change what arrives, and the receiver pulls them with `spt fetch` when it - wants them. `--ttl` sets how long an attachment stays available (default 30 - days, a unit is required); expired attachments answer 404 immediately. -- Every message now carries a short ID. `spt msg show ` prints a message by - it, and `spt send --reply-to ` marks what a message answers. Messages - from another machine resolve through the machine that holds them. -- `spt serve lan --bootstrap` hands the spt binary to a machine that is not yet - a node, over the local network on port 5470. It is off by default and off - again after every daemon restart, serves only the binary, its release sidecar - and an install command, and refuses to start by name when the applied update - set is not signed or does not match. It prints a checksum per platform, and - `spt install` gains `--expect-sha256` and `--release-json` so what was - downloaded can be compared before it is run. The documentation port is - untouched and stays on loopback. -- `[adapter].docs_dir` publishes an adapter's own documentation at the `docs` - segment of its address, beside the adapter's output directory rather than in - place of it. A missing key, an unreadable manifest, a path that escapes the - adapter's directory, and a directory that has since been removed all answer - the same 404 naming what was asked for. -- The changelog is now one of the pages the local documentation server offers. -- A harness is told when a message it receives refers to a file the reader - cannot open — an attachment to pull, or a path that belongs to another - machine — instead of leaving the reader to discover it. For a path on - another machine, that machine can be asked to publish it and answers with a - link; it honors such a request only for its own endpoints, so no third - machine can have someone else's file exposed. ### Changed @@ -79,14 +78,6 @@ handed the binary over the local network. The `XFER` access surface is retired. and `?json` views. `/` redirects to `//`; canonical docs URLs live at `//docs/`, while existing bare docs paths remain compatibility aliases. -### Removed - -- The `XFER` access surface is retired: attachments replaced the transfer it - gated, so the entry leaves the access-control vocabulary along with the - transfer itself. An existing rule that names `XFER` is kept and reported at - load rather than dropped silently; it no longer governs anything. The shell - channel's own transfer progress is unaffected. - ### Fixed @@ -94,16 +85,6 @@ handed the binary over the local network. The `XFER` access surface is retired. history and last-message state, for both queued backlog and live TCP delivery. A later hook poll does not record those deliveries again; filtered notifications are not recorded as delivered messages. -- A node's message and activity history reads back correctly once its log grows - past a quarter of a megabyte. Positions no longer restart from the beginning, - so a request for everything since a given point no longer comes back empty or - repeats entries that were already seen, and an oversized position is answered - with the current head instead of skipping a page. -- An agent's own saved context is no longer overwritten, unread, by the - automatic summary that follows it. The automatic summary is filed directly - and never writes to the file an agent saves its own context to. -- `spt api bind` no longer prints an engine-room probe line on a machine that - has no engine room. ## [0.67.1] - 2026-09-06 diff --git a/Cargo.lock b/Cargo.lock index 0a8a5d86..4d191b93 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2387,7 +2387,7 @@ dependencies = [ [[package]] name = "mock-adapter" -version = "0.68.0" +version = "0.67.1" dependencies = [ "hmac", "sha2 0.10.9", @@ -4096,7 +4096,7 @@ dependencies = [ [[package]] name = "spt" -version = "0.68.0" +version = "0.67.1" dependencies = [ "chrono", "clap", @@ -4131,7 +4131,7 @@ dependencies = [ [[package]] name = "spt-daemon" -version = "0.68.0" +version = "0.67.1" dependencies = [ "avt", "base64", @@ -4162,7 +4162,7 @@ dependencies = [ [[package]] name = "spt-live" -version = "0.68.0" +version = "0.67.1" dependencies = [ "serde", "spt-msg", @@ -4174,7 +4174,7 @@ dependencies = [ [[package]] name = "spt-msg" -version = "0.68.0" +version = "0.67.1" dependencies = [ "libc", "spt-proto", @@ -4186,7 +4186,7 @@ dependencies = [ [[package]] name = "spt-net" -version = "0.68.0" +version = "0.67.1" dependencies = [ "chacha20", "data-encoding", @@ -4210,7 +4210,7 @@ dependencies = [ [[package]] name = "spt-poolguard" -version = "0.68.0" +version = "0.67.1" dependencies = [ "spt-procident", "tempfile", @@ -4218,14 +4218,14 @@ dependencies = [ [[package]] name = "spt-procident" -version = "0.68.0" +version = "0.67.1" dependencies = [ "libc", ] [[package]] name = "spt-proto" -version = "0.68.0" +version = "0.67.1" dependencies = [ "ed25519-dalek 2.2.0", "rand_core 0.6.4", @@ -4233,7 +4233,7 @@ dependencies = [ [[package]] name = "spt-runtime" -version = "0.68.0" +version = "0.67.1" dependencies = [ "regex", "schemars", @@ -4248,7 +4248,7 @@ dependencies = [ [[package]] name = "spt-store" -version = "0.68.0" +version = "0.67.1" dependencies = [ "fs2", "libc", @@ -4268,7 +4268,7 @@ dependencies = [ [[package]] name = "spt-term" -version = "0.68.0" +version = "0.67.1" dependencies = [ "avt", "portable-pty", @@ -4284,7 +4284,7 @@ dependencies = [ [[package]] name = "spt-test-support" -version = "0.68.0" +version = "0.67.1" dependencies = [ "tempfile", ] @@ -5634,7 +5634,7 @@ dependencies = [ [[package]] name = "xtask" -version = "0.68.0" +version = "0.67.1" dependencies = [ "fs2", "serde_json", diff --git a/Cargo.toml b/Cargo.toml index 6a5bbe24..51cadcb4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -16,7 +16,7 @@ members = ["crates/spt-proto", "crates/spt-procident", "crates/spt-poolguard", " # version stays its own counter (REQ-ARCH-3) and the SignedRelease `version` # is the separate monotonic release counter; neither conflates with this. [workspace.package] -version = "0.68.0" +version = "0.67.1" # W3 REQ-SCREENGRID-WIDTH width policy: ONE display-width authority for the whole # tree. spt-term's ScreenGrid and the ratatui picker (ratatui 0.29 resolves diff --git a/crates/spt-daemon/tests/mesh.rs b/crates/spt-daemon/tests/mesh.rs index bda7c9ce..f205ee19 100644 --- a/crates/spt-daemon/tests/mesh.rs +++ b/crates/spt-daemon/tests/mesh.rs @@ -102,34 +102,14 @@ fn connect_retry(name: &str) -> Brain { /// Poll `probe` until true; panic with `what` on timeout (15s — green paths /// exit early, so the long bound only pays on the *red* mesh assertions). -/// the pump's REGISTRY round cadence, which is what this file's convergences -/// wait on (the rig sets `registry: Duration::from_millis(100)`). -const PUMP_REGISTRY_CADENCE: Duration = Duration::from_millis(100); - -/// Budget for a convergence in this file, DERIVED rather than guessed: the old -/// `for _ in 0..600 { sleep 25ms }` was a hardcoded 15.0 s copied into four -/// test files across 31 call sites, tuned against one green run and sitting -/// inside the CI box's own measured variance — a golden attempt lost it by -/// 0.7 s. This is 300 x [`PUMP_REGISTRY_CADENCE`] = 30 s: HALF the suite's 60 s SLOW line -/// (`.config/nextest.toml`), so it is generous against a slow box while still -/// refusing a convergence that has genuinely stopped. -const CONVERGE_BUDGET: Duration = Duration::from_millis(PUMP_REGISTRY_CADENCE.as_millis() as u64 * 300); - -/// NOTE, deliberate and stated rather than silent: the call sites in this file -/// still hand back a bare `bool`, so the failure reports the budget, the -/// elapsed time and the sample count but NOT which predicate was false. The -/// file whose cell actually red (`mesh_recovery.rs`) names its per-site state -/// (`healed=… live=…`); doing the same here is mechanical and unreviewed, so it -/// is left for whoever next has a red in this file to name — with the loop and -/// the budget already correct underneath them. fn converge(what: &str, mut probe: impl FnMut() -> bool) { - spt_test_support::converge_within(CONVERGE_BUDGET, what, &mut || { + for _ in 0..600 { if probe() { - Ok(()) - } else { - Err(String::from("predicate false (this site does not name its state yet)")) + return; } - }); + thread::sleep(Duration::from_millis(25)); + } + panic!("never converged: {what}"); } /// The inert/registry-only cadence: only registry feeds flow (notif/sync/ diff --git a/crates/spt-daemon/tests/mesh_recovery.rs b/crates/spt-daemon/tests/mesh_recovery.rs index d08b9b5a..d561b040 100644 --- a/crates/spt-daemon/tests/mesh_recovery.rs +++ b/crates/spt-daemon/tests/mesh_recovery.rs @@ -87,28 +87,14 @@ fn connect_retry(name: &str) -> Brain { panic!("brain could not connect"); } -/// The rig's dialer-side bound, named ONCE and used in BOTH places it matters: -/// every `net_broker_with_dial_timeout` knob that imposes it, and the -/// convergence budget derived from it below. It was a bare -/// `Duration::from_millis(1500)` at two call sites with the budget hardcoded -/// elsewhere as `0..600` x 25 ms — three literals with no stated relationship, -/// which is how a budget stops tracking the mechanism it must outlast. -const TEST_DIAL_BOUND: Duration = Duration::from_millis(1500); - -/// What a convergence in this file races: a dial that fails at -/// [`TEST_DIAL_BOUND`], is retried by the pump, then reconciled and written -/// back. Budget = that bound x enough attempts to be a MARGIN rather than a -/// coin toss, kept under the suite's 60 s SLOW line (`.config/nextest.toml`) — -/// past that a convergence is a different bug than a slow box. -/// -/// The old fixed 15.0 s was ~1.5x a FAST observation (9.8 s / 7.2 s) on a box -/// that ran 19 of 73 cells >= 1.5x slower in one attempt, and it lost by 0.7 s. -/// 20 x the dial bound is 30 s: twice the old budget, half the SLOW line, and -/// derived from the thing it waits on instead of tuned against one green run. -const CONVERGE_BUDGET: Duration = Duration::from_millis(TEST_DIAL_BOUND.as_millis() as u64 * 20); - -fn converge(what: &str, mut probe: impl FnMut() -> Result<(), String>) { - spt_test_support::converge_within(CONVERGE_BUDGET, what, &mut probe); +fn converge(what: &str, mut probe: impl FnMut() -> bool) { + for _ in 0..600 { + if probe() { + return; + } + thread::sleep(Duration::from_millis(25)); + } + panic!("never converged: {what}"); } /// Run the production dispatcher on the serving node (pump legs pull against @@ -256,7 +242,7 @@ fn roster_route_survives_a_transient_dial_failure_with_discovery_disabled() { // A's dials fail fast (1.5s) — the everyday transient must not pin the // test on the production 10s bound. let _a_broker = - net_broker_with_dial_timeout(&a_name, dir.path(), Some(TEST_DIAL_BOUND)); + net_broker_with_dial_timeout(&a_name, dir.path(), Some(Duration::from_millis(1500))); let _b_broker = net_broker(&b_name, dir.path()); // The stale address: a socket that WAS a real endpoint and is now dead // (host dropped, port closed) — the field shape (machine moved/rebooted). @@ -317,13 +303,7 @@ fn roster_route_survives_a_transient_dial_failure_with_discovery_disabled() { ); let healed = matches!(s.valid_route(&b_hex), Some(v) if *v != stale); let live = PumpHealth::load_from(&health_path).live_peers > 0; - // Both predicates stay REQUIRED (the 07-16 gate refuted two - // trivially-true ones); the Err arm only reports which half is missing. - if healed && live { - Ok(()) - } else { - Err(format!("healed={healed} live={live}")) - } + healed && live }); stop.store(true, Ordering::Relaxed); dstop.store(true, Ordering::Relaxed); @@ -382,17 +362,11 @@ fn poison_cache_row_is_repaired_from_the_roster_at_startup() { // Repaired-or-rejected: whatever is under B's key must satisfy the // invariant (never the poison) — and a LIVE peer proves the repaired // route carried a real QUIC connect, not just a rewritten file. - let invariant = s - .get(&b_hex) + s.get(&b_hex) .map(|v| spt_store::peeraddrs::addr_peer_id(v) == Some(b_hex.as_str())) - .unwrap_or(false); - let routed = s.valid_route(&b_hex).is_some(); - let live = PumpHealth::load_from(&health_path).live_peers > 0; - if invariant && routed && live { - Ok(()) - } else { - Err(format!("invariant={invariant} routed={routed} live={live}")) - } + .unwrap_or(false) + && s.valid_route(&b_hex).is_some() + && PumpHealth::load_from(&health_path).live_peers > 0 }); stop.store(true, Ordering::Relaxed); dstop.store(true, Ordering::Relaxed); @@ -448,16 +422,10 @@ fn health_degrades_during_all_peer_failure_and_recovers_on_real_progress() { // fingerprint that made the field incident invisible). converge("health degrades during the outage", || { let h = PumpHealth::load_from(&health_path); - let degraded = matches!( + matches!( h.verdict(), spt_daemon::pump::health::HealthVerdict::Degraded { .. } - ); - let reason = h.failing.get(&b_hex).map(String::as_str); - if degraded && reason == Some("address-resolution") { - Ok(()) - } else { - Err(format!("degraded={degraded} reason={reason:?}")) - } + ) && h.failing.get(&b_hex).map(String::as_str) == Some("address-resolution") }); let hb: u64 = std::fs::read_to_string(&heartbeat_path) .expect("heartbeat exists") @@ -479,12 +447,7 @@ fn health_degrades_during_all_peer_failure_and_recovers_on_real_progress() { *discovered.lock().unwrap() = Some(b_addr); converge("health recovers on real progress", || { let h = PumpHealth::load_from(&health_path); - let verdict = h.verdict(); - if verdict == spt_daemon::pump::health::HealthVerdict::Healthy { - Ok(()) - } else { - Err(format!("verdict={verdict:?}")) - } + h.verdict() == spt_daemon::pump::health::HealthVerdict::Healthy }); stop.store(true, Ordering::Relaxed); dstop.store(true, Ordering::Relaxed); @@ -512,7 +475,7 @@ fn a_dead_roster_address_does_not_strand_the_peer_from_discovery() { let (a_name, b_name) = (unique_name(), unique_name()); // A's dials fail fast — the dead roster address must cost ~1.5s, not 10. let _a_broker = - net_broker_with_dial_timeout(&a_name, dir.path(), Some(TEST_DIAL_BOUND)); + net_broker_with_dial_timeout(&a_name, dir.path(), Some(Duration::from_millis(1500))); let _b_broker = net_broker(&b_name, dir.path()); // The dead address: a real endpoint that is gone (host moved/rebooted) — // the same construction the trap test uses, for the same reason. @@ -598,12 +561,7 @@ fn a_dead_roster_address_does_not_strand_the_peer_from_discovery() { ); converge("discovery reached after the roster address failed", || { - let live = PumpHealth::load_from(&health_path).live_peers; - if live > 0 { - Ok(()) - } else { - Err(format!("live_peers={live}")) - } + PumpHealth::load_from(&health_path).live_peers > 0 }); // [int->REQ-DIAL-ROUTE-LEG-ATTRIBUTION] The verdict bound to the scenario, @@ -723,15 +681,9 @@ fn a_frozen_self_row_heals_at_boot_and_readvertises_the_live_address() { converge("self row healed and re-advertised", || { let r = spt_store::roster::RosterStore::load_from(&roster_path); let Some(me) = r.find("meshrec", &a_hex) else { - return Err("no self row in the roster yet".into()); + return false; }; - let bumped = me.lease_epoch > FROZEN_LEASE; - let advertised = me.address.as_ref() == Some(&a_addr); - if bumped && advertised { - Ok(()) - } else { - Err(format!("lease_bumped={bumped} advertised={advertised}")) - } + me.lease_epoch > FROZEN_LEASE && me.address.as_ref() == Some(&a_addr) }); assert!( EpochSource::load_from(&node_counter).current() > FROZEN_LEASE, diff --git a/crates/spt-daemon/tests/pump.rs b/crates/spt-daemon/tests/pump.rs index 3c83bec1..eb80b9b4 100644 --- a/crates/spt-daemon/tests/pump.rs +++ b/crates/spt-daemon/tests/pump.rs @@ -131,34 +131,14 @@ fn stamped(body: &str) -> String { } /// Poll `probe` until it returns true, panicking with `what` on timeout. -/// the dialer-side bound this file's rig imposes via `set_quic_op_timeout`, -/// which is what a convergence here waits out before the pump retries. -const TEST_DIAL_BOUND: Duration = Duration::from_millis(500); - -/// Budget for a convergence in this file, DERIVED rather than guessed: the old -/// `for _ in 0..600 { sleep 25ms }` was a hardcoded 15.0 s copied into four -/// test files across 31 call sites, tuned against one green run and sitting -/// inside the CI box's own measured variance — a golden attempt lost it by -/// 0.7 s. This is 60 x [`TEST_DIAL_BOUND`] = 30 s: HALF the suite's 60 s SLOW line -/// (`.config/nextest.toml`), so it is generous against a slow box while still -/// refusing a convergence that has genuinely stopped. -const CONVERGE_BUDGET: Duration = Duration::from_millis(TEST_DIAL_BOUND.as_millis() as u64 * 60); - -/// NOTE, deliberate and stated rather than silent: the call sites in this file -/// still hand back a bare `bool`, so the failure reports the budget, the -/// elapsed time and the sample count but NOT which predicate was false. The -/// file whose cell actually red (`mesh_recovery.rs`) names its per-site state -/// (`healed=… live=…`); doing the same here is mechanical and unreviewed, so it -/// is left for whoever next has a red in this file to name — with the loop and -/// the budget already correct underneath them. fn converge(what: &str, mut probe: impl FnMut() -> bool) { - spt_test_support::converge_within(CONVERGE_BUDGET, what, &mut || { + for _ in 0..600 { if probe() { - Ok(()) - } else { - Err(String::from("predicate false (this site does not name its state yet)")) + return; } - }); + thread::sleep(Duration::from_millis(25)); + } + panic!("never converged: {what}"); } // [int->REQ-INST-5] [int->REQ-INST-7] [int->REQ-NOTIF-1] the full self-drive: @@ -520,7 +500,7 @@ fn pump_survives_a_black_holing_peer_heartbeat_advances_no_restart() { // A's broker: requires the seed-proof + a SHORT QUIC-op bound, so each // dead-peer dial fails fast (≈ the bound) instead of pinning the test ~10s. let a_host = NetHost::start(proving()).expect("a host"); - a_host.set_quic_op_timeout(TEST_DIAL_BOUND); + a_host.set_quic_op_timeout(Duration::from_millis(500)); let a_broker = Broker::bind_in_with_net(&a_name, dir.path().join("a-eff.log"), Some(a_host)) .expect("bind a"); { @@ -710,7 +690,7 @@ fn pump_w2_live_peer_advertised_amid_dead_peers_no_restart() { // A: proving broker with a SHORT dial bound so each dead-peer dial fails fast. let a_host = NetHost::start(proving()).expect("a host"); - a_host.set_quic_op_timeout(TEST_DIAL_BOUND); + a_host.set_quic_op_timeout(Duration::from_millis(500)); let a_broker = Broker::bind_in_with_net(&a_name, dir.path().join("a-eff.log"), Some(a_host)) .expect("bind a"); { diff --git a/crates/spt-daemon/tests/registry_lifecycle.rs b/crates/spt-daemon/tests/registry_lifecycle.rs index 1f11bab5..f7559d27 100644 --- a/crates/spt-daemon/tests/registry_lifecycle.rs +++ b/crates/spt-daemon/tests/registry_lifecycle.rs @@ -15,7 +15,6 @@ //! filter strings. use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; -use std::sync::mpsc; use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; @@ -88,53 +87,16 @@ fn stamped(body: &str) -> String { } /// Poll `probe` until it returns true, panicking with `what` on timeout. -/// the pump's REGISTRY round cadence, which is what this file's convergences -/// wait on (the rig sets `registry: Duration::from_millis(100)`). -const PUMP_REGISTRY_CADENCE: Duration = Duration::from_millis(100); - -/// Budget for a convergence in this file, DERIVED rather than guessed: the old -/// `for _ in 0..600 { sleep 25ms }` was a hardcoded 15.0 s copied into four -/// test files across 31 call sites, tuned against one green run and sitting -/// inside the CI box's own measured variance — a golden attempt lost it by -/// 0.7 s. This is 300 x [`PUMP_REGISTRY_CADENCE`] = 30 s: HALF the suite's 60 s SLOW line -/// (`.config/nextest.toml`), so it is generous against a slow box while still -/// refusing a convergence that has genuinely stopped. -const CONVERGE_BUDGET: Duration = Duration::from_millis(PUMP_REGISTRY_CADENCE.as_millis() as u64 * 300); - -/// NOTE, deliberate and stated rather than silent: the call sites in this file -/// still hand back a bare `bool`, so the failure reports the budget, the -/// elapsed time and the sample count but NOT which predicate was false. The -/// file whose cell actually red (`mesh_recovery.rs`) names its per-site state -/// (`healed=… live=…`); doing the same here is mechanical and unreviewed, so it -/// is left for whoever next has a red in this file to name — with the loop and -/// the budget already correct underneath them. fn converge(what: &str, mut probe: impl FnMut() -> bool) { - spt_test_support::converge_within(CONVERGE_BUDGET, what, &mut || { + for _ in 0..600 { if probe() { - Ok(()) - } else { - Err(String::from("predicate false (this site does not name its state yet)")) + return; } - }); + thread::sleep(Duration::from_millis(25)); + } + panic!("never converged: {what}"); } -/// Budget for the PUMP'S OWN EXIT once its stop flag is set (the a4 fix). -/// MIRRORS the product's private `PUMP_PEER_IO_TIMEOUT` (30 s, `pump/mod.rs`), -/// the bound on any single reply read — the pump splits its carrier at -/// construction precisely so a black-holed peer cannot wedge the loop. The -/// mirror is unavoidable (that const is private; this is a test-only change), -/// so the coupling is STATED rather than hidden: IF THAT CONSTANT MOVES, MOVE -/// THIS ONE. Drift makes this budget wrong, never silently permissive, because -/// the failure below names the budget it waited. -/// -/// HONEST LIMIT: this does NOT bound a pathological round. A round runs five -/// legs and may contain several bounded reads, so a worst-case round exceeds -/// one timeout. It bounds what this rig can actually produce — a live loopback -/// peer whose reads return in milliseconds — and its job is to turn a HANG into -/// a NAMED FAILURE, which is precisely what the prior comment here (correctly) -/// said a bare unbounded join could not do. -const PUMP_EXIT_BUDGET: Duration = Duration::from_secs(30); - fn pump_paths(root: &std::path::Path) -> PumpPaths { PumpPaths { owlery: root.join("owlery"), @@ -198,38 +160,6 @@ fn gauges(probe: &mut Brain) -> (u64, u64) { (s.streams_held, s.stream_seats) } -/// B's live stream table, formatted for a failure message so the next red NAMES -/// the row instead of leaving a bare count. It reports what the table actually -/// carries: `net-status` gauges are TOTALS, and `NetStreamInfo` has no FAMILY -/// field — family is a registry-row concept here, so the caller prints -/// per-family row counts alongside this. Called only from a panic arm. -fn stream_table(probe: &mut Brain) -> String { - match probe.net_streams() { - Ok(r) if r.streams.is_empty() => String::from(" (stream table EMPTY)"), - Ok(r) => r - .streams - .iter() - .map(|s| { - format!( - " stream {} conn {} remote {} next_seq {} finished {} local_open {} floor {} lifetime {:?}", - s.stream_id, - s.conn_id, - &s.remote_id_hex[..s.remote_id_hex.len().min(12)], - s.next_seq, - s.finished, - s.initiated_locally, - s.floor_seq, - s.lifetime, - ) - }) - .collect::>() - .join("\n"), - // UNREADABLE is its own value: never render a failed probe as an empty - // table, or the next reader diagnoses "no streams" from "could not ask". - Err(e) => format!(" (stream table UNREADABLE: {e})"), - } -} - // [int->REQ-HAZARD-REGISTRY-STALL] [int->REQ-ONEWAY-STREAM-TERMINAL] // [int->REQ-SEAT-LIFETIME-BOUNDED] the plateau seam, hertz shape verbatim: // N registry rounds (plus sync pulls — the CROSS-FAMILY correction: retention @@ -358,12 +288,7 @@ fn oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing() { // ── The pump: production loop, ONE persistent conn to B, registry + // sync legs at a fast cadence (many rounds inside the window). ────── let pump_stop = Arc::new(AtomicBool::new(false)); - // THE HANDLE IS KEPT (a4 fix). It used to be dropped on the floor by a bare - // `thread::spawn` in a block — so the rig had no way to order the pump's - // last round before anything it later sampled. See the quiesce below for - // why a keeping a handle is not the same as a bare join. - let (pump_exited_tx, pump_exited_rx) = mpsc::channel::<()>(); - let pump = { + { let name = a_name.clone(); let reg = Arc::clone(&a_registry); let flag = Arc::clone(&pump_stop); @@ -400,13 +325,8 @@ fn oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing() { PumpHooks::default(), &flag, ); - // Sent AFTER `run_peer_pump` returns, so a successful recv proves - // the loop is OVER — not merely that the flag was observed. The - // send is deliberately last and its result ignored: if the rig has - // already dropped the receiver the pump must still exit cleanly. - let _ = pump_exited_tx.send(()); - }) - }; + }); + } // Both legs proved live: B holds A's advertised row; A pulled B's mind. converge("B registry holds reglc-ag@A", || { @@ -452,48 +372,18 @@ fn oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing() { // ── Quiesce: stop the pump, let in-flight serves finish — physical // rows and seats return to the clean baseline on BOTH ends of the // still-open connection (no conn-close sweep involved). ───────────── - // ORDER THE PUMP'S EXIT BEFORE ANYTHING SAMPLES B. This block used to - // argue AGAINST joining the pump, and the argument was sound about the - // thing it named: a BARE, UNBOUNDED join can hang, and a hung test reports - // nothing at all — strictly worse than the assertion it would protect. - // What it missed is that "bounded" and "joined" are separable, so the - // choice was never join-or-converge. - // - // WHY IT HAD TO CHANGE (golden #272 r2 attempt 4, 25e60015): the drains - // below converge on A's and B's GAUGES, which prove the observable state - // is clean — they do NOT prove the pump THREAD is done. The flag is read - // at the top of `run_peer_pump`'s loop, so a round already in flight when - // the flag is set still runs to completion, and its last feed can land on - // B AFTER the gauges have read clean. That is one held row plus one - // in-flight seat arriving after `b_held_before` is sampled — the exact - // `held 0 -> 1, seats 1` red, on the assertion ~120 lines below. - // - // WHAT IS BOUNDED, measured rather than assumed: the loop condition is - // `while !stop.load(...)`, so the flag IS honoured at a round boundary; - // and every reply read is bounded by the product's `PUMP_PEER_IO_TIMEOUT` - // (30 s) because the pump splits its carrier at construction so a - // black-holed peer cannot wedge it. A round runs five legs and may contain - // several such reads, so a PATHOLOGICAL round can outlast any single - // timeout — see `PUMP_EXIT_BUDGET`, which says so out loud. - // - // So: a BOUNDED RENDEZVOUS, not a bare join. `recv_timeout` waits for the - // pump to actually return and FAILS BY NAME if it does not, which is the - // one thing the unbounded join could not do. The join that follows is then - // provably prompt — the thread has already signalled that it returned. - // Rule of thumb, updated: join what you can bound, BOUND what you cannot - // join promptly, converge on state you can only observe — never sleep. + // The pump is stopped and then WAITED FOR VIA AN OBSERVABLE, not joined. + // Deliberate, and the distinction is the point of this fix: `converge` + // polls a real gauge (seats back to 0, rows back to baseline) with a + // bounded budget, so it proves the drained STATE rather than guessing at a + // duration — that is the good pattern, already here. A join would be + // stronger still, but `run_peer_pump`'s round can block on network work, + // and an unbounded join that hangs is a worse failure than the assertion + // it would protect: a hung test reports nothing at all. The dispatcher + // below IS joined because its 25ms flag-poll makes the join provably + // prompt. Rule of thumb: join what you can bound, converge on what you + // cannot — never sleep on either. pump_stop.store(true, Ordering::Relaxed); - match pump_exited_rx.recv_timeout(PUMP_EXIT_BUDGET) { - Ok(()) => {} - Err(e) => panic!( - "the pump did not exit within {PUMP_EXIT_BUDGET:?} of its stop flag ({e:?}). \ - The loop checks the flag at a round boundary, so this means a leg is still \ - running: either a reply read is wedged past the product's PUMP_PEER_IO_TIMEOUT, \ - or a round is chaining more bounded reads than this budget covers. Both are real \ - findings — do NOT raise this budget to make them go away." - ), - } - pump.join().expect("pump thread parks once it has signalled its exit"); converge("A drains to baseline (rows + seats)", || { let (held, seats) = gauges(&mut a_probe); seats == 0 && held <= a_held_0 + IN_FLIGHT_SLACK @@ -621,22 +511,11 @@ fn oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing() { // inside the window would pass here. That is acceptable only because the // re-apply bound would still catch the writes such a spike produced. let (b_held_gen2, b_seats_gen2) = gauges(&mut b_probe); - // THE CONDITION IS UNTOUCHED — only the MESSAGE grew. The gauges are - // TOTALS, so `held 0 -> 1, seats 1` said a row arrived without saying WHICH, - // leaving the next reader to re-derive it from scratch. The table names the - // row; per-family counts sit beside it because `NetStreamInfo` carries no - // family field (family is a registry-row concept here). Evaluated only on - // failure: `assert!` does not touch its format arguments on the pass path. assert!( b_held_gen2 <= b_held_before && b_seats_gen2 == 0, "the refresh must subscribe NO historical rows (zero replay churn): held \ {b_held_before} -> {b_held_gen2}, seats {b_seats_gen2} (growth is the violation; a \ - drop is a straggler retiring)\n\ - B stream table at failure (gauges are totals; these name the rows):\n{}\n\ - B registry rows by family: reglc-ag={} ling-reglc={}", - stream_table(&mut b_probe), - b_registry.rows(subnet, "reglc-ag").len(), - b_registry.rows(subnet, "ling-reglc").len(), + drop is a straggler retiring)" ); // Park gen-2 before the TempDir drops: a dispatcher still polling into a diff --git a/crates/spt-daemon/tests/twohost.rs b/crates/spt-daemon/tests/twohost.rs index 141c9afc..cd835133 100644 --- a/crates/spt-daemon/tests/twohost.rs +++ b/crates/spt-daemon/tests/twohost.rs @@ -518,57 +518,6 @@ fn rig_wait(what: &str, deadline: Duration, mut probe: impl FnMut() -> bool) { panic!("never converged on the rig: {what}"); } -/// Accept the reply set the WIRE CONTRACT actually promises for a rest op. -/// -/// These rungs used to assert `matches!(out, Edge(_))` — treating the reply -/// DISCRIMINANT as the witness that the state changed. The product declines to -/// promise that, in two places at this sha: -/// -/// `resthost.rs` module doc, "## Idempotence over the wire": -/// "`daemon_rest_event` is idempotent by the pure transition table (a -/// replayed event against a state it does not move is a no-edge), so a -/// redelivered rest op is naturally a `no_edge` reply — report, don't -/// dedup. The request's `op_id` is correlation/tracing only." -/// -/// `request_rest`'s own doc: -/// "`open_op` is the durable id for the exactly-once stream open (the -/// attach discipline); the request line itself is unjournaled — a -/// redelivered request is a natural `no_edge` at the target's pure table, -/// reported not deduped." -/// -/// Exactly-once covers the STREAM OPEN, explicitly not the request line. So a -/// `no_edge` reply is a CONTRACT-CONFORMING answer to a request whose edge was -/// applied by an earlier delivery — and asserting `Edge(_)` makes the rung red -/// on behaviour the product documents as normal. -/// -/// THE WITNESS IS THE DURABLE OBSERVABLE, which every one of these rungs -/// already waits on immediately after the call (the registry row advertising -/// Suspended/Active at A; `read_rest` at B). Nothing is lost by accepting both -/// replies: the state assertion is unchanged and still does all the proving. -/// -/// Fails only on the two outcomes that mean the op did NOT land: `Failed` (the -/// target's transition host errored) and `NoReply` (gate-refused or dropped). -/// The reply that arrived is PRINTED with a timestamp, because which one -/// arrived is real information about delivery even though it is not a verdict. -// [int->REQ-INST-6] -fn accept_rest_reply(what: &str, out: &spt_daemon::resthost::RestRequestOutcome) { - use spt_daemon::resthost::RestRequestOutcome as O; - match out { - O::Edge(detail) => { - println!("TWOHOST REST {what}: EDGE at {} — {detail}", now_ms()); - } - O::NoEdge => { - println!( - "TWOHOST REST {what}: NO_EDGE at {} — contract-conforming (a redelivered \ - request answers no_edge; the durable observable below is the witness)", - now_ms() - ); - } - O::Failed(e) => panic!("{what}: the target's transition host FAILED: {e}"), - O::NoReply => panic!("{what}: NO REPLY — gate-refused or dropped, the op did not land"), - } -} - fn stamped(node: &str, body: &str) -> String { format!( "\n{body}", @@ -2270,7 +2219,10 @@ fn two_host_ladder_role_a() { MintedOp::new(Minter::Cli, op()), ) .expect("suspend round trip"); - accept_rest_reply("remote suspend", &out); + assert!( + matches!(out, spt_daemon::resthost::RestRequestOutcome::Edge(_)), + "remote suspend applied an edge at B: {out:?}" + ); rig_wait("rest: B's row advertises Suspended at A", rig.wait, || { registry .rows(&rig.subnet, ID_B) @@ -2286,7 +2238,10 @@ fn two_host_ladder_role_a() { MintedOp::new(Minter::Cli, op()), ) .expect("wake round trip"); - accept_rest_reply("remote wake", &out); + assert!( + matches!(out, spt_daemon::resthost::RestRequestOutcome::Edge(_)), + "remote wake applied an edge at B: {out:?}" + ); rig_wait( "rest: B's row advertises Active again at A (with a presence stamp)", rig.wait, @@ -2538,7 +2493,10 @@ fn two_host_ladder_role_a() { MintedOp::new(Minter::Cli, op()), ) .expect("A-3 setup suspend"); - accept_rest_reply("A-3 setup suspend", &out); + assert!( + matches!(out, spt_daemon::resthost::RestRequestOutcome::Edge(_)), + "A-3 setup suspend applied an edge at B: {out:?}" + ); rig_wait("A-3: B advertises Suspended at A", rig.wait, || { registry.rows(&rig.subnet, ID_B).iter().any(|i| { i.node == rig.b_hex() && i.status == spt_net::net::registry::Status::Suspended @@ -2579,7 +2537,10 @@ fn two_host_ladder_role_a() { MintedOp::new(Minter::Cli, op()), ) .expect("A-3 routed wake"); - accept_rest_reply("A-3 routed wake", &out); + assert!( + matches!(out, spt_daemon::resthost::RestRequestOutcome::Edge(_)), + "A-3 routed wake applied an edge at B: {out:?}" + ); rig_wait("A-3: B advertises Active again at A", rig.wait, || { registry.rows(&rig.subnet, ID_B).iter().any(|i| { i.node == rig.b_hex() && i.status == spt_net::net::registry::Status::Active diff --git a/crates/spt-daemon/tests/twohost_web.rs b/crates/spt-daemon/tests/twohost_web.rs index 9f6c19a2..0eb2371c 100644 --- a/crates/spt-daemon/tests/twohost_web.rs +++ b/crates/spt-daemon/tests/twohost_web.rs @@ -45,7 +45,7 @@ //! daemons); the two-box climb is the gate's field leg. use std::io::{Read, Write as _}; -use std::net::{IpAddr, SocketAddr, TcpStream}; +use std::net::{IpAddr, SocketAddr, TcpStream, UdpSocket}; use std::path::Path; use std::sync::atomic::{AtomicBool, AtomicU64}; use std::sync::Arc; @@ -120,6 +120,16 @@ const RANGE_OFFSET: u16 = 1; const DENY_OFFSET: u16 = 2; /// The requester-cell offset the helper cell uses — B dials this one A-ward. const HELPER_OFFSET: u16 = 3; +/// The INBOUND PROBE's offset. Clear of the four requester cells (0..=3) and +/// inside the same `port_a + offset` range the rig already uses, so whatever +/// rule lets the ceremony's ports through covers this one too — a probe on a +/// port outside that range would vouch for a path nothing else takes. +const PROBE_OFFSET: u16 = 9; +/// How long A listens and B sends. Ten seconds against the ceremony's 900: the +/// point of the probe is that a blocked link is answered in the time it takes +/// to notice, not in the time it takes to give up. +const PROBE_WINDOW: Duration = Duration::from_secs(10); +const PROBE_MAGIC: &[u8] = b"SPT-TWOHOST-INBOUND-PROBE-v1"; const HELPER_BYTES: &[u8] = b"# the file the user pointed at\n"; const OPEN_ENDPOINT: &str = "w1-open"; @@ -303,6 +313,149 @@ fn helper_envelope(quoted: &str) -> String { /// production loop; the gate it runs is the production gate. // [int->REQ-WEB-CROSS-NODE-PROXY] // [int->REQ-WEB-ACCESS-SURFACE] +// ══ THE INBOUND PROBE ═══════════════════════════════════════════════════════ +// A golden run once spent 900 s on each half discovering that B's datagrams +// never reached A: every QUIC dial died on the 10 s bound — 75 of them, cadenced +// at exactly 12.00 s, never one ADMIT. +// +// TWO LAYERS were behind it and EITHER ALONE produces the same silence, which is +// why the first fix looked wrong when it changed nothing: +// 1. the host firewall — BlockInbound on every profile, and its allow rules are +// PER-EXECUTABLE, so no rule covered the test binary and the runner (a +// service) never shows an Allow dialog; +// 2. the TAILNET ACL, asymmetric — a member device may open flows TO a TAGGED +// resource and the reverse is denied, so the receiver permits inbound from a +// list the sender is simply absent from. +// Every face that ever appeared to work rode RETURN traffic of flows the receiver +// had opened; a COLD claim in this direction can never cross. The ceremony cannot +// tell that apart from a pairing fault: at B a datagram dropped by a firewall +// looks exactly like one that arrived and was ignored, so the failure was +// triaged as product, then rig, before it was measured as box. +// +// These two cells answer that question FIRST, in ten seconds, in their own +// words. They are run by their own `cargo test` invocation before the ceremony +// step (see golden.yml) — inside one invocation the cells are threads and +// nothing orders them. +// +// WHY THIS LIVES IN THIS BINARY rather than in a shell script beside it, which +// would have been the smaller change: the Windows rule is PER-EXECUTABLE. An +// allow rule naming pwsh, python, or a bench script says nothing about +// `twohost_web-.exe`, so a probe written in any of them would today agree +// with reality by coincidence and would go GREEN the moment anyone allowed the +// prober — certifying the exact failure it exists to catch. Only a probe that +// IS the test binary carries the test binary's firewall identity. (It is also +// why the operator rule must be PORT+REMOTE scoped: a program-scoped rule is +// orphaned by the next rebuild's hash, exactly as the dead `_work\spt-core` +// rules were. This probe stays correct under either.) +// +// THE ASSERTION LIVES ON A, deliberately. B cannot know whether its datagram +// arrived, so a B-side "I sent it" proves nothing about the path. +#[test] +fn two_host_inbound_probe_role_a() { + let Some(rig) = Rig::from_env("a") else { + return; + }; + let port = rig.port_a + PROBE_OFFSET; + let socket = UdpSocket::bind(SocketAddr::new(IpAddr::from([0, 0, 0, 0]), port)) + .unwrap_or_else(|e| panic!("INBOUND_PROBE: role A could not bind udp {port}: {e}")); + socket + .set_read_timeout(Some(Duration::from_millis(250))) + .expect("probe socket takes a read timeout"); + println!("TWOHOST-WEB probe role A: listening on udp {port} for {PROBE_WINDOW:?}"); + + let deadline = Instant::now() + PROBE_WINDOW; + let mut buf = [0u8; 64]; + let mut seen = 0usize; + while Instant::now() < deadline { + match socket.recv_from(&mut buf) { + Ok((n, from)) if buf[..n] == *PROBE_MAGIC => { + seen += 1; + println!("TWOHOST-WEB probe role A: datagram {seen} from {from}"); + break; + } + // A stray datagram is not this probe's business; keep waiting rather + // than passing on someone else's traffic. + Ok((n, from)) => println!("TWOHOST-WEB probe role A: {n} foreign bytes from {from}"), + Err(_) => {} + } + } + + // ITS OWN OUTCOME, never the ceremony's error: this reds as INBOUND_BLOCKED + // and names the peer, the port, the window and the fix, so the next box + // regression is answered by the failure text instead of a 900 s investigation. + assert!( + seen > 0, + "INBOUND_BLOCKED: role A received NO probe datagram on udp {port} from {} within {PROBE_WINDOW:?}. +This is a BOX rule, not a product or rig fault: the ceremony below would spend the full +SPT_TWO_HOST_WAIT_SECS budget dialling a host that cannot receive, and would report it as a pairing +failure. + +THERE ARE TWO LAYERS AND EITHER ONE ALONE PRODUCES THIS EXACT SILENCE. Fixing only the first is the +mistake this text exists to prevent: it is NECESSARY, NOT SUFFICIENT, and the run reds identically +afterwards. + + LAYER 1 - the Windows host firewall on the receiver. BlockInbound on every profile, and its allow + rules are PER-EXECUTABLE, so a rule naming pwsh or python says nothing about this test binary and + the runner (a service) never shows an Allow dialog. Fix, operator, elevated - allow INBOUND UDP + {}-{} from {} only: PORT+REMOTE scoped, never program-scoped, or the next rebuild binary hash + orphans the rule the way the dead _work/spt-core rules were orphaned: + New-NetFirewallRule -DisplayName 'spt twohost rig inbound' -Direction Inbound -Protocol UDP -LocalPort {}-{} -RemoteAddress {} -Action Allow + + LAYER 2 - the TAILNET ACL, which is asymmetric and denies this direction. The sender is a TAGGED + resource owned by another tailnet user; a member device may open flows TO it, and the reverse is + denied, so the receiver permits inbound from a list this sender is simply absent from. Every + helper-stall face that ever appeared to work rode RETURN traffic of flows the receiver opened; a + COLD claim in this direction can never cross. Fix, operator - grant it in the tailnet policy + (asked on releases#272: src tag:eye-tracking-resource -> dst this host, udp 7460-7499). Verify with + `tailscale debug netmap` on the RECEIVER: the sender IP must appear among the PacketFilter + permitted inbound sources. + +THE DISCRIMINATOR, ten seconds, run it before assuming which layer bit: bind the SAME listener and +send to it once over the LAN and once over Tailscale. LAN receives and Tailscale does not = LAYER 2, +the host firewall is not your problem. Neither receives = LAYER 1 (or both). Solicited return traffic +works under either fault, so an echo reply proves nothing about this direction. + +Port range above is exactly what THIS rig binds (port_a..probe); releases#272 carries the wider +7460-7499, which covers it. Apply the board's range if they differ - this message and that comment +must never become two different commands.", + rig.peer_ip, + rig.port_a, + rig.port_a + PROBE_OFFSET, + rig.peer_ip, + rig.port_a, + rig.port_a + PROBE_OFFSET, + rig.peer_ip, + ); + println!("TWOHOST-WEB probe role A: INBOUND OK on udp {port}"); +} + +#[test] +fn two_host_inbound_probe_role_b() { + let Some(rig) = Rig::from_env("b") else { + return; + }; + let target = SocketAddr::new(rig.peer_ip, rig.port_a + PROBE_OFFSET); + let socket = UdpSocket::bind(SocketAddr::new(IpAddr::from([0, 0, 0, 0]), 0)) + .expect("probe sender binds an ephemeral port"); + println!("TWOHOST-WEB probe role B: sending to {target} for {PROBE_WINDOW:?}"); + + // Send for the WHOLE window rather than once: A's job may still be starting, + // and the question is whether ANY datagram crosses, not whether the first one + // happened to. This cell CANNOT FAIL on delivery — only A can see that — so a + // send error is reported and the window continues. + let deadline = Instant::now() + PROBE_WINDOW; + let mut sent = 0usize; + while Instant::now() < deadline { + match socket.send_to(PROBE_MAGIC, target) { + Ok(_) => sent += 1, + Err(e) => println!("TWOHOST-WEB probe role B: send error (continuing): {e}"), + } + thread::sleep(Duration::from_millis(500)); + } + println!("TWOHOST-WEB probe role B: {sent} datagrams sent to {target}"); + assert!(sent > 0, "role B could not send a single probe datagram to {target}"); +} + #[test] fn two_host_web_role_b() { let Some(rig) = Rig::from_env("b") else { diff --git a/crates/spt-test-support/src/lib.rs b/crates/spt-test-support/src/lib.rs index f6e940cb..9250c41b 100644 --- a/crates/spt-test-support/src/lib.rs +++ b/crates/spt-test-support/src/lib.rs @@ -88,58 +88,6 @@ fn restore_var(name: &str, value: Option) { } } -/// Poll `probe` until it converges, or fail with WHAT WAS ACTUALLY SEEN. -/// -/// Four test files in `spt-daemon` each carried a private copy of this loop -/// with the SAME hardcoded budget — `for _ in 0..600 { sleep 25ms }`, 15.0 s — -/// across 31 call sites. That number was never derived from anything: the cell -/// it was tuned for ran 9.8 s and 7.2 s, so 15 s was about 1.5x a FAST -/// observation on a box measured at >= 1.5x slower for 19 of 73 cells in one -/// attempt. It sat inside the machine's own variance, and a golden attempt lost -/// it by 0.7 s. Four private copies is also why fixing the cell that red would -/// have left 25 other carriers of the same defect. -/// -/// So the loop lives here ONCE and the BUDGET IS A PARAMETER, derived at the -/// call site from whatever that site races — a dial bound, a pump cadence — -/// named as a const beside the rig knob that sets it, so the budget moves when -/// the mechanism moves instead of being re-guessed. -/// -/// `probe` returns `Ok(())` when converged and `Err(state)` when not yet, where -/// `state` DESCRIBES what it saw. The last `Err` is printed on failure, because -/// the old panic said only `never converged: ` and left the reader to -/// reconstruct the run from the daemon's stderr. A guard that reds must say -/// what it saw. -/// -/// Keep budgets under the suite's 60 s SLOW line (`.config/nextest.toml`): -/// generous is fine, but a convergence that legitimately needs longer than that -/// is a different bug than a slow box. -pub fn converge_within( - budget: std::time::Duration, - what: &str, - mut probe: impl FnMut() -> Result<(), String>, -) { - const SAMPLE: std::time::Duration = std::time::Duration::from_millis(25); - let started = std::time::Instant::now(); - let mut samples = 0usize; - let mut last = String::from(""); - while started.elapsed() < budget { - samples += 1; - match probe() { - Ok(()) => return, - Err(state) => last = state, - } - std::thread::sleep(SAMPLE); - } - panic!( - "never converged: {what} - budget {budget:?} (derived at the call site; suite SLOW line is 60s) - elapsed {:?} over {samples} samples at {SAMPLE:?} - last saw {last}", - started.elapsed() - ); -} - - #[cfg(test)] mod tests { use super::*; diff --git a/crates/spt/tests/common/mod.rs b/crates/spt/tests/common/mod.rs index 312d1ce5..a5da2f8f 100644 --- a/crates/spt/tests/common/mod.rs +++ b/crates/spt/tests/common/mod.rs @@ -76,99 +76,6 @@ pub fn daemon_stderr_panel(pre_redirect: &Path) -> String { ) } -/// What [`preserve_stderr_evidence`] saved, and the part of it a CI log can show. -pub struct PreservedEvidence { - /// Where the whole sinks were copied. Named in the panic so a local run can - /// open them; on CI the directory usually dies with the runner, which is why - /// `tail` exists beside it. - pub dir: std::path::PathBuf, - /// A BOUNDED tail of the daemon/brain stderr sink, for pasting straight into - /// a panic message. Bounded because a panic that dumps an unbounded log is a - /// panic nobody reads. - pub tail: String, - /// What was copied, or why it was not. Never silent: a preservation step - /// that fails quietly is worse than none, because the failure text will - /// promise evidence that is not there. - pub note: String, -} - -/// Copy the daemon/brain stderr sinks OUT of a run's temp home before the -/// assertions can abort the test. -/// -/// THE MECHANISM THIS EXISTS FOR: the run's `SPT_HOME` is a `TempDir`. A failing -/// assertion unwinds, the `TempDir` drops, and the drop deletes the directory — -/// so the failure that most needs the brain's own stderr is precisely the one -/// that destroys it. Copying is not tidiness; it is the difference between a -/// leak red you can diagnose and one you can only re-run. Do not "simplify" this -/// away: the panic message that cites `dir` is written on the assumption the -/// copy already happened. -/// -/// Destination: `/test-artifacts///`, overridable with -/// `SPT_TEST_ARTIFACTS`. Under `target/` on purpose — it is the tree `cargo -/// clean` already owns, it is per-checkout, and it is neither a scratchpad (not -/// preservation) nor `.spt/preserved` (that is for gated evidence a human filed, -/// not for every local run). -pub fn preserve_stderr_evidence(home: &Path, rig: &str) -> PreservedEvidence { - const TAIL_LINES: usize = 200; - - let stamp = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_millis().to_string()) - .unwrap_or_else(|_| "unknown-epoch".to_string()); - - // `target/` from THIS test binary rather than from a guessed path: the exe - // is `//deps/`, so two hops up is the profile dir and - // three is the target root, whatever `CARGO_TARGET_DIR` was spelled as. - let root = std::env::var_os("SPT_TEST_ARTIFACTS") - .map(std::path::PathBuf::from) - .or_else(|| { - std::env::current_exe().ok().and_then(|exe| { - exe.ancestors() - .nth(3) - .map(|target| target.join("test-artifacts")) - }) - }) - .unwrap_or_else(std::env::temp_dir); - let dir = root.join(rig).join(&stamp); - - let sink = spt_daemon::stderrlog::sink_path(home); - let pre_redirect = home.join("daemon.stderr.log"); - - let mut notes = Vec::new(); - if let Err(error) = std::fs::create_dir_all(&dir) { - notes.push(format!("create_dir_all({}) FAILED: {error}", dir.display())); - } - for src in [&sink, &pre_redirect] { - let Some(name) = src.file_name() else { continue }; - match std::fs::copy(src, dir.join(name)) { - Ok(bytes) => notes.push(format!("{} -> {bytes} bytes", src.display())), - // An ABSENT sink is a finding, not a non-event: it means the daemon - // never wrote one, which is itself the answer to some leak reds. - Err(error) => notes.push(format!("{} NOT copied: {error}", src.display())), - } - } - - let tail = match std::fs::read_to_string(&sink) { - Ok(text) => { - let lines: Vec<&str> = text.lines().collect(); - let skipped = lines.len().saturating_sub(TAIL_LINES); - let body = lines[skipped..].join("\n"); - if skipped == 0 { - body - } else { - format!("… {skipped} earlier line(s) elided …\n{body}") - } - } - Err(error) => format!("daemon/brain stderr sink UNREADABLE at {}: {error}", sink.display()), - }; - - PreservedEvidence { - dir, - tail, - note: notes.join("; "), - } -} - fn terminate_captured_tree(child: &mut Child) { #[cfg(windows)] { diff --git a/crates/spt/tests/common/reap.rs b/crates/spt/tests/common/reap.rs index 7a6d83af..7e84d63e 100644 --- a/crates/spt/tests/common/reap.rs +++ b/crates/spt/tests/common/reap.rs @@ -250,56 +250,6 @@ fn under_root(exe: &Path, root: &Path) -> bool { } } -/// One process that outlived the test, stamped with what a reader needs to -/// IDENTIFY it without a second hunt. -/// -/// A bare `(pid, exe)` names a number and an image, and both are the parts that -/// go stale fastest: by the time the panic is read the pid may belong to someone -/// else, and the image is the same `spt` binary every other process in this -/// suite runs. `started_at` pins WHICH process held the number, and `parent` -/// says who was still holding it open — a leaked service whose parent is our own -/// daemon is a teardown-order defect, and one whose parent is gone (or is the -/// OS) is an orphan, which is a different bug with a different fix. Naming the -/// leaked child of a previous run cost a separate investigation; a survivor that -/// carries its own ancestry answers that in the failure text. -#[derive(Clone, PartialEq, Eq)] -pub struct Survivor { - pub pid: u32, - pub exe: PathBuf, - /// Creation time, `None` when the identity oracle could not answer — which - /// is itself worth printing: an unpinned survivor cannot be re-identified. - pub started_at: Option, - /// The parent AS THE TABLE READS IT NOW. `None` means the table had no entry - /// for this pid by the time we stamped it (it exited between selection and - /// stamping), not that it is parentless. - pub parent: Option, - /// The parent's image, when readable. An unreadable parent path is normal - /// for OS-owned parents and is NOT evidence of anything. - pub parent_exe: Option, - /// Which selector found it: `"staged-root"` (image under this run's own - /// staged root — needs no pid) or `"ancestry"` (descends from a pid observed - /// while it was provably ours). Says WHY we claim it, so a disputed survivor - /// can be argued with. - pub via: &'static str, -} - -/// One line per survivor, because these are read in a CI log where a derived -/// multi-line struct dump buries the pid that matters. -impl std::fmt::Debug for Survivor { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "pid={} via={} exe={}", self.pid, self.via, self.exe.display())?; - match self.started_at { - Some(t) => write!(f, " started_at={t}")?, - None => write!(f, " started_at=UNPINNED")?, - } - match (self.parent, &self.parent_exe) { - (Some(ppid), Some(pexe)) => write!(f, " parent={ppid} ({})", pexe.display()), - (Some(ppid), None) => write!(f, " parent={ppid} (image unreadable)"), - (None, _) => write!(f, " parent=UNKNOWN"), - } - } -} - /// What a POPULATION sweep found — and, separately, what it could not SEE. /// /// The distinction is the whole point. Both selectors below match on the @@ -311,7 +261,7 @@ impl std::fmt::Debug for Survivor { #[derive(Clone, Debug, Default)] pub struct Population { /// Still running, and its image authenticates it as ours. - pub survivors: Vec, + pub survivors: Vec<(u32, PathBuf)>, /// In ancestry scope, still running, image UNREADABLE. No knowledge — never /// scored as "not ours". pub unreadable: Vec, @@ -378,27 +328,6 @@ pub fn population_sweep( let mut counted: std::collections::HashSet = std::collections::HashSet::new(); let me = std::process::id(); - // ONE snapshot of pid -> ppid for the whole sweep. Read once, deliberately: - // stamping each survivor from its own fresh read would let the table move - // between survivors and produce a parent set that never existed at any - // single instant. - let parents: std::collections::HashMap = - spt_store::proc::process_table().into_iter().collect(); - let stamp = |pid: u32, exe: PathBuf, via: &'static str| -> Survivor { - let parent = parents.get(&pid).copied(); - Survivor { - pid, - exe, - started_at: match spt_store::proc::process_identity(pid) { - ProcIdentity::Present(t) => Some(t), - _ => None, - }, - parent, - parent_exe: parent.and_then(spt_store::proc::exe_path), - via, - } - }; - // ── Half 1: image path under the run's own staged root. ── let root = std::fs::canonicalize(staged_root).unwrap_or_else(|_| staged_root.to_path_buf()); for (pid, _) in spt_store::proc::process_table() { @@ -412,7 +341,7 @@ pub fn population_sweep( continue; }; if under_root(&exe, &root) && counted.insert(pid) { - found.survivors.push(stamp(pid, exe, "staged-root")); + found.survivors.push((pid, exe)); } } @@ -436,7 +365,7 @@ pub fn population_sweep( match spt_store::proc::exe_path(pid) { Some(exe) if same_image(&exe, seed_image) => { if counted.insert(pid) { - found.survivors.push(stamp(pid, exe, "ancestry")); + found.survivors.push((pid, exe)); } } // Descends from something that WAS ours and we cannot read what diff --git a/crates/spt/tests/resident_service_e2e.rs b/crates/spt/tests/resident_service_e2e.rs index 76114fd5..415c505a 100644 --- a/crates/spt/tests/resident_service_e2e.rs +++ b/crates/spt/tests/resident_service_e2e.rs @@ -385,47 +385,14 @@ fn a_declared_service_rises_with_the_daemon_and_reaches_the_cli() { ); // ── (8) Reap the whole tree SCOPED, before any assertion can abort us. ── - let dir_teardown = spt(&["shell", "teardown", "DirScout", "--owner", "doyle"]); - // OBSERVE the graceful stop instead of discarding it. `daemon stop --force` - // is the step that is SUPPOSED to take the daemon and everything it - // supervises down; the leak assertion below fires when it did not. Throwing - // its result away (`let _ =`) means the one command whose failure explains - // the leak leaves no trace — and a non-zero exit here is not hypothetical: - // it is what a daemon that never came up, or one already reaped by an - // earlier arm, returns. Printed unconditionally, because a stop that - // SUCCEEDED and leaked anyway is the more interesting finding of the two. - let daemon_stop = spt(&["daemon", "stop", "--force"]); - eprintln!( - "=== resident-service teardown: dir_teardown={:?} daemon_stop={:?} ===\n\ - --- dir teardown stderr ---\n{}\n\ - --- daemon stop stdout ---\n{}\n\ - --- daemon stop stderr ---\n{}", - dir_teardown.status.code(), - daemon_stop.status.code(), - String::from_utf8_lossy(&dir_teardown.stderr), - String::from_utf8_lossy(&daemon_stop.stdout), - String::from_utf8_lossy(&daemon_stop.stderr), - ); + let _ = spt(&["shell", "teardown", "DirScout", "--owner", "doyle"]); + let _ = spt(&["daemon", "stop", "--force"]); // Every breadcrumb pid goes through the AUTHENTICATED reaper, each with the // image IT runs. A breadcrumb is a NUMBER, not an identity: the writer exits, // the OS re-mints the number (2.2 s minimum on this box), and a bare // `taskkill /F /T` on it tree-kills whoever inherited it — a concurrent test, // dying with a bare exit 1 and nothing to blame. let mut verdicts: Vec<(&'static str, Verdict)> = Vec::new(); - - // THE SUPERVISOR HOST GOES FIRST. The ServiceSet lives in the DAEMON, not in - // the brain (`daemon.rs` spawns the service host there deliberately, so a - // routine brain restart does not bounce every resident service through the - // orphan path), and `servicehost.rs`'s Relaunch arm re-mints a service pid - // when one dies un-asked. Killing a supervised service while its supervisor - // is still up therefore does not reduce the population — it ROTATES it: the - // service comes back as a NEW pid that no per-pid check in this test holds, - // which is exactly the leak shape this rig keeps catching. Order is the fix; - // the broker's `Child` handle is authenticated by construction and stays the - // mechanism, unchanged. - let _ = broker.kill(); - let _ = broker.wait(); - for (label, pid, expected, observed) in [ ("resident/boot", boot_pid, &boot_exe, boot_obs), ("resident/rel", rel_service_pid, &rel_exe, rel_obs), @@ -440,9 +407,10 @@ fn a_declared_service_rises_with_the_daemon_and_reaches_the_cli() { reap::authenticated_kill("resident/brain", brain, &spt_bin, brain_obs), )); } - // (The broker was killed ABOVE, before the services it supervises — see the - // ordering note there. It is held by its `Child` handle, authenticated by - // construction, never a breadcrumb.) + // The broker is held by its `Child` handle — authenticated by construction, + // never a breadcrumb. Unchanged, and deliberately so. + let _ = broker.kill(); + let _ = broker.wait(); // ── (8b) What SURVIVED, asked of the OS rather than of our own bookkeeping. ── // @@ -473,18 +441,6 @@ fn a_declared_service_rises_with_the_daemon_and_reaches_the_cli() { let population = sweep(); eprintln!("=== resident-service teardown: verdicts={verdicts:?} population={population:?} ==="); - // PRESERVE THE SINKS BEFORE ANY ASSERTION CAN ABORT US. `home` is a - // `TempDir`: a failing assertion unwinds, the drop deletes the directory, - // and the brain/daemon stderr that explains a leak goes with it. This copy - // is why the leak assertion below can cite a path AND paste a tail; remove - // it and the failure text starts promising evidence that no longer exists. - let evidence = common::preserve_stderr_evidence(home.path(), "resident_service_e2e"); - eprintln!( - "=== resident-service evidence: dir={} ===\n{}", - evidence.dir.display(), - evidence.note - ); - std::env::remove_var("SPT_HOME"); // ── ASSERTIONS ── @@ -718,16 +674,8 @@ fn a_declared_service_rises_with_the_daemon_and_reaches_the_cli() { after {settle_waited:?} of a {SWEEP_SETTLE_BUDGET:?} budget — went_clean=false \ at the full budget is a leak that never drained; a wait far SHORTER than the \ budget with survivors still listed means they appeared after the sweep read \ - clean, which is a different defect from termination still being in flight\n\ - \n Each survivor above carries started_at and its CURRENT parent: a survivor \ - whose parent is this run's own daemon is a teardown-ORDER defect, one whose \ - parent is gone is an orphan, and `via=` says which selector claimed it. That \ - is the identification, in the failure text, rather than a hunt afterwards.\n\ - \n evidence copied to {} ({})\n --- daemon/brain stderr sink, last lines ---\n{}", - population.survivors, - evidence.dir.display(), - evidence.note, - evidence.tail + clean, which is a different defect from termination still being in flight", + population.survivors ); assert!( population.unreadable.is_empty(), diff --git a/crates/spt/tests/webserve_attachment_e2e.rs b/crates/spt/tests/webserve_attachment_e2e.rs index c1516a82..676f1936 100644 --- a/crates/spt/tests/webserve_attachment_e2e.rs +++ b/crates/spt/tests/webserve_attachment_e2e.rs @@ -532,6 +532,8 @@ fn an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message() { std::fs::create_dir_all(work.join("two")).unwrap(); std::fs::write(&first_of_two, b"alpha bytes").unwrap(); std::fs::write(&second_of_two, b"beta bytes").unwrap(); + let before_two = run(home, &["serve", "list", "--json"], None); + let before_count = stdout(&before_two).matches("\"served_name\"").count(); let two = run( home, &[ @@ -546,31 +548,11 @@ fn an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message() { assert_eq!(urls.len(), 2, "one ATTACHED line per attachment: {:?}", urls); assert_ne!(urls[0], urls[1], "two files are two links, never one reused"); let after_two = run(home, &["serve", "list", "--json"], None); - // IDENTITY, not a count. A before/after total over `serve list` cannot carry - // this claim, and the count it replaces was not merely weak but WRONG: ARM 10 - // deliberately leaves an EXPIRED-but-unreaped entry behind (`--ttl 1s`, then a - // 1500 ms sleep), `ServeRequest::List` answers `registry.entries()` UNFILTERED - // so that corpse is still counted, and the attachment TTL reaper rides the - // daemon's 5 s pulse. A sweep landing between the two lists retires the corpse - // and the delta reads +1 for two registrations that BOTH happened. Measured at - // c4919243 on Windows: 4 reds in 15 runs, every one exactly -1 - and the - // FASTEST run of the batch was among them, so this is the pulse's PHASE, not a - // slow box losing a margin. ARM 12 below already refuses to depend on ARM 10's - // entry for exactly this reason; this arm no longer does either. - // - // Naming the two entries is also the claim the arm actually makes, so it is - // immune to the retirement of any entry it did not create. - let listed: serde_json::Value = - serde_json::from_str(&stdout(&after_two)).expect("the serve list json parses"); - let rows = listed.as_array().expect("serve list --json is an array of entries"); - for url in &urls { - let name = url.rsplit('/').next().expect("an ATTACHED url ends in its served name"); - assert!( - rows.iter().any(|row| row["served_name"] == *name), - "each attachment registered its OWN entry - {name} is missing from {}", - stdout(&after_two) - ); - } + assert_eq!( + stdout(&after_two).matches("\"served_name\"").count(), + before_count + 2, + "each attachment registered its OWN entry" + ); let alpha_back = work.join("alpha-back.md"); let beta_back = work.join("beta-back.md"); assert!(run(home, &["fetch", &urls[0], alpha_back.to_str().unwrap()], None).status.success()); @@ -614,84 +596,29 @@ fn an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message() { &[ "send", TARGET, "--from", SENDER, "--attachment", doomed_source.to_str().unwrap(), - // A LIFETIME THAT CANNOT EXPIRE DURING THIS TEST. The arm used to - // mint a short-lived entry and race it: 1s, then widened to 5s with - // the note that "five seconds cannot expire before the capture - // below". On a loaded golden runner five seconds DID expire before - // the capture — the entry was already reaped when this arm read the - // registry, and it panicked as "not in the registry", which reads as - // a product defect rather than as the rig's own clock. - // - // Widening the window again would be the SAME FIX THAT ALREADY - // FAILED ONCE, just with a bigger number: any wall-clock margin is a - // bet against the slowest machine that will ever run this. So the - // lifetime is now long enough to be irrelevant, and the EXPIRY is - // driven deliberately below, through the registry writer this arm - // already uses to plant its guard entry. No wall clock decides - // anything here any more. - "--ttl", "3600s", + // FIVE seconds, not one. A 1s lifetime can expire while the send's + // own child process is still running, so a reaper tick can retire + // the entry BEFORE this arm ever reads its snapshot path — measured, + // one red then a pass on re-run, which is the signature of a race + // rather than a wrong name. Five seconds cannot expire before the + // capture below and still expires long inside the wait. + "--ttl", "5s", ], Some("this one is for the reaper"), ); - // Elapsed since the send RETURNED, so any future red carries a number instead - // of a guess. It bounds the CAPTURE only — the registration happens inside - // the daemon before the send's delivery leg finishes, so this is not the - // register-to-read gap and must not be quoted as one. Bound at the point of - // use rather than pre-declared: the instant must still be taken AFTER `run` - // returns, and a late-initialised binding buys nothing for that. - let since_send = std::time::Instant::now(); assert!(doomed_send.status.success(), "ttl send: {}", stderr(&doomed_send)); let doomed_snapshot = snapshot_of("reap-me.md").unwrap_or_else(|| { let listed = run(home, &["serve", "list", "--json"], None); panic!( - "the attachment this arm just registered is not in the registry {} ms after the send RETURNED (this bounds the capture, not the register-to-read gap: the daemon's registration precedes the send's own delivery leg). + "the attachment this arm just registered is not in the registry. send stderr: {} registry: {}", - since_send.elapsed().as_millis(), stderr(&doomed_send), stdout(&listed) ) }); assert!(doomed_snapshot.exists(), "its snapshot bytes are on disk before the reap"); - // NOW retire it, deliberately, instead of waiting for a clock. `scope_entry` - // addresses an entry by its served NAME and rewrites its ttl; expiry is - // `registered_at_ms + ttl <= now`, so 1 ms against a registration already in - // the past is expired by construction, on the reaper's next tick. - // - // `scope_entry` also sets `audience` unconditionally from its argument, so - // passing None CLEARS any audience. reap-me.md has none, which makes that a - // no-op HERE — but do not lift this loop onto an audienced entry without - // passing its audience back in, or the scope goes with the ttl. - // - // The loop re-reads because the daemon is the registry's other writer. Be - // precise about what that buys: it detects the entry being ABSENT, not a - // clobber — a daemon save landing between this load and this save would be - // overwritten by this snapshot and nothing here would notice. That hazard is - // idle in this arm by arithmetic rather than by vigilance: every other live - // entry carries 30 d or 3600 s, so the only concurrent writer has nothing to - // retire and nothing to write, and this snapshot equals its state. - let mut retired = false; - for attempt in 0..5 { - let mut registry = spt_store::serving::ServingRegistry::load_at(®istry_path).unwrap(); - if registry - .scope_entry("reap-me.md", Some(1), None, None) - .is_some() - { - registry.save_at(®istry_path).unwrap(); - retired = true; - break; - } - // Gone already means the daemon retired it between our read and now, - // which is the outcome this loop is trying to cause — not a failure. - if snapshot_of("reap-me.md").is_none() { - retired = true; - break; - } - assert!(attempt < 4, "could not backdate the doomed entry's ttl"); - } - assert!(retired, "the doomed entry was given an already-expired lifetime"); - // A reference entry, already expired, planted for the guard. Re-read after // planting: the daemon is the registry's other writer and its own tick could // land between this load and this save. diff --git a/docs-site/src/changelog.md b/docs-site/src/changelog.md index c52bb265..21074f52 100644 --- a/docs-site/src/changelog.md +++ b/docs-site/src/changelog.md @@ -12,17 +12,40 @@ breaks something, or changes the observable behavior of existing surfaces broadly; **patch** for fixes, and for additive opt-in capability — a new key, flag, or page that no existing user can encounter without opting into it. -## [0.68.0] - 2026-09-08 - -Web serving. Files, directories, adapter documentation and the changelog are -reachable at node-prefixed addresses on the local server, and an address that -names another machine in the subnet is answered by the machine that owns it. -Messages can carry attachments the receiver pulls on demand, every message -gets a short ID to show or reply to, and a machine with no spt on it can be -handed the binary over the local network. The `XFER` access surface is retired. +## [Unreleased] ### Added +- `spt serve lan --bootstrap` hands the spt binary to a machine that is not + yet a node. It is a SECOND listener on port 5470, bound on all interfaces, + off by default and off again after every daemon restart; `--stop` takes it + down and a bare `spt serve lan` reports its state. It serves only + `/bin//spt[.exe]`, that file's `.release.json` sidecar, and an + `/install` page - everything else answers not-found naming the listener, and + the docs server on 5474 stays loopback-only throughout. While it is up, + anyone who can reach the socket may pull the binary, so the now-signal + carries a standing LAN-EXPOSED line for the whole window. +- The bootstrap listener serves the artifacts of the applied signed release + set or it refuses to start, naming why + (`LAN_BOOTSTRAP_REFUSED:unsigned-exe`, `:sha-mismatch`, `:set-not-applied`). + Each platform is re-checked as it is served, so a tampered or missing + artifact answers `LAN_BOOTSTRAP_TRIPLE_UNAVAILABLE` for that platform alone + while the others keep serving. Start prints one `sha256 ` line + per platform, and `spt install` gains `--expect-sha256` (refuses anything but + those exact bytes, printing what it downloaded first) and `--release-json` + (re-verifies the signature and digest against its own built-in release keys) + so the two machines' hashes can be compared by eye. +- The changelog is now a page of the local docs (`spt docs url`, then + **Changelog** under Reference), so what a release changed can be read on + the machine being updated without a repository checkout. The page is + generated from `CHANGELOG.md` and checked against it in CI. +- An adapter can publish its own documentation: the new optional + `[adapter].docs_dir` manifest key names a directory inside the adapter, + and it is served at `//a//docs/`. Without the key that URL + answers not-found naming the facet; it never serves anything else in its + place. The key is adapter-relative — an absolute path or a path that + leaves the adapter's own directory is refused by name, both when the + manifest is read and again on every request. - `spt serve add`, `spt serve rm`, and `spt serve list --json` manage live file and directory references. Same-name registrations receive stable numbered suffixes; removing an entry never deletes its source. Only the same absolute @@ -42,34 +65,6 @@ handed the binary over the local network. The `XFER` access surface is retired. cached; `HEAD` and `Range` requests are honored by the owner. An owner that refuses answers 403 naming `WEB`; an owner that cannot be reached answers 502 naming the node, within a bounded time. -- `spt send --attachment ` sends a file with a message. The file's bytes - are captured as they are at send time, so later edits or a deletion do not - change what arrives, and the receiver pulls them with `spt fetch` when it - wants them. `--ttl` sets how long an attachment stays available (default 30 - days, a unit is required); expired attachments answer 404 immediately. -- Every message now carries a short ID. `spt msg show ` prints a message by - it, and `spt send --reply-to ` marks what a message answers. Messages - from another machine resolve through the machine that holds them. -- `spt serve lan --bootstrap` hands the spt binary to a machine that is not yet - a node, over the local network on port 5470. It is off by default and off - again after every daemon restart, serves only the binary, its release sidecar - and an install command, and refuses to start by name when the applied update - set is not signed or does not match. It prints a checksum per platform, and - `spt install` gains `--expect-sha256` and `--release-json` so what was - downloaded can be compared before it is run. The documentation port is - untouched and stays on loopback. -- `[adapter].docs_dir` publishes an adapter's own documentation at the `docs` - segment of its address, beside the adapter's output directory rather than in - place of it. A missing key, an unreadable manifest, a path that escapes the - adapter's directory, and a directory that has since been removed all answer - the same 404 naming what was asked for. -- The changelog is now one of the pages the local documentation server offers. -- A harness is told when a message it receives refers to a file the reader - cannot open — an attachment to pull, or a path that belongs to another - machine — instead of leaving the reader to discover it. For a path on - another machine, that machine can be asked to publish it and answers with a - link; it honors such a request only for its own endpoints, so no third - machine can have someone else's file exposed. ### Changed @@ -77,30 +72,12 @@ handed the binary over the local network. The `XFER` access surface is retired. and `?json` views. `/` redirects to `//`; canonical docs URLs live at `//docs/`, while existing bare docs paths remain compatibility aliases. -### Removed - -- The `XFER` access surface is retired: attachments replaced the transfer it - gated, so the entry leaves the access-control vocabulary along with the - transfer itself. An existing rule that names `XFER` is kept and reported at - load rather than dropped silently; it no longer governs anything. The shell - channel's own transfer progress is unaffected. - ### Fixed - Messages delivered by `spt api listen` now reach the receiver's incoming-message history and last-message state, for both queued backlog and live TCP delivery. A later hook poll does not record those deliveries again; filtered notifications are not recorded as delivered messages. -- A node's message and activity history reads back correctly once its log grows - past a quarter of a megabyte. Positions no longer restart from the beginning, - so a request for everything since a given point no longer comes back empty or - repeats entries that were already seen, and an oversized position is answered - with the current head instead of skipping a page. -- An agent's own saved context is no longer overwritten, unread, by the - automatic summary that follows it. The automatic summary is filed directly - and never writes to the file an agent saves its own context to. -- `spt api bind` no longer prints an engine-room probe line on a machine that - has no engine room. ## [0.67.1] - 2026-09-06 diff --git a/docs/FLAKE-LEDGER.md b/docs/FLAKE-LEDGER.md index ee98261c..3373e613 100644 --- a/docs/FLAKE-LEDGER.md +++ b/docs/FLAKE-LEDGER.md @@ -39,7 +39,7 @@ contemporaneous one is a row that will be cited as if it were. | `spt-daemon::servicehost_supervision_e2e bits_swap_under_the_hold_and_the_new_ones_come_up_on_release` | 1× hfenduleam Windows golden (2026-08-01 run 30720655534, sha 13e94d5) | **TEST DEFECT, not a load flake — the only entry here whose mechanism is the assert itself.** The new-process leg asserted identity by PID INEQUALITY (`assert_ne!(second, Some(first))`); Windows recycles pids aggressively enough under suite churn to hand the SAME pid back to the replacement, and the run failed with both sides `Some(47100)` while the log SELF-PROVED the swap (SERVICE_STARTED 47100 → SERVICE_QUIESCED cooperative exit under hold → SERVICE_STARTED 47100). The immediately-preceding asserts — `ServiceOutcome::Started` on release and the swapped-in binary's beacon — had already passed, so the functional swap was never in doubt: a pid was standing in for an identity it cannot carry past any exit (the KNOWN-HAZARDS `pid_started_at` class). File untouched by DOORBELL (last touch 1488e39, already on main) — pre-existing surface | **HARDENED (2026-08-01, releases#93):** the leg now compares the original's BIRTH STAMP (`proc::process_identity`, the `(pid, start-time)` pair) captured while it is alive against the same pid re-read after the ceremony — `Absent` when nothing took the pid, a DIFFERENT `Present` when something did, and only the original still running reads equal. Guarded by an INSTRUMENT assert that the stamp was readable at capture, because `Unproven != Unproven` is false and would have passed the leg by having nothing to compare. Mutation-proved both ways: an original that survives the ceremony REDS the identity assert, an unreadable stamp REDS the instrument assert. **No flake-registry.json entry** — its `retire_when` is "after the owning defect is fixed", the fix ships in the same commit, so an entry would be born retired; and registry entries require a same-SHA-rerun confirmation, which this never had (doyle's RCA proved it from the log, not from a rerun) | | `spt::psyche_real_bound_kill_soft_budget_e2e` | assert-FAIL 1× hfenduleam Windows golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :309 — expected `Some(Ok)`, got `Timeout` 1s×11 on summarizer.bat); LEAK 7/7 across recent golden Windows legs (7th on the att3 GREEN leg, 17.497s) | **two defects, not one signal** (hertz 2026-08-04): (1) assert arm — the positive control (:309) fires the FAST body through the SAME 1s kill bound; `wait_bounded` starts its deadline AFTER spawn (runtime.rs:941) so 1s covers the child's whole startup, and a trivial `@echo` bat can exceed it under load (mechanism, not yet a measurement — the control leg's wall time is the instrument); (2) leak arm is STRUCTURAL and distinct — `wait_bounded` kills the DIRECT child only (child.kill() :949 on cmd.exe); the sleeper is a GRANDCHILD (`cmd /C summarizer.bat` → ping), so ten kills leave ten ~5s survivors against nextest's 100ms leak timeout; no bound-shortening can fix it. USHER-chain delta-eliminated by dependency direction (spt-daemon below spt, deployah source-verified). Kin: releases#90 echo-commune spawn-killed-at-bound | DISPATCHED hertz 2026-08-04 (pkg item 1): control-leg bound via `BrainLifecycle::refresh_manifest` (lifecycle.rs:681) SAME host — budgets are host state (`Arc>` :672) so the standing latch and the `stamp()==None` clear row stay load-bearing; a second host would make that row vacuous. Leak fix lane-split doyle-ruled: TEST-ONLY sleeper fixture bin in spt-daemon = hertz now; PROD `wait_bounded` tree-kill-on-timeout = EVAL request to board, not dispatched | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_comes_back_when_knocked_awake` | 1× kitsubito Linux golden (2026-08-04 run 30940180764 att1, sha 17f95a7, :1198 — erhost not up in 30s; brain stderr EMPTY vs named `ENGINE_ROOM_SPAWN_FAIL` emitter) | test file brand-new on the USHER branch. The chain's own suspect commit d449ce5 (exit-in-thread) source-eliminated on two strong legs: TEMPORAL (rc blocked in the bring_up RPC the whole window, rig-killed after — teardown never reached) + ARTIFACT OWNERSHIP (the awaited artifact is the broker's in-memory `sessions.insert`, broker.rs:6138-6153 — daemon-side chain, no CLI participant). Offline arm differs from the passing control arm ONLY by prior warm bring-up + `endpoint stop` | observed 1×; did NOT repeat at att3 same-sha (heavy-leg population 177 matched, the test RAN and passed — non-vacuous green). Declared read: next occurrence gets the stop-survivor broker-state RCA (wake_inflight gate broker.rs:5782-5852, `WAKE_DEDUP_WAIT=2s`; sessions reap; ceremony/perch re-read :5600-5604), not a rerun | -| `spt::resident_service_e2e` teardown LEAK, the assert rendered :664 then **:670** on main (moved by 8d10b280, text byte-identical — the row is keyed by TEST NAME, never position; grep both) (second svcmock outlived the sweep) | 4 classified hfenduleam Windows golden occurrences: 2026-08-04 run 30940180764 att2 @17f95a7; 2026-08-30 run 33296634901 att1 @da71b785; 2026-09-06 run 34017906638 att3 @04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64; 2026-09-09 run 34310511612 att1 @f6110c2a12df0dd50b87dfb60a2ec4120b5cf98d. First: second `svcmock.exe` pid 14452 outlived teardown; functional half all-TRUE, 3 reap verdicts Killed | teardown/zombie class (IR-34 kin). DISTINCT from this test's :382 PRECONDITION signature (ir15 att1, daemon never came up in 124s — that one is IR-17's family and carries no defect evidence): 2-of-11 recent Windows victim legs by test, 1-of-11 by this assert | DISPATCHED hertz 2026-08-04 (pkg item 4). Did not recur at att3 (passed, no leak row). +1 UNCLASSIFIED sighting 2026-08-17: `a_declared_service_rises…` FAILED 25.7s in a 745-full-parallel local sweep (nameplate-asm @2564f93, agent env scrubbed) — failure body NOT captured (`tail` ate it; labelled hole, no signature to compare), targeted rerun ×3 green. Counted as a sighting of SOME row of this test, attributable to none. **+1 CLASSIFIED 2026-08-30 (v0.67.0 golden r1, run 33296634901 att1 @da71b785): SECOND occurrence of THIS row** — same assert :664, same box, functional half all-TRUE, 3 reap verdicts Killed; NEW data the 08-04 occurrence lacked: the survivor is specifically the svcBOOT mock (pid 54000) and `went_clean=false after 60.31s of a 60s budget` — the settle ran to FULL budget, i.e. the leak never drained rather than appearing post-sweep. Signature predates the head's entire diff (row classified at 17f95a7); ruled ledgered-class at triage, same-sha rerun att2 = cell re-executed PASS 5.160s, 9/9. Gate closed, ROW STAYS OPEN; artifact: spt-preserve\v0670-golden-r1\ (deployah, log capture + SHA256SUMS). **+1 CLASSIFIED 2026-09-06: THIRD occurrence**, [run 34017906638 att3, job 101463810585](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101463810585), sha `04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64`: same :664 assertion, functional half green, three named reap verdicts Killed; survivors **svcboot pid 50484 + relshell pid 41892**, `went_clean=false after 60.1700792s of a 60s budget`. QUIET box per doyle: zero builder load, todlando's last activity 09:08Z; Phase A 197.840s versus att2 251s. The run's final orphan cleanup still named both pids at 09:44:19Z; temp-sandbox cleanup had failed on the relshell executable. This is the Windows face of hertz's daemon-leak cluster; the W2 rig-fixups PR must name this keyed row. ROW STAYS OPEN — recording a recurrence is not a repair. **+1 CLASSIFIED 2026-09-09 (v0.68.0 / #272 golden r3, run 34310511612 att1, job 102336053348, sha `f6110c2a`, Phase B 69/234, FAIL 71.881 s): FOURTH occurrence of THIS row.** Same assertion, now rendered `:670` (moved from `:664` by 8d10b280, text byte-identical; test file identical to main `e4444413`, zero riders touch it). Functional half all TRUE; three named reap verdicts Killed (boot 53100, rel 25012, brain 42716). Survivor: svcboot `svcmock.exe` pid **25596 — a DIFFERENT pid from the 53100 the test started and killed**, `went_clean=false after 60.6070532s of a 60s budget` (full budget, never drained). NEW evidence this face adds: (a) the brain — host of the service supervisors — was ALIVE when the boot service was killed (its own REAP afterwards reports SUCCESS on 42716 AND on a child 50480, identity not captured); (b) the test DISCARDS the `daemon stop --force` result (`let _ =`, :389) and kills children before the supervisor host; (c) `servicehost.rs` relaunches an un-asked exit (`on_exit -> ExitAction::Relaunch`, backoff base 1 s). So a supervisor-relaunched svcboot in the boot-kill..brain-kill window fits every fact — but the SERVICE_EXIT line that would PROVE it lived in the brain's stderr sink inside the temp sandbox, which the job's cleanup removed, so that link is UNPROVEN and is what the dispatched lane exists to close. Ruled LEDGERED-CLASS at triage (doyle); same-sha rerun att2 ordered and #272's own Windows cells were green in the same leg. Preserved: `.spt/preserved/golden-272-r3-drive/r3-run-34310511612.log`. DISPATCHED hertz 2026-09-09, test-only: observe the daemon-stop result, kill the supervisor host BEFORE the supervised children, stamp the survivor's start time and parent pid, preserve the brain sink on the leak path, identify child 50480. **ROW STAYS OPEN until that lane lands and a Windows golden passes through it.** | +| `spt::resident_service_e2e` teardown LEAK at :664 (second svcmock outlived the sweep) | 3 classified hfenduleam Windows golden occurrences: 2026-08-04 run 30940180764 att2 @17f95a7; 2026-08-30 run 33296634901 att1 @da71b785; 2026-09-06 run 34017906638 att3 @04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64. First: second `svcmock.exe` pid 14452 outlived teardown; functional half all-TRUE, 3 reap verdicts Killed | teardown/zombie class (IR-34 kin). DISTINCT from this test's :382 PRECONDITION signature (ir15 att1, daemon never came up in 124s — that one is IR-17's family and carries no defect evidence): 2-of-11 recent Windows victim legs by test, 1-of-11 by this assert | DISPATCHED hertz 2026-08-04 (pkg item 4). Did not recur at att3 (passed, no leak row). +1 UNCLASSIFIED sighting 2026-08-17: `a_declared_service_rises…` FAILED 25.7s in a 745-full-parallel local sweep (nameplate-asm @2564f93, agent env scrubbed) — failure body NOT captured (`tail` ate it; labelled hole, no signature to compare), targeted rerun ×3 green. Counted as a sighting of SOME row of this test, attributable to none. **+1 CLASSIFIED 2026-08-30 (v0.67.0 golden r1, run 33296634901 att1 @da71b785): SECOND occurrence of THIS row** — same assert :664, same box, functional half all-TRUE, 3 reap verdicts Killed; NEW data the 08-04 occurrence lacked: the survivor is specifically the svcBOOT mock (pid 54000) and `went_clean=false after 60.31s of a 60s budget` — the settle ran to FULL budget, i.e. the leak never drained rather than appearing post-sweep. Signature predates the head's entire diff (row classified at 17f95a7); ruled ledgered-class at triage, same-sha rerun att2 = cell re-executed PASS 5.160s, 9/9. Gate closed, ROW STAYS OPEN; artifact: spt-preserve\v0670-golden-r1\ (deployah, log capture + SHA256SUMS). **+1 CLASSIFIED 2026-09-06: THIRD occurrence**, [run 34017906638 att3, job 101463810585](https://github.com/BigscreenVR/spt-bs-core/actions/runs/34017906638/job/101463810585), sha `04e32c8c95cf09ddc2a44a51cd0233b0d13bdc64`: same :664 assertion, functional half green, three named reap verdicts Killed; survivors **svcboot pid 50484 + relshell pid 41892**, `went_clean=false after 60.1700792s of a 60s budget`. QUIET box per doyle: zero builder load, todlando's last activity 09:08Z; Phase A 197.840s versus att2 251s. The run's final orphan cleanup still named both pids at 09:44:19Z; temp-sandbox cleanup had failed on the relshell executable. This is the Windows face of hertz's daemon-leak cluster; the W2 rig-fixups PR must name this keyed row. ROW STAYS OPEN — recording a recurrence is not a repair. | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1212 MINTED-NOT-RESUMED (sid identity) | 1× hfenduleam local assembly gate (2026-08-17, worktree nameplate-asm @2564f93, QUIET targeted round 1-of-3, 6.85s; rounds 2-3 green; +1 earlier FAIL 53.9s in the same tree's 745-full-parallel sweep, body tail-eaten — that occurrence carries no signature). Hot reproducer: **4–5/20** in quiet single-cell isolation at `f2d215a`; baseline arm at `60d74ea`: **5/20**. The cross-sha rate proves the family pre-existed the milestone head. | **TEST DEFECT — assert-side sampling race, not stopped-seat re-adoption.** Failure captures held `sid_after == sid_before` while broker truth crossed `None → Some(2)`, warm bring-up spawned, and stop returned `STOPPED`. The engine-room path unconditionally mints a fresh label (`broker.rs:5643`) and launches it with `is_resume=false`; the broker inserts its numeric PTY session into `sessions` before replying `Spawned`, while the child stamps the minted label only on its later self-bind. The test waited only for broker-table visibility and immediately sampled `info.json`, so it intermittently read the stopped life’s deliberately retained sid before the fresh bind rotated it. The alternate wake/reconcile path is ineligible: `endpoint stop` terminal-normalizes to `offline` + `suspended`, while `resume_woken_endpoint` requires active rest intent. Probe runs ended less than one second after admission and showed zero seat-gate involvement, matching the boundary. NAMEPLATE remains source-eliminated: no broker/session-mint/erhost file in that delta. | **CLOSED — GATE PASS** on `test/keystone-182-family-b` @`541e56f` (code @`a58a559`): after broker truth appears, poll under a hard 10s bound for the observable sid rotation before killing the controller. A fresh self-bind passes; unreadable, never-bound, or genuinely re-adopted records return the still-equal observed sid and remain red. Focused cell 1/1 green (6.97s), focused Clippy `-D warnings` green, traceability 776/776. The pre-registered signature-classified gate on the lane tip ran the exact cell ×20: **family-B sid signature 0/20** versus the hot 4–5/20 baseline, proving the fix effective. All four failures were the independent pre-#197 family-A throttle signature at 61–63s — expected on this `f2d215a`-based lane and not counted against this fix. Rides the next assembly as the test lane. | | `spt::engine_room_bringup_e2e a_cleanly_offline_engine_room_is_still_brought_up_by_its_own_gate` — :1150 PRECONDITION (warm bring-up never spawns) | 3×/3 hfenduleam main-checkout pool (2026-08-17, main @27d40b9, agent env scrubbed, 186s each — consistent, not intermittent) | **TEST INFRASTRUCTURE DEFECT, not product evidence (IR-21/IR-39 class).** The clean main target had no `target/debug/mock-session.exe`: `cargo test -p spt` builds the `spt` test target but not another package's `mock-adapter` binary. `engine_room_bringup_e2e` derived that absent sibling path without asserting it, registered it as the harness command, then spent the full launch bound waiting for a program that never existed. This fully explains `warm bring-up spawned=false`, `endpoint stop` → `NO_SUCH_ENDPOINT`, the empty brain stderr, the 186s duration, and why a warm assembly pool passed or reached a later assertion. Explicitly building `cargo build -p mock-adapter --bin mock-session` restored the real loop; the earlier environment-sensitivity hypothesis is retired. Distinct from the genuine offline/control-projection race and from row 41's Linux occurrence. | CLOSED by test-only hardening: fail immediately with the missing path plus actionable build command, and prebuild the cross-package fixture before the golden suite. No product policy changed. RCA independently source-confirmed by doyle 2026-08-17 | | `spt-daemon daemon::tests::a_tree_teardown_reaches_a_grandchild_the_service_spawned` | 2× hfenduleam Windows golden: 10.176s run 30776330383 @184f2ac (2026-08-03); 10.225s run 30873007187 att1 @4b37512 (2026-08-04) | **TWO mechanisms wearing one assert string — do not count as a cross-sha repeat** (doyle source verification 2026-08-04): red 1 was the bare-pid `provably_gone(grandchild)` poller reading a pid-reuse stranger — it is the cited motivating incident in the pin comment (daemon.rs:2726-2737) and was RETIRED by a5042ec ("poll the process we pinned, not the number it happens to hold"); red 2 is POST-fix (pin present at 4b37512:2371) — a pinned, stamp-checked identity stayed not-provably-gone for the FULL 10s bound (daemon.rs:2757), mechanism OPEN (H1 unauthenticated SELECTION / H2 enrollment gap — see INFRA-REGISTER IR-17 sub-observation). Both reds sit AT the bound; todlando's off-CI discriminator ran 0/200 under live-fleet load with every pass 1-4s (never near the bound; filter positive-controlled via `nextest list` = exactly one test). The lesson this row carries: assert-body diffing across shas carried the string across a mechanism change — identity must include the polled predicate's SUBJECT, not just the message | INSTRUMENTED-AWAITING-FIRE (hertz selection probe, `SELECTION_PROBE` line rides main; a green retires nothing — doyle-ruled). Green at att3 17f95a7. The bound question (is 10s right for a loaded runner) parked with hertz pkg item 4; kill path and spawn flags untouched until the probe fires (#131 stays todlando's, operator-triage-gated). **GREEN-CAPTURE BASELINE (hertz item 3, 2026-08-04, doyle-scoped: run-local flags, N=20, this row only).** Repro: `cargo nextest run -p spt-daemon --lib --success-output immediate --test-threads 1 -E 'test(a_tree_teardown_reaches_a_grandchild_the_service_spawned)'`, ×20, on a released box (no CI worker, no release build), agent env stripped, lane `test/probe-green-capture` @`60ca8f8` base `6ec5237` — source-identical to the v0.54.0 tag `86f0d84` for this row (the range is the release commit only: Cargo.toml/Cargo.lock/CHANGELOG.md, **zero `.rs`**). **field 4 (ppid-match count) = 2 on 20 of 20 runs. min 2 / median 2 (lower-middle, convention pre-registered BEFORE the capture) / max 2. Raw: 2×20. ZERO variance.** All 20 exits 0; durations 1.86–2.79s (consistent with todlando's 1–4s passes, nowhere near the 10s bound). Population asserted per run (`Starting 1 test` ×20) and the 20 probe lines carry 20 UNIQUE candidate pids, so these are 20 distinct runs, not one line re-read. Both sweep populations 0 after: pool-scoped and the System32 `cmd`/`ping` victims. **What this establishes:** a second process in the child's parentage is STRUCTURAL here, not an excursion — hertz's prediction (pre-registered with doyle before the capture, falsifier `min=median=1`) SURVIVED. Mechanism named in the same lane's comment fix: this row spawns WINDOWLESS, which masks `DETACHED_PROCESS` back off (daemon.rs:1218-1219) so the child owns a console. **What it does NOT establish, and must not be read as:** (a) the companion's IDENTITY is UNMEASURED — the probe renders only the CHOSEN candidate and the reject census prints only on the panic path, so "the second match is a conhost" remains inference, not data; a red is what would print it. (b) The floor here is **2**, so the incident's `3` is an excursion above 2, **not** above the `1` the comment and register carry — on this box, at this sha, "it had always read 1" is not what the scan reads. That figure needs re-deriving on the hfenduleam runner before anything is built on it; a 20/20 zero-variance local read cannot speak for a different box. (c) A green still retires NOTHING (doyle's standing rule) — this is a baseline SHAPE, not a fix, and the row stays INSTRUMENTED-AWAITING-FIRE | @@ -49,8 +49,3 @@ contemporaneous one is a row that will be cited as if it were. | `spt-daemon::twohost_web two_host_web_role_b` — the A_CELLS completion witness (:257-284) | 1x, the FIRST-EVER real pair run (one box, 2026-09-07 02:56Z, `.worktrees/gate-w1-26a96d58/.spt/twohost-web/none2/`). No pair had ever run role B before — kitsubito's leg was the env-gated no-op, so this cell had never once executed for real. | **NOT A FLAKE — a DETERMINISTIC completion-witness defect, and it would have failed every pair run forever.** Filed here because doyle routed it here; the distinction matters to whoever reads this row next, because a reruns-sometimes-passes reading would be wrong. B's witness polled `brain.net_streams()` every 250 ms for peer-initiated rows and timed out at `role B saw only 0 of 3 requester streams within 240s` — while THE PRODUCT WAS GREEN: A's three cells all passed (403 naming WEB / byte-equal / 206 with Content-Range, 0.14-0.23 s each) and B's own breadcrumbs proved it served all three (`WEB_STREAM stream=1 refused ACCESS_DENIED`, `stream=2 sent 200 35 bytes`, `stream=3 sent 206 4 bytes`). MECHANISM, source-confirmed at 8d980fdf, TWO removal paths not one: a served stream leaves the table either by `retire_stream` (the `retired` flag, filtered out in `stream_infos_filtered`, nethost.rs:1956-1961) or by `retire_stream_terminal` (nethost.rs:2053, `streams.remove`) — and the dispatcher's own worker performs that retirement in its `DispatchOutcome::Served` arm the moment the serve completes. On a loopback pair a cell finishes far inside the 250 ms gap, so the row is gone before the next sample. SAME CLASS AS ROW 46 (a witness reading an INSTANTANEOUS level the product is entitled to lower), reached by a different removal path — second instance in one day. | **FIXED** on `build/ws272-w1` as a fixup: new `run_dispatch_loop_observed` seam exposes a MONOTONIC `served` counter (`AtomicU64`, incremented in the `Served` arm BEFORE the retirement that removes the row), `run_dispatch_loop` delegates to it so all ~8 existing call sites are untouched; role B waits on the counter and no longer calls `net_streams`. Counts every answer the owner produced — dispatch.rs's `StreamFamily::Web` arm maps `Sent`, `Refused` (the 403) and `Failed` (the 502) alike to `Served`, reserving `DispatchOutcome::Failed` for a transport error, so the deny cell counts and a retryable transport failure cannot inflate the total. NOT a shorter poll: a race with a smaller window is the same race. Re-gate is the pair only (doyle). | | `spt::resident_service_e2e a_declared_service_rises_with_the_daemon_and_reaches_the_cli` — the **:453 PRECONDITION** assert (a THIRD distinct signature for this test: distinct from its :664 teardown-LEAK row above, and the direct successor of that row's noted ":382 PRECONDITION" signature, which is this assert at its old line) | 1× kitsubito Linux (2026-09-07, the W1 #249 builder battery AT the gated sha `8d980fdf`, gate dir `~/spt-w1/.worktrees/ws272-w1/.spt/ws272-w1-gate/`; nextest exit 100, ONE Summary `2999 tests run: 2997 passed (8 slow, 1 leaky), 2 failed, 1 skipped`, `panicked at` = 2, FAIL at 53s). Raw preserved at `.spt/preserved/w1-kitsubito-8d980fdf/nextest.raw`, sha256 `9c456e21a6a0f75d…3b51e`, verified against the remote by hash | **The witness expired; the daemon did not fail to come up — the assert's own words are false as rendered.** `daemon_up = wait_until(45s, brain_ready(&ready_path).is_some())` (:207) went false, and the panel that same assert prints shows the tree UP: broker generation 0 pid 3300168, `BRAIN_UP` pid 3300200, `BRAIN_PHASE:announce done in 1ms`, `BRAIN_PHASE:resume done in 0ms`, `SERVICE_BOOT:svcboot: Started`, `SERVICE_STARTED` for BOTH svcboot and relshell, and both services reaped `verdict=KILLED` with an empty survivor set. Load: a 2999-test Phase-A pool with 8 slow siblings, on a binary that spawns a REAL `spt daemon run` tree and was NEVER in the HEAVY class — the 2026-07-15 rotating-victim mechanism verbatim, same 45s-deadline family as the `resume_no_control_steal_e2e` row below (its co-victim in this very Summary). **UNMEASURED, not concluded:** where the 45s went. Nothing stamps the interval between the daemon child's spawn and the brain's first log line, so the ~10.1s exe-hash on the ready path (measured v0.66.0, a different lane) is a CANDIDATE here and nothing more. **Named and EXCLUDED:** the panel carries `DOCS_SERVER_BIND_FAIL: port 5474: Address already in use (os error 98)` twice — the rig daemon lost the well-known docs port to kitsubito's resident perch daemon. `EADDRINUSE` returns immediately and the daemon continues by design, so that is a rig-hygiene defect (fixed below), NOT this red's cause; it is recorded here because it is the loudest line in the panel and would otherwise be re-derived by the next reader | **HARDENED AT THE RECIPE (2026-09-07), not at the test.** Added to both `` strings under the HEAVY-AT-BIRTH ruling, together with 10 other never-swept daemon-tree binaries — the census (method, all eleven names, and the byte-equality assertion on the two copies) is the stanza in `.config/nextest.toml`. The 45s budget is deliberately UNTOUCHED: retuning a deadline is the same race with a different number. Separately and independently of this red, every rig `spt daemon run` spawn in `crates/spt/tests` now sets `SPT_TEST_EPHEMERAL_ADVISORY_PORTS=1` (37 sites across 31 files; 2 sites already had it) so no test tree competes with a resident fleet daemon for 5474 again. **ROW STAYS OPEN** — the harden is a prediction until this binary is seen executing in Phase B; a red on the quiet serialized box would refute the pool mechanism and re-open the box/product question | | `spt::resume_no_control_steal_e2e brain_respawn_keeps_every_session_controller_and_still_promotes` — keyed on the MESSAGE `the trial candidate never stamped brain.ready`, NOT on the rendered line. The panic renders at `:358`, which is `teardown_panic`'s funnel `panic!` shared by every failing arm in the file (:397 and :490 today); a row keyed on :358 would silently absorb an unrelated future red | 1× kitsubito Linux (2026-09-07, same battery, same Summary and same preserved raw as the `resident_service_e2e` :453 row above — the two are co-victims of ONE window; FAIL at 46s) | The `:490` call site: `wait_ready_pid(&ready_path, 45s)` returned `None` — the trial brain candidate never stamped `brain.ready` inside the budget. Panel: `CONN_WRITE_POISONED conn=3 … wall_ms=1788745673787 mono_ms=2784` (the 800ms brain-write bound this test itself sets via `SPT_BRAIN_WRITE_DEADLINE_MS=800`), then `BRAIN_SUBSCRIBER_STALL_EVICT` 1/2/3 at mono 7327-7335 and all three controller conns `event=writer-exit … reason=write-failed kind=TimedOut` at 7343 — every controller conn retired long before the 45s expired. Same 45s-ready-deadline family and same never-swept-HEAVY recipe as the row above: a real `supervise_brain` + `spt daemon brain` tree at full Phase-A parallelism. **DISCRIMINATOR, and the reason this is not filed as a bare rerun-clear:** doyle's control leg re-ran BOTH reds isolated in the same pool — exit 0, ONE Summary `2 run: 2 passed`, at 10.47s and 11.92s against budgets they had just consumed 46s and 53s of. At-budget under load vs. 4x under budget alone is the load signature; a logic defect does not respect pool occupancy. **NOT claimed:** that the 800ms write bound CAUSED the missing ready stamp — the poisoned conns belong to the OLD generation's controllers and are equally readable as a co-symptom of the same starvation; distinguishing them needs a stamp this rig does not have | **HARDENED AT THE RECIPE (2026-09-07)** with the row above and the same nine other binaries — see the `.config/nextest.toml` census stanza. Neither the 45s wait nor the 800ms knob was retuned. **ROW STAYS OPEN**, same pre-registered discriminator: serialized in Phase B this binary runs alone, and a red there refutes the pool mechanism and re-opens the product question | -| `spt-daemon::mesh_recovery roster_route_survives_a_transient_dial_failure_with_discovery_disabled` — "never converged: roster route converged after the transient" (`mesh_recovery.rs:97` at `25e60015`; `converge()` = 600 × 25 ms = 15.0 s wall-clock budget) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 3 (2026-09-09 00:43:52Z, job 102283894969, Phase B serialized, cell 15.715 s, 233/234). Same cell PASSED at the same sha on attempts 1 and 2 (9.801 s, 7.172 s). | **BOX CONTENTION, not product — budget sat inside the box's own variance:** the pump/dial path is untouched by #272 (diff v0.67.0..25e60015 on `crates/spt-daemon/src/pump` + `crates/spt-net` = webmsg/xfer only). Stderr: cache-leg `PUMP_PEER_FAIL` at the 1.5 s test dial bound as designed, roster-leg `PUMP_DIAL_SUBMIT`, then no line for ~13.8 s until the panic — a burst ate the 5-8 s of headroom the cell had. Box evidence: Phase A (pure unit) slowed monotonically 448.7 → 495.1 → 542.6 s across the three attempts at ONE sha; per-cell attempt3/attempt2 over 73 Phase B cells ≥ 1 s: median 1.05×, 19 cells ≥ 1.5×, worst 5.2×; cargo/rustc/nextest 0 and ~1.1 of 16 cores busy at census; no other CI run on either runner. Mechanism candidate (todlando, measured by doyle 01:17Z): Windows Defender real-time ON, MsMpEng at 68 % of a core on the idle box, a fresh 35 MB exe pays 1.0-2.1 s on first execution vs 20-260 ms warm; exclusion list unreadable unelevated. Rotating single victim across attempts (ttl 5 s cell on a2, this 15 s cell on a3) = RANDOM-VICTIM signature: one env cause, N apparent flaky tests. Same-sha rerun-failed ruled once more (attempt 4, last), stop on any Phase B red. | OPEN — hertz rider 5 ordered 2026-09-09 01:20Z (post-v0.68.0 thin PR, base e4444413): the `for _ in 0..600 { sleep 25 ms }` shape is 31 sites across 4 test files (mesh.rs 9, mesh_recovery.rs 6, pump.rs 7, registry_lifecycle.rs 9; closed family repo-wide); derive every budget from the named bound it races (test-local const shared with the rig's `set_quic_op_timeout`), keep it under the 60 s nextest SLOW line, print elapsed / samples / last predicate state at the panic; predicates untouched. Runner-desktop contention registered as an INFRA entry (Defender exclusion = operator). | -| `spt-daemon::registry_lifecycle oneway_rounds_plateau_rows_seats_and_a_refresh_replays_nothing` — "the refresh must subscribe NO historical rows (zero replay churn): held 0 -> 1, seats 1" (`registry_lifecycle.rs:514` at `25e60015`; the directional held/seats assertion after the gen-2 refresh) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:36Z, job 102306494097, Phase B serialized, cell 11.594 s, 233/234). Same cell PASSED at the same sha on attempts 1-3 (10.817 s, 10.518 s, 23.344 s). The two earlier victims of this arc (arm 12 ttl race; mesh_recovery converge budget) both PASSED on this attempt. | **RIG SAMPLE RACE, load-widened — not product:** A's pump is a bare `thread::spawn` (:307) stopped by the `pump_stop` flag (:386) and NEVER JOINED; the drains converge on B's gauges, gen-1's dispatcher is joined (:414), `b_held_before` is sampled, gen-2 starts — and the pump's last 100 ms-cadence round can still deliver ONE feed to B after that sample, which B then holds as one row with one in-flight seat. One row, not a replay storm: the re-apply bound one assertion earlier (:497) PASSED, and a replay regression re-subscribes the whole history. The test's own comment (:500-506) records this single-straggler face ("the extra held row was ONE straggler feed … mis-attributed to gen-2"). Product path untouched by the milestone (`registryhost.rs`/`pump` no diff v0.67.0..25e60015; `dispatch.rs` changes are the Web stream family; `broker.rs` a test accessor). Stderr carries only `CONN_LIFECYCLE` lines: a stream-8 subscription attached at mono 8.1 s and a serve on it at 8.7 s, neither released before the panic at ~11.45 s; conn ids are not attributable to A vs B from the log. Box: per-cell attempt4/attempt2 median 1.00× but 12 of 72 cells ≥ 1.5×, worst 4.75× — bursty. Third distinct single victim in four attempts at one sha = one environment cause (random-victim family); STOP ruled, sha retired. | OPEN — hertz rider ordered 2026-09-09 02:47Z (test-only, stacked on b359e40e, rides r3's head): `spawn_pump` returns its JoinHandle and the test JOINS it after `pump_stop` before the drains/sample (verify `run_peer_pump` exits at a round boundary on the flag); the :514 panic prints the gauges AND B's stream table (ids/families); audit the file for other flag-stopped-never-joined actors sampled by gauges (:417 documents the gen-1 worker case); predicates and both directional assertions untouched. | -| `spt-daemon::twohost two_host_ladder_role_a` — "A-3 setup suspend applied an edge at B: NoEdge" (`twohost.rs:2496` at `25e60015`; the rung asserts `matches!(out, RestRequestOutcome::Edge(_))` on the wire reply) | 1× golden twohost-a (hfenduleam role A / kitsubito role B), #272 WEBSERVE r2 run 34262154550 attempt 4 (2026-09-09 02:44:22Z, job 102316461906, 19.95 s; twohost-b job 102316461873 then burned its 900 s on the wake anchor A never sent — collateral, one transaction). The same rung PASSED at attempts 1-3 (attempt 3: A-3 reached B ~4.5 s after ID_B's wake; attempt 4: 10.3 s, the toast rung's replication wait ran 7.8 s vs 1.0 s). | **RIG ASSERTS THE WIRE DISCRIMINANT AGAINST THE DOCUMENTED CONTRACT; the double delivery is a pre-existing dispatcher overlap, not this milestone.** Contract (`resthost.rs:21-27`, `:198-202`): exactly-once covers the rest STREAM OPEN only; the request line is unjournaled and a redelivered rest request is a natural `no_edge`, "report, don't dedup". Evidence: B's stored intent for ID_B (`read_rest` = info.json `rest_state`, no derivation) flipped Active → Suspended inside [22.348, 22.460] (B polled its anchor at 250 ms from 14.593Z, passed 22.598Z), the window of A's single request (`request_rest` is one-shot), which was answered `no_edge` at 22.4617Z and by the transition guard wrote nothing — so a second Suspend was applied at B in that instant. B's `CONN_LIFECYCLE` shows the Rest worker's fresh-query-then-subscribe pair TWICE for stream 85 (conn 119/120 @22.4327-.4330; conn 121/122 @22.4579-.4586, conn 122's close = A's reply; conn 120 closes 22.4973). One dispatcher generation cannot re-serve at 25 ms (InFlight claim held until the worker returns, retry floor 500 ms), so two dispatcher instances = two brain processes against B's broker; the claim path is untouched by #272 (one counter line, a thin wrapper, a comment word in the Rest arm). Auto-suspend (node knob off, no daemon.json on the box, tick fires only from Dormant, per-endpoint override never set) and the liveness-derived state (never persisted) are ruled out from source. Dispatcher generation is not logged on a clean serve. | OPEN — hertz rider 4 ordered 2026-09-09 03:11Z (test-only, rides r3's head): the four wire-Edge assertions in `twohost.rs` (:2223, :2242, :2497, :2541) accept Edge or NoEdge, fail only on Refused/Failed/NoReply/BadRequest, print which reply arrived, and witness the DURABLE observable each rung already waits on (registry row at A; `read_rest` at B). Two post-publish product lanes seeded for todlando: a served-path `dispatch_event` naming its dispatcher generation; the two-dispatcher overlap hazard (a3's B log: 27 streams attached by 2+ connections; a4's: 3 — the overlap may be constant and normally masked by retire-before-second-poll). **CORRECTION 2026-09-09T03:22Z (todlando split, manifest-verified a3 B log): lane-2 data = a3 25 non-controller query+subscribe pairs (2 of the 27 were controller handovers: streams 9, 82), gaps 0-21 ms median 4 + one 127 s long-lived-row outlier; a4 2 pairs (stream 12 was a handover), gaps 25.3/25.5 ms. Scatter => fixed-period phase-offset story REFUTED and withdrawn. Standing read: two dispatcher workers, constant overlap, normally masked by the pre-serve 'row already gone' arm (dispatch.rs:1053-1060, unlogged); load stretches the mask window. Lane-2 instrument = generation named on the Served path, count distinct generations.** | -| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` — ARM 12 "the attachment this arm just registered is not in the registry" (`webserve_attachment_e2e.rs:612` at `25e60015`) | 1× hfenduleam Windows golden, #272 WEBSERVE r2 run 34262154550 attempt 2 (2026-09-08 20:33:40Z, job 102216542171, Phase B serialized, cell 11.733 s, 233/234). The emitter's OWN comment records an earlier occurrence at ttl 1 s during lane development (one red then a pass), widened to 5 s — this is the same signature one size up. | **RIG WALL-CLOCK RACE, not product — two product mechanisms EXCLUDED at the sha before the word:** (1) an in-daemon clobber between the register write and the reaper tick: `livehost.rs:1244 reap_expired_attachments` runs under `servehost::with_registry_write` and `servehost::apply_at` takes `REGISTRY_WRITE`; the CLI registers THROUGH the daemon socket (`attach.rs:94`), so one writer process, lock holds; (2) a torn read: `ServingRegistry::save_at` is `atomic_write_bytes_durable`. POSITIVE EVIDENCE (deployah 20:52Z): the SAME cell PASSED at the SAME sha on attempt 1 of the same run (Phase B 234/234, 18:47-19:31Z), nothing between the two attempts but a pool reap and a queue — a pass-then-fail at one sha is the race signature, not a deterministic defect. The cell's own envelope was 11.733 s against a 5 s ttl, so the register-to-read gap had room to cross it. Mechanism: the arm mints a 5 s attachment, the reaper ticks every 5 s, and the spawned `spt send` child plus the read landed past expiry on the loaded golden runner; the daemon's own `ATTACHED … ttl 5000ms` line is in the panic, so the serve path DID register it. Elapsed time UNMEASURED: the Windows sandbox cleanup step removed the rig home at 20:41:09Z before anyone could read a `SERVE_REAP` stamp. Same-sha rerun-failed ruled once (rate rerun, said so), non-vacuity = the cell re-executes and passes. | OPEN — hertz rider 3 BUILT 88625fa0 (green Linux + Windows, unpushed, post-v0.68.0 thin PR; ordered 2026-09-08 20:50Z): make ARM 12 deterministic (long ttl + expire under the daemon's own writer path, or capture the snapshot path from the send's stderr) and print elapsed ms at the panic so the next occurrence carries its own number. | -| `spt::webserve_attachment_e2e an_attachment_is_snapshot_served_fetched_back_and_named_by_its_message` ARM 11 — `webserve_attachment_e2e.rs:551` "each attachment registered its OWN entry" left 4 right 5 | Windows at c4919243/tree be2184af: 4 reds in 15 runs (x5: FAIL 14.591 s; x10: FAIL runs 4/8/9 at 17.707/22.834/24.498 s), every red exactly -1; fastest run of 15 is a red, slowest a pass — no elapsed correlation. Linux 5/5 at 9.9 s flat (window proportionally smaller; not evidence of absence). | **COUNT-DELTA ARM RACING A DELIBERATE CORPSE AGAINST THE 5 s PULSE.** ARM 10 mints a real `--ttl 1s` attachment and sleeps 1.5 s; `ServeRequest::List` (servehost.rs:224) returns entries unfiltered so the corpse is in `before_count`; `reap_expired_attachments()` rides the 5 s reconcile pulse (livehost.rs); a pulse inside the before..after window makes the delta +2-1. Sibling of the arm-12 race afb711c9 retired, one arm up. Census at the sha: ARM 11 is the only count-delta arm after ARM 10. | FIXED test-only by rider 5 f6110c2a (parent c4919243): ARM 11 asserts identity (each ATTACHED url's served name present in `serve list --json`), before_count and the +2 delta removed, no Reconcile between arms. _SIBLING ROW (arm 11) 2026-09-09 03:5xZ, found by hertz's r3 rider proof legs, verified by doyle from source at c4919243:_ | diff --git a/docs/INFRA-REGISTER.md b/docs/INFRA-REGISTER.md index 39908031..4a008d61 100644 --- a/docs/INFRA-REGISTER.md +++ b/docs/INFRA-REGISTER.md @@ -4660,414 +4660,3 @@ the wrapper's harness children orphaned + running") — so IR-80 ships callers, - **Kin:** [[IR-81]] and IR-80 Amendment 1 (keying on a mutable identity), the F17 riders (a witness that never executed the direction it certifies). - -### IR-85 — the Windows self-hosted box runs its fs-heavy tests 3-4x slower than a week ago; two CI wall clocks were sized for the old box, and a red run kept hiding it - -- **Status:** OPEN, filed by hertz 2026-09-09 at the #272/v0.68.0 golden r3/r4 arc, on doyle's - dispatch. Number ruled by doyle from his own census of main@`a2f335f8` (register ended at IR-83; - IR-84 was branch-claimed by name only, with no entry text in any register file in any worktree). - **Absorbs two drafts:** doyle's `IR-DRAFT-windows-fs-heavy-slowdown-and-golden-wall` (his IR-A + - IR-B) and his earlier `IR-NEXT` (operator-desktop load), both retired by reference — this is the - single entry. Every job/step and per-test timing below is **doyle's**, read via the job/step API. - · **Origin:** #272 golden r3 attempt 2 was CANCELLED by a wall clock with both test phases green. -- **⚠ THE CAP CHANGE IN `a2f335f8` IS NOT THE FIX AND MUST NOT BE READ AS ONE.** It bounds the run; - it repairs nothing. A future reader who finds an 80-minute wall and no entry here would reasonably - conclude the problem was solved. It was only made visible. - -- **A GREEN RUN COSTS MORE THAN A RED ONE, WHICH IS WHY THIS WENT UNSEEN.** A failing run - short-circuits past the wall a green run has to cross. r3 att1 finished in **48m39s only because - it FAILED at Phase B**; att2, green, hit 49m59s and was cancelled. Every earlier docs-drift skip - therefore presented as "upstream failure" — the wall was never the reported cause of anything, and - the surviving evidence systematically flattered the budget. - -- **THE WALL.** Windows golden `test` job (48 steps), fully GREEN: **30m53s** (2026-08-30, run - 33296634901) and **33m21s** (2026-09-06, run 34017906638). At the v0.68.0 head against the - then-current 50-minute cap — **r3 att2**, job 102352551368 at `f6110c2a`: Phase A 17m14s, Phase B - 20m25s, doctests 1m04s, clippy 3m31s, **CANCELLED at step 30 at 49m59s with everything green**; - steps 31-42 (installer, docs floor, both docs-drift gates, dormancy) never ran. r2 att1 - (`25e60015`, 09-08 18:47Z) reached step 34 at 44m30s. Steps 30..39 cost ~3m30s on the 09-06 green - (notify 47s, installer 13s, docs-drift 1m54s). Predicted green need ~**56 min**; cap raised to - **80** (need + ~40% for day-to-day variance) in `a2f335f8`, ci unit 25 -> 40 in the same commit. -- **r4, GREEN — the first complete measurement at the head: 54m35s** (07:12:26Z -> 08:07:01Z), - **25m25s** headroom under 80; docs-drift (step 38) 2m23s; Phase A 15m51s, Phase B 22m56s. The ~56 - prediction held to within 1.5 min, so the cap is sized on evidence, not generosity. **What that - does NOT establish:** it was a sizing forecast, and its holding says nothing about the cause - diagnosis below. Nor are att2-vs-r4 per-phase deltas a trend — att2 was cancelled mid-run, so only - its phase legs are comparable at all, and the spread they sit in is the same variance the 40% is - there to absorb. - -- **THE SLOWDOWN, per test, same box, same tests.** `spt-daemon::sync` concurrent_writes - **22.4s -> 74.8s**, two_tier_sync 18.7 -> 67.4; `spt-store` monic clone_copies 17.7 -> 62.9, - different_monics 18.2 -> 64.9; syncmerge reconciled_write 27.3 -> 51.2. **~25 spt-store/spt-daemon - tests now exceed 30s on Windows Phase A at `f6110c2a`, against 1-2s each on kitsubito.** - Phase-level: Phase A nextest 170s (09-06) -> 642s (att1) -> 1034s (att2); Phase B 710s -> 1164s. - The suite did not get more expensive — Linux is unchanged. - -- **DATING SAYS ENVIRONMENT DOMINATES, AND BY HOW MUCH.** main's own ci Windows `unit` job: - 11-12 min on 09-06 (runs 34040416870, 34041526195, 34043323577) -> 13-20 min on 09-07 -> 12-24 min - on 09-08, hitting **22 min at `e4444413`** (run 34261096301) three minutes under its own 25-minute - wall. **Gradual over days, under no single gate change, is the shape of an environment term, not a - commit's.** Head growth is real but MINOR: +138 Phase A tests, +20 Phase B, HEAVY 34 -> 35. - -- **THE BOX IS AN OPERATOR DESKTOP (folded in from the retired IR-NEXT), and that turns every fixed - wall-clock budget in the Windows suite into a coin.** From the r2 arc, three attempts at ONE sha - (run 34262154550 @ `25e60015`): Phase B red each time on a **different single cell or none** — a1 - 234/234 (job red on disk floors only), a2 `spt::webserve_attachment_e2e` arm 12, a3 - `spt-daemon::mesh_recovery roster_route_survives_a_transient_dial_failure_with_discovery_disabled` - (15.0s `converge()` budget, cell 15.715s; the same cell 9.8s / 7.2s on a1/a2). Phase A — 3,346 - spawn-dominated unit cells — slowed **monotonically 448.7 -> 495.1 -> 542.6s at that one sha**. - Per-cell a3/a2 over 73 Phase B cells >= 1s: median 1.05x, mean 1.34x, 19 cells >= 1.5x, worst 5.2x - (`endpoint_lifecycle poll_vs_reap` 1.1 -> 5.8s) — **BURSTY, not uniform**. A rotating single victim - across attempts at one sha is ONE environment cause; hardening victims one at a time never closes - it (paid before: `e2e-leaked-daemons-shared-box`). The budgets that lost were ~1.5x the fast - observation — inside the box's measured variance, so they were coins that had been landing right. -- **Box census (doyle 01:13-01:19Z, no cargo/rustc/nextest running, CPU 31%, ~1.1 of 16 cores busy):** - `qbittorrent.exe` seeding since 09-08 10:20Z (box tx 21.3 MB/s over 5s); fleet `spt daemon brain` - with 516 GB read since 09-07 08:03Z (~3.5 MB/s steady); Defender real-time ON, `MsMpEng` at - 67.8/62.8/48.9/32.2/12.2% of a core over 5s **on the idle box**; a fresh 35 MB exe pays - 2092/994/1171/1043 ms on FIRST execution against 31/263/19/260 ms on the second — **and every CI - attempt rebuilds every test binary fresh.** At 06:45Z on 09-09, with the twohost legs running: - MsMpEng 89% CPU / 991 MB WS, qbittorrent pid 47056 holding 7817 CPU-seconds (2.2 h) since - 09-08 03:20, free 134 GiB (275 -> 197 -> 131 across the three r3 dispatches). -- **⚠ AN UNREADABLE ROW IS NOT AN ABSENT ONE.** The Defender exclusion list cannot be read - unelevated on this box: `Get-MpPreference` returns the literal string - `N/A: Must be an administrator to view exclusions` **as the ExclusionPath VALUE**, so a - `-contains` test reads ABSENT and is meaningless; the HKLM `Windows Defender\Exclusions\Paths` - read throws `SecurityException`. Nobody may report the runner directory as unexcluded from an - unelevated shell. - -- **HYPOTHESIS ALREADY KILLED, so nobody re-runs it:** the Windows-only `ADAPTER_WEB_PENDING` - reconcile failure (servehost nudge) **cannot** explain this — spt-store monic and contextstore - never touch that path. - -- **WHAT IS STILL NOT ESTABLISHED (labelled, so it is not inherited as fact).** The dating argument - establishes environment-DOMINANT and bounds head growth as the minor term. It does **not** - apportion the environment term itself: Defender vs the third-party torrent load vs the fleet - brain's steady read vs anything else is unsplit, and **"MsMpEng at 89%" remains a correlate - measured beside the slowdown, not a proven cause.** The discriminator lane below is what settles - head-vs-environment on evidence rather than on the shape of a drift curve. - -- **REMEDY — none landed. This entry is the debt, and its middle arms need an operator.** Sequence - matters; run them in this order: - 1. **DISCRIMINATOR LANE (hertz, one box, ~1.5-2 h).** Run the five named tests at `04e32c8c` and - at `f6110c2a` on hfenduleam. **Same-slow at both = environment; slow only at the head = head - growth.** Cheap and decidable, and it must precede any operator ask — do not spend an elevation - request on a hypothesis a one-lane measurement can test. Design note, because it is the part - that makes the number trustworthy: the arms run **INTERLEAVED** (A/B/A/B/A/B, 3 reps each), not - all-A-then-all-B, so drift that hits the whole box cancels in the comparison instead of landing - on one arm. **A measurement that only works if everyone behaves is not a measurement** — the - torrent client and Defender are running throughout and are the thing under test, not noise - anyone gets to remove. Confounder already excluded: the three test-bearing files - (`spt-daemon/tests/sync.rs`, `spt-store/src/monic.rs`, `spt-store/src/syncmerge.rs`) are - **byte-identical blobs at both shas**, so a difference cannot be the tests changing; the two - crates around them are not (+11,388 lines over 48 files). - 2. **Remove the third-party load first.** No torrent client on the CI box during golden windows. - It is the cheapest variable to remove, and removing it first makes arm 3's benefit measurable - instead of confounded. - 3. **OPERATOR ASK — Defender path exclusions** for `C:\actions-runner\_work` and the gate pools, - read from an ADMIN shell first (see the unreadable-row warning above), then re-measure the - first-touch tax on a file **under that path** — the scratchpad measurement is outside the - runner dir, so it proves the mechanism's size, not the runner's exposure. Unsettable - unelevated; same shape as **IR-89**'s elevated firewall rule. - 4. **CI census, cheap and durable:** print `Get-MpComputerStatus` RealTimeProtectionEnabled plus - the ExclusionPath read **verbatim, refusal text included**, in the Windows job's census step, - so a run's own record says what Defender it ran under. - 5. **Re-measure both caps after any arm lands — the arm a future reader will skip, so it is loud - here.** A cap sized against a degraded box is correct only while the box stays degraded. - Leaving 80 in place after a repair silently restores the original hazard: a wall so generous it - no longer catches a wedged test, which is the job the golden cap was added to do in the first - place (the 2026-06-03 handoff.rs ConPTY stall, 22 unbounded hosted minutes). -- **Kin:** [[IR-76]] (the golden runner IS the builders' box — the structural reason a desktop's - load reaches CI at all), [[IR-64]] (box-level facts only the operator can move), [[IR-89]] (the - other elevated box-rule ask), `defender-first-touch-tax-on-fresh-test-binaries` and - `e2e-leaked-daemons-shared-box` (memory). -- **Ripe when:** arm 1 now; arms 2-4 on the operator's answer; arm 5 at the next `golden.yml` touch - after any of them. · **Size:** arm 1 a measurement, arm 4 one census line, arm 5 two literals. -- **AMENDMENT 2026-09-09 (two terms this entry did not name when it landed at `d32d5c4c`).** - 1. **`ci.yml`'s `changes` job runs `unit` on BOTH runners for every push to `main`.** The classify - step at `.github/workflows/ci.yml:49` emits `code=true` for every non-`pull_request` event, so - the docs-only skip that PR runs #206/#207 demonstrated is **`pull_request`-only** — read from - `ci.yml` at `main` by doyle, who names it his own error after twice ruling the opposite from the - PR runs alone. Consequence for this entry's wall clock: run **34337797758** (the ff of - `b66a9612`, a docs-only delta over golden-green `a2f335f8`) ran `unit (Windows)` 09:59:57Z → - 10:40:48Z and was **CANCELLED at the 40-minute job wall** (job timeout; conclusion `cancelled`, - step 6 `Unit tests` 10:08:23 → 10:40:03) — a red on `main` at a sha whose content cannot fail a - unit test. It is an IR-85 face, not a flake row. - 2. **A near-full volume is an environment term for the slowdown**, alongside the Defender - first-touch tax and the operator-desktop load already recorded. **Deliberately unquantified:** - the discriminator window that would have apportioned it was VOID, because the one leg that - completed ran under both a CI job and a falling disk. See **IR-90**. -- **WHAT THE VOIDED DISCRIMINATOR ESTABLISHED, negatively (hertz, 2026-09-09).** The fs-heavy - slowdown REPRODUCES at `04e32c8c`, which predates the +11,388-line head growth: spt-daemon - `concurrent_writes` 105.420 s, `two_tier_sync` 58.362 s, spt-store `clone_copies` 65.657 s, - `different_monics` 73.967 s, `syncmerge reconciled_write` 86.765 s, against the 09-06 baseline of - 22.4 / 18.7 / 17.7 / 18.2 / 27.3 s — **3.2x to 4.7x on all five, at the OLD sha**. So head growth - is **not NECESSARY** for the slowdown. That is the whole of it: it does NOT measure how much the - environment explains, and it is not evidence about the head arm at all — that leg ran inside run - 34337797758's window and on a volume that reached 0.018 GiB free, and the head arm never produced - a comparable pair. **Arm 1 stays OPEN**, its re-run deferred until **IR-90**'s guard exists and a - window with no `main` push and no CI job on the box can be scheduled. - -### IR-86 — golden's 32 GiB floor is BELOW the measured 67.4 GiB Windows suite footprint, so a start floor passes a box that cannot finish - -- **Status:** OPEN, filed by doyle 2026-09-08 at the #272/v0.68.0 golden r2 triage. · **Origin:** - run 34262154550 @ `25e60015` (shaped on `e4444413`), Windows test job 102182665033: every product - step green (Phase A 3346/3346, Phase B 234/234, Summary 2, FAIL 0), the ONLY reds were the two - in-job floor gates. `FLOOR_START` 97,649,786,880 B PASS at 18:47:23Z -> `FLOOR_DOCS` - 25,284,501,504 B RED at 19:31:55Z -> `FLOOR_END` RED. Consumed in-job: 67.4 GiB. -- **Mechanism:** `golden.yml` derives 32 GiB as "the observed tens-of-GB full-suite footprint - rounded up to the next binary boundary, preserving headroom for one complete run" (comment - above the Windows start floor). The measured footprint is 67.4 GiB, so a start reading anywhere - in [32, ~100) GiB passes and the job then walks under the floor by construction: box baseline - 91 GiB minus 67.4 = 23.5 < 32. This is a shortfall of the DERIVATION, not a rate, not a product - red, and not the instant-vs-sustained face IR-46 files (that face is fixed by the FLOOR_DOCS - re-read, which is what caught it). The start floor must be footprint + end-floor (~100 GiB) or - it asserts nothing about finishing. -- **Coverage consequence (the load-bearing part):** the Windows `Docs drift gate (CLI ref + llms - links)` step is sequenced BEHIND `FLOOR_DOCS`, so it SKIPPED and r2 held NO Windows CLI-ref - axis at that sha until the rerun. A floor red that skips a gate is a coverage hole wearing a - disk red. -- **Kin:** [[IR-46]] (a floor asserts an instant), [[IR-59]] (pool arithmetic, LNK-class reds - that name no disk), [[IR-64]] (the operator-payload reservoir that sets the box baseline; its - ripe-when arm — a golden that dies at the floor despite adopted pool discipline — FIRED here - in the in-job form), [[IR-73]] (second half BUILT: the within-job re-read exists and is what - produced the FLOOR_DOCS reading), REQ-CI-FREE-SPACE-PREFLIGHT. -- **Coupling (deployah 23:54Z, from `golden.yml` at `25e60015`):** the Windows `DISK docs floor` - (:616) and `Docs drift gate — windows` (:629) carry only `if: runner.os == 'Windows'`, no - `always()`, while `DISK end floor` does (`always() && runner.os == 'Windows'`, ~:705). Actions' - default is "previous steps succeeded", so ANY Phase B red skips the docs axis by construction — - observed three times at this sha (floor red, ttl cell, ttl cell). The docs gate is therefore never - independent evidence: it can only be banked when Phase B is green in the same job. Remedy line: - gate the two docs steps on `!cancelled()` plus the success of the steps they actually depend on - (checkout/build), not on Phase B — a rig change, new sha, hertz workflow rider slot. -- **Remedy shape:** re-derive the start floor from the measured footprint (67.4 GiB Windows, - Linux to be measured) plus the 32 GiB end floor, print the footprint (start minus end free) - on every run so the number re-measures itself, and keep the 32 GiB literal only for - `FLOOR_END`. -- **Ripe when:** next `golden.yml` touch (hertz workflow rider slot). · **Size:** small — one - literal per start-floor site + one printed subtraction. - -### IR-87 — two-host ceremony halves are INDEPENDENT jobs on DIFFERENT runners (A Windows/hfenduleam, B Linux/kitsubito); B's 900 s budget runs on B's own clock, so any older queued Windows job starves A and reds B deterministically, and the panic text blames pairing - -- **Status:** OPEN, filed by doyle 2026-09-08 at the #272 golden r2 triage. · **Origin:** - run 34262154550, twohost-b job 102207296960 (log 81,692 B, sha256 67403f04…8030): role B built, - rostered, polled PAIR_MEET_UP every 30 s from 19:34:30Z, and hit the rig_wait barrier - "pairing: A rostered via the daemon-hosted responder" at its 900 s deadline 19:49:17Z - (`crates/spt-daemon/tests/twohost.rs`, exit 101, 2 passed 1 failed). twohost-a (102207296985) - was QUEUED from 19:32:13Z with no runner: the Windows runner went to thin ci run 34261096301 - (post-merge on main @ `e4444413`) the second the golden test job released it at 19:31:56Z. -- **Mechanism:** `golden.yml` sequences twohost-a and twohost-b after `test`, but they are two - jobs on two runners (measured 2026-09-08 20:56Z via the jobs API `runner_name`: twohost-a on - hfenduleam, twohost-b on kitsubito), and GitHub's queue does not honour a workflow's intra-run - job ordering against a FOREIGN run on either runner. B starts, its whole budget ticks against a partner that has not been - scheduled, and `rig_wait` panics `never converged on the rig: {what}` — a text that reads as a - pairing defect. The lanes were not at fault, and twohost-a's outcome was predetermined once B - exited (a later A pass would be vacuous). -- **Kin:** [[IR-76]] (hfenduleam is golden runner AND builders' box; the job-start census cannot - see a competing GitHub run either), [[IR-62]] (rig collisions on shared runners), the W1 two-host - rig, `a-barrier-cannot-ride-a-carrier-its-sender-outlives` (memory, same family: a barrier needs - the receiver's own answer). -- **Remedy shape (pick at rider time):** (1) start barrier on B keyed to A's job having STARTED - (a run-scoped artifact or a `needs`+matrix collapse into ONE job that spawns both roles), or - (2) B's deadline clock starts at first A-rostered signal with a separate, longer "A never - appeared" budget, or (3) the panic names the missing partner's job state so triage does not - chase pairing. Any option: the two-host halves must never share a runner queue with a foreign - run while one half waits. -- **Ripe when:** next `golden.yml`/twohost rig touch. · **Size:** small-medium (job shape or - barrier rework + panic text). - -### IR-88 — `pool-release` via `cargo run -p xtask` REBUILDS xtask INTO the pool it is releasing, so a reaped pool regrows ~2.8 GB in silence - -- **Status:** OPEN, filed by doyle 2026-09-08 from hertz's reap report (20:00Z) during the #272 - golden r2 repair. · **Origin:** hertz reaped hertz-lane4 / hertz-repin / hertz-percell-id target - subtrees (Length-sum upper bounds 68.16 / 47.11 / 8.79 GB) and then ran `cargo run -p xtask -- - pool-release …` from each worktree: cargo rebuilt xtask into the just-reaped `target/` (2.8 GB - back in lane4, partial in repin), nothing printed said so, and the descendancy stopper killed - the second before it finished. Removed by hand afterwards; the post-reap free reading on the - record (275.95 GiB at 19:55:51Z) predates the regrowth. -- **Mechanism:** the documented verb (AGENTS.md: "drop it with `pool-release`") is a `cargo run`, - and cargo's default target for that run IS the pool being released. The verb's own success path - recreates the state the operator just measured away; a teardown step that runs it AFTER a reap - undoes part of the reap and leaves the recorded free reading stale. -- **Remedy shape:** either (a) `pool-release` documented and scripted as a PREBUILT `xtask.exe` - invoked against `--pool ` from another pool, or (b) the verb refuses when its own - `CARGO_TARGET_DIR`/default target equals `--pool` and prints the prebuilt form, or (c) the reap - recipe orders release BEFORE the subtree delete. Memory banked 2026-09-08 - (`pool-release-rebuilds-xtask-into-the-pool-you-just-reaped`); this entry is the durable home. -- **POSITIVE CONTROL (hertz, 2026-09-09, ruling-3 gate-r3 teardown):** the same release run - from a PREBUILT `xtask.exe` invoked from outside the pool regrew NOTHING — the pool's own - `xtask.exe` still carried its original 21:00:42 mtime afterwards, and the reap that followed - reclaimed 63.71 GiB (132.03 -> 195.74 GiB free) against a 64.88 GiB Length-sum. So the - regrowth is a property of the `cargo run` VEHICLE, not of the verb, and remedy (a) is proven - rather than merely proposed. -- **Kin:** [[IR-31]] (pool budgeting), [[IR-56]] (pool-claim identity from CWD — same verb - family, same "the tool acts on where it stands" shape), releases#103. -- **Ripe when:** next xtask pool-verb touch or the teardown runbook edit. · **Size:** tiny — - one guard or one doc line. - -### IR-89 — hfenduleam's Windows Firewall drops cold inbound UDP to the runner-built test exes, so the two-host rig's first B→A claim (W2 helper) reds for its whole 900 s budget and the panic blames A - -- **Status:** BOX HALF **APPLIED** 2026-09-08, WORKFLOW HALF **OPEN**. Filed by doyle 2026-09-08 - at the #272 golden r2 terminal triage; the operator-blocked flag is retired here, because - the operator acted. - The operator applied BOTH layers that evening — the Windows inbound rule (~23:41Z) and the - tailnet ACL grant (~23:56Z) — and the re-probe with the same rule-less pwsh listener read - **3/3 over Tailscale on 7483 AND 3/3 on 7489 at 23:57:50Z** from `100.98.197.12`, with a - post-terminal probe 3/3 on both ports at attempt 4. v0.68.0 then shipped at `a2f335f8` with - both two-host legs GREEN on golden r4, so the box half is closed by measurement and not by - assertion. **STILL OPEN, and it is the durable half:** the WORKFLOW guard (hertz) — the - pre-ceremony B→A UDP probe at `d882297f` that reds `INBOUND_BLOCKED` in 10 s with its own - name instead of burning 900 s blaming pairing, plus retiring the dead `_work\spt-core` - rules note in the runner runbook. Until that lands, the next box (or the next ACL edit) - re-opens the 900 s face with nothing to name it. · **Origin:** run 34262154550 @ `25e60015`, twohost-a 102229928746 - (`two_host_web_helper_role_a`, :968, 900.40 s) and twohost-b 102229928689 - (`two_host_web_role_b`, :559, 910.20 s): B logged 75 × "A not ready … broker QUIC op exceeded - the 10s bound (peer unresponsive)" from 20:49:49Z to 21:04:37Z and was never ADMITTED. Ladder - green both sides; both halves started 20:41:23Z on their own runners (not IR-87's starvation). -- **Mechanism (MEASURED 21:11Z):** the box's firewall policy is `BlockInbound,AllowOutbound` on - all three profiles (Ethernet and Tailscale both Private). Inbound allow rules exist only for - exes under `Documents\projects\spt-core\target` (interactive Allow clicks) and under the DEAD - pre-rename runner path `_work\spt-core\spt-core`; exactly ONE under `_work\spt-bs-core` (its - `target\debug\spt.exe`, the CLI, not the binder — deployah's count, mine had said zero), none for - any `twohost_web-*.exe`. `Get-NetFirewallProfile` shows `DefaultInboundAction = NotConfigured`, - whose effective default IS block: the block is Windows' default, not a configured policy, so there - is no policy to "restore". The runner is a service (`.\decid`), so the Allow dialog that minted the - old rules cannot appear. A's helper cell 3 bound `broker udp 7483` inside - `_work\spt-bs-core\spt-bs-core\target\debug\deps\twohost_web-f73bc44be13353fb.exe`; B dialled - `100.68.35.65:7483` cold. Probe: a rule-less pwsh UDP listener on hfenduleam:7483 received 0 of - 3 datagrams from kitsubito `100.98.197.12`; the reverse control (python listener on - kitsubito:7483) received 3 of 3 from hfenduleam. -- **Why three faces from one cause:** the shared-key rig (pre-`e4444413`) reached A only through - holes A's OUT-dialling cells (7480–7482) had opened for `id_a` — six 10 s bounds until B's - magicsock fell onto one of those paths, then ADMITTED (the 63 s stall, helper-stall memory), or - onto a live same-key sibling that never replied (the 21 min hang in 34239258523). Per-cell A - identity (`id_a_for`, `e4444413`) removed the accidental route by construction, leaving only - the cold path, which the firewall drops. hertz's one-box discriminator "vanishes" was true on - one box because one box never crosses the firewall. Product and rig are both exonerated; the - rig's `a_addr` comment already names W2 as the first A-ward claim. -- **Remedy (two halves):** (1) BOX, operator-owned, elevated on hfenduleam: - `New-NetFirewallRule -DisplayName "spt-ci two-host rig UDP-In (kitsubito only)" -Direction Inbound -Action Allow -Protocol UDP -LocalPort 7460-7499 -RemoteAddress 100.98.197.12 -Profile Any` - (rig ports at the sha: ladder 7460/7461; web `PORT_OFFSET` 20 → A cells 7480–7483, B 7481; the - A-ward surface today is exactly UDP 7480–7483 — deployah's tighter range — and the whole range - moves with any `SPT_TWO_HOST_PORT_A` override, so the rule must move with it); - program-path rules are the wrong shape because the exe hash changes per build. Verify with the - same probe (3/3) before any rerun. (2) WORKFLOW, hertz: a B→A UDP probe step in the twohost - jobs before the ceremony, so this reds in 10 s with its own name instead of 900 s blaming - pairing; and retire the dead `_work\spt-core` rules note in the runner runbook. -- **Kin:** [[IR-83]] (the tailnet-ACL face of a cross-box inbound block — KIN, **NOT the same - finding**: that entry is the tailnet policy layer, this one is the Windows firewall layer, and - the SECOND LAYER note below is exactly the measurement that separates them), [[IR-87]] (the - other way a half waits 900 s on a partner), [[IR-76]] (golden runner - is the operator's box), [[IR-64]] (box-level facts only the operator can move), - `twohost-web-helper-stall-shared-a-identity-stale-path` (memory; its CONFIRMED arm is re-read - by this entry). -- **Ripe when:** the WORKFLOW half, at `d882297f`'s PR. (The box half was ripe "now" against - `25e60015` and is done; #272 golden acceptance is no longer blocked by it.) · **Size:** one - workflow step remaining; the elevated command is spent. -- **SECOND LAYER (measured 23:41–23:50Z, after the operator applied the Windows rule exactly as - asked and the re-probe STILL read 0/3):** the tailnet ACL. The same `python.exe` listener (own - program rule) received 3/3 from kitsubito over the LAN (`192.168.1.168 → 192.168.1.81:7483`) and - 0/3 over Tailscale (`100.98.197.12 → 100.68.35.65`), TCP connect over Tailscale times out too, - solicited return over Tailscale works, ShieldsUp false both ends. `tailscale debug netmap` on - hfenduleam: one PacketFilter rule, 18 permitted inbound sources, kitsubito ABSENT; kitsubito is - `tag:eye-tracking-resource` owned by a different tailnet user, and its own filter DOES permit - hfenduleam. Asymmetric policy: member device → tagged resource allowed, reverse denied. The - Windows rule was NECESSARY (rule-less exe, LAN probe 0/2 earlier) and NOT SUFFICIENT. Remedy - half (1) gains an ACL grant, operator-owned in the admin console: - `{"action":"accept","src":["tag:eye-tracking-resource"],"dst":["hfenduleam:7460-7499"],"proto":"udp"}`. - Rider 4's failure text names both layers and the LAN-vs-Tailscale probe as the discriminator. - Lesson for the entry: a cross-box "unresponsive" has at least THREE layers (Windows rule, tailnet - policy, the exe's own bind); probe through the SAME path the rig uses, and re-probe after every - single change — the first fix reading as "applied" is not the probe reading 3/3. -- **Corroboration + reading traps (deployah 23:50Z, from the box):** Windows rule found present by - FILTER search (`netsh advfirewall firewall show rule name=all dir=in verbose`), not by name — the - operator's name is hyphenated "two-host" and a `twohost` grep reads it ABSENT. Netmap: the single - filter permits ALL ports and ALL protocols from its 18 sources, so the grant is SOURCE-scoped, not - port-scoped — "this peer is not a permitted source at all", which is why TCP timed out beside UDP. - Two permitted sources (100.98.213.33, 100.98.214.87) share kitsubito's 100.98/16 and read as hits - to an eyeball scan for "100.98."; test membership of the full /32, never a prefix. His first netmap - read extracted a non-existent field (`SrcIPs`; the real key is `Srcs`), printed an empty list and - minted a confident "ABSENT" — an empty extraction cannot witness absence; the 18-count is from the - corrected read. Prefs on the box: ShieldsUp false, NoStatefulFiltering true, RouteAll true, - NetfilterMode 2 — nothing there explains the drop. -- **Probe-rider design rule (hertz, d882297f):** the pre-ceremony probe reds `INBOUND_BLOCKED` for - EITHER layer and does not pretend to know which — its job is "the box cannot receive, stop triaging - the product"; the layer is named only by the one measurement that decides it (same listener, LAN - vs Tailscale), which the failure text prints as a recipe. A guard that names a cause it did not - measure is the defect class this whole entry documents. - -### IR-90 — a full disk on the self-hosted box reds a rig as a PRODUCT refusal, not a build error, and no rig or gate records the free space that would falsify it - -- **Status:** OPEN, filed by hertz 2026-09-09 from the disk-full incident on hfenduleam. · - **Origin:** the IR-85 discriminator lane's head arm, killed by the volume rather than by the code - under test. -- **Symptom:** `spt-daemon::sync concurrent_writes_reconcile_on_elected_node_and_converge` FAILED at - 67.588 s with a panic in our own test at `crates\spt-daemon\tests\sync.rs:198`: - `pull: Custom { kind: Other, error: "sync refused: bundle failed: git -C \tracked-b\.seed.git - bundle create \scratch\serve\serve-pull-6.bundle ^d12a7134... a-doyle failed (exit Some(1)): - fatal: sha1 file '' write error. Out of diskspace\nerror: pack-objects died" }`, then a - second panic at `:213` (`pull thread: Any { .. }`) as the harness thread unwound. -- **Cause:** `C:` was at **0.018 GiB free of 1862.02 GiB (0.00%)** at that instant - (`Win32_LogicalDisk`, 10:57Z). `git bundle create` could not write; the daemon's serve path turned - that into its real product refusal string `sync refused: bundle failed`; the test asserted on the - refusal. **Every layer behaved correctly, and the report reads as a sync regression at the sha - under test.** -- **Why it is worse than the disk faces already known:** the two banked faces - (`disk-full-reds-as-lnk1318-pdb-error`) are TOOLCHAIN costumes — `LNK1318` at link, and rustc - I/O before any link — and both route a reader to "the box is sick". This one routes to a CODE - OWNER: it names our file, our line and our refusal string, and the disk word sits in the FOURTH - nested clause behind a git exit code. It arrived mid-discriminator with an old sha and a new sha - side by side, where the cheapest reading — "the head arm failed, the old arm passed" — is a - head regression that does not exist. -- **Blast radius, same incident:** the volume also carried a live `unit (self-hosted, Windows, - hfenduleam)` job (run 34341010297) compiling into it, which completed FAILURE on its own and was - ruled VOID after the fact. doyle's cancel of THAT run returned `Cannot cancel a workflow run that - is completed` — recorded because a register that credits a controlled action nobody performed - teaches the next reader that the box was under control. (The earlier run 34337797758, in - **IR-85**'s amendment, is a different run with a different ending: CANCELLED at the job wall. One - register putting one cancel string on the wrong run is the same defect, one run over.) `spt daemon status` went to peer pump last tick 185 s with 24 brain - subscribers stall-evicted (last evict 10:51:22Z, inside the disk window), with an operator restart - under consideration — a restart that risks the releases#287 shell stranding and that, had it - appeared to help, would have taught everyone the wrong cause. -- **⚠ Two things that looked like disk fallout and are NOT**, recorded so this entry does not - overclaim. (a) `serve list` returning `SERVE_UNCONFIRMED`: doyle read it from source — - `servehost.rs` and `KIND_SERVE_REQUEST` are ABSENT at 0.67.0/0.67.1 and were minted by 0.68.0, and - the resident broker is still the 0.67.0 image after the brain-only flip, so the CLI's 10 s bound - times out **by design** against an older daemon. (b) `peer reachability: DEGRADED, 4 of 7 peers - unreachable for 882775 s` — 10.2 days, predating the incident entirely. Even the pump term is - claimed as CONCURRENT, not caused: hertz called the pump recovered off ONE post-reclaim sample - (54 s) and doyle falsified it two minutes later (133 s), because a single reading of a monotonic - "last tick N s ago" counter cannot separate STILL TICKING from TICKED ONCE, and a fresh project - index proves the coordinator loop, not the pump. -- **Remedy — one guard, two placements, and it REFUSES rather than runs:** - 1. **Rig start.** For the two-host rig and any test rig that shells out to `git bundle` or writes a - store: read free space on the volume holding the rig's temp root and `target`, and under a floor - fail immediately with a message that says DISK and prints the number — never enter the - ceremony. Floor: start at **10 GiB**, above the largest single artifact these rigs write and far - below any healthy state of this box; tune only with a measurement. - 2. **Failure text**, for the case where space runs out MID-run and no start check can catch it: - when a shelled-out git/store operation fails, append the current free space to the error before - it becomes a product refusal string, so the panic that reaches a human already carries the - falsifier. -- **WHY BOTH, measured rather than argued (hertz, 2026-09-09, the #209 lane):** a floor read ONCE at - rig start would have PASSED that run and the run still ended nearly empty — `cargo check` started - at **136.32 GiB** free, the test-profile build of 221 binaries took it to **40.18 GiB in ten - minutes**, the legs finished at **16.24 GiB**, and it read **12.27 GiB** ninety seconds later. A - start-only guard is a guard against yesterday's disk. That last ~4 GiB fell with those legs already - terminal; the candidates are the runner's CI job and its `_work` tree, unsampled and therefore - UNATTRIBUTED. -- **Explicitly NOT the remedy:** a bigger disk, or a reap schedule. ~195 GiB of headroom was consumed - by ordinary work, ACCRUED across a ~45-minute build phase; **no reading bounds a rate** — the - free-space observations either side are endpoints of an accrual, and a rate derived from them - retargets a hunt (deployah's went to runaway logs, VSS and torrent preallocation on the strength of - a `>150 MB/s` derived that way, while the measured live box-wide write rate was ~1.5 MB/s). What - consumed it: two `cargo nextest run -E <5-test filter>` lanes at **82.88** and **64.44 GiB** — a - filter narrows the RUN, never the BUILD, and both built 221 test binaries under the `test` profile - — plus 40.93 GiB of `C:\actions-runner\_work` and 9.8 GiB of `%TEMP%`. Any headroom this box - has is two cold pools away from gone, so the guard must be a REFUSAL, not a budget. -- **Kin:** **IR-85** (the same box's fs-heavy slowdown; a near-full volume is now a named environment - term there), **IR-86** (a start floor that passes a box which cannot finish — the same defect one - layer up, and this entry is its mid-run half), **IR-76** (the golden runner is the operator's - desktop), `disk-full-reds-as-lnk1318-pdb-error` (memory; this is its third face and its standing - gap), `test-profile-pool-outgrows-the-disk-floor`, `free-space-floor-blocks-golden`. -- **Ripe when:** now. · **Size:** one assertion plus one error-context append, both test-side. diff --git a/docs/RELEASE-RUNBOOK.md b/docs/RELEASE-RUNBOOK.md index c0fde01a..a4f7b38d 100644 --- a/docs/RELEASE-RUNBOOK.md +++ b/docs/RELEASE-RUNBOOK.md @@ -149,22 +149,6 @@ The gater hands the assembled head to the release driver, who verifies it asking — PORTER's hand-off omitted it and the driver had to ask, which worked only because the driver knew to. - **Name the changed seam's CONSUMERS beside it (#272 r1, 2026-09-08).** The - never-executed list enumerates cells the batch ADDED; it cannot see a - pre-existing cell whose denominator went stale because a composer under it - changed. The first golden of #272 (run 34239258523) went red on three `spt` - e2e cells that had last passed at W1: W2 changed what every message envelope - carries (its own commit message predicted "cells that assert a raw spool - will red, and each one gets repinned"), nobody re-ran the `spt` e2e crate - after it, and the thin-CI unit job runs libs only. The second hand-off - artifact is therefore: for every `pub fn` the batch's diff touched in a shared - composer/renderer/parser, which test crates drive it, and the sha of each - crate's last full execution (from the preserved raws, the same source the - never-executed list reads). A crate whose last full run predates the change - is run in full on both OSes BEFORE the head is handed off — the gater's - targeted legs are a claim that the untargeted cells cannot see the change, - and a commit that predicts reds has refuted that claim in writing. - **The push run IS the golden run.** Pushing the head to `golden/**` runs the full suite; dispatching afterwards runs an identical second one, and per-ref concurrency (`cancel-in-progress: false`) makes them serial, so the duplicate [exited with code 0]