--- bounded wait (fc5887b1) 8195: // Bounded wait for the exit-waiter to reap the table row; 8196- // belt: a wedged/missed waiter (root already dead) gets a 8197- // direct de-table + stamp clear so the gate can converge. 8198- let reap_deadline = Instant::now() + Duration::from_secs(5); 8199- loop { 8200- if !recover(&self.sessions).contains_key(&sid) { 8201- break; 8202- } 8203- if Instant::now() >= reap_deadline { 8204- let removed = recover(&self.sessions).remove(&sid); 8205- if let Some(h) = removed { 8206- recover_log(&h.log).stamp_reaped(); 8207- spt_proto::emit_line_err!( 8208- "SPAWN_ZOMBIE_FORCE_DETABLE:{}: exit-waiter never \ 8209- reaped dead session {sid} — de-tabled directly", --- nextest.toml @main # cargo-nextest configuration. # # CI runs the workspace test suite under `cargo nextest run` in TWO PHASES (see the # `test` job in .github/workflows/golden.yml — the full suite is the GOLDEN lane's; # ci.yml is the thin lane and runs no phase split): # Phase A `-E 'not ( )'` — the light pool, full parallelism, fast. # Phase B `-E ''` — the heavy real-broker/PTY/daemon-tree class, # run AFTER Phase A so the box is QUIET, and # serialized by the test-group below. # now lives in exactly TWO places — golden.yml's job-level `HEAVY` env (from # which BOTH phases compose, Phase A as `not ( $HEAVY )`) and the test-group override # at the bottom of this file. Keep those two in sync. It was three until the # releases#108 bench wiring hoisted the workflow's pair into one env var; entries # below that say "added to all three strings" record what was true when they # were written, and mean these two plus the workflow's second copy. # See docs/RELEASE-RUNBOOK + ci-kitsubito. [profile.default] # A hung test (e.g. a blocking PTY/socket read that never completes) must FAIL FAST, # never wedge the CI job up to its wall-clock timeout. Warn once a test exceeds 60s, # then SIGKILL + fail it after 4 such periods (4 minutes). No legitimate test in this # workspace runs that long; this is purely the no-infinite-wedge backstop. slow-timeout = { period = "60s", terminate-after = 4 } # Concurrency cap for the heavy real-broker / PTY / daemon-tree integration tests. # # WHY: nextest fans a phase into ONE parallel pool sized to the runner's core count. # The "heavy" tests each spin a REAL broker, forkpty a REAL PTY child, and/or spawn a # REAL `spt daemon`/brain process tree. Two failure modes were observed on the shared # self-hosted runners: # 1. heavy-vs-heavy contention: ~16 real brokers launching at once contend for the # scheduler / PTY allocation / socket setup, producing wait-deadline / PTY-spawn # / child-reap TIMING flakes whose victim rotates run to run (spawn_env on Linux # forkpty, g4 on Windows, controller_writer, attach_survives, …). # 2. heavy-starved-by-light: even with the heavy class serialized, the ~1200 LIGHT # tests still ran at full parallelism in the SAME pool and saturated the box, so # a heavy test's helper thread lost the CPU race (a_journaled_input_wedge # subscribed=false; job_escape_e2e tree-reap timeout). The two-phase split fixes # THIS: when Phase B runs, Phase A is done and the box is idle. # # The test-group then serializes the heavy class WITHIN Phase B so failure mode 1 # can't bite either, even on the idle box. max-threads = 1 is contention-free and # deterministic (a cap > 1 still lets two real brokers race = lucky, not reliable). # Group membership == the phase filter. (Driver-pacing in inject_control_wedge # additionally removes that gate's self-inflicted CPU storm — a correctness nicety on # top of the structural split.) [test-groups] heavy-broker-pty = { max-threads = 1 } # — the heavy real-broker/PTY/daemon-tree class, by package+binary. Swept by # concrete evidence: a binary is HEAVY iff some test in it spawns a real PTY child # (a `SpawnReq`/`spawn_program` with a `cat`/`findstr`/`yes`/`sleep`/`ping`/FLOOD # program) OR a real `spt daemon`/brain process tree (`supervise_brain`, CARGO_BIN_EXE # daemon run). Pure in-process loopback-QUIC wire tests (net_broker with NO PTY child: # netbroker, netstream, mesh, sync, replicate, presence, …) are LIGHT and stay in # Phase A. All spt-term test binaries forkpty a real PTY child, so the whole package # is heavy. ONE deliberate exception to the sweep criterion: `net_worker_starve` is # loopback-QUIC (no PTY/daemon tree) but its <300ms scheduling CANARY is the point of # the test — on a loaded Phase-A pool the canary reads box saturation, not worker # starvation (F-039 rider, doyle-ruled: isolate the rig, NEVER loosen the bound — a # loosened canary stops being a canary). It rides Phase B's quiet box, serialized by # the group. Keep this string IDENTICAL to both -E filters in ci.yml. # # THE HEAVY-AT-BIRTH RULING (W4 gate, doyle RCA 2026-07-15) — this stanza is its canonical # text; cite it as "the HEAVY-AT-BIRTH ruling", never as a row number. The sighting behind it # is FLAKE-LEDGER: the daemon-tree e2e bring-up row (2026-07-15), which was BACKFILLED in # 2026-08-02 (releases#95) — the commit that minted the "#15" citation said it had appended a # ledger entry and had not, and that unclaimed number was later squatted by an unrelated # 2026-08-01 row. Read that row for the provenance limits: four spt e2e binaries that spawn a # REAL `spt daemon run` tree (CARGO_BIN_EXE daemon run + ~10s bringup) sat in the LIGHT # Phase-A pool — a 16-way daemon storm on kitsubito blew the 30s brain-readiness # deadline probabilistically. Six PRECONDITION brain-never-came-up sightings across the # W3+W4 gates (two independent runs), ROTATING membership (the load-flake tell); # run_no_dup_session SOLO on the warm tree = PASS in 10.7s; no OOM/fork/TasksMax events # on box (v6-first DNS with no default v6 route noted as a boot pad — the # broken-IPv6-iroh tell). All four QUALIFY as HEAVY under this file's own sweep # criterion and postdate the last sweep: bind_honest_cross_perch_e2e, # live_adapt_translation_swap_e2e, multi_subnet_bringup_e2e, run_no_dup_session_e2e — # added to all three strings. Plus daemon_refresh_e2e, NEW in this same W4 # branch: it spawns the identical real `spt daemon run` + endpoint-bringup tree, so it # qualifies at birth — classed heavy NOW rather than after its first Phase-A flake. # (translate_proof stays Phase A: one sighting, single-child spawn, watching. # composite_e2e, also new in W4, stays LIGHT: daemonless apply, no daemon tree.) # # PROJECT-INDEX W2 (todlando 2026-07-15): projindex_writer_e2e is NEW and spawns a # real `spt daemon run` tree TWICE (cold + warm start legs) — HEAVY AT BIRTH per the # HEAVY-AT-BIRTH ruling (classify at file creation, not at first flake). Added to # all three strings. # # PROJECT-INDEX W3 (todlando 2026-07-15): projindex_reader_e2e is NEW and spawns a # real `spt daemon run` tree (the daemon-maintained-index reader legs) — HEAVY AT # BIRTH, same ruling. Added to all three strings. # # REDISPATCH-TRUTH W1 (todlando 2026-07-16): redispatch is NEW — each test spawns # two hermetic QUIC brokers + killable dispatch_fixture child processes (real # dispatcher generations) — HEAVY AT BIRTH, same ruling. Added to all three # strings. # # REDISPATCH-STALL W1 (todlando 2026-07-16): redispatch_stall is NEW — real hermetic # QUIC brokers per test, several legs spawn real PTY children (echo child) and/or # killable dispatch_fixture generations, and legs set process-wide env knobs # (SPT_BRAIN_WRITE_DEADLINE_MS / SPT_TEST_STREAM_OPENER_UNSUPPORTED) that require # process-per-test isolation — HEAVY AT BIRTH, same ruling. Added to all three # strings. # # REGISTRY-LIFECYCLE W1 (todlando 2026-07-17): registry_lifecycle is NEW — real # hermetic QUIC brokers per test running PRODUCTION pump + dispatcher loops (the # long-lived-carrier plateau seam, KH 7.44), and one leg sets the process-wide # SPT_DISPATCH_INJECT_REGISTRY_SEATBLOCKED valve (process-per-test isolation) — # HEAVY AT BIRTH, same ruling. Added to all three strings. # # REGISTRY-LIFECYCLE W2 (todlando 2026-07-17): endpoint_lifecycle is NEW — the # KH 7.45 matrix: real brokers, real PTY children (killed mid-test), a served # attach viewport torn down raw, and a process-global SPT_HOME — HEAVY AT # BIRTH, same ruling. Added to all three strings. # # REGISTRY-LIFECYCLE W3 (todlando 2026-07-17): spawn_truth is NEW — real # brokers + PTY children (incl. a 2-level wrapper→client tree killed # mid-test) and process-wide env knobs (SPT_SPAWN_CLIENT_GRACE_MS / # SPT_TEST_SPAWN_FRESH_UNSUPPORTED) needing process-per-test isolation — # HEAVY AT BIRTH, same ruling. Added to all three strings. # # RC-RENDER-TRUTH W1 (todlando 2026-07-18): rc_attach_truth is NEW — the # KH 7.46 umbrella matrix: real `spt daemon run` trees + dummy-harness # bringups + a zombie wrapper child + a process-global SPT_HOME — HEAVY AT # BIRTH, same ruling. Added to all three strings. Its in-crate # sibling (rc::tests::qualified_targets_attach_with_bare_wire_id, a # two-broker QUIC attach rig) is kind(bin) and rides the FLAKE-LEDGER: # attach_viewport_reconnects_across_a_broker_bounce override below instead. # # RC-RENDER-TRUTH W2 (todlando 2026-07-18): controller_lease is NEW — the # KH 7.48 matrix: real brokers + PTY echo children, a deliberately WEDGED # controller conn, and the process-wide SPT_BRAIN_WRITE_DEADLINE_MS knob # (process-per-test isolation) — HEAVY AT BIRTH, same ruling. Added to all # three strings. # # RC-RENDER-TRUTH W3 (todlando 2026-07-18): render_lifecycle is NEW — the # KH 7.47 output-before-exit leg: a real broker + a short-lived PTY child # through the production serve pump — HEAVY AT BIRTH, same ruling. Added to # all three strings. # # PORTER (todlando 2026-08-21, releases#201): exit_every_reap is NEW — a real # broker + PTY echo child driven through the production attach->serve pump, # twice, and it sets the process-wide SPT_TEST_EXIT_WAIT_ERR valve around one of # the two spawns (process-per-test isolation) — HEAVY AT BIRTH, same ruling. # Added to both strings. # # RC-RENDER-TRUTH v0.38.1 leg 3 (todlando 2026-07-19): endpoint_survival gains a # ready_agent resume-rig test (a 2nd real broker + real HARNESS_CMD ping/sleep child) # for REQ-HOSTING-AUTHORITY-CONTROLLABLE. The file ALREADY qualified as HEAVY under this # file's own sweep criterion (its live_agent resume test spawns the identical real # broker + harness tree) but had sat in Phase A — a latent miscategorization. Classed # HEAVY now (HEAVY-AT-BIRTH ruling: real-daemon-tree = heavy, before a Phase-A flake). # Added to all three strings. # # TEARDOWN-AUTHORITY W1 (todlando 2026-07-19): endpoint_teardown_authority_e2e is NEW — # real `spt daemon run` broker + a dummyharness endpoint + a DESCENDANT process, brought # up, torn down and brought up AGAIN per test (the ADR-0045 subtree-reap legs), plus a # process-global SPT_HOME. It qualified as HEAVY AT BIRTH under this file's own sweep # criterion and I shipped it into Phase A anyway — a CLASSIFY-AT-BIRTH MISS BY THE RULE'S # OWN AUTHOR, caught only when the test went red in CI at 53.9s against a 7-10s quiet-box # cost. Recorded in those words because the failure mode of this rule is not ignorance of # it; it is attention keyed on the code under test while the rule sits one layer out. # Added to all three strings. # # RC-RESIZE-GEOMETRY-EPOCH (todlando 2026-07-21): resize_geometry_epoch is NEW — a real # broker + a REAL ConPTY child (a PowerShell painter) driven across a resize, twice per # leg, plus the process-wide SPT_TEST_RESIZE_WINDOW_MS valve — HEAVY AT BIRTH, same # ruling. Added to all three strings. # # RC-RESIZE-PRESENTATION-BARRIER (todlando 2026-07-21): resize_presentation_barrier is # NEW — the LIVE-delivery successor rig: TWO real brokers + real ConPTY painters per leg # (a control-run authority session plus the resized session), the same window valve — # HEAVY AT BIRTH, same ruling. Added to all three strings. # # DAEMON-LIFECYCLE W2 Leg A (todlando 2026-07-22): restart_replay_lifetime is NEW — a # real broker + a real PTY echo child + TWO real `dispatch_fixture` CHILD PROCESSES (the # dispatcher generations), plus a process-global SPT_HOME. It qualifies as HEAVY AT BIRTH # under this file's own sweep criterion on every axis at once, and is classed here in the # SAME commit that introduces it — the TEARDOWN-AUTHORITY W1 miss (a classify-at-birth # miss by the rule's own author, caught only by a 53.9s CI red) is the reason that timing # is now the discipline rather than the intent. Added to all three strings. # # DAEMON-LIFECYCLE W2 Leg B (todlando 2026-07-22): transport_death_eof is NEW — TWO real # brokers with REAL QUIC endpoints, a real PTY echo child and a real `dispatch_fixture` # CHILD generation PER LEG, plus a process-global SPT_HOME. It is also SLOW BY # CONSTRUCTION: the torn leg waits out the QUIC idle timeout (MESH_MAX_IDLE = 60s), and # that duration IS the measurement, so it can never be tuned down to fit a light pool. # HEAVY AT BIRTH, classed in the SAME commit that introduces it. Added to all three # strings. # # REMOTE-TRUTH A-2 completion (todlando 2026-07-30): wake_resume_bind_e2e is NEW — a real # detached `spt daemon run` (broker + supervised brain), a real broker-PTY harness bringup # that BINDS, a killed harness, a real daemon-driven resume through [session.resume], and a # nested psyche host, all under a process-global SPT_HOME. It is also slow by construction: # the negative control deliberately WAITS OUT several reconcile ticks to prove a suspended # cold endpoint is left alone, and that quiet interval IS the measurement. HEAVY AT BIRTH, # classed in the SAME commit that introduces it. Added to all three strings. # # DOORBELL W5 gate (doyle 2026-08-01): knock_approve_not_hostage_e2e is NEW — approving a # knock whose knocker lives on another node reaches `wansend::answer_send_routed` and # `knock_send_remote`, both of which call `spt_daemon::daemon::ensure_running`, so the test # spawns a REAL `spt daemon` tree (broker + supervised brain) into its isolated SPT_HOME and # then drives a real QUIC dial at an unreachable peer. It is also SLOW BY CONSTRUCTION: the # receipt leg fails by exceeding the broker's 10s per-peer QUIC bound, and that timeout IS # the failure being measured — a shorter one would stop being the ordinary unreachable-peer # shape the row is about. ~11s on a quiet box. HEAVY AT BIRTH, classed in the SAME commit # that introduces it. Added to all three strings. # # …and commune_two_cwd_e2e, NEW in the same commit, which I first classed LIGHT on the # strength of its VERBS — `api psyche-download` and `endpoint list --json` are pure local # store reads, and the sweep criterion asks what a binary SPAWNS. It spawns a daemon: EVERY # `spt api` invocation calls `ensure_daemon` (api/mod.rs `run`, REQ-DAEMON-3), so the rig # owns a real daemon tree the verb under test never asked for. Caught by counting surviving # `spt.exe` processes after a run, not by reading — recorded in those words because the # criterion is easy to apply to the code under test and easy to miss one layer out, which is # the same failure mode the TEARDOWN-AUTHORITY W1 note above records. # # The other two W5 siblings stay in Phase A and are verified so by the same count: # human_redeem_msg_code_e2e (`knock new-code` / `redeem` on the LOCAL route) and # hide_new_remote_rows_e2e (`endpoint list --json`) add no `spt.exe` at all. Recorded # because "it is a W5 int row" is not the criterion; spawning one is. # # BAROMETER H2 (hertz 2026-08-02, releases#101): knock_mutual_cross_node_e2e is NEW and # spawns TWO real `spt daemon run` trees at once — one per SPT_HOME — because both # requirements it carries are statements about WHICH NODE holds a record, and a rig with # one home cannot express them. It also holds a process-global SPT_HOME. Twice the daemon # tree of any binary already on this list, so it qualifies twice over. ~6s on a quiet box. # HEAVY AT BIRTH, classed in the SAME commit that introduces it. Added to all three # strings. # # NOT slow by construction and deliberately so: its two barriers are OBSERVED STATE # (`brain.ready`, then `net_status` reporting a bound endpoint), never a wall-clock guess, # and its daemons rendezvous on directly-seeded addresses with the relay off, so nothing # in it waits out a discovery or dial timeout. A `PRECONDITION: brain never came up` or a # `never reported a net endpoint` panic here is therefore a BOX signal — the Phase-A # daemon-storm shape from FLAKE-LEDGER #15 — and not a defect in what it asserts. # # W4 / releases#177 (todlando 2026-08-19): er_brief_once_per_session_e2e is NEW and # spawns a real `spt daemon run` tree plus a real PTY child through a genuine # engine-room bring-up, then takes a SECOND seat on it — the same shape as its two # er_briefing_* siblings already on this list, one seat heavier. HEAVY AT BIRTH, # classed in the SAME commit that introduces it. Added to both strings (this # one and golden.yml's job-level `HEAVY` env — the two copies are compared # byte-for-byte, not by eye: the filter values were extracted from both files and # asserted equal before AND after this edit, because a filter that silently # repartitions the suite does not red, it just runs the test in the wrong phase). # W2 / releases#208 (todlando 2026-08-21): er_briefing_session_scoped_e2e is NEW and # spawns a real `spt daemon run` tree plus a real PTY child through a genuine engine-room # bring-up, then reads what the session was DELIVERED — the same shape as the er_briefing_* # siblings already on this list. HEAVY AT BIRTH, classed in the SAME commit that introduces # it, and added to both strings (this one and golden.yml's job-level `HEAVY` env). # The two copies were extracted and asserted byte-equal BEFORE and AFTER the edit, not # eyeballed: a filter that silently repartitions the suite does not red, it just runs the # test in the wrong phase. # v0.67.1 golden r1+r2 (doyle RCA-274-R1-LINUX, 2026-09-06): brain_resume_conn_deadlock # was NEVER a member — born 03c71093 (2026-07-09), SIX DAYS before the HEAVY-AT-BIRTH ruling, # and the sweep was only ever applied forward from that date. Each of its two timing cells # spins a REAL broker + SIX real PTY `yes` flood children and the two are byte-identical # rigs, so in the light pool they run CONCURRENTLY (12 floods + 2 brokers) inside a 3186- # tests-in-48s Phase A. Golden 34014574926 @34fdb848: att1 97 ticks vs the >=100 floor (twin # PASS in the same window), att2 88 and 97 (BOTH twins). The floor guards a deadlock whose # signature is ~0 ticks; 88-97 is a load-margin miss ~100x above it, and the deterministic # zero-subscriber guard in the same binary PASSED both attempts. HEAVY under this file's # own criterion; added to both strings (this one and golden.yml's job-level `HEAVY` # env), extracted and asserted byte-equal before and after the edit by the edit script. # PRE-REGISTERED DISCRIMINATOR: serialized in Phase B on the quiet box these cells run one # at a time — a red THERE refutes the pool mechanism and re-opens the box/product question. # DRIFT REPAIR, found by the check this lane adds and not by a red (hertz 2026-09-07): # `twohost_web` (spt-daemon) was added to golden.yml's job-level HEAVY env by W1 `8d980fdf` # and NOT to this file's test-group override, so for one commit golden ran it in Phase B # UNSERIALIZED — excluded from Phase A by the workflow's `not ( $HEAVY )`, then admitted to # Phase B without the `heavy-broker-pty` cap this file grants. That is precisely the failure # the byte-equality note above names: it does not red, it just runs the test in the wrong # phase. Added here; `check_heavy_integration_classification` now asserts the two copies # match, so the next divergence fails `xtask check` instead of waiting for a flake. # W1 #249 kitsubito battery (hertz RCA 2026-09-07, evidence # `.spt/preserved/w1-kitsubito-8d980fdf/nextest.raw`, sha256 9c456e21…, ONE Summary, # exit 100): TWO daemon-tree binaries red at the same 45s brain-readiness deadline — # `resident_service_e2e` :453 (53s) and `resume_no_control_steal_e2e` (46s). Both are the # 2026-07-15 rotating-victim mechanism verbatim, and BOTH were non-members: like # brain_resume_conn_deadlock above, the sweep was only ever applied to binaries someone # happened to look at. So this entry is a CENSUS, not a sighting list: every # `crates/spt/tests/*.rs` was read mechanically for the two criteria this stanza already # states (an `.args(["daemon", "run"])` spawn, or a real brain tree via `supervise_brain` / # `spt daemon brain`) and cross-checked against the member list. ELEVEN qualifying binaries # were outside it: activity_link_push_e2e, coordinator_image_e2e, endpoint_autostart_e2e, # idle_edge_drain_e2e, idle_edge_seal_e2e, n1_compat, resident_service_e2e, # resume_no_control_steal_e2e, trial_drain_drive_e2e, twohost_cli, webserve_cross_node_e2e — # all eleven added to both strings (this one and golden.yml's job-level `HEAVY` env), # extracted and asserted byte-equal BEFORE and AFTER by the edit script. # TWELFTH, added by hand in WEBSERVE W2 and NOT by the criterion: webserve_attachment_e2e. # It spawns a daemon and no source pattern can see it — measured against # `check_heavy_integration_classification`'s own predicate, it scores ZERO on all three # literals because it autostarts through ORDINARY verbs (`send`, `serve list`, `fetch`). # The only daemon literal it carries is `["daemon", "stop"]`, its REAP. That is the shape # of the gap rather than an accident of this one file: a source criterion reads the place a # well-behaved rig is guaranteed to look innocent, and the rig that LEAKS carries no daemon # literal at all — so any future widening keys on a CREATION signal or on runtime, never on # the reap. Until then this name is held here by hand, and losing it silently reclasses a # daemon-spawning binary into the full-parallel pool. # NINE of the eleven carry NO sighting: they are classified by the criterion, which is what # HEAVY-AT-BIRTH says to do — a first flake is not the evidence the ruling waits for. Note # idle_edge_drain_e2e and idle_edge_seal_e2e were BOTH `SLOW [> 60.000s]` in that same Phase-A # pool, beside the two reds; that is consistent with starvation and is NOT offered as proof of # it. PRE-REGISTERED DISCRIMINATOR: serialized on the quiet box these eleven run one at a time — # a red THERE refutes the pool mechanism for that binary and re-opens the box/product question. # CORROBORATION found while censusing, not sought: `.github/ci/flake-registry.json` already # carries `spt::activity_link_push_e2e::activity_frames_reach_a_linked_shell_through_the_real_daemon` # as an ACTIVE linux flake (from red sha ffc7e9e, 2026-07-29) — a registered Linux flake that has # been a light-pool daemon-tree binary the whole time. It is one of the eleven. That is one more # reading of the same evidence, NOT a second mechanism, and the registry entry stays as it is: # retiring it needs the 30-green run it names, which this reclass has not yet bought. # WHAT THIS DOES NOT FIX: the 45s deadlines themselves are untouched (a budget retune is the # same race with a different number), and nothing here instruments WHERE the 45s went — no # stamp exists between the daemon child's spawn and the brain's first log line. [[profile.default.overrides]] filter = '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' test-group = 'heavy-broker-pty' # FLAKE-LEDGER: attach_viewport_reconnects_across_a_broker_bounce (W-3, todlando # 2026-07-06): the rc real-broker UNIT tests inside the # `spt` BINARY escape the integration-binary HEAVY filter above — they are `kind(bin)` # (the binary's own `#[cfg(test)]` module), not `kind(test)` — so they ran in the LIGHT # Phase-A pool at full parallelism. `rc::tests::attach_viewport_reconnects_across_a_broker_bounce` # (a real broker + a broker-bounce reconnect) then TIMED OUT @240s under a 360-way # `-p spt --bin spt` run; 1/1 ISOLATED green in ~1.3s — the 4th distinct full-parallel # trip, so the remedy is RECIPE hardening (serialize the class), not the test. Fold the # rc broker/bounce class into the same heavy group so it can never be starved again; # EXTEND this name set on each new sighting (the FLAKE-LEDGER discipline). [[profile.default.overrides]] filter = 'package(spt) & kind(bin) & test(/^rc::tests::(attach_viewport_reconnects_across_a_broker_bounce|broker_host_child|qualified_targets_attach_with_bare_wire_id|reconnect_class_only_severed_transport_redrives)$/)' test-group = 'heavy-broker-pty' # FLAKE-LEDGER: attach_viewport_reconnects_across_a_broker_bounce, SECOND HALF # (doyle 2026-07-20, from the W5 gate loops). That entry's # text names a CLASS — real-broker units that escape the integration-binary filter # because they are not `kind(test)` — and then enumerated four `rc::tests::` names. # THE ENUMERATION IS WHAT SHIPPED, so the identical shape in the spt-daemon LIB was # left in the light pool for a year: `applyhost::tests` stands up a real # `Broker::bind` + `serve()` thread in 10 of its 13 units (`served_broker`, # applyhost.rs), and `apply_staged_update_set_refuses_broker_touching_payloads` # duly TIMED OUT @240s on hfenduleam — at v0.32.0, where it was diagnosed as # exactly this and the remedy was never landed, and again under the W5 gate. # # THE DEFECT IS PROVABLE FROM THIS FILE ALONE, with no timing evidence: before # this stanza there were TWO overrides, neither matching `kind(lib)`. Timing only # ever estimated the RATE. Recorded because the seed was first filed the other way # round and had to be re-filed. # # SCOPED BY MODULE, NOT BY TEST NAME — deliberately, since name-scoping is the # defect above. `livehost::tests` (livehost.rs:~1700) and `pump::tests` # (pump/mod.rs:~2030) bind real brokers too; they were found only because fixing # `applyhost` alone would have repeated the attach_viewport_reconnects_across_a_broker_bounce # entry's mistake one package over. The cheap # units serialized alongside them cost far less than the next 240s timeout. # # `xtask check` now ENFORCES this (`check_heavy_unit_classification`): a lib module # that binds a real broker in its `#[cfg(test)]` block and is missing from this # filter fails the build. The rule stops depending on the next person reading it. # # DOORBELL W1 (todlando 2026-08-01): `broker::tests` joins the list — the engine-room # bring-up arc's end-to-end row stands up a REAL `Broker::bind_in` + `serve()` thread # and a REAL PTY child (the bring-up spawns the engine room through its bound adapter), # which is this class exactly. Classed in the SAME commit that introduces it rather # than after a Phase-A timeout, per the HEAVY-AT-BIRTH ruling. SCOPED BY MODULE, not by test # name, per this stanza's own ruling: the other 74 broker units are cheap and cost far # less serialized than the next 240s timeout costs. [[profile.default.overrides]] filter = 'package(spt-daemon) & kind(lib) & test(/^(applyhost|broker|livehost|pump)::tests::/)' test-group = 'heavy-broker-pty' # …and `wansend`, found BY THE CHECK ABOVE rather than by a person, which is the # whole argument for having it. `crates/spt/src/wansend.rs` binds two real brokers # and drives a production `run_dispatch_loop` in its `#[cfg(test)]` block — in the # `spt` BINARY, so it is neither `kind(test)` nor `spt-daemon`, and it sat outside # every filter. It already had a Phase-A sighting on 2026-07-16 # (`remote_rest_suspends_and_wakes_with_the_deferred_gate`, NoReply @1.82s vs # 0.28-0.32s solo) whose standing ruling was "move it into the heavy group IF IT # REPEATS". Classified now on the SHAPE instead — waiting for a second sighting is # how the first one gets re-argued as a flake (HEAVY-AT-BIRTH ruling: classify at birth). [[profile.default.overrides]] filter = 'package(spt) & kind(bin) & test(/^wansend::tests::/)' test-group = 'heavy-broker-pty' # LISTEN-HOSTING-TOPOLOGY (todlando 2026-07-21): `api::startup::tests` gains the # bind -> listen -> control -> detach regression, which binds a REAL broker and spawns a # REAL PTY child in the `spt` BINARY's own `#[cfg(test)]` block — the same class as the # two stanzas above, and `xtask check`'s `check_heavy_unit_classification` fails the # build without this entry. SCOPED BY MODULE, not by test name, per the # attach_viewport_reconnects_across_a_broker_bounce entry's second half: # the cheap startup units serialized alongside it cost far less than the next 240s # timeout, and a name list is the defect that ruling exists to stop repeating. [[profile.default.overrides]] filter = 'package(spt) & kind(bin) & test(/^api::startup::tests::/)' test-group = 'heavy-broker-pty' # ───────────────────────── Windows Phase-A parallelism bound ───────────────────────── # # LOAD-FLAKE FAMILY, cause-side leg 2 (todlando 2026-07-22; deployah option (c), # doyle-ratified). hfenduleam is a self-hosted runner that SHARES ITS HOST WITH THE LIVE # AGENT FLEET. Running the ~1881-test Phase-A battery at full parallelism there saturates # a box that is already carrying ~43 live spt-family processes, and the observed result is # not a flaky test — it is ONE environmental cause picking a RANDOM VICTIM per run: on # release PR #56, four runs over byte-identical code (zero `.rs` delta) killed a DIFFERENT # single daemon-spawning test each time (brain_decouple twice, then adapter_translate, then # adapter_digest), each dying PROCESS-LEVEL with a bare exit 1 and no assertion output while # its siblings in the same family passed alongside it. # # So Phase A gets a deliberate parallelism cut ON THE WINDOWS LEG ONLY. The Linux leg # (kitsubito) does not host the fleet and keeps full parallelism — it is a different box # with a different problem, and F-039's ruling applies here too: bound the rig, do not # loosen anything that would stop being a signal. # # WHY A PROFILE AND NOT A SECOND PHASE-A STEP: a per-OS pair of `cargo nextest run` steps # would put a FOURTH copy of the filter string in the tree, and this file's own # header already warns that keeping THREE in sync is the standing hazard. ci.yml instead # selects this profile with a step-level NEXTEST_PROFILE expression, so the filter string # stays at three copies. Phase B keeps running under `default`. # # NON-INHERITANCE, AND WHY THE FOUR OVERRIDES BELOW ARE COPIED (doyle gate-check # 2026-07-22 — the argument cuts both ways and he was right to demand it discharged): # a custom nextest profile inherits scalar settings from `default` (slow-timeout carries # over) but it DOES NOT inherit `profile.default.overrides`. Of the five overrides above, # exactly ONE — the by-binary integration filter — is `kind(test)` and therefore # lives entirely in Phase B, which never runs under this profile. The other FOUR are the # FLAKE-LEDGER: attach_viewport_reconnects_across_a_broker_bounce class: real-broker units # that are `kind(bin)`/`kind(lib)` and so escape # the integration-binary filter, which means THEY RUN IN PHASE A and depend on this group # to stay serialized. Dropping them here would have silently un-serialized on Windows the # exact class whose un-serialization caused two separate 240s timeouts. They are mirrored # verbatim below, and `xtask check`'s `check_phase_a_profile_parity` now FAILS THE BUILD if # a future non-`kind(test)` override is added to `default` without being mirrored — the # same posture as that entry's second half: the rule must not depend on the next person reading # this comment. # # [impl->REQ-CI-WINDOWS-PHASE-A-BOUND] [profile.ci-windows] # CHOSEN, NOT MEASURED — recorded in those words so the next person does not treat it as # an optimum. hfenduleam has 16 logical cores; 8 is half, leaving headroom for the live # fleet co-tenant. No measurement exists that separates 8 from 6 or 12, and none was run: # the ratified fix is "stop saturating the box", not "find the saturation knee". Retuning # is this one integer. test-threads = 8 [[profile.ci-windows.overrides]] filter = 'package(spt) & kind(bin) & test(/^rc::tests::(attach_viewport_reconnects_across_a_broker_bounce|broker_host_child|qualified_targets_attach_with_bare_wire_id|reconnect_class_only_severed_transport_redrives)$/)' test-group = 'heavy-broker-pty' [[profile.ci-windows.overrides]] filter = 'package(spt-daemon) & kind(lib) & test(/^(applyhost|broker|livehost|pump)::tests::/)' test-group = 'heavy-broker-pty' [[profile.ci-windows.overrides]] filter = 'package(spt) & kind(bin) & test(/^wansend::tests::/)' test-group = 'heavy-broker-pty' [[profile.ci-windows.overrides]] filter = 'package(spt) & kind(bin) & test(/^api::startup::tests::/)' test-group = 'heavy-broker-pty' --- #295 nextest.toml + golden.yml diff diff --git a/.config/nextest.toml b/.config/nextest.toml index 722c0ffe..903f464e 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -315,7 +315,7 @@ heavy-broker-pty = { max-threads = 1 } # same race with a different number), and nothing here instruments WHERE the 45s went — no # stamp exists between the daemon child's spawn and the brain's first log line. [[profile.default.overrides]] -filter = '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' +filter = '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|daemon_status_pid_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' test-group = 'heavy-broker-pty' # FLAKE-LEDGER: attach_viewport_reconnects_across_a_broker_bounce (W-3, todlando diff --git a/.github/workflows/golden.yml b/.github/workflows/golden.yml index f45c86cc..454493a7 100644 --- a/.github/workflows/golden.yml +++ b/.github/workflows/golden.yml @@ -155,7 +155,7 @@ jobs: # daemon tree, a real PTY child, and two genuine engine-room seats. Recorded # here for the same standing check — a per-phase count that shifts by more # than this is a repartition, not this lane. - HEAVY: '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' + HEAVY: '(package(spt-daemon) & kind(test) & binary(/^(attach|brain_resume_conn_deadlock|brain_swap|broker|budget|conn_blackhole_lifecycle|controller_lease|daemon_e2e|daemon_lifecycle_real_brain|digest|dispatch|driven_by_selfheal|endpoint_lifecycle|endpoint_survival|exit_every_reap|handoff|idempotent|inject_control_wedge|input_ack_deadlock|mesh_recovery|net_worker_starve|redispatch|redispatch_stall|registry_lifecycle|render_lifecycle|resize_geometry_epoch|resize_presentation_barrier|restart_replay_lifetime|resume|shellchan|spawn_truth|transport_death_eof|two_origin_spanning|twohost|twohost_web)$/)) | (package(spt) & kind(test) & binary(/^(activity_link_push_e2e|attach_link_push_e2e|attach_wedge_e2e|bind_cwd_project_e2e|bind_honest_cross_perch_e2e|brain_respawn_rename|brain_split|brain_survive|commune_two_cwd_e2e|coordinator_image_e2e|daemon_refresh_e2e|daemon_status_pid_e2e|dummy_harness_e2e|endpoint_autostart_e2e|endpoint_teardown_authority_e2e|engine_room_bringup_e2e|er_brief_once_per_session_e2e|er_briefing_presentation_e2e|er_briefing_presented_e2e|er_briefing_session_scoped_e2e|er_sequestered_cwd_e2e|gateway_owner_shell_e2e|idle_edge_drain_e2e|idle_edge_seal_e2e|job_escape_e2e|knock_approve_not_hostage_e2e|knock_mutual_cross_node_e2e|live_adapt_translation_swap_e2e|livehost_bootgate_e2e|multi_subnet_bringup_e2e|n1_compat|n1_pairing|oneliner_e2e|projindex_reader_e2e|projindex_writer_e2e|psyche_download_e2e|rc_attach_truth|resident_service_e2e|resume_no_control_steal_e2e|resume_template_e2e|run_no_dup_session_e2e|trial_drain_drive_e2e|twohost_cli|wake_resume_bind_e2e|webserve_attachment_e2e|webserve_cross_node_e2e)$/)) | (package(spt-term) & kind(test))' # A wedged test must not block the single self-hosted runner slot (the # 2026-06-03 handoff.rs ConPTY stall burned 22 hosted minutes unbounded). # 80 (was 50; the note this replaces narrated kitsubito at 25 -> 50): --- conn_blackhole_lifecycle.rs header @19a90979 //! MSG-IDENTITY W6 / F-039 leg (e) — the DETERMINISTIC BLACK-HOLED-CONTROLLER //! LIFECYCLE HARNESS (REQ-CONN-BLACKHOLE-LIFECYCLE-HARNESS), on a REAL broker //! hosting REAL PTY children over the real local IPC. //! //! ## What this is //! //! This is hertz's RCA fix-shape item 5 made executable (`.claude/ //! hertz-pty-freeze-rca.md` §Proposed fix shape, item 5): the standing //! conformance rig for the round-4 `SharedSend` fix class. It reproduces the //! EXACT Windows named-pipe black-hole (a controller that stops draining while a //! finite burst floods its socket, so the broker's writer parks inside //! `write_frame` on `WriteFileEx`+`SleepEx(INFINITE, alertable)` — no supported //! write timeout) and, against CURRENT v0.30.6 semantics (`crates/spt-daemon/ //! src/conn.rs` bounded/cancelable/poison-on-failure writes, landed fc048bb), //! asserts hertz's FIVE invariants for ONE incident: //! //! 1. UNRELATED SESSIONS CONTINUE — a second, independent session's viewer keeps //! receiving output throughout the black-hole incident (the fan-out never //! blocks the whole broker on the one wedged conn); //! 2. THE BAD PHYSICAL CONNECTION IS CANCELED/CLOSED WITHIN THE BOUND — a write //! asserts, and the deterministic black-hole only reproduces on the Windows //! named-pipe path (`interprocess 2.4.2` → `WriteFileEx`+`SleepEx(INFINITE)`, //! `set_timeout` → `Unsupported`) — the exact field failure. On Unix the //! poison/retire path is a `shutdown(SHUT_RDWR)` abort of a buffer-fill block: a //! genuinely DIFFERENT mechanism, not this incident. Faking a Unix companion //! would test a different thing, so — mirroring the r4 gate's own //! `#[cfg(windows)]` leg — the entire harness is Windows-only. On other targets //! this test binary compiles to zero tests (no fakery). //! //! ## RUNNER CONTRACT — run process-per-test (nextest) or single-threaded //! //! Shares ONE process-global `SPT_HOME` + `SPT_BRAIN_WRITE_DEADLINE_MS`, spins a //! REAL broker + TWO PTY children. HEAVY class. Run isolated with //! `cargo nextest run -p spt-daemon --test conn_blackhole_lifecycle` (CI's path) //! or `cargo test --test conn_blackhole_lifecycle -- --test-threads=1`. NEVER a //! bare `cargo test --test conn_blackhole_lifecycle` on Windows. Every read in //! this test rides a bounded thread+channel or a polled deadline; the test //! captures plain values as it goes and only asserts AFTER teardown, so it can //! never hang even on broken code. #![cfg(windows)] use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; use std::sync::Arc; use std::thread; use std::time::{Duration, Instant}; use interprocess::local_socket::Stream; use spt_daemon::codec::{read_frame, write_frame}; use spt_daemon::frame::{Envelope, Role}; use spt_daemon::msg::{ decode_bytes, encode_bytes, InputReq, OutputEvent, SessionsReply, SpawnReq, Spawned, StallEvictsReply, SubscribeOutcome, SubscribeReq, SubscribedReply, KIND_INPUT, KIND_OUTPUT, KIND_SESSIONS, KIND_SESSIONS_REPLY, KIND_SPAWN, KIND_SPAWNED, KIND_STALL_EVICTS, KIND_STALL_EVICTS_REPLY, KIND_SUBSCRIBE, KIND_SUBSCRIBED, }; use spt_daemon::transport::{send_hello, LocalSocketTransport}; use spt_daemon::Broker; use spt_net::net::attach::AttachIntent; use spt_test_support::TestHome; static SEQ: AtomicU32 = AtomicU32::new(0); fn unique_name() -> String { let n = SEQ.fetch_add(1, Ordering::Relaxed); format!("spt-daemon-blackhole-lc-{}-{}.sock", std::process::id(), n) } fn init_home() -> TestHome { TestHome::new() } fn kill_pid(pid: u32) { --- test fns 3:101:fn unique_name() -> String { 7:106:fn init_home() -> TestHome { 11:110:fn kill_pid(pid: u32) { 15:116:fn send(conn: &mut Stream, kind: &str, payload: serde_json::Value) { 19:120:fn connect(name: &str) -> Stream { 23:134:fn flood_spawn_req(endpoint: &str) -> SpawnReq { 27:158:fn seed_burst_spawn_req(endpoint: &str) -> SpawnReq { 31:180:fn spawn_session(name: &str, req: SpawnReq) -> (Stream, u64) { 35:196:fn stall_evict_count(name: &str, deadline: Duration) -> u64 { 39:224:fn query_sessions(name: &str, deadline: Duration) -> Option { 43:247:fn blackholed_controller_lifecycle_five_invariants() {