+ src.join("manifest.toml"), + format!( + "[adapter]\nname = \"mock-owner-online\"\nkind = \"shell\"\nversion = \"1\"\n\ + min_spt_core_version = \"0\"\n\n[shell]\nspawn = '{noop}'\n\ + persistent = true\n" + ), + ) + .unwrap(); + spt_runtime::registry::register(&adapters_dir, &src, 1).unwrap(); + let registered = spt_runtime::registry::registered(&adapters_dir); + + // The restart shape, built with NO suspend/rest path anywhere: recorded + // ONLINE over a dead pid, with a parked launch stamp. + let mk = |launched_ms: u64| -> String { + let id = spawn_record(owlery, "ling", "mock-owner-online", None).unwrap(); + let perch = spt_store::perch::resolve_shell_perch_path_in(owlery, "ling", &id); + let mut i = shellinfo::read_shell_info(&perch).unwrap(); + i.status = SHELL_STATUS_ONLINE.to_string(); + shellinfo::write_shell_info(&perch, &i).unwrap(); + std::fs::write(perch.join(shellinfo::SHELL_PID_FILE), "2000000000").unwrap(); + let launch = shellinfo::ShellLaunch { + pid_started_at: Some(1), + launched_ms, + }; + std::fs::write( + perch.join(shellinfo::SHELL_LAUNCH_FILE), + serde_json::to_string(&launch).unwrap(), + ) + .unwrap(); + id + }; + let stranded = mk(boot.saturating_sub(600_000)); // 10 min BEFORE boot + let force_killed = mk(boot + 120_000); // 2 min AFTER boot + + // The daemon's own order: seed the edge, then spend the sweep. + let mut edge = OwnerOnlineEdge::new(); + edge.seed(owlery); + let swept = restore_persistent_shells_at_boot(owlery, ®istered, &adapters_dir); + assert!( + swept.is_empty(), + "the sweep is SPENT and restored nothing: its owner-online conjunct \ + refused, correctly, and it never runs again this generation: {swept:?}" + ); + + // The owner comes online. No rest edge is emitted by bringup — this write is + // the endpoint record changing, nothing more. + std::fs::write(owner_dir.join("info.json"), "{}").unwrap(); + + let set = Arc::new(WakeSet::new()); + let restored = + restore_persistent_shells_on_owner_online(owlery, ®istered, &adapters_dir, &mut edge, &set); + + assert!( + restored.contains(&format!("ling/{stranded}")), + "the owner came online after the sweep, so THIS is the trigger that owes \ + the restore — the seam releases#228 reported: {restored:?}" + ); + assert!( + !restored.contains(&format!("ling/{force_killed}")), + "an instance launched THIS boot is a force-kill, refused by the same \ + discriminant the sweep uses — re-evaluating can only refuse MORE: \ + {restored:?}" + ); + + // ...and the trigger is an EDGE: the owner stays online, a NEW casualty is + // stranded, and the next pass does nothing. A level trigger would take it. + let late = mk(boot.saturating_sub(600_000)); + let again = + restore_persistent_shells_on_owner_online(owlery, ®istered, &adapters_dir, &mut edge, &set); + assert!( + again.is_empty(), + "no NEW owner-online transition ⇒ no restore attempt at all; this is what \ + bounds a failing launch to one attempt per owner-online event instead of \ + one every five seconds forever (ling/{late} stayed down): {again:?}" + ); + } + fn fast_params() -> WakeParams { WakeParams { backoff_base_ms: 10, =====DOCS===== diff --git a/CONTEXT.md b/CONTEXT.md index fecd99c0..3503eaef 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -383,7 +383,8 @@ The manifest value is the **floor**: a node/endpoint setting may **tighten** (de A shell automatically goes **offline when its owner endpoint goes offline**. Two complementary manifest options bring it (and its owner) back: -- **`persistent`** (bool): the shell is automatically brought **online whenever its owner endpoint is online**. Covers "owner already up." **Node-qualified once instances span nodes** (ruled 2026-07-24, #70 grill): when an owner comes online at a given node, only the **latest** instances qualified to that *endpoint@node* pairing come online with it — a stale instance of the same shell left on another node does not auto-wake alongside. **A node restart is covered by a boot sweep, not by the owner's own online edge** (BAROMETER W2, releases#78): a machine death breaks no link, so the instance's record survives the reboot still saying `online` and the wake cascade — which reads the *recorded* field — skips it, stranding every `persistent` instance on the node at once. The daemon therefore (i) heals a recorded `online` its derived status contradicts, every reconcile cycle, writing only when the value actually changes, and (ii) runs a **once-per-daemon-generation boot sweep** that relaunches an instance when **all four** conditions hold: the adapter declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing); the instance is down **in fact**; and its **recorded launch predates the boot instant**. A launch parks a **birth stamp** beside the pid to make that last question answerable — **no stamp ⇒ not restored** (the sweep cannot prove the corpse predates boot, and leaving a shell down beats relaunching a binary an operator may have killed deliberately), and a platform with no boot oracle restores nothing. A shell force-killed *during* this boot is launched after it, can never satisfy the predicate, and so is never spontaneously relaunched — the deploy/quarantine ruling of KNOWN-HAZARDS 2.6 preserved by construction. +- **`persistent`** (bool): the shell is automatically brought **online whenever its owner endpoint is online**. Covers "owner already up." **Node-qualified once instances span nodes** (ruled 2026-07-24, #70 grill): when an owner comes online at a given node, only the **latest** instances qualified to that *endpoint@node* pairing come online with it — a stale instance of the same shell left on another node does not auto-wake alongside. **A node restart is covered by a boot sweep, and an owner that comes online after that sweep by an owner-online edge** (BAROMETER W2 releases#78; CONDUIT W2 releases#228): a machine death breaks no link, so the instance's record survives the reboot still saying `online` and the wake cascade — which reads the *recorded* field — skips it, stranding every `persistent` instance on the node at once. The daemon therefore (i) heals a recorded `online` its derived status contradicts, every reconcile cycle, writing only when the value actually changes, and (ii) runs a **once-per-daemon-generation boot sweep** that relaunches an instance when **all four** conditions hold: the adapter declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing); the instance is down **in fact**; and its **recorded launch predates the boot instant**. A launch parks a **birth stamp** beside the pid to make that last question answerable — **no stamp ⇒ not restored** (the sweep cannot prove the corpse predates boot, and leaving a shell down beats relaunching a binary an operator may have killed deliberately), and a platform with no boot oracle restores nothing. A shell force-killed *during* this boot is launched after it, can never satisfy the predicate, and so is never spontaneously relaunched — the deploy/quarantine ruling of KNOWN-HAZARDS 2.6 preserved by construction. + **Daemons boot before endpoints do, so the sweep is not the only trigger** (releases#228): an owner that is not yet online when the sweep runs is skipped by that very conjunct, and — the sweep being once per daemon generation, and bringup from offline emitting no rest edge — nothing revisited it, so its shells stayed down until a human relinked. The reconcile loop therefore runs the **same four conditions** for every owner that has gone **offline→online since the last pass**. The trigger is that **edge, never the level “the owner is online”**: a successful restore restamps itself out of eligibility, but a *failing* launch under a level trigger would be retried every reconcile tick for as long as the owner stayed up, where the edge bounds it to one attempt per owner-online event. Re-evaluating later in a generation can only refuse **more**, never less — every launch path restamps the birth stamp, a kill never does, and the boot instant is fixed for the machine boot — so the force-kill ruling above is preserved identically on the second trigger. - **`wake_command`** (template): a long-running **wake-watcher** process spt-core runs *while the shell is offline* **and eligible** (the eligibility rule is below — an offline instance sitting on a same-boot corpse arms no watcher). Its sole job is to fire a wake. It runs on the **shell's node** (where the platform wake-event originates — e.g. interacting with a "shut-down" avatar). Covers "owner is down, wake it from outside." **Online/offline are mutually-exclusive processes:** online ⇒ the shell binary runs (no watcher); offline ⇒ the wake-watcher runs (no shell binary). spt-core flips between them. **Offline does not by itself arm the watcher — a third outcome exists** (BAROMETER W2, releases#78): an instance that is offline over a **corpse launched during this boot** (a force-kill, a crash) is deliberately held out of watcher eligibility, so *neither* process runs until recovery is demanded — `relink`, or a `shell cmd` that wakes. Eligibility is recorded-`offline` **AND** (no corpse, **or** the corpse's launch predates the boot instant); an instance whose corpse cannot be dated — no birth stamp, or no boot oracle on the platform — is treated as ineligible rather than guessed at. Only a cleanly-closed instance and a **restart casualty** arm a watcher. The reason is the deploy case: on Windows an operator kills a shell precisely to free its exe for overwrite, and a watcher armed there would re-lock the file under them. diff --git a/docs-site/src/harness-contract/manifest.md b/docs-site/src/harness-contract/manifest.md index 2b4fd4d9..e8f53337 100644 --- a/docs-site/src/harness-contract/manifest.md +++ b/docs-site/src/harness-contract/manifest.md @@ -687,7 +687,7 @@ broadcast = "subnet" # "subnet" | "same-node" | "none" (discovery scop command_receipt = "stdin" # "http" | "stdin" | "relay" (how commands arrive) pre_close = "park-and-save" # optional instruction sent on link-break close_timeout_ms = 3000 # graceful-termination window -persistent = true # auto-online whenever the owner endpoint is online; across a node restart a boot sweep restores it (owner online + launch predates boot; an instance with no launch stamp is left down) +persistent = true # auto-online whenever the owner endpoint is online; across a node restart a boot sweep restores it, and an owner that comes online after that sweep restores it on that transition (owner online + launch predates boot; an instance with no launch stamp is left down) wake_command = "my-waker --link {link_token}" # wake-watcher, run while offline AND eligible — an offline instance sitting on a same-boot corpse arms no watcher and waits for relink. Exit code 86 = wake. Program token + {adapter_dir} resolve like spawn (since v0.44.0); {perch_dir} never fills here can_shutdown = false # may the shell fire `api owner-shutdown`? require_approval = "none" # "none" | "remembered" | "always" (per-spawn gate) diff --git a/docs-site/src/shells/overview.md b/docs-site/src/shells/overview.md index 2d61f9ff..b80a1aec 100644 --- a/docs-site/src/shells/overview.md +++ b/docs-site/src/shells/overview.md @@ -95,15 +95,22 @@ with `can_shutdown = true` may suspend its own owner (`api owner-shutdown`) Two lifecycle facts worth knowing before you rely on either: -- **A node restart brings `persistent` instances back by a boot sweep**, not by - the owner's own online edge. A machine death breaks no link, so the instance's - record survives the reboot claiming it is still up; the daemon corrects that - record and then, once per daemon generation, relaunches an instance whose - adapter is `persistent`, whose **owner endpoint is online**, that is down in - fact, and whose recorded launch **predates the boot instant**. An instance last - launched before this behavior shipped carries no launch stamp, so the sweep - cannot date its corpse and leaves it down — it is covered from its next launch - onward. +- **A node restart brings `persistent` instances back by a boot sweep, and an + owner that comes online after that sweep by an owner-online edge.** A machine + death breaks no link, so the instance's record survives the reboot claiming it + is still up; the daemon corrects that record and then, once per daemon + generation, relaunches an instance whose adapter is `persistent`, whose **owner + endpoint is online**, that is down in fact, and whose recorded launch + **predates the boot instant**. An instance last launched before this behavior + shipped carries no launch stamp, so the sweep cannot date its corpse and leaves + it down — it is covered from its next launch onward. Daemons start before + endpoints do, so those same four conditions are re-run for **any owner that + goes offline→online** between reconcile passes: without that, a shell whose + owner was still down when the sweep ran would stay down until someone relinked + it by hand. It is that transition — not the standing state "the owner is + online" — that triggers the retry, which is what keeps a launch that keeps + failing to one attempt per owner-online event rather than one every reconcile + tick. - **Offline alone does not arm the watcher.** A shell force-killed (or crashed) *during the current boot* is offline with no watcher running, and comes back only on demand — `spt shell relink `, or a `shell cmd` that wakes it. This diff --git a/docs/KNOWN-HAZARDS.md b/docs/KNOWN-HAZARDS.md index 85c9b48c..07b9aa6b 100644 --- a/docs/KNOWN-HAZARDS.md +++ b/docs/KNOWN-HAZARDS.md @@ -85,9 +85,9 @@ Hard-won edge cases harvested from the sister project (`claude_skill_owl`, ~80 c - **Failure:** the contract's own sentence says a `persistent` shell is **online whenever its owner endpoint is online**, and a node restart is the one case where that promise was never kept. A machine death breaks no link, so `close_shell` never runs and 2.6's stale record survives the reboot; when the owner comes back online, the wake cascade reads the **recorded** field, sees `online`, and skips the relaunch of a binary that has not existed since the previous boot. Nothing else corrects it: the instance is down forever, with no error anywhere, until a human happens to run `relink`. Every `persistent` instance on the node is stranded by one restart, together. - **The owner-facing surfaces HIDE the fault**, which is why it survived unreported for days: every display path (`shell list`, its `--json` twin, the shell-context render) routes through the ONE discovery seam, which **derives** status (2.6) and so correctly reports the binary as offline — while the only consumer whose decision matters, the wake cascade, reads the RECORDED field that no owner-facing surface shows. Absence of a visible symptom is therefore **not** evidence of absence of this fault, and a clean-looking context render must never be taken as proof the class did not occur. The correct falsifier is the **on-disk record**. - **The existing cascade test cannot fail on it — its SETUP SUPPRESSES THE FAILING ARM.** `rest_edges_cascade_shells_with_divergence` establishes its fixture by driving the suspend path, which itself writes the `offline` the code under test is supposed to encounter — so the restart shape (a record still saying `online` over a corpse, reached with **no** suspend edge) is unreachable from that fixture BY CONSTRUCTION. The test is not weak; it gives a true answer to the wrong question, so a reader asking "is the cascade covered?" is told yes. This is the third suppressed-arm instance found in one week, and the pattern is the same each time: **a fixture that establishes the precondition the code under test is supposed to establish.** -- **Invariant:** a node restart must not strand a `persistent` instance. Two legs, and each is worthless alone. **(a) The record stops lying:** the daemon-side reconciler heals a recorded `online` that 2.6's derivation contradicts, writing `offline` — every cycle, **guarded on an actual change** (a heal that rewrote each tick would be a stream of identical writes and would destroy the record's mtime as a signal). **(b) The restart shape reaches a restoration path:** a once-per-daemon-generation boot sweep relaunches an instance when **all four** hold — the adapter section declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing — that is the line between healing a record and resurrecting a shell nobody is there to drive); the instance is down **in fact** (the derived read, made trustworthy across a pid reuse by 2.6's birth stamp); and its recorded launch **predates the boot instant**. No stamp ⇒ **not restored** (the sweep cannot prove it predates boot); no boot oracle on the platform ⇒ nothing restored at all. The boot instant is derived (Linux `btime`; Windows now − `GetTickCount64`), so a named slack constant absorbs derivation jitter and keeps a process launched moments after boot from reading as a corpse from the previous one. +- **Invariant:** a node restart must not strand a `persistent` instance. Two legs, and each is worthless alone. **(a) The record stops lying:** the daemon-side reconciler heals a recorded `online` that 2.6's derivation contradicts, writing `offline` — every cycle, **guarded on an actual change** (a heal that rewrote each tick would be a stream of identical writes and would destroy the record's mtime as a signal). **(b) The restart shape reaches a restoration path:** a once-per-daemon-generation boot sweep — and, for an owner that comes online after that sweep has run, an **owner offline→online edge** in the reconcile loop running the identical body (releases#228) — relaunches an instance when **all four** hold — the adapter section declares `persistent`; the **owner endpoint is online** (an offline owner is owed nothing — that is the line between healing a record and resurrecting a shell nobody is there to drive); the instance is down **in fact** (the derived read, made trustworthy across a pid reuse by 2.6's birth stamp); and its recorded launch **predates the boot instant**. No stamp ⇒ **not restored** (the sweep cannot prove it predates boot); no boot oracle on the platform ⇒ nothing restored at all. The boot instant is derived (Linux `btime`; Windows now − `GetTickCount64`), so a named slack constant absorbs derivation jitter and keeps a process launched moments after boot from reading as a corpse from the previous one. **The once-ness of the sweep is scheduling, not safety** (ruled releases#228): the predates-boot discriminant is what forbids relaunching a force-killed shell, and it is monotone within a machine boot — every launch restamps, a kill never does — so a second trigger evaluating the same conjuncts can only refuse more. The second trigger is an **edge**, not the level “owner is online”, which is what bounds a failing launch to one attempt per owner-online event instead of one per reconcile tick. - **Regression shape (the row the suppressed fixture cannot express):** construct the restart shape **without a suspend edge** — an instance whose record says `online` over a corpse that predates boot, with an online owner — and assert it is restored. A fixture that suspends first re-creates the suppressed arm and proves nothing about this class. -- **spt-core mapping:** `spt_daemon::shellwake::{heal_stale_online_records, restore_persistent_shells_at_boot, launch_predates_boot, BOOT_RESTORE_SLACK_MS}`; `spt_store::proc::boot_instant_ms`; `spt_store::shellinfo::{ShellLaunch, record_shell_launch, read_shell_launch}` (the birth stamp, parked at the two production pid-write sites the launch-site census identified). +- **spt-core mapping:** `spt_daemon::shellwake::{heal_stale_online_records, restore_persistent_shells_at_boot, restore_persistent_shells_on_owner_online, OwnerOnlineEdge, launch_predates_boot, BOOT_RESTORE_SLACK_MS}`; `spt_store::proc::boot_instant_ms`; `spt_store::shellinfo::{ShellLaunch, record_shell_launch, read_shell_launch}` (the birth stamp, parked at the two production pid-write sites the launch-site census identified). - **Source:** spt-core, BAROMETER W2 (releases#78) — field specimen `liam/alchemy-0`, a genuinely stranded record: `online` over a dead pid, no launch stamp, days old. @@ -990,7 +990,7 @@ The kill-path rule above generalizes: `daemon.pid` is not authority for *"which | 2.4 | gen_start = now() on cold-start + handoff | per-instance generation | | 2.6 | A shell's ONLINE-ness is DERIVED (recorded status AND a not-provably-dead `shell.pid`) — an abruptly-killed binary breaks no link, so `close_shell`'s offline flip never runs and the record lies forever; derive at the gates/renders, keep the recorded field at the writers, the suspend-cascade close, and the wake reconciler — whose "no spontaneous relaunch" (deploys and quarantine own that decision) is since W2 the stated corpse-predates-boot predicate, not the stale record holding it open, and whose recycled-pid blindness is closed by the launch birth stamp wherever one is parked. Not only a crash edge: the GRACEFUL daemon-stop path kills bound shells at stop-begin with `close_shell` unrun (field 2026-07-25 — shells died abruptly, `info.json` still `online`, while the broker drained on 2m10s), so every daemon restart manufactures these stale records routinely; the derivation heals them at the gates | `spt_store::shellinfo::{shell_pid_provably_dead,effective_status,is_shell_online}`, `linkhost` relink/drive/cmd-wake, `shelldisc::discover`, `activity::observe_links` | -| 2.7 | A node restart must not permanently strand every `persistent` shell: (a) the daemon heals a recorded `online` the derivation contradicts, write-guarded on an actual change, and (b) a once-per-generation boot sweep relaunches an instance only when ALL of — adapter `persistent`, owner endpoint online, down in fact, recorded launch predates the boot instant (slack-absorbed) — hold; no stamp or no boot oracle ⇒ not restored. The owner-facing surfaces derive and look clean while the cascade reads the recorded field, so a clean render is NOT evidence the class did not occur — the falsifier is the on-disk record. The pre-existing cascade test's setup suppresses the failing arm (it suspends first), so the restart shape must be asserted on a fixture that never calls the suspend path | `shellwake::{heal_stale_online_records,restore_persistent_shells_at_boot,launch_predates_boot}`, `proc::boot_instant_ms`, `shellinfo::{record_shell_launch,read_shell_launch}` | +| 2.7 | A node restart must not permanently strand every `persistent` shell: (a) the daemon heals a recorded `online` the derivation contradicts, write-guarded on an actual change, and (b) a once-per-generation boot sweep PLUS an owner offline->online edge in the reconcile loop (releases#228 — the sweep alone strands the shells of any owner not yet online when it ran) relaunch an instance only when ALL of — adapter `persistent`, owner endpoint online, down in fact, recorded launch predates the boot instant (slack-absorbed) — hold; no stamp or no boot oracle ⇒ not restored. The owner-facing surfaces derive and look clean while the cascade reads the recorded field, so a clean render is NOT evidence the class did not occur — the falsifier is the on-disk record. The pre-existing cascade test's setup suppresses the failing arm (it suspends first), so the restart shape must be asserted on a fixture that never calls the suspend path | `shellwake::{heal_stale_online_records,restore_persistent_shells_at_boot,restore_persistent_shells_on_owner_online,OwnerOnlineEdge,launch_predates_boot}`, `proc::boot_instant_ms`, `shellinfo::{record_shell_launch,read_shell_launch}` | | 3.1 | Ephemeral perch cleanup on all exit paths | `ring` (RAII guard) | | 3.4 | A ring never adopts (so never deletes) a perch it did not create — probe the DIR, not the ready marker; record/spool/unreadable = occupied, empty = refused too; deliver + loud `RING_PERCH_EXISTS`/`RING_STALE_DIR` instead of block-waiting | `spt_msg::ring` probe + `create_dir` leaf | | 3.5 | A perch GC classifies on record PRESENCE only and never asks a liveness resolver — `is_perch_alive` is INVERTED here (recordless residue reads ALIVE, an offline endpoint reads DEAD: 24 vs 6 measured on HFENDULEAM 2026-08-04), so `!is_perch_alive` authorizes exactly the inverse set; the registry is never asked either (offline endpoints are legitimately absent from it) | `spt_store::perchgc::sweep`, `spt endpoint gc` | diff --git a/docs/MANIFEST.md b/docs/MANIFEST.md index b05f66a5..927bc5a9 100644 --- a/docs/MANIFEST.md +++ b/docs/MANIFEST.md @@ -720,7 +720,7 @@ broadcast = "subnet" # subnet | same-node | none command_receipt = "http" # http | stdin | relay (how it receives agent commands) pre_close = "park-and-save" # optional instruction sent to the binary on link-break close_timeout_ms = 3000 # graceful termination window before force-close -persistent = true # auto-online whenever the owner endpoint is online; a node restart is covered by the boot sweep (see Sleep/wake) +persistent = true # auto-online whenever the owner endpoint is online; a node restart is covered by the boot sweep, and an owner coming online after it by the owner-online edge (see Sleep/wake) wake_command = "gamerobot-waker --link {link_token}" # long-running wake-watcher run WHILE offline AND eligible; exit(wake-opcode) ⇒ revive can_shutdown = false # if true, the shell may fire `api owner-shutdown` to suspend its owner directly require_approval = "none" # none | remembered | always — per-spawn user-approval gate (floor; node/endpoint may tighten) @@ -739,7 +739,7 @@ types = ["image", "sound", "event"] # "event" = arbitrary descriptive payloads The capability + sensory vocabularies live **here**, not on the perch — spt-core resolves them by `adapter_name`. Command-delivery (`command_receipt`) reuses the agent inject-input modes. The shell binary binds via `api bind` (type=Shell, owner from the link) and pushes sensory via `api emit --type `. -**Sleep/wake:** when offline **and eligible** (+ `wake_command` set), spt-core runs the wake-watcher (mutually exclusive with the shell binary); `exit(wake-opcode)` → wake resolution; crash-exit → respawn with exponential backoff + give-up (until the next relink/launch). **The wake opcode is exit code 86.** The wake-watcher template fills `{id}`, `{adapter_name}`, `{link_token}`, `{adapter_dir}` (NOT `{perch_dir}` — that key is spawn-only; the waker fill site carries no perch context), and its `{link_token}` is minted *unparked* — an offline link has no live credential (the close retired it); a waker wakes by exit code, never by driving the link. **Eligibility (BAROMETER W2, releases#78):** a watcher is armed for an instance recorded `offline` that either carries no corpse (cleanly closed, or never launched) or whose corpse's recorded launch **predates the current boot** — a restart casualty. An instance whose binary died **during this boot** (force-kill, crash) arms no watcher and returns only on demand (`relink`, or a waking `shell cmd`), because killing a shell to overwrite its exe is a routine deploy step and a watcher armed underneath would re-lock the file; an instance whose corpse cannot be dated (no launch stamp, or no boot oracle on the platform) is treated the same way rather than guessed at. `persistent` auto-onlines the shell with its owner, and across a **node restart** — where nothing breaks the link, so the instance's record survives the reboot still claiming it is up — the daemon heals that record and a **once-per-daemon-generation boot sweep** relaunches the instance if its adapter is `persistent`, its owner endpoint is online, it is down in fact, and its recorded launch predates the boot instant. `can_shutdown` authorizes `api owner-shutdown`. Cross-node *fresh-spawn* to wake an owner requires the owner's `shell_wake_spawn_anywhere` settings flag (pre-consent). +**Sleep/wake:** when offline **and eligible** (+ `wake_command` set), spt-core runs the wake-watcher (mutually exclusive with the shell binary); `exit(wake-opcode)` → wake resolution; crash-exit → respawn with exponential backoff + give-up (until the next relink/launch). **The wake opcode is exit code 86.** The wake-watcher template fills `{id}`, `{adapter_name}`, `{link_token}`, `{adapter_dir}` (NOT `{perch_dir}` — that key is spawn-only; the waker fill site carries no perch context), and its `{link_token}` is minted *unparked* — an offline link has no live credential (the close retired it); a waker wakes by exit code, never by driving the link. **Eligibility (BAROMETER W2, releases#78):** a watcher is armed for an instance recorded `offline` that either carries no corpse (cleanly closed, or never launched) or whose corpse's recorded launch **predates the current boot** — a restart casualty. An instance whose binary died **during this boot** (force-kill, crash) arms no watcher and returns only on demand (`relink`, or a waking `shell cmd`), because killing a shell to overwrite its exe is a routine deploy step and a watcher armed underneath would re-lock the file; an instance whose corpse cannot be dated (no launch stamp, or no boot oracle on the platform) is treated the same way rather than guessed at. `persistent` auto-onlines the shell with its owner, and across a **node restart** — where nothing breaks the link, so the instance's record survives the reboot still claiming it is up — the daemon heals that record and a **once-per-daemon-generation boot sweep** relaunches the instance if its adapter is `persistent`, its owner endpoint is online, it is down in fact, and its recorded launch predates the boot instant. Because daemons boot before endpoints do, the same four conditions are re-run for any owner that goes **offline→online since the last reconcile pass** (releases#228) — without that edge, an owner that was still down when the sweep ran keeps its shells down until someone relinks by hand. It is an edge and not the level “the owner is online” so that a failing launch is attempted once per owner-online event rather than once per tick. `can_shutdown` authorizes `api owner-shutdown`. Cross-node *fresh-spawn* to wake an owner requires the owner's `shell_wake_spawn_anywhere` settings flag (pre-consent). **Instantiation governance:** `require_approval` gates `shell spawn` (reusing the consent grant store — `remembered` persists a grant via allow-always, `always` prompts every spawn); `max_instances_per_owner` + `over_cap` cap how many instances one owner may hold (online + offline both count toward the cap). An instance's **alias** is **not** a manifest field — it is per-instance runtime state on the shell perch's `info.json` (set at `shell spawn --alias` / `shell rename`), since the manifest describes the *adapter*, not individual instances. diff --git a/traceable-reqs.toml b/traceable-reqs.toml index e6c82faa..66bc4a81 100644 --- a/traceable-reqs.toml +++ b/traceable-reqs.toml @@ -4105,6 +4105,11 @@ id = "REQ-SHELL-PERSISTENT-BOOT-RESTORE" title = "A `persistent` SHELL COMES BACK AFTER A NODE RESTART, WHICH IS THE ONE CASE THE CONTRACT'S OWN SENTENCE PROMISES AND THE IMPLEMENTATION NEVER HANDLED. CONTEXT.md states it plainly -- a persistent shell is automatically brought online whenever its owner endpoint is online -- and the shipped code honors that across a graceful suspend->wake rest edge ONLY. A machine restart produces no such edge and there is no boot-time restoration at all, so after a reboot every persistent shell on the node stays down until a human runs `relink` by hand, with no surface saying so. Field, HFENDULEAM 2026-08-01: the fleet came up at 01:11 and NOT ONE persistent shell came back; the two instances that did were relaunched by a human at 01:27, sixteen minutes later. TWO INDEPENDENT GATES, both of which must fall or the promise stays broken. GATE 1, no restoration path: the five launch sites all require either an explicit command, a rest edge, or a `wake_command` the adapter may deliberately not declare, and daemon boot starts only the servicehost sweep and the wake reconciler. GATE 2, the stale record blocks the cascade even when an edge DOES arrive: the to_active arm gates on the RECORDED status being offline, and a machine death breaks no link, so the close that would write offline never runs and the record says online over a corpse forever. LEG (a) HEALS THE RECORD TO THE TRUTH, with a BIRTH-STAMPED probe rather than bare pid liveness -- a pid namespace resets at boot, so a corpse pid can be REUSED by an unrelated process, and a reused pid reads alive, makes the derived status read online, and makes `relink` refuse SHELL_ALREADY_ONLINE: the one command that recovers the instance refusing on the very state that is wrong. TWO STAMPS ARE PARKED, NOT ONE, AND THE REASON IS A PLATFORM ASYMMETRY THAT WOULD OTHERWISE SHIP SILENTLY BROKEN ON LINUX: `process_started_at` returns a Windows FILETIME (absolute, comparable across boots) but on Linux returns field 22 of /proc//stat, which is JIFFIES SINCE BOOT and resets every boot, so a value recorded before a restart is indistinguishable from a fresh one after it. The NATIVE stamp is therefore used for one question only -- the same-pid-different-process PAIR test within a single boot, mirroring `liveness::relay_liveness` -- and an ABSOLUTE epoch-ms launch stamp is parked beside it for the cross-boot question. LEG (b) IS A ONCE-PER-DAEMON-GENERATION BOOT SWEEP relaunching persistent instances whose owners are online, and its discriminant is THE LAUNCH STAMP PREDATES THE BOOT INSTANT, which preserves the class-c force-kill ruling BY CONSTRUCTION rather than by a second rule: a force-kill during steady-state operation leaves a corpse launched AFTER boot, so it is never spontaneously relaunched and the mid-deploy exe-overwrite hazard that ruling protects is untouched. A NAMED SLACK CONSTANT absorbs boot-instant jitter so a process launched moments after boot is never misclassified as a corpse. Gate: impl -- the two stamps parked at the two production pid-write sites (the census is authoritative: five LAUNCH sites funnel through exactly TWO pid writes), the birth-stamped death probe, the record heal in the daemon-side reconciler guarded on an actual change of the recorded value, the boot-instant helper on both platforms, and the boot sweep with its predates-boot discriminant plus slack; unit -- a reused pid (same pid, different birth stamp) reads GONE where bare-pid liveness reads alive (the row the shipped probe cannot pass), a same-pid-same-birth process reads HELD, the heal writes offline over a stale online record and writes NOTHING when the record already matches the truth (the guard asserted by counting writes, since a heal that rewrites every cycle passes any state-only assertion), a corpse launched BEFORE the boot instant is eligible for restore while one launched AFTER it is NOT (the class-c pair, asserted together since either alone passes an implementation that restores everything or nothing), and an instance whose OWNER is offline is not restored even though its corpse predates boot." required_stages = ["doc", "impl", "unit"] # doc ACTIVATED BAROMETER W2 doc leg (doyle-ruled, this milestone): the CONTEXT.md `persistent` amendment states the boot sweep and its four conjunct gates incl. no-stamp => not restored, which IS this requirement's doc evidence; leaving the stage inactive would have made it evidence the registry cannot see. +[[requirements]] +id = "REQ-SHELL-OWNER-ONLINE-RESTORE" +title = "A `persistent` SHELL WHOSE OWNER ENDPOINT WAS NOT YET ONLINE WHEN THE BOOT SWEEP RAN IS RESTORED WHEN THAT OWNER COMES ONLINE, RATHER THAN STAYING DOWN UNTIL A HUMAN RELINKS IT. ROOT (releases#228): daemons boot before endpoints do, so the boot sweep of REQ-SHELL-PERSISTENT-BOOT-RESTORE correctly SKIPS an owner that is not up yet -- an offline owner is owed nothing -- and then nothing ever revisits it: the sweep is once per daemon generation, and bringup from offline produces no rest edge, so the ADR-0048 cascade never fires either. The shell stays down until a `shell cmd` wakes it or an operator relinks by hand, and whether a given node restart strands its shells is an ordering accident. THE ONCE-NESS IS SCHEDULING, NOT SAFETY (doyle, releases#228, re-measured at head): the `launch_predates_boot` corpse-boot discriminant is what preserves the force-kill ruling, and it is MONOTONE WITHIN A MACHINE BOOT -- every launch path restamps `launched_ms`, a kill never restamps, `boot_ms` is the machine boot instant -- so re-evaluating the conjuncts later in a generation can only refuse MORE, never less. FIX: the reconcile loop carries the SAME per-owner restore body the boot sweep runs, triggered on owner offline->online transitions since the last pass. THE TRIGGER IS AN EDGE, NOT A LEVEL, and the distinction is the whole bound: success self-limits because the launch restamps the instance out of eligibility, but a FAILING launch under a level trigger -- owner is online -- retries every five seconds for as long as the owner stays up, while an edge bounds it to one attempt per owner-online event. THE REFUSAL ARMS ARE CARRIED BY CONSTRUCTION, not by restatement: both triggers call one shared per-owner body, so persistent-only, owner online, down in fact via the derived read with the pair test, stamp present, launch predates boot, and no boot oracle means restore nothing can never drift apart between the two paths. THE EDGE STATE IS SEEDED BEFORE THE SWEEP, deliberately: seeded after, an owner that comes online in the window between the sweep and the first tick reads as already-online and never fires -- the exact miss this requirement exists to close -- while seeding before can at worst re-attempt an instance the sweep already restored, which the RESTAMPED BIRTH STAMP refuses -- and NOT the down-in-fact arm, which is the arm that reads like the answer and is not: a launch is not the online switch, so a just-restored instance keeps an `offline` record until its bind-shell handshake and still reads as down in fact; what refuses the second attempt is that the launch moved `launched_ms` past the boot instant. Each trigger names ITSELF in its event line, because the field evidence that diagnosed this ticket was a count of those events and a shared name would leave the next investigation unable to say which trigger fired. Kin REQ-SHELL-PERSISTENT-BOOT-RESTORE (the boot trigger and the shared arms), REQ-HAZARD-RESTART-STRANDS-PERSISTENT-SHELLS (the invariant this closes the remaining seam of), REQ-SHELL-2 (the online/offline mutual exclusivity the edge path holds by stopping an armed watcher before it launches -- mid-generation a watcher CAN be armed for the casualty, unlike at boot)." +required_stages = ["doc", "impl", "unit"] # ACTIVATED CONDUIT W2 in the commit landing the evidence (doyle ruled shape (c) with conditions on releases#228; (a) not built, (b) rejected). doc = the CONTEXT.md `persistent` amendment plus KNOWN-HAZARDS 2.7 and MANIFEST.md Sleep/wake, all of which stated the boot sweep as the ONLY trigger and went false the moment a second trigger existed. MINTED CONDUIT W2 (todlando, 2026-08-28), releases#228. + [[requirements]] id = "REQ-SHELL-KILL-AUTHENTICATED" title = "A RECORDED PID IS NOT KILL AUTHORITY ON THE PRODUCT PATH EITHER: no shell-lifecycle kill fires until the pid+birth PAIR proves the process at that pid is still the one we launched. BAROMETER W2 authenticated the DERIVATION path (`shellinfo::shell_pid_provably_dead` runs the pair test) and left the KILL path believing a bare pid, so the same recycled-pid state that the derived read correctly calls a corpse still routes to `taskkill /PID /T /F` (unix `kill -9`) against whatever unrelated process now wears that number -- and `/T` takes its whole subtree with it, on the user's machine. Proven reachable by probe, not inferred: a live stranger's pid parked beside a foreign birth stamp is killed by `kill_shell_at` while `shell_pid_provably_dead` calls the same pair a corpse. SITE SHAPE IS TWO CHOKEPOINTS, NOT THREE CALL SITES: `shellhost::kill_shell_at` (reached by `close_shell` and the `spt shell teardown` force path) and `shellwake::kill_waker_at` (reached by `launch_shell`, `close_shell`, `stop_watcher`, and the reconciler), both funnelling into the single `kill_shell_pid` taskkill; the gate therefore lands in the two chokepoints and NO caller changes. THE WAKER HALF NEEDED A WRITE-SIDE MINT BEFORE ANY GATE COULD MEAN ANYTHING: `waker.pid` was written bare, and `relay_liveness(live_pid, None)` is HELD by design (pre-stamp records fail toward alive), so a read-side gate there would have returned Held for every record that exists today and every one written tomorrow -- a VACUOUS gate that proves green. The pair is therefore minted in the waker's single existing write as a two-line `waker.pid` (`\\n`): ONE file, ONE write, so the pair is atomic -- a second file has a torn window in which the pid is on disk and the stamp is not, and that window reads Held, which is the vacuous arm reintroduced as a race. Write and every reader move through ONE shared record/read function pair, mirroring `record_shell_launch`/`read_shell_launch`. FAIL-TOWARD-ALIVE IS THE PARSE ARM TOO: an unparseable record reads None and REFUSES the kill while LEAVING the record, because kill-nothing-and-retire-the-pid-file-anyway orphans a live waker while presenting success -- worse than the defect being fixed. UNPROVEN REFUSES: the destructive arm treats unknown as do-not-fire (consequence asymmetry -- a mis-fire force-kills a stranger's tree, a missed kill leaves our own binary running and observable), which FALSIFIES the shipped unconditional promise that a link-break's force-close always kills and \"the binary never survives\", so `close_shell`'s step 3, the REQ-SHELL-2 language, the CONTEXT lifecycle line and KNOWN-HAZARDS 7.58 are amended together with the code rather than one comment patched. THE REFUSAL IS LOUD: every declined kill names the reason and the pid on the channel the close path already speaks, and the bounded confirm loop reports a binary that outlived it -- a quiet wrong state is the un-RCA-able one, and the harm-asymmetry argument has to hold in the log and not only in principle. THIS NARROWS THE MIS-FIRE, IT DOES NOT ELIMINATE IT, AND THE TEXT SAYS SO: on Linux `process_started_at` reads /proc//stat field 22, which counts CLK_TCK=100 jiffies, so two processes whose starts fall in the SAME 10ms tick carry an identical stamp and the pair cannot separate them. The window closes from ANY recycled pid to A PID RECYCLED INTO THE SAME 10ms TICK AS THE RECORDED START -- which requires our process to die AND the OS to hand its number to a new process inside one tick, a precondition that is its own mitigation. Windows is unaffected (FILETIME, far finer resolution). Measured by hertz on kitsubito; recorded verbatim so nobody re-derives it. THE ASYMMETRY IS WHY THIS IS THE KILL PATH'S SENTENCE TO WRITE: the same predicate with the same imprecision is SAFE on the derivation side and UNSAFE here -- Held means 'stays stale' for a derived read and 'fire the kill' for a destructive one, so only the kill side carries the residual harm and only the kill side has to declare it. Image-path corroboration (which would unify this rule with the rig-side technique in KNOWN-HAZARDS 7.58) is filed past as releases#120, deliberately not taken in a closing milestone. MIGRATION WINDOW, NAMED NOT SILENT: pre-W2 shell records and pre-fix waker records carry no stamp, read Held on existence alone, and keep today's bare-pid kill until their next launch writes a pair. DOWNGRADE DIRECTION, NAMED: an OLD binary reading the NEW two-line `waker.pid` parse-fails to None, which reads as no-pid and refuses the kill -- missed-kill, not mis-fire, the safe polarity. Gate: impl -- the pair gate in both chokepoints, the shared waker record/read pair with the two-line format, every waker.pid reader moved onto it, the loud refusal arms, and the confirm-loop survivor report; unit -- a recycled pid (live, birth mismatch) is NOT killed where the shipped code kills it, a genuine own-process pair (Held) IS still killed so the gate is not vacuous, an absent stamp keeps the pre-stamp kill, Unproven refuses, an unparseable waker record refuses AND leaves the record in place (asserted together, since retire-anyway passes a kill-count assertion alone), and the refusal is observable rather than silent; int -- hertz's probe inverted: a live stranger parked beside a foreign stamp SURVIVES `kill_shell_at` while `shell_pid_provably_dead` still calls the pair a corpse (the control), with the victim checked through `try_wait` on the held handle rather than a pid-existence probe, which reads alive for an already-killed process while the handle is open."