--- name: failed-job-is-not-a-test-victim description: Counting failed CI jobs as test victims fabricates them — classify each into deterministic code red / non-test step red / one-off victim using the nextest summary and the steps API metadata: node_type: memory type: feedback originSessionId: 3b6f6e5f-c7fa-447c-b4bc-bf7a1df9707a modified: 2026-08-04T18:16:24.260Z --- 2026-08-04: doyle asked which tests were the victims across the LOCKSMITH-window golden failures. The window held 8 failed runs / 12 failed jobs. Reading "failed job" as "a test lost" would have filed FOUR phantom victims and buried the three real ones. The three classes, each with the probe that separates it: - **Deterministic code red** — the same test names fail on BOTH boxes across runs. Cross-platform reproduction disqualifies a victim reading. (7a3c08c, 8688e19: linkhost, shell_relink, activity_link_push, drive_e2e.) - **Non-test step red** — the job is red but its nextest summary reads `2605 tests run: 2605 passed`. The failure is a workflow STEP: `gh api repos/O/R/actions/jobs/` and read `.steps[] | select(.conclusion!="success")`. Here: `Real-shell E2E (notify) - windows` twice, `Clippy (deny warnings)` once (job exit 101), `Two-host role A` once. - **One-off victim** — 1 failed of ~2605, ONE box, a DIFFERENT test each time, not reproducing elsewhere. Only three qualified, and they are the endemic-intermittency evidence. **Why:** ⭐⭐ a count without its classification is not a finding, and here the two readings pointed at opposite conclusions — 12 "victims" says the box is on fire, 3 says there is a real but narrow intermittent. Also: the rerun behaviour has to be stated PER SHA, not generalised. Two of the three cleared on a rerun of the identical sha; the third never got a green golden at its sha, so it is 2-of-3, not 3-of-3, and writing "all cleared on rerun" would have been a fabricated regularity. **How to apply:** for every failed job, pull the log via `gh api repos/O/R/actions/jobs//logs` (never `gh run view --log`, which refuses mid-run — [[gh-job-log-refuses-while-run-in-progress]]), grep `FAIL \[` AND the `Summary \[` line, and print the match COUNT beside the job. Zero FAIL matches with an all-pass summary means go to the steps API, not "unknown". Before reporting that a named test is ABSENT from the victim list, positive-control it: grep the test name and confirm it appears as a PASS in those same logs, so the zero is competent rather than an empty filter ([[zero-match-filter-reads-as-absent]]). And when comparing an old victim to a new one, compare SIGNATURES not just counts — an in-test panic frame at a named source line is a different mechanism from a child exiting nonzero with no panic at all, however similar the 1-of-2600 arithmetic looks.