---
name: runner-online-but-deaf
description: Self-hosted runner shows online+idle in API while jobs sit queued — long-poll can be dead; check runner _diag SSL/connect errors before blaming the queue
metadata: 
  node_type: memory
  type: project
  originSessionId: 650aa13f-e98d-4c25-995f-c75873b0b298
  modified: 2026-07-27T10:53:54.960Z
---

A self-hosted runner can report `status=online busy=false` via the GitHub API while its job long-poll is DEAD — queued jobs then sit forever with an "idle" runner. 2026-07-19 case: GitHub let the `*.actions.githubusercontent.com` TLS cert expire (`NotTimeValid`), runner retried silently; cancel/force-cancel/rerun on the run all left jobs queued.

**Why:** API "online" reflects the registration session, not the dispatch long-poll. Silence looks like idleness.

**⚠ COUNTER-PROBE FIRST — RUN THIS BEFORE THE PLAYBOOK (todlando 2026-07-27, doyle-ratified; a queue-idle observation goes STALE fast, and force-cancelling a draining runner kills a healthy in-flight job).** Ask "is it draining right now?" on the box before touching anything:
1. `ps -o pid,ppid,etime,args -p <Runner.Worker pid>` — **PPID must be the CURRENT `Runner.Listener` pid** and **ELAPSED small** ⇒ freshly spawned, the runner is working. An orphan from a pre-update listener has a stale/dead PPID.
2. `pgrep -P <worker pid> -a` — **a live child step** (`bash -e .../_work/_temp/<id>.sh`) ⇒ a real job is executing. Childless ⇒ suspicious.
3. `ls -lt ~/actions-runner/_diag/Worker_*.log | head` — **workers cycling seconds/minutes apart** ⇒ jobs are being taken.
4. Cheapest external confirmation: a leg of YOUR OWN pr passing on that runner (`gh pr checks`).
❌ **The version-dir path proves NOTHING** — my own false start: worker running from `bin.2.336.0` while the listener runs from `bin/` looks like a stale post-update orphan, but `bin`, `bin.2.335.1`, `bin.2.336.0` coexist by design. Only PPID + elapsed + live-child answer the question.

**How to apply:** when jobs queue >5 min with an idle runner AND the counter-probe above says it is NOT draining: (1) ssh the runner box, `journalctl -u actions.runner.*` + newest `_diag/Runner_*.log`, grep connect/SSL errors; (2) probe the exact failing host with openssl/curl and READ THE CERT DATES; (3) check box clock/CA store per [[measure-the-box-before-the-instrument]]; (4) if cert dates are wrong at the server end, it is a GitHub incident — check githubstatus.com, wait, don't churn reruns. `gh api .../actions/runs/<id>/force-cancel` unsticks a zombie queued run that plain cancel won't.
**VARIANT (2026-07-27): cancel-mid-claim zombies BOTH runners.** Cancelling a run while its
twohost jobs were mid-claim left each box's Runner.Worker hung at spawn (0.00s CPU, parent run
completed/cancelled server-side) — both runners busy=true with ZERO in-progress runs, whole queue
starved. force-cancel API = 409 on a completed run (no help). Unelevated Stop-Process = denied
(runner service elevated). Cleared after ~20min (operator kill or token expiry). Prevention: never
cancel a main run while twohost legs are claiming; diagnosis signature = busy=true on all runners
+ empty non-terminal run list + Worker with zero CPU.