--- name: screenshot-capture description: | Capture, process, and analyze screenshots of any running application on Windows 10/11. Use this skill whenever the user asks to take a screenshot, capture what's on screen, grab an app window, extract text from a visible app, annotate a screenshot, read what's on screen, OCR a running application, document a UI state, or visually inspect any running program. Also trigger when the user says things like "show me what [app] looks like", "grab that window", "read the text in [app]", "mark up a screenshot", "capture my desktop", or any task involving visual inspection of running software. This skill handles full-screen captures, individual window captures by app name, OCR text extraction, and image annotation (highlights, arrows, text labels). Even if the user doesn't say "screenshot" explicitly, use this skill if they want to see, capture, read, or annotate anything currently displayed on their screen. --- # Screenshot Capture & Processing for Windows This skill gives you the ability to capture screenshots of any running application on a Windows 10/11 system, extract text via OCR, and annotate the images — all from Claude Code's terminal. ## Overview The workflow has three main capabilities: 1. **Capture** — Take a screenshot of the full screen or a specific application window 2. **OCR** — Extract readable text from a captured screenshot 3. **Annotate** — Add highlights, arrows, bounding boxes, and text labels to screenshots These can be combined freely: capture a window, OCR it, then annotate regions of interest. ## Dependencies Before doing anything, ensure the required Python packages are installed. Run the setup script: ```bash python "/scripts/setup.py" ``` This installs `Pillow`, `mss`, `pywin32`, `pytesseract`, and `pygetwindow` if they're missing. It also checks whether Tesseract OCR is installed on the system and tells you if you need to install it manually. Tesseract is the one dependency that can't be pip-installed — it's a system binary. If it's not present, tell the user they need to install it from https://github.com/UB-Mannheim/tesseract/wiki and make sure the install path is on their system PATH (the default installer usually handles this). Without Tesseract, OCR won't work, but capture and annotation still will. ## Step 1: Discover Running Windows Before capturing, you usually want to know what's available. Use the list script: ```bash python "/scripts/list_windows.py" ``` This prints a JSON array of all visible windows with their titles, process names, and geometry (position + size). Use this to find the window the user is asking about. Match by process name or title substring — users will say things like "grab Chrome" or "capture the VS Code window", so do a fuzzy match. The output looks like: ```json [ {"title": "My Document - Google Chrome", "process": "chrome.exe", "x": 0, "y": 0, "width": 1920, "height": 1040}, {"title": "main.py - Visual Studio Code", "process": "Code.exe", "x": 100, "y": 50, "width": 1600, "height": 900} ] ``` ## Step 2: Capture ### Full-screen capture ```bash python "/scripts/capture.py" --output screenshot.png ``` Captures the entire primary monitor. ### Specific window by title substring ```bash python "/scripts/capture.py" --window "Google Chrome" --output chrome.png ``` The `--window` flag does a case-insensitive substring match against window titles. If multiple windows match, it captures the first one found and warns you. If no match is found, it exits with an error and lists available windows so you can retry. ### Options - `--output PATH` — Where to save the PNG (required) - `--window TITLE` — Substring to match against window titles (optional; omit for full screen) - `--delay SECONDS` — Wait before capturing, useful if the user needs time to arrange windows (default: 0) - `--activate` — Bring the target window to the foreground before capturing (only with `--window`) The capture script brings the window to the foreground (if `--activate` is used), waits for the specified delay, then grabs the window region using `mss` for speed and reliability. The output is always a PNG. ## Step 3: OCR (Extract Text) Once you have a screenshot, extract text: ```bash python "/scripts/ocr.py" --input screenshot.png ``` This runs Tesseract on the image and prints the extracted text to stdout. It also writes a structured JSON result to `_ocr.json` with bounding boxes for every detected text block. ### Options - `--input PATH` — The screenshot PNG to process (required) - `--region X,Y,W,H` — Only OCR a specific rectangular region of the image (optional) - `--output PATH` — Save the JSON result to a custom path (default: `_ocr.json`) - `--lang LANG` — Tesseract language code (default: `eng`) The JSON output includes both the full extracted text and per-block coordinates, which is useful for targeted annotation later: ```json { "full_text": "File Edit View ...", "blocks": [ {"text": "File", "x": 12, "y": 5, "width": 30, "height": 16, "confidence": 95.2}, {"text": "Edit", "x": 50, "y": 5, "width": 28, "height": 16, "confidence": 93.8} ] } ``` ## Step 4: Annotate Add visual annotations to a screenshot: ```bash python "/scripts/annotate.py" --input screenshot.png --output annotated.png --annotations annotations.json ``` The annotations file is a JSON array of annotation objects. Supported types: ### Rectangle highlight ```json {"type": "rect", "x": 100, "y": 200, "width": 300, "height": 50, "color": "red", "thickness": 3} ``` ### Arrow ```json {"type": "arrow", "from_x": 100, "from_y": 100, "to_x": 300, "to_y": 200, "color": "red", "thickness": 3} ``` ### Text label ```json {"type": "text", "x": 100, "y": 150, "text": "Bug is here!", "color": "red", "size": 20} ``` ### Circle / Ellipse highlight ```json {"type": "circle", "cx": 200, "cy": 200, "radius": 50, "color": "yellow", "thickness": 2} ``` ### Filled rectangle (semi-transparent overlay) ```json {"type": "highlight", "x": 100, "y": 200, "width": 300, "height": 50, "color": "yellow", "opacity": 0.3} ``` Colors can be common names ("red", "blue", "green", "yellow") or hex codes ("#FF5500"). Thickness defaults to 2 if not specified. ## Putting It Together: Common Workflows ### "Capture Chrome and tell me what's on screen" 1. Run `list_windows.py` to find the Chrome window 2. Run `capture.py --window "Chrome" --activate --output chrome.png` 3. Read `chrome.png` with the Read tool (Claude can see images natively) 4. Describe what you see to the user ### "Read the text in my terminal" 1. Run `list_windows.py` and find the terminal/PowerShell/cmd window 2. Run `capture.py --window "PowerShell" --output terminal.png` 3. Run `ocr.py --input terminal.png` to extract text 4. Return the extracted text to the user ### "Highlight the error message in the screenshot" 1. If you already have a screenshot, run `ocr.py` to find the error text and its coordinates 2. Create an annotations JSON file with a rect around the error region and a text label 3. Run `annotate.py` to produce the annotated image 4. Share the annotated screenshot with the user ### "Take a screenshot every 5 seconds for 30 seconds" Use a bash loop: ```bash for i in $(seq 1 6); do python "/scripts/capture.py" --output "frame_$i.png" sleep 5 done ``` ## Tips - Always run `list_windows.py` first when the user asks about a specific app — don't guess window titles. - If a window capture comes back oddly cropped or black, try adding `--activate --delay 1` to give the window time to render after being brought to the foreground. - For OCR on small text, the results improve if you capture at the highest resolution available. Don't resize images before OCR. - When the user says "read this" while pointing at their screen, capture full-screen, OCR it, and then describe what you find. - Annotation coordinates come from either the OCR JSON (for text regions) or from your own visual analysis of the image. - All scripts are standalone Python — they print errors to stderr and results to stdout, and return non-zero exit codes on failure.