# Phase 2: Window and Region Targeting - Research

**Researched:** 2026-04-12
**Domain:** Window enumeration, targeted screen capture, region cropping
**Confidence:** HIGH

## Summary

Phase 2 extends the capture pipeline from desktop-only to window-specific and region-specific targeting. The existing `CaptureTarget` interface is well-designed for extension -- three new target classes (`WindowTarget`, `RegionTarget`, `WindowRegionTarget`) implement the same `capture()` contract. The `node-screenshots` library provides all needed APIs: `Window.all()` for enumeration, `window.captureImageSync()` for window capture, and `Image.cropSync()` for region cropping.

Key technical findings from live testing: (1) `Window.all()` is the only way to find windows -- there is no `Window.fromId()`, so re-lookup by ID requires filtering the `all()` array each frame; (2) minimized windows with zero size throw "Zero width not allowed" on `toPngSync()`; (3) minimized windows with non-zero size return tiny taskbar-button-sized images (useless); (4) `Image.cropSync()` clips to image bounds silently but returns 0-width images for fully out-of-bounds coordinates. The scheduler needs try/catch with skip logic for graceful degradation.

**Primary recommendation:** Implement three CaptureTarget subclasses following the DesktopTarget pattern, add a `list_windows` tool, extend `start_capture` with target discriminated union, and add try/catch frame skipping to the scheduler.

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions
- **D-01:** Extend existing `start_capture` tool with optional target parameters rather than creating separate tools
- **D-02:** Target specification via discriminated union: `target` field accepts "desktop" (default), "window", or "region"
- **D-03:** Window target params: `window_handle` (number, exact match) OR `window_title` (string, substring match); at least one required when target="window"
- **D-04:** Region target params: `x`, `y`, `width`, `height` (all numbers, pixels, screen-absolute coordinates)
- **D-05:** Window-relative region params: `window_handle` or `window_title` + `region_x`, `region_y`, `region_width`, `region_height` (relative to window top-left)
- **D-06:** New `list_windows` MCP tool returns array of visible windows with: `handle` (number), `title` (string), `processName` (string), `x`, `y`, `width`, `height`
- **D-07:** Support both handle-based (exact, from list_windows) and title substring match (case-insensitive)
- **D-08:** If title matches multiple windows, capture the first match and include matched window title in response metadata
- **D-09:** If no window matches, return error immediately (don't start a session that will fail)
- **D-10:** When target window closes or minimizes during active capture: skip that frame, record null/placeholder entry in frames array
- **D-11:** Session metadata includes `skippedFrames` count and reasons
- **D-12:** Capture continues after skip -- don't abort the whole session for a transient window state
- **D-13:** If ALL frames would be skipped (window gone from start), error the session after 3 consecutive failures
- **D-14:** New `WindowTarget` class implementing CaptureTarget interface
- **D-15:** New `RegionTarget` class -- uses Monitor capture + sharp crop for screen regions
- **D-16:** New `WindowRegionTarget` class -- captures window then crops to sub-region
- **D-17:** All targets return PNG buffers for uniform pipeline processing

### Claude's Discretion
- Window enumeration filtering (whether to exclude zero-size or invisible windows)
- Error message wording for window not found, region out of bounds
- Internal caching strategy for window handle lookups
- Whether to add `stop_capture` tool in this phase or defer

### Deferred Ideas (OUT OF SCOPE)
None -- discussion stayed within phase scope
</user_constraints>

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| CAPT-02 | Capture specific application window by title or handle | WindowTarget class using node-screenshots Window.captureImageSync(); D-03, D-07, D-08 |
| CAPT-03 | Capture rectangular screen region (x, y, width, height) | RegionTarget class using Monitor.captureImageSync() + Image.cropSync(); D-04 |
| CAPT-04 | Capture region relative to window position | WindowRegionTarget class using Window.captureImageSync() + Image.cropSync(); D-05 |
| CAPT-05 | List all visible windows with title, handle, process info | list_windows tool using Window.all(); D-06 |
| TIME-05 | Handle target window closing/minimizing gracefully | Scheduler try/catch with skip logic, 3-consecutive-failure abort; D-10 through D-13 |
</phase_requirements>

## Standard Stack

### Core (already installed)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| node-screenshots | 0.2.8 | Window enumeration + capture | Already in use; provides Window.all(), Window.captureImageSync(), Image.cropSync() [VERIFIED: node_modules/node-screenshots/index.d.ts] |
| sharp | 0.34.5 | Image processing (not needed for basic region crop but used in grid pipeline) | Already in use for grid compilation [VERIFIED: package.json] |

### Supporting
No new dependencies needed. All functionality is provided by existing libraries.

**Installation:** No new packages required.

## Architecture Patterns

### New Files
```
src/
  capture/
    targets/
      capture-target.ts      # existing interface (unchanged)
      desktop-target.ts       # existing (unchanged)
      window-target.ts        # NEW: WindowTarget implements CaptureTarget
      region-target.ts        # NEW: RegionTarget implements CaptureTarget
      window-region-target.ts # NEW: WindowRegionTarget implements CaptureTarget
      window-utils.ts         # NEW: shared window lookup helpers
  types.ts                    # MODIFY: extend CaptureConfig.target union, add target schemas
  server.ts                   # MODIFY: register list_windows, extend start_capture
  capture/
    scheduler.ts              # MODIFY: add try/catch frame skip logic
```

### Pattern 1: Window Lookup Helper (shared by WindowTarget and WindowRegionTarget)
**What:** Centralized function to find a window by handle or title substring
**When to use:** Every frame capture in WindowTarget/WindowRegionTarget
**Example:**
```typescript
// Source: verified against node-screenshots index.d.ts
import { Window } from "node-screenshots";

export function findWindow(
  handle?: number,
  title?: string,
): Window | null {
  const windows = Window.all();
  if (handle !== undefined) {
    return windows.find((w) => w.id() === handle) ?? null;
  }
  if (title !== undefined) {
    const lower = title.toLowerCase();
    return windows.find((w) => w.title().toLowerCase().includes(lower)) ?? null;
  }
  return null;
}
```

**Key insight:** `Window.all()` is the ONLY way to find windows -- there is no `Window.fromId()`. Each call re-enumerates all windows from the OS. This is fast enough for capture intervals >= 100ms but should not be called unnecessarily. [VERIFIED: live test of Window static methods]

### Pattern 2: Target Factory in server.ts
**What:** Create the appropriate CaptureTarget based on parsed input
**When to use:** In start_capture handler, after validating input
**Example:**
```typescript
function createTarget(args: StartCaptureInput): CaptureTarget {
  switch (args.target) {
    case "window":
      return new WindowTarget(args.window_handle, args.window_title);
    case "region":
      return new RegionTarget(args.x!, args.y!, args.width!, args.height!);
    case "window_region":
      return new WindowRegionTarget(
        args.window_handle, args.window_title,
        args.region_x!, args.region_y!, args.region_width!, args.region_height!,
      );
    case "desktop":
    default:
      return new DesktopTarget();
  }
}
```

### Pattern 3: Scheduler Frame Skip
**What:** Wrap capture() in try/catch, skip frame on failure, abort after 3 consecutive failures
**When to use:** Modified scheduler tick function
**Example:**
```typescript
let consecutiveFailures = 0;

async function tick(): Promise<void> {
  try {
    await captureFrame();
    consecutiveFailures = 0; // reset on success
    // ... check limits and schedule next
  } catch (err) {
    consecutiveFailures++;
    const elapsed = Date.now() - startTime;
    skippedFrames.push({ index: frameIndex, elapsedMs: elapsed, reason: String(err) });
    frameIndex++; // advance index even on skip

    if (consecutiveFailures >= 3) {
      // Abort: target is gone
      session.state = "error";
      session.error = `Target unavailable: ${consecutiveFailures} consecutive capture failures`;
      // ...
      return;
    }

    scheduleNext(); // continue trying
  }
}
```

### Pattern 4: Zod Discriminated Union for Target Types
**What:** Extend start_capture input schema with target-specific fields
**When to use:** In types.ts schema definition
**Example:**
```typescript
// Base fields (existing)
const baseFields = {
  interval_ms: z.number().min(100),
  max_frames: z.number().min(1).max(50).default(20),
  duration_ms: z.number().min(100).optional(),
  jpeg_quality: z.number().min(1).max(100).default(80),
};

// Target-specific optional fields added to flat schema
// (simpler than true discriminated union for MCP tool interface)
const targetFields = {
  target: z.enum(["desktop", "window", "region", "window_region"]).default("desktop"),
  window_handle: z.number().optional(),
  window_title: z.string().optional(),
  x: z.number().optional(),
  y: z.number().optional(),
  width: z.number().optional(),
  height: z.number().optional(),
  region_x: z.number().optional(),
  region_y: z.number().optional(),
  region_width: z.number().optional(),
  region_height: z.number().optional(),
};
```

**Note:** MCP tool schemas use flat zod objects (not nested discriminated unions). Validation of required-per-target fields should be done in the handler with clear error messages. [ASSUMED -- MCP SDK registerTool accepts flat zod shapes; nested discriminated unions may or may not work with the SDK's schema generation]

### Anti-Patterns to Avoid
- **Caching Window objects across frames:** Window objects from `Window.all()` are snapshots -- they go stale if the window moves, resizes, or closes. Always re-enumerate per frame.
- **Using async capture in hot loop:** `captureImageSync()` is faster and simpler for the scheduler pattern. Only use async if you need to interleave with other I/O.
- **Silently returning empty buffers:** A 0x0 capture (minimized window) should throw, not return an empty buffer that breaks downstream grid compilation.

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Window enumeration | Win32 API bindings | `Window.all()` from node-screenshots | Already handles cross-platform differences |
| Region cropping | Manual pixel buffer manipulation | `Image.cropSync()` from node-screenshots | Native Rust implementation, handles bounds |
| DPI-aware coordinates | Manual DPI calculations | node-screenshots handles DPI internally | Monitor.scaleFactor() is informational; capture coordinates are in screen pixels [VERIFIED: live test at scale=1] |

## Common Pitfalls

### Pitfall 1: Minimized Window Capture Failure
**What goes wrong:** `window.captureImageSync()` on a minimized window returns a 0x0 Image. Calling `toPngSync()` on it throws "Zero width not allowed".
**Why it happens:** Windows does not render minimized windows to a buffer -- the capture returns an empty image.
**How to avoid:** Check `window.isMinimized()` before capture, OR catch the error in the scheduler and treat as a skipped frame. The catch approach is more robust since the window could minimize between the check and the capture.
**Warning signs:** "Zero width not allowed" error from node-screenshots.
**Verified:** [VERIFIED: live test on Windows 11 -- minimized VoiceMeeter window returned 0x0 image]

### Pitfall 2: Window Gone Between Frames
**What goes wrong:** Window is found in frame N but not in frame N+1 (user closed it).
**Why it happens:** `Window.all()` is a point-in-time snapshot. The window can disappear at any time.
**How to avoid:** `findWindow()` returns null when window is gone. Treat null as a skipped frame. The D-13 rule (3 consecutive failures = abort) handles permanent disappearance.
**Warning signs:** Sudden null returns from findWindow after successful captures.

### Pitfall 3: Crop Out-of-Bounds Returns Zero-Width Image
**What goes wrong:** `Image.cropSync()` with x >= image.width returns a 0-width image. `toPngSync()` then fails.
**Why it happens:** cropSync clips to bounds silently rather than throwing.
**How to avoid:** Validate region coordinates against actual image dimensions before cropping. Clamp or error if region is fully outside the image.
**Verified:** [VERIFIED: live test -- cropSync(width+10, 0, 100, 100) returned 0x100 image]

### Pitfall 4: Stale Module-Level Target Singletons
**What goes wrong:** Current server.ts uses a module-level `desktopTarget` singleton. With per-session targets, this pattern breaks -- each session needs its own target.
**Why it happens:** Phase 1 only had one target type. Phase 2 needs per-session target creation.
**How to avoid:** Create target in start_capture handler, pass to scheduler. Remove or keep desktopTarget as default only. The target must be passed per-session, not stored globally.

### Pitfall 5: Title Substring Matching Ambiguity
**What goes wrong:** Title "Chrome" matches "Google Chrome", "Chrome DevTools", etc.
**Why it happens:** Case-insensitive substring match is intentionally loose for agent convenience.
**How to avoid:** Per D-08, use first match and include the matched window title in response metadata so the agent knows which window was selected. The agent can then refine with a more specific title or use the handle directly.

## Code Examples

### list_windows Tool Registration
```typescript
// Source: existing server.ts pattern + node-screenshots index.d.ts
server.registerTool(
  "list_windows",
  {
    title: "List Windows",
    description: "List all visible application windows with their handles, titles, and positions.",
    inputSchema: {},
  },
  async () => {
    const windows = Window.all()
      .filter((w) => w.title() !== "" && w.width() > 0 && w.height() > 0 && !w.isMinimized())
      .map((w) => ({
        handle: w.id(),
        title: w.title(),
        processName: w.appName(),
        x: w.x(),
        y: w.y(),
        width: w.width(),
        height: w.height(),
      }));

    return {
      content: [{ type: "text" as const, text: JSON.stringify({ windows }) }],
    };
  },
);
```

### WindowTarget Class
```typescript
// Source: capture-target.ts interface + node-screenshots API
import { Window } from "node-screenshots";
import type { CaptureTarget } from "./capture-target.js";
import { findWindow } from "./window-utils.js";

export class WindowTarget implements CaptureTarget {
  readonly name: string;
  private readonly handle?: number;
  private readonly titleMatch?: string;

  constructor(handle?: number, title?: string) {
    this.handle = handle;
    this.titleMatch = title;
    this.name = `window:${handle ?? title}`;
  }

  async capture(): Promise<Buffer> {
    const win = findWindow(this.handle, this.titleMatch);
    if (!win) {
      throw new Error(`Window not found: ${this.handle ?? this.titleMatch}`);
    }
    if (win.isMinimized()) {
      throw new Error(`Window is minimized: "${win.title()}"`);
    }
    const image = win.captureImageSync();
    if (image.width === 0 || image.height === 0) {
      throw new Error(`Window capture returned empty image: "${win.title()}"`);
    }
    return image.toPngSync();
  }
}
```

### RegionTarget Class
```typescript
import { Monitor } from "node-screenshots";
import type { CaptureTarget } from "./capture-target.js";

export class RegionTarget implements CaptureTarget {
  readonly name: string;

  constructor(
    private readonly x: number,
    private readonly y: number,
    private readonly width: number,
    private readonly height: number,
  ) {
    this.name = `region:${x},${y},${width}x${height}`;
  }

  async capture(): Promise<Buffer> {
    const monitor = Monitor.fromPoint(this.x, this.y);
    if (!monitor) {
      throw new Error(`No monitor found at point (${this.x}, ${this.y})`);
    }
    const image = monitor.captureImageSync();
    // Adjust coordinates relative to monitor origin
    const relX = this.x - monitor.x();
    const relY = this.y - monitor.y();
    const cropped = image.cropSync(relX, relY, this.width, this.height);
    if (cropped.width === 0 || cropped.height === 0) {
      throw new Error(`Region crop produced empty image`);
    }
    return cropped.toPngSync();
  }
}
```

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Single DesktopTarget singleton | Per-session target creation | Phase 2 | server.ts needs refactoring to create target per session |
| Scheduler assumes capture always succeeds | Try/catch with skip logic | Phase 2 | Enables graceful degradation for window targeting |

## Assumptions Log

| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | MCP SDK registerTool handles flat zod schemas with optional fields correctly for the discriminated union approach | Architecture Patterns - Pattern 4 | Would need to restructure the schema or use .refine() for cross-field validation |
| A2 | Window.all() performance is acceptable when called every 100ms+ | Architecture Patterns - Pattern 1 | Would need to cache Window objects and handle staleness differently |
| A3 | Monitor.fromPoint() correctly identifies the monitor containing the given screen coordinate on multi-monitor setups | Code Examples - RegionTarget | Region capture on secondary monitors could fail; would need Monitor.all() scan fallback |

## Open Questions

1. **DPI Scaling Behavior**
   - What we know: At scale=1, coordinates match pixel-for-pixel. Monitor.scaleFactor() is available.
   - What's unclear: At 150% or 200% scaling, are Window.x()/y()/width()/height() in logical or physical pixels? Does cropSync use the same coordinate space as the captured image?
   - Recommendation: Works at scale=1 (this machine). Flag for testing on high-DPI systems. STATE.md already notes this as a research flag.

2. **stop_capture Tool**
   - What we know: D-discretion allows deferring this.
   - What's unclear: Whether agents need to abort sessions before they complete.
   - Recommendation: Defer to a later phase. Sessions have max_frames and duration_ms limits. Agents can ignore sessions they don't want.

## Discretion Recommendations

Based on research findings, here are recommendations for Claude's discretion areas:

1. **Window enumeration filtering:** Filter out windows with empty title, zero width/height, and minimized state from `list_windows` results. [VERIFIED: live test shows empty-title window is the taskbar, zero-size windows are background processes -- neither useful for agents]

2. **Caching strategy:** Do NOT cache Window objects across frames. Re-enumerate via `Window.all()` each capture to get fresh state. Cache only the lookup criteria (handle/title) not the Window reference. [VERIFIED: Window objects are snapshots that go stale]

3. **stop_capture tool:** Defer to a later phase. Not needed for Phase 2 functionality.

## Sources

### Primary (HIGH confidence)
- node-screenshots index.d.ts -- full API surface for Window, Monitor, Image classes
- Live testing on Windows 11 -- verified Window.all(), captureImageSync(), isMinimized(), cropSync() behaviors
- Existing codebase (capture-target.ts, desktop-target.ts, server.ts, scheduler.ts, types.ts) -- verified current patterns

### Secondary (MEDIUM confidence)
- node-screenshots README.md -- usage examples and API overview

### Tertiary (LOW confidence)
- None

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH -- no new dependencies, all APIs verified via live testing
- Architecture: HIGH -- follows existing CaptureTarget pattern, all edge cases tested
- Pitfalls: HIGH -- all five pitfalls verified through live testing on Windows 11

**Research date:** 2026-04-12
**Valid until:** 2026-05-12 (stable -- no fast-moving dependencies)