# extended-mode-display — repo study

- **Remote:** https://github.com/BigscreenVR/extended-mode-display.git
- **Branches:** `main` only (tracks `origin/main`, up to date). No other local or remote branches. Tags: `v1.0.0`–`v1.0.3`.
- **Author/timeline:** Single developer, Philip Krejov (`philip@bigscreen.com`), 53 commits, 2024-08-22 → 2025-05-11. Last release v1.0.3 (2025-05-11).
- **Language/shape:** Small Python package (`extended_mode_display/`, 9 modules, ~660 LOC) wrapping two prebuilt Windows .exe tools in `resources/`. PyInstaller builds via `build.spec`; GitHub Actions release workflow (`.github/workflows/build_and_release.yml`, artifact name `camera-apps-*`).

## 1. Purpose and status

**This is NOT a consumer "Beyond as desktop monitor" product.** It is an internal factory/lab tool for putting a Bigscreen Beyond into a known raw-panel state and pushing **static test images** to it for optical/display **measurement and calibration** (camera-rig based — see workflow artifact name `camera-apps`, entry point `prepare-headset`, README: "preparing a headset for extended-mode-display measurement").

Components (per `pyproject.toml` `[project.scripts]` and `.vscode/launch.json`):
- `prepare-headset` (`extended_mode_display/prepare_headset_for_measurement.py`) — HID bring-up + direct-mode image display.
- `display-server` (`extended_mode_display/display_server.py`) — Pyro4 RPC server on `localhost:8081` exposing `DirectModeDisplay.im_show` so a separate measurement process can push images (port moved from one clashing with `rerun`, commit `c56a0ba`).
- `bars_gen.py` (repo root) — NiceGUI web UI for hand-tuning periodic vertical-bar (column banding) correction per RGB channel and writing it into SteamVR's per-unit config (see §5).

Status: **achieved its goal; maintained through May 2025; not abandoned.** No TODO debt except one (`prepare_headset_for_measurement.py:25`: "move the remaining hid functions to hid_commands.py"). `bars_gen.py` is bitrotted: it calls `srgb_to_linear`/`linear_to_srgb` which are commented out since numba removal (commit `25e7d65`) — currently a NameError; evidence it was a one-off tuning tool. `extended_mode_display/process.py` (raw `CreateProcessW` unbuffered-pipe process launcher) is dead code, never imported — leftover from getting `display_image.exe` stdin control working.

**Key pivot in history:** Original flow (Oct 2024, commit `0cc8436`) put the headset into **extended desktop mode** and drew with an OpenCV fullscreen window. On 2025-02-20 (`7cfd2a6`, `9d0eaad`) the measurement flow switched to staying in **NVAPI direct mode** and presenting via `display_image.exe` (D3D12). `enable_extended_mode()` remains exported but is no longer called by the prepare flow. Speculation: direct mode chosen for reliability/exactness (the extended path needs polling loops to force window position/fullscreen, `extended_mode_display.py:35-55`) and pixel-exact presentation without DWM.

## 2. Architecture — two display paths

### Path A: Extended mode (`extended_mode_display/extended_mode_display.py`)
- Beyond appears as a normal Windows desktop monitor. Located by **matching exact resolution** via `screeninfo.get_monitors()` (`filter_displays`, errors if 0 or >1 match).
- An OpenCV `WINDOW_NORMAL` window is moved to that monitor's origin, set `WND_PROP_FULLSCREEN`, then position/size verified in a `polling2` retry loop (5 s timeout) — acknowledges Windows window-placement flakiness.
- `display_side_by_side(left, right)` — `cv2.hconcat` two per-eye images into one surface; the monitor is **both eyes side-by-side on a single scanout**.
- 10-minute display timeout to prevent OLED **burn-in** (`_display_timeout = 10*60`, commit `9ec79c2` "Added timeout to prevent burnin").

### Path B: Direct mode (`extended_mode_display/direct_mode_display.py` + `resources/display_image.exe`)
- `DirectModeDisplay` spawns `display_image.exe -image <png>` as a child process and keeps it running; new images are pushed by writing `F<absolute/path>\n` to its stdin (forward slashes required); `q\n` quits. ~6 s startup sleep. Images passed as PNG files written to a temp dir, cached by content hash — a **file-based, seconds-latency** path, fine for static test patterns, useless as a video pipeline.
- `display_image.exe` (374 KB) is a modified NVIDIA sample: strings show `D3D12RenderImage::UpdateImageTexture`, `nvapi_QueryInterface`, and the full usage text of NVIDIA's directmode demo. `resources/readme.md`: it comes from **`https://github.com/BigscreenVR/mmi_test`** `directmode/x64/Release/display_control.exe`, "modified to display multiple images". → `mmi_test` is the first-party repo holding the C++ source of this direct-mode D3D12 presenter.

### Direct-mode plumbing (`extended_mode_display/direct_mode.py` + `resources/direct_mode_dx12.exe`)
- `direct_mode_dx12.exe` (458 KB) is the **NVIDIA VRWorks direct-mode demo, DX12 build** — embedded PDB paths: `C:\dvs\p4\build\sw\devrel\VRWorks\graphics\rel_3_4\direct_mode_dx\demo\dx12\triangle_vs.hlsl`. Pure **NVAPI** ("NVIDIA DirectMode VR", `NV_DIRECT_MODE_INFO`), options: `-enable`/`-disable` (DirectMode on/off, "may require a reboot"), `-enum_disp`, `-enum_mode`, `-present_test`, `-dsc_present_test <display> <mode> <type> <notify> <dsc_version> <dsc_slicecount> <dsc_outputbppx16>`. Also has interactive runtime commands (strings): `1: turn on the display, 2: turn off the display, 3: blank the display, 4: unblank the display`. Handles HDCP status/link-fail and hotplug event callbacks.
- `direct_mode.py` sets env **`DM_VENDOR_ID=0x2709`** before calling it. The exe reads this env var to select which EDID vendor counts as a VR display ("Used for all API calls; ex. 0xAC10").
  - **Decoded:** 0x2709 read as the little-endian uint16 of EDID bytes 8–9 → big-endian word 0x0927 → PNP letters **"BIG"** (B=00010, I=01001, G=00111). Sanity check: NVIDIA's example 0xAC10 → 0x10AC → "DEL" (Dell). So the **Beyond's EDID manufacturer ID is "BIG"**, and 0x2709 is the value any NVAPI direct-mode client must pass to acquire it.
- `enable_extended_mode()` (`direct_mode.py:28-46`) — the **"extended-mode unlock" recipe**:
  1. `direct_mode_dx12.exe -enable` (whitelist Beyond as direct-mode display);
  2. `-dsc_present_test 0 1 1 0 0x11 4 128` = display #0, **mode #1**, event-based present, no notify, **DSC v1.1, 4 slices, 8 bpp output** (128/16), then send `q` to exit;
  3. `direct_mode_dx12.exe -disable` (release back to the OS).
  After this dance Windows can drive the Beyond as a plain extended desktop monitor. Speculation: the DSC present pass initializes the VXR7200 bridge/panels (display-on, link training with DSC) into a state where normal desktop scanout subsequently works; a cold Beyond does not light up as a desktop monitor otherwise.

## 3. HID control protocol (the crown jewels)

`extended_mode_display/hid_commands.py` + `prepare_headset_for_measurement.py`. Device: **hidapi, VID `0x35BD`, PID `0x0101`** (Beyond's control HID interface). All commands are `send_feature_report` with report ID 0, then an ASCII command char, then args:

| Report bytes | Function | Notes |
|---|---|---|
| `0, '-', 0` | display power OFF | `disable_display_power()` |
| `0, '+', 0` | display power ON | `enable_display_power()`; `DisplayPowerManager` waits 3 s after on |
| `0, 'd', rate` | **set EDID to native**; rate: 0=BOTH, 1=90 Hz, 2=75 Hz | **"resets the vxr7200 and takes a few seconds to come back up"** — 5 s sleep (`prepare_...py:31`). Firmware can republish different EDIDs on command. |
| `0, 'p'` | disable proximity sensor | so panels stay on without the HMD being worn |
| `0, 'o', eye, 1, 0x29` | "Display_On command" to left (0)/right (1) display | 0x29 = MIPI **DCS set_display_on** opcode — direct DCS pass-through to each panel |
| `0, 'L', r, g, b` | decorative LED color | dim green `(0,1,0)` used as "prepared" indicator |
| `0, 'I', hi, lo` | **display duty** (persistence/brightness): `0x03FF` = 100%, `0x0066` = 10% | 10-bit duty value; measurement uses 100% (full persistence); normal VR low-persistence is low duty |

Prepare flow (current, `prepare_headset_for_measurement():86-102`): set EDID native (default **75 Hz**; "90hz has not been tested, use at own risk", commit `814c1ee`) → start `DirectModeDisplay` → LED dim green → duty 100% → disable proximity.

## 4. Display modes / resolutions observed

- `bars_gen.py:112`: `ExtendedModeDisplay(resolution=(1920*2, 1920))` → in extended mode (Nov 2024 era) the Beyond appeared as a **3840x1920** desktop monitor (**1920x1920 per eye**); `mura.png` (committed sample output) is 1920x1920 8-bit grayscale; all `test_images/*.png` are 1920x1920.
- `test_images/gen_flare_image.py`: `IMAGE_WIDTH = 2544*2, IMAGE_HEIGHT = 2544` → **5088x2544 (2544x2544 per eye)** for direct-mode display. Suggests the direct-mode DSC scanout is 5088x2544 (panel spec is nominally 2560x2560/eye — discrepancy unresolved; speculation: 2544 is the DSC-sliced active area, or images are letterboxed/scaled by the renderer).
- Refresh: 75 Hz validated; 90 Hz explicitly untested for this pipeline.
- Speculation: full-res scanout needs DSC (only available via the NVAPI direct-mode path here); the 3840x1920 extended mode is a lower-bandwidth EDID mode that fits DP without DSC.

## 5. Mura / banding correction and SteamVR linkage

`bars_gen.py` exists to correct **periodic vertical column banding**: per-channel parameters offset/repeat/drop/width generate a 1920x1920 multiplicative correction in linear light (`gen_bars`, `apply_correction` — sRGB→linear→correct→sRGB), previewed live on the headset (left eye only), then:

```python
cv2.imwrite("mura.png", bars[:, :, channel])
dst_path = "C:/Program Files (x86)/Steam/config/lighthouse/lhr-13a0784a/userdata/mura.mc"
```

→ **SteamVR's per-unit mura correction file `mura.mc` is a single-channel PNG** living under `Steam/config/lighthouse/<device-serial>/userdata/` (`lhr-13a0784a` = a specific Beyond unit). SteamVR's compositor (or the Beyond driver) applies it at runtime. Implication: per-unit panel-uniformity correction is part of the production image chain, applied by SteamVR — anything replacing SteamVR must reapply it.

## 6. What is NOT here (answers to the brief)

- **No lens distortion correction** of any kind: no warp mesh, no shader, no per-unit optical calibration files, no IPD handling, no chromatic aberration. Intentional — measurement wants raw panel pixels. (The warp lives in the SteamVR driver / elsewhere.)
- **No head tracking** (not even 3DOF), no virtual-screen positioning, no IMU access over the HID interface in this code.
- **No desktop/screen capture**, no live video path — static PNGs only.
- **No SteamVR runtime dependency** (only writes into SteamVR's config dir as a side channel).
- **Platforms:** Windows-only (`msvcrt` in process.py, .exe tools, NVAPI, screeninfo/OpenCV on Win32). **NVIDIA-only** for everything direct-mode (NVAPI VRWorks). No AMD/Intel/Linux paths anywhere.
- **Brightness/persistence:** only the coarse duty command (§3); no per-nit calibration.

## 7. Linkbox / video path facts revealed

- The DP→panel bridge is an **Analogix VXR7200** (DP 1.4 → MIPI, VR-specific part) — named in code comment; EDID switch resets it (~5 s).
- The headset firmware controls EDID contents, panel power, per-panel DCS commands, duty cycle, proximity gating — all over USB HID (VID 0x35BD), i.e., **the display pipe is configurable without SteamVR**.
- Working DSC config on NVIDIA: **DSC 1.1, 4 slices, 8 bpp**.
- HDCP is present in the path (demo reports HDCP status; "HDCP is disabled for the GPU" message exists).
- Direct mode hand-off (`-enable` … `-disable`) can demand a reboot in some cases (usage string).

## Relevance to sauna

**Reuse directly:**
1. **HID bring-up vocabulary** (§3) — VID/PID and feature reports for EDID mode switch, panel power, display-on (DCS 0x29), duty/persistence, proximity disable, LED. This is exactly the no-SteamVR headset bring-up layer sauna needs. Port to hidapi on Linux trivially.
2. **EDID-switch-by-firmware** (`'d'` command) — the Beyond can present itself as a *plain monitor with a chosen mode* on command. For sauna's "just a display" UX this is the cleanest lever: pick the 75 Hz native EDID, let the OS mode-set normally. 90 Hz unvalidated — start at 75.
3. **Extended-mode unlock recipe** (§2): EDID native → NVAPI DSC present test → release. If a cold Beyond won't light up as a desktop monitor, this is the known-working sequence (NVIDIA-only). Investigate whether the DSC present step is truly required or whether newer firmware/EDID modes make the headset enumerate cleanly — that determines if AMD/Intel/Linux extended mode is even possible without an equivalent.
4. **DM_VENDOR_ID 0x2709 ("BIG")** — required for any custom NVAPI direct-mode compositor; also the EDID match key for detecting the Beyond as a display on any OS (e.g., Linux DRM + `VK_EXT_direct_mode_display`/`VK_EXT_acquire_drm_display` can match on it).
5. **Known-good link parameters:** DSC 1.1 / 4 slices / 8 bpp; observed surfaces 5088x2544 (direct) and 3840x1920 (extended); 75 Hz.
6. **Burn-in + persistence handling patterns:** mandatory idle timeout; duty register for brightness/persistence trade-off.
7. **`BigscreenVR/mmi_test`** — fetch it: contains the C++ D3D12 NVAPI direct-mode presenter source (`directmode/`), the natural skeleton for sauna's custom compositor on Windows/NVIDIA.

**Dead ends / cautions:**
- The OpenCV-fullscreen-window-on-extended-desktop approach needed polling hacks and was dropped for direct mode even for static images; a sauna compositor should use exclusive fullscreen/DXGI flip or direct mode, not a desktop window.
- The PNG-file + stdin image push and Pyro4 server are measurement-tool conveniences — nothing there is a latency-viable video path.
- Everything direct-mode here is NVAPI: locked to Windows+NVIDIA. For sauna's multi-platform goal, this repo offers no AMD/Intel/Linux direct-mode prior art.
- No distortion correction exists here — sauna's warp mesh must come from the SteamVR Beyond driver / Bigscreen's per-unit calibration service (different repo); do not look for it in this codebase.
- The 2544-vs-2560 per-eye discrepancy and the meaning of EDID mode list per rate setting need resolution against firmware/EDID dumps (this repo never dumps the EDID it sets, though `direct_mode_dx12.exe -enum_disp` prints raw EDID — a handy tool for that).