mcap/video-capture-plan.html
Musa Mahmood 9953ba6f1f [0.1.4] Video capture phase 2: Windows.Graphics.Capture, for when Desktop Duplication cannot run
- modules/Screen_Record/wgc_source.jai: the screen through Windows.Graphics.Capture, with
  hand-written bindings for its WinRT interfaces (capture item interop, item, free-threaded frame
  pool, frame, session 1-3, IDirect3DDxgiInterfaceAccess, IClosable). A small COM handler on
  FrameArrived sets an event per frame; when Windows refuses it the pool is polled each
  millisecond. The cursor is drawn by Windows; the yellow border is turned off where the system
  allows (Windows 11). A screen that changes size makes the pool again. WinRT's own calls are
  loaded from combase.dll at run time: runtimeobject.lib names no DLL for #library.
- capture_gpu.jai: what both sources share (the device on the screen's adapter, the region copy,
  the staging ring), so each is only "the next image's region into a texture".
- Record_Options.capture: AUTO (Desktop Duplication, else Windows.Graphics.Capture when it says
  fall_back), DXGI or WGC. Sources open on the capture thread, as WinRT objects belong to their
  apartment; record_start waits for it to say how that went. Record_Options.testing simulates the
  two-GPU failure, forces polling, or keeps the border, for record-probe.
- record-probe: -capture auto|dxgi|wgc, -simulate-unsupported, -poll, -border-test with
  -keep-border as its control, -cursor-test; DPI awareness set already is no longer reported as
  a failure.
- On the RTX 5080 laptop: auto falls back on the simulated failure; WGC records 4K at 60 fps with
  nothing dropped; its cursor and the one cursor.jai draws for duplication are the same; the
  border is off for an unpackaged app (0 yellow edge pixels, 14,214 with it kept);
  WDA_EXCLUDEFROMCAPTURE is respected; polling holds 30 fps. The plan records the results.
- README: Screen_Record's line, which held a carriage return in place of "\r" since 0.1.3.
2026-10-10 07:03:47 +01:00

422 lines
36 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Video Capture Plan</title>
<style>
:root {
--bg: #fbfaf8; --surface: #ffffff; --text: #1d1d1f; --dim: #5d5d66; --line: #e3e1dc;
--accent: #3a5bd9; --accent-soft: #e9edfb; --warn: #9a5b00; --warn-soft: #fbf1e1;
--code-bg: #f2f0ec; --ok: #2e7d4f; --ok-soft: #e5f3ea;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
}
:root[data-theme="dark"] {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
* { box-sizing: border-box; }
body {
margin: 0; background: var(--bg); color: var(--text);
font: 16px/1.6 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
}
main { max-width: 860px; margin: 0 auto; padding: 40px 16px 80px; }
header p { color: var(--dim); margin-top: 4px; }
h1 { font-size: 2rem; line-height: 1.2; margin: 0; }
h2 { font-size: 1.35rem; margin: 48px 0 8px; padding-top: 16px; border-top: 1px solid var(--line); }
h3 { font-size: 1.05rem; margin: 28px 0 6px; }
p, li { max-width: 72ch; }
a { color: var(--accent); }
code, pre { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; font-size: 0.88em; }
code { background: var(--code-bg); padding: 1px 5px; border-radius: 4px; }
pre { background: var(--code-bg); padding: 12px 14px; border-radius: 8px; overflow-x: auto; line-height: 1.45; }
pre code { background: none; padding: 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 0.94rem; }
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
th { color: var(--dim); font-weight: 600; }
.table-wrap { overflow-x: auto; }
.summary { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 16px 20px; margin-top: 24px; }
.summary ul { margin: 6px 0 0; padding-left: 20px; }
.note { background: var(--accent-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk { background: var(--warn-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk strong { color: var(--warn); }
.tag { display: inline-block; font-size: 0.78rem; font-weight: 600; padding: 1px 8px; border-radius: 99px; background: var(--accent-soft); color: var(--accent); }
.tag.later { background: var(--code-bg); color: var(--dim); }
.tag.decided { background: var(--ok-soft); color: var(--ok); }
nav ol { columns: 2; column-gap: 32px; padding-left: 20px; margin: 8px 0 0; }
@media (max-width: 600px) { nav ol { columns: 1; } h1 { font-size: 1.6rem; } }
.phase { border-left: 3px solid var(--accent); padding: 2px 0 2px 14px; margin: 16px 0; }
.phase h3 { margin-top: 0; }
figure { margin: 18px 0; }
figcaption { color: var(--dim); font-size: 0.88rem; margin-top: 6px; }
svg text { fill: var(--text); font: 13px system-ui, sans-serif; }
svg .dim { fill: var(--dim); }
svg .box { fill: var(--surface); stroke: var(--line); }
svg .hl { fill: var(--accent-soft); stroke: var(--accent); }
svg .arrow { stroke: var(--dim); fill: none; marker-end: url(#a); }
</style>
</head>
<body>
<main>
<header>
<h1>Screen video capture in mcap</h1>
<p>Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · <strong>phases 1 and 2 built, mcap 0.1.4</strong></p>
</header>
<div class="summary">
<strong>In short</strong>
<ul>
<li><span class="tag decided">Decided</span> <strong>DXGI Desktop Duplication first.</strong> If it can't run, for example on a laptop with two GPUs where it fails with <code>DXGI_ERROR_UNSUPPORTED</code>, recording falls back to <strong>Windows.Graphics.Capture</strong>. Both sit behind one interface, so everything after the capture is shared. It's more code than Windows.Graphics.Capture alone, but each method covers the cases where the other fails.</li>
<li><span class="tag decided">Decided</span> <strong>Encoding uses the FFmpeg mcap already ships.</strong> It tries NVENC first, then AMD AMF, Intel QSV, Media Foundation and <code>libx264</code>. The output is H.264 in MP4, with frames written only when the screen changes. That FFmpeg build already includes every one of these encoders.</li>
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen starts a recording.</strong> It opens the same frozen-screen overlay as stills, in a video mode. The region must be at least <strong>240 × 180</strong> pixels, which Snagit doesn't require.</li>
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen again stops it</strong> and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.</li>
<li><span class="tag decided">Decided</span> <strong>A recording becomes an <code>.mcap</code></strong> in <em>Screen Captures</em>, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.</li>
<li>It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.</li>
<li><span class="tag decided">Done</span> <strong>Phase 1</strong> (2026-10-10, mcap 0.1.3): <code>modules/Screen_Record</code> and <code>record-probe</code>. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see <a href="#phases">phase 1's result</a>).</li>
<li><span class="tag decided">Done</span> <strong>Phase 2</strong> (2026-10-10, mcap 0.1.4): the Windows.Graphics.Capture fallback. Automatic mode falls back to it on a simulated <code>DXGI_ERROR_UNSUPPORTED</code>. It records 4K at 60 fps with nothing dropped, draws the cursor exactly as Desktop Duplication's code does, and the yellow border can be turned off on Windows 11.</li>
<li><span class="tag later">Later</span> Audio is phase 5. Recording one window, and regions that span two screens, come after.</li>
</ul>
</div>
<nav>
<ol>
<li><a href="#scope">Scope</a></li>
<li><a href="#using">Using it</a></li>
<li><a href="#sources">Grabbing frames</a></li>
<li><a href="#pipeline">The frame pipeline</a></li>
<li><a href="#encoding">Encoding</a></li>
<li><a href="#library">Into the library</a></li>
<li><a href="#hotkeys">Hotkeys</a></li>
<li><a href="#audio">Audio</a></li>
<li><a href="#phases">Phases</a></li>
<li><a href="#risks">Risks</a></li>
<li><a href="#questions">Open questions</a></li>
</ol>
</nav>
<h2 id="scope">1. Scope</h2>
<p><strong>In:</strong></p>
<ul>
<li>Recording a rectangle on one screen, or a whole screen, to H.264 MP4 at up to 60 frames a second, with the mouse cursor.</li>
<li>Pause and resume, stop, and discard.</li>
<li>Saving into the library with the window and app it was recorded from.</li>
<li>System audio and the microphone, in the last phase.</li>
</ul>
<p><strong>Out, for now:</strong></p>
<ul>
<li>Recording one window that follows it as it moves.</li>
<li>Regions that span two screens.</li>
<li>Webcam overlay.</li>
<li>Trimming and editing.</li>
<li>HDR output.</li>
<li>macOS and Linux.</li>
</ul>
<p>Windows only, like the stills overlay. It needs Windows 10 version 2004 or later; the Windows.Graphics.Capture features used below need that version.</p>
<h2 id="using">2. Using it</h2>
<ol>
<li><strong>Ctrl+PrintScreen</strong> freezes the screens and shows the overlay from stills, in its video mode. The outline and the magnifier are red, not orange, so you can see it will record.</li>
<li><strong>Drag a region.</strong> The magnifier's label shows its size.
<ul>
<li>The drag is kept on the screen where it started (see <a href="#sources">3.4</a>).</li>
<li><strong>Below 240 × 180</strong>, the outline is drawn dashed and the label reads <code>320 × 120 · at least 240 × 180</code>. Letting go then starts nothing, and the overlay stays open to drag again.</li>
<li>A click without dragging, or Enter, takes the whole screen, as for stills.</li>
<li>Escape or a right click cancels, as for stills.</li>
</ul></li>
<li><strong>Recording starts</strong> as soon as you let go, with no countdown (see <a href="#questions">open question 4</a>).
<ul>
<li>A thin red border is drawn just <em>outside</em> the region, so it's never in the video.</li>
<li>A small toolbar sits beside the region, outside it where there's room. It shows the time, the size and frames dropped, with <em>Pause</em>, <em>Stop</em> and <em>Discard</em>.</li>
<li>Both the border and the toolbar are kept out of the capture with <code>SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE)</code>, which needs Windows 10 2004. Whether Desktop Duplication respects this is checked in phase 1. The border is outside the region either way, so only the toolbar depends on it.</li>
</ul></li>
<li><strong>Ctrl+PrintScreen</strong> again, or <em>Stop</em>, ends the recording at once:
<ul>
<li>The encoder is flushed and the file closed.</li>
<li>The file becomes an <code>.mcap</code>.</li>
<li>mcap comes to the front showing it in the Capture window, as it does after a still.</li>
</ul></li>
</ol>
<p>Dimensions are rounded down to even numbers after the minimum is checked, since H.264's 4:2:0 colour needs that. 240 × 180 is already even, so a region that passes the check stays at least 240 × 180.</p>
<h2 id="sources">3. Grabbing frames</h2>
<p>One interface, two implementations, chosen when the recording starts:</p>
<pre><code>Capture_Source :: struct {
start: (source: *Capture_Source, screen: Screen_Ref, region: Rect, cursor: bool) -> ok: bool, error: string, fall_back: bool;
next_frame: (source: *Capture_Source, timeout_ms: u32) -> Captured_Frame, status: Frame_Status; // NEW, SAME, LOST, FAILED
stop: (source: *Capture_Source);
name: string; // "DXGI" or "WGC", for the log and the toolbar's tooltip
}
Captured_Frame :: struct {
texture: *ID3D11Texture2D; // the region only, region-sized, owned by the source until the next call
time: s64; // QueryPerformanceCounter ticks: when it was on screen
cursor: Cursor_State; // DXGI only: where, visible, and the shape when it changed
}</code></pre>
<p>Both run on a capture thread of their own, with <strong>their own D3D11 device</strong>, not the app's. The app's device starts on WARP and moves to the GPU later (<code>#WarpFirstStartup</code> in <code>graphics/DX11.jai</code>). Desktop Duplication also needs a device on the GPU the screen is connected to, which may not be the one the app draws with.</p>
<h3>3.1 DXGI Desktop Duplication (first choice)</h3>
<ul>
<li><strong>Finding the screen:</strong> <code>CreateDXGIFactory1</code>, then <code>EnumAdapters1</code> and <code>EnumOutputs</code>, to find the output whose desktop rectangle holds the region. The device is created on <em>that</em> output's adapter, with <code>D3D11CreateDevice(adapter, D3D_DRIVER_TYPE_UNKNOWN, ...)</code>. The jai <code>dxgi</code> module already has every type needed: <code>IDXGIOutput5</code>, <code>DuplicateOutput1</code>, <code>IDXGIOutputDuplication</code>, <code>DXGI_OUTDUPL_FRAME_INFO</code>.</li>
<li><strong>Duplicating:</strong> <code>IDXGIOutput5::DuplicateOutput1</code> with <code>B8G8R8A8_UNORM</code>. mcap is per-monitor DPI aware (v2), and the plain <code>DuplicateOutput</code> is documented to fail for such processes. Phase 1 confirms this, and falls back to <code>IDXGIOutput1::DuplicateOutput</code> where <code>IDXGIOutput5</code> is missing.</li>
<li><strong>Each frame:</strong>
<ul>
<li><code>AcquireNextFrame(timeout)</code> waits for one.</li>
<li><code>LastPresentTime == 0</code> means only the mouse moved: the cursor is updated and no new image is copied.</li>
<li><code>CopySubresourceRegion</code> copies the region into a region-sized texture.</li>
<li><code>ReleaseFrame</code> is called straight after the copy. Holding a frame stalls the desktop.</li>
</ul></li>
<li><strong>The cursor isn't drawn into the image.</strong> Its position comes with each frame, and its shape from <code>GetFramePointerShape</code> when <code>PointerShapeBufferSize</code> is set. It is drawn onto the copy on the CPU, after it's read back (<a href="#pipeline">4</a>). There are three kinds of shape to handle: colour, masked colour, and monochrome. The monochrome kind (the text I-beam) inverts what's under it.</li>
<li><strong>Errors while recording:</strong>
<ul>
<li><code>DXGI_ERROR_ACCESS_LOST</code> comes from a display mode change, the UAC or lock screen, or a game switching to fullscreen. The duplication is made again, retrying every 250 ms, and the recording goes on. The toolbar shows the gap.</li>
<li><code>DXGI_ERROR_WAIT_TIMEOUT</code> means nothing changed.</li>
<li>Anything else ends the recording, keeping what was captured.</li>
</ul></li>
</ul>
<h3>3.2 When to fall back</h3>
<p>The choice is made once, at the start, and logged:</p>
<div class="table-wrap"><table>
<tr><th>Desktop Duplication says</th><th>Meaning</th><th>Then</th></tr>
<tr><td><code>DXGI_ERROR_UNSUPPORTED</code> from <code>DuplicateOutput1</code></td><td>Two GPUs, and the process or device isn't on the one driving the screen. Also returned when no driver can duplicate.</td><td>Windows.Graphics.Capture</td></tr>
<tr><td><code>DXGI_ERROR_NOT_CURRENTLY_AVAILABLE</code></td><td>Too many duplications of this screen at once (other recorders running)</td><td>Windows.Graphics.Capture</td></tr>
<tr><td><code>E_ACCESSDENIED</code></td><td>The secure desktop (UAC, lock screen) is showing</td><td>Retry for 2 s, then tell the user</td></tr>
<tr><td>Screen rotated (<code>DXGI_OUTDUPL_DESC.Rotation</code> isn't identity)</td><td>Frames would come sideways</td><td>Windows.Graphics.Capture, which hands them over upright</td></tr>
<tr><td>No output holds the region</td><td>The screens changed since the freeze</td><td>Error, nothing recorded</td></tr>
</table></div>
<p>A setting forces one method, for testing and for machines where one misbehaves: <code>capture: auto</code> (the default), <code>dxgi</code> or <code>wgc</code>, under a new <code>[video]</code> section of <code>settings.config</code>.</p>
<h3>3.3 Windows.Graphics.Capture (fallback)</h3>
<ul>
<li><strong>Setup:</strong> the WinRT runtime is started on the capture thread (<code>RoInitialize</code>, multithreaded).
<ul>
<li><code>IGraphicsCaptureItemInterop::CreateForMonitor</code> makes the item for the screen.</li>
<li><code>CreateDirect3D11DeviceFromDXGIDevice</code> wraps the capture thread's device.</li>
<li><code>Direct3D11CaptureFramePool::CreateFreeThreaded</code> makes a pool of 2 BGRA frames.</li>
</ul></li>
<li><strong>Polling, not events:</strong> <code>TryGetNextFrame</code> is called at the frame rate, with the same timeout as DXGI. This avoids implementing WinRT event-handler objects in Jai.</li>
<li><strong>Cursor and border:</strong> <code>IsCursorCaptureEnabled</code> draws the cursor into the image, so the cursor step in 3.1 is skipped. <code>IsBorderRequired = false</code> removes the yellow border on Windows 11; whether an unpackaged app may do that without asking is checked in phase 2. Windows 10 always shows the border.</li>
<li><strong>Bindings:</strong> roughly six WinRT interfaces, written by hand as COM vtables, plus HSTRINGs (<code>WindowsCreateString</code>). This is the bulk of phase 2.</li>
</ul>
<h3>3.4 One screen per recording</h3>
<p>Desktop Duplication works one output at a time, and Windows.Graphics.Capture one monitor at a time. A region over two screens would need two captures stitched together, possibly at different DPI and refresh rates. In video mode the overlay keeps the drag on the screen where it started. Spanning screens is <span class="tag later">Later</span>.</p>
<h2 id="pipeline">4. The frame pipeline</h2>
<pre><code>capture thread encoder thread
next_frame ─► region texture ─► staging ring (3) ─► queue (≤ 8 frames) ─► AVFrame bgr0 ─► encoder ─► MP4
Map the one from drops the oldest when full,
2 frames back, copy counted as dropped
out, draw the cursor</code></pre>
<ul>
<li><strong>Reading back without stalls:</strong> three staging textures are used in turn. The CPU reads the one copied two frames ago, so <code>Map</code> never waits on the GPU. At 4K and 60 frames a second, about 2 GB/s is read back, well within what the hardware can do.</li>
<li><strong>Timestamps</strong> come from when each frame was on screen (<code>LastPresentTime</code>, or the system-relative time of a Windows.Graphics.Capture frame), in a 1/90000 timebase. Time paused is subtracted.</li>
<li><strong>The frame rate</strong> is capped at the setting (30 by default, or 60). When nothing changes, no frame is written, apart from one each second so players can seek. A still screen costs almost nothing.</li>
<li><strong>Buffers</strong> come from a fixed pool the size of the region, made at the start. Nothing is allocated per frame.</li>
</ul>
<h2 id="encoding">5. Encoding</h2>
<ul>
<li><strong>The encoder</strong> is the first of <code>h264_nvenc</code>, <code>h264_amf</code>, <code>h264_qsv</code>, <code>h264_mf</code> and <code>libx264</code> that opens at the region's size. Each is opened once, so a driver that's present but broken falls through to the next. The one used is logged and stored with the recording.</li>
<li><strong>The input</strong> is BGRA as <code>AV_PIX_FMT_BGR0</code>. NVENC converts it to its own colour format on the GPU. For the CPU encoders, <code>swscale</code> converts it to <code>yuv420p</code> first.</li>
<li><strong>Quality</strong> is constant quality, about CQ 23 for NVENC (CRF 23 for <code>libx264</code>), with a keyframe every 2 s. NVENC uses preset <code>p4</code>. Screen content (text, flat colour) compresses well at this setting. Phase 1 measures the file sizes.</li>
<li><strong>Crash-safe files:</strong> the MP4 is written fragmented (<code>movflags=frag_keyframe+empty_moov</code>) to <code>%TEMP%\mcap\recording-*.mp4</code>. If mcap or Windows crashes mid-recording, the file still plays up to the last fragment. On the next start, a leftover file is offered for import.</li>
<li><strong>Bindings:</strong> <code>modules/ffmpeg</code> already binds everything encoding needs (<code>avcodec_find_encoder_by_name</code>, <code>avcodec_send_frame</code>, <code>avcodec_receive_packet</code>, <code>avformat_alloc_output_context2</code>, <code>avformat_write_header</code>, <code>av_interleaved_write_frame</code>, <code>av_write_trailer</code>, <code>avio_open</code>). <code>swresample</code> ships, but has no bindings yet. Audio needs them (phase 5).</li>
</ul>
<h2 id="library">6. Into the library</h2>
<ul>
<li><strong>Like an import:</strong> on stop, the MP4 goes through the path imported videos take (<code>import_video_mcap</code> and <code>mcap_write_streamed</code>, <code>video_poster</code> for the thumbnail frame) into <code>Screen Captures</code>. The temporary file is then deleted.</li>
<li><strong>metadata.json</strong> is a still's: the title and app of the window under the region's middle when recording started, and when it started.</li>
<li><strong>capture.json</strong> is a still's (region, position, app path) plus <code>Duration</code>, <code>FrameRate</code>, <code>VideoCodec</code>, <code>Encoder</code> (<code>h264_nvenc</code> ...), <code>Capture</code> (<code>DXGI</code> or <code>WGC</code>), <code>DroppedFrames</code> and <code>Paused</code> (seconds).</li>
<li><strong>The catalog</strong> today marks a capture as a video only from <code>VideoCodec</code> in <code>import.json</code> (<code>app/catalog.jai</code>). It will read the same keys from <code>capture.json</code>, so recordings show as videos, play with Open, and are skipped by segmentation.</li>
<li><strong>The thumbnail frame:</strong> <code>video_poster</code> picks a frame a tenth of the way in, at most 3 s, to skip a phone video's black start. A screen recording has no black start, but the first second is often the region being chosen. The same rule is kept unless it picks badly.</li>
</ul>
<h2 id="hotkeys">7. Hotkeys</h2>
<div class="table-wrap"><table>
<tr><th>Action in <code>[global_hotkeys]</code></th><th>Default</th><th>What it does</th></tr>
<tr><td><code>capture_region</code> (stills, built)</td><td><code>PrintScreen</code></td><td>Opens the overlay for a still.</td></tr>
<tr><td><code>record_region</code></td><td><code>Ctrl-PrintScreen</code></td><td>Opens the overlay in video mode. While a recording runs, it stops and saves it.</td></tr>
</table></div>
<ul>
<li><strong>One key for start and stop.</strong> It's registered all the time, as <code>capture_region</code> is, and only <code>global_hotkey_fired</code> needs to know whether a recording is running. No key is borrowed from the app being recorded: a plain Escape, or Windows' own Ctrl+Escape, would be.</li>
<li><strong>Escape inside the overlay</strong>, before recording starts, cancels as it does for stills.</li>
<li><strong>Saving and discarding:</strong> stopping always saves. Discarding is the toolbar's <em>Discard</em> button, which asks first.</li>
<li><strong>PrintScreen during a recording</strong> still takes a still. The overlay would freeze the screen, and that freeze would end up in the video, so the still is taken without the overlay: the whole screen under the cursor, saved as usual. Open question 1 asks whether that's right.</li>
<li><strong>Settings files:</strong> <code>record_region</code> is a new <code>windows_only</code> action. Existing settings files get it appended on the next launch, as <code>capture_region</code> was.</li>
</ul>
<h2 id="audio">8. Audio <span class="tag later">Phase 5</span></h2>
<ul>
<li><strong>Sources:</strong> system audio comes from WASAPI loopback on the default output device, and the microphone from WASAPI capture. Each is optional, set in the toolbar and remembered under <code>[video]</code>.</li>
<li><strong>Mixing:</strong> both are mixed to 48 kHz stereo float with <code>swresample</code> (bindings to generate), and encoded as AAC at 160 kb/s into the same MP4.</li>
<li><strong>Sync:</strong> each audio packet's QPC time is lined up with the video's.</li>
<li><strong>Silence:</strong> loopback sends nothing while nothing plays, so silence is written to fill the gaps.</li>
</ul>
<h2 id="phases">9. Phases</h2>
<p>Each phase ends with something to run and check, and is a commit (0.1.3 onwards).</p>
<div class="phase">
<h3>Phase 1: <code>record-probe</code> with Desktop Duplication <span class="tag decided">Done</span></h3>
<p>A command-line tool, <code>tools/record_probe.jai</code>, built with <code>jai first.jai - record_probe</code>:</p>
<ul>
<li>It lists adapters and outputs, picks a screen or region, and duplicates it on the output's adapter.</li>
<li>It records N seconds through the pipeline (section 4) and encoder (section 5) to an MP4.</li>
<li>It prints which method and encoder it used, frames captured, written and dropped, time per stage, and the file size.</li>
</ul>
<p>It answers four questions on your laptop:</p>
<ul>
<li>Does <code>DuplicateOutput1</code> succeed on the internal screen, and on an external one?</li>
<li>Does it succeed with CUDA loaded in the same process? mcap loads onnxruntime with CUDA, which may keep the process on the NVIDIA GPU.</li>
<li>Does a window with <code>WDA_EXCLUDEFROMCAPTURE</code> stay out of the frames?</li>
<li>What does 4K at 60 frames a second cost?</li>
</ul>
<p><strong>Done when:</strong> 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A <code>DXGI_ERROR_UNSUPPORTED</code> is also an answer: it means phase 2 is needed on this machine.</p>
<div class="note"><strong>Result (2026-10-10).</strong> Release build, RTX 5080 laptop. The screens are a 3840 × 2160 external and the 2560 × 1600 internal; DXGI lists both on the NVIDIA adapter.
<ul>
<li><strong>The done-when run:</strong> 60.2 s of the 4K screen at 30 fps, with a square moving across it the whole time (<code>-animate</code>):
<ul>
<li>3,602 images taken from the screen and 1,802 frames sent, all of them encoded with NVENC and none dropped.</li>
<li>18 MB a minute.</li>
<li>FFmpeg reads it back as 60.07 s at 30 fps.</li>
<li>Per frame: 4.4 ms to read back and 13.4 ms to encode, on separate threads.</li>
</ul>
The frame decoded back matches the one captured, cursor included, by eye. Playing it in VLC and Media Player is still to check.</li>
<li><strong>60 fps:</strong> all 59.3 changes a second are kept. On a fixed tick grid, a 60 Hz screen beat against the 60 fps cap and lost about 7% of frames. Frames are now sent as soon as their turn is due, on a schedule that advances by exactly one frame each time. Caps of 30, 60 and 24 give 29.8, 59.3 and 23.8 frames a second.</li>
<li><strong>The questions:</strong>
<ul>
<li><strong>Both screens</strong> duplicate with <code>DuplicateOutput1</code>.</li>
<li><strong>The plain <code>DuplicateOutput</code></strong> works too in a per-monitor DPI-aware process. The documented failure doesn't happen here; <code>DuplicateOutput1</code> stays the first choice.</li>
<li><strong>With a CUDA context in the process</strong>, duplication still works.</li>
<li><strong><code>WDA_EXCLUDEFROMCAPTURE</code> is respected:</strong> the excluded square is 0% present in the frame and the control square 100%.</li>
<li><strong>The two-GPU failure</strong> wasn't reproduced. Task Manager shows the Intel GPU, but no screen is attached to it now: the laptop is in discrete-GPU mode. Testing it means switching the laptop to Optimus (hybrid) mode in the NVIDIA Control Panel or the BIOS, which needs a restart.</li>
</ul></li>
<li><strong>Encoders</strong>, 4K at 30 fps, 8 s of the same motion. All of them keep up:
<div class="table-wrap"><table>
<tr><th>Encoder</th><th>Encode per frame</th><th>MB a minute</th></tr>
<tr><td><code>h264_nvenc</code></td><td>13.7 ms</td><td>23.4</td></tr>
<tr><td><code>h264_qsv</code> (the Intel GPU, though no screen is attached to it)</td><td>12.3 ms</td><td>16.7</td></tr>
<tr><td><code>h264_mf</code></td><td>16.8 ms</td><td>24.0</td></tr>
<tr><td><code>libx264</code> (veryfast, CRF 23)</td><td>14.9 ms</td><td>11.4</td></tr>
<tr><td><code>h264_amf</code></td><td colspan="2">Not available: no AMD driver (<code>amfrt64.dll</code>), so it falls through to the next</td></tr>
</table></div>
Most of NVENC's 13.7 ms is FFmpeg copying the 33 MB frame before the GPU sees it. Handing it the frame without the copy is a later optimisation, not needed at 4K 60.</li>
<li><strong>A crash mid-recording:</strong> killed about 7 s in, the fragmented MP4 still reads back with 6 s of video. At most the last keyframe interval (2 s) is lost.</li>
<li><strong>The minimum size:</strong> 239 × 300 is refused, and 241 × 181 records at 240 × 180. The minimum is checked before rounding to even.</li>
</ul>
<p><strong>Changes from the plan:</strong></p>
<ul>
<li><strong>The DPI check:</strong> once, a process that isn't DPI aware saw the 4K screen as 1920 × 1080, and the region would have been the wrong part of it. This couldn't be reproduced afterwards. <code>dxgi_open</code> now compares the duplication's size with the desktop's and refuses when they differ.</li>
<li><strong>The probe's extra options:</strong> <code>-animate</code>, which moves a square for measuring throughput, and <code>-check FILE</code>, which reads a recording back.</li>
</ul></div>
</div>
<div class="phase">
<h3>Phase 2: Windows.Graphics.Capture fallback <span class="tag decided">Done</span></h3>
<p>The second source behind the same interface: WinRT bindings, polling <code>TryGetNextFrame</code>, the cursor and border settings. The fallback rules from 3.2, and <code>-capture dxgi|wgc|auto</code> on the probe.</p>
<p><strong>Done when:</strong></p>
<ul>
<li>The probe records the same screens with <code>-capture wgc</code>.</li>
<li><code>auto</code> picks Desktop Duplication where it works and Windows.Graphics.Capture where it doesn't. The latter is shown either on the two-GPU laptop or on a rotated screen.</li>
<li>Whether the yellow border can be turned off is recorded.</li>
</ul>
<div class="note"><strong>Result (2026-10-10).</strong> Release build, the same laptop and screens as phase 1.
<ul>
<li><strong>Built:</strong>
<ul>
<li><code>wgc_source.jai</code>: hand-written bindings for eight WinRT interfaces.</li>
<li><code>capture_gpu.jai</code>: what both sources share, namely the device on the screen's adapter, the region copy and the staging ring. Each source is now only "the next image's region into a texture".</li>
<li><code>Record_Options.capture</code>: <code>.AUTO</code>, <code>.DXGI</code> or <code>.WGC</code>.</li>
<li>The probe's <code>-capture auto|dxgi|wgc</code>.</li>
</ul></li>
<li><strong>The fallback:</strong>
<ul>
<li><code>auto</code> picks Desktop Duplication where it works.</li>
<li>With <code>-simulate-unsupported</code>, which makes duplication fail with <code>DXGI_ERROR_UNSUPPORTED</code> as on a two-GPU laptop, <code>auto</code> logs why and records with Windows.Graphics.Capture.</li>
<li><code>-capture dxgi</code> with the same simulated failure refuses, with exit code 3.</li>
<li>The real two-GPU case is shelved: this laptop's GPU is pinned in the BIOS.</li>
</ul></li>
<li><strong>Recording:</strong> 4K at 60 fps with motion: 56.6 frames a second taken and every one encoded, none dropped. Windows.Graphics.Capture delivers a little under the screen's 60 Hz; Desktop Duplication delivered 59.3. A 641 × 481 region on the internal screen gives the same frame as Desktop Duplication's in phase 1.</li>
<li><strong>The border:</strong> <code>IsBorderRequired = false</code> works for mcap as an unpackaged app on Windows 11, with no access request. A GDI sample of the screen's outer 6 pixels found no yellow pixels with the border off, and 14,214 with it left on (<code>-keep-border</code>, the control).</li>
<li><strong>The cursor:</strong> with the mouse put in the region (<code>-cursor-test</code>), Windows.Graphics.Capture's own cursor and the one <code>cursor.jai</code> draws into Desktop Duplication's frames are the same, pixel for pixel, by eye at 4×.</li>
<li><strong><code>WDA_EXCLUDEFROMCAPTURE</code></strong> is respected by Windows.Graphics.Capture too: the excluded square is 0% present and the control square 100%.</li>
<li><strong>Waiting for frames:</strong> each new frame sets an event, through a small COM handler for <code>FrameArrived</code>, which Windows took. The polling fallback, for when Windows refuses the handler (<code>-poll</code>), takes 41 frames a second of a 60 Hz screen, enough for a 30 fps recording (29.0 sent), at a 1 ms timer resolution.</li>
</ul>
<p><strong>Changes from the plan:</strong></p>
<ul>
<li><strong>Events, not polling.</strong> The plan said to poll <code>TryGetNextFrame</code>, to avoid writing a WinRT event handler in Jai. The handler took about 30 lines, so it's the default, and polling is the fallback.</li>
<li><strong>The source opens on the capture thread</strong>, for both methods. WinRT objects belong to the apartment that made them, so <code>record_start</code> starts the capture thread and waits for it to say whether its source opened.</li>
<li><strong>The WinRT calls</strong> (<code>RoInitialize</code>, <code>RoGetActivationFactory</code>, HSTRINGs) are loaded from <code>combase.dll</code> at run time. Their import library, <code>runtimeobject.lib</code>, names no DLL of its own, which Jai's <code>#library</code> needs.</li>
<li><strong>Test switches:</strong> <code>Record_Options.testing</code> holds three, for the probe: simulate the two-GPU failure, force polling, keep the border.</li>
</ul></div>
</div>
<div class="phase">
<h3>Phase 3: Recording in the app</h3>
<p>The pieces:</p>
<ul>
<li>The <code>record_region</code> action, which starts a recording or, during one, stops it.</li>
<li>The overlay's video mode: red outline, the 240 × 180 minimum, the drag kept to one screen.</li>
<li>The capture and encoder threads in the app.</li>
<li>Stopping, and the file into the library (section 6).</li>
<li>The catalog reading <code>capture.json</code>.</li>
<li>A shutdown step that finishes the file.</li>
<li>Tests of the minimum-size and even-size rules, and of the <code>capture.json</code> round trip.</li>
</ul>
<p><strong>Done when:</strong></p>
<ul>
<li>Ctrl+PrintScreen, a drag, then Ctrl+PrintScreen again leaves a playable video in Screen Captures, shown as a video in the grid and opened in the Capture window.</li>
<li>A 239 × 300 drag doesn't start a recording.</li>
<li>Quitting mid-recording keeps it.</li>
</ul>
</div>
<div class="phase">
<h3>Phase 4: Border, toolbar, pause and recovery</h3>
<p>The pieces:</p>
<ul>
<li>The border and toolbar windows, kept out of the capture.</li>
<li>Pause, resume, and Discard with a confirmation.</li>
<li>The dropped-frame count.</li>
<li>Recovery from <code>ACCESS_LOST</code>.</li>
<li>Offering a leftover temporary file for import at startup.</li>
<li>The <code>[video]</code> settings: frame rate, cursor, capture method.</li>
</ul>
<p><strong>Done when:</strong> neither the border nor the toolbar appears in a recording. Pausing for 10 s leaves no gap in the video. A UAC prompt or a resolution change mid-recording leaves a video that plays through.</p>
</div>
<div class="phase">
<h3>Phase 5: Audio</h3>
<p>Section 8: <code>swresample</code> bindings, loopback and microphone, mixing, AAC.</p>
<p><strong>Done when:</strong> a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.</p>
</div>
<h2 id="risks">10. Risks</h2>
<div class="risk"><strong>Two GPUs.</strong> This is the reason for the fallback. One step beyond it: with the NVIDIA control panel set to the high-performance GPU for everything, both methods might be affected. Phase 1 tests your laptop with and without that setting.</div>
<div class="risk"><strong>HDR screens.</strong> Asking for 8-bit BGRA from an HDR screen gives Windows' own tone mapping, which can look washed out. Recording HDR (16-bit float frames, HEVC 10-bit) is out of scope. The toolbar will say when the screen is HDR.</div>
<div class="risk"><strong>Protected content</strong> (DRM video in browsers, some streaming apps) records as black with either method. Nothing can be done about it, so it's documented in the README.</div>
<div class="risk"><strong>Laptop power.</strong> 60 frames a second at 4K keeps the GPU awake. The default is 30, and the probe measures the cost.</div>
<h2 id="questions">11. Open questions</h2>
<ol>
<li><strong>PrintScreen during a recording:</strong> take the whole screen without the overlay (this plan), ignore it, or freeze anyway and accept the frozen moment in the video?</li>
<li><strong>The folder:</strong> recordings in <em>Screen Captures</em> beside stills, or a folder of their own (<em>Screen Recordings</em>)? This plan uses <em>Screen Captures</em>, so the grid shows both together.</li>
<li><strong>A time limit:</strong> none by default, apart from stopping when the disk has less than 2 GB free?</li>
<li><strong>A countdown</strong> (3, 2, 1) before recording starts, as Snagit has? This plan starts at once.</li>
<li><strong>Mouse clicks:</strong> should they be shown in the video, for example as a ring at each click, as Snagit can?</li>
</ol>
</main>
</body>
</html>