[0.1.2] Video capture plan: Desktop Duplication with a Windows.Graphics.Capture fallback, NVENC, Ctrl+PrintScreen
- video-capture-plan.html: recording a region of one screen to H.264 MP4 and into the library as a video .mcap. DXGI Desktop Duplication first, on a device of its own on the screen's adapter; Windows.Graphics.Capture when duplication cannot run (two GPUs, rotated screens, too many duplications). Encoders tried in turn: NVENC, AMF, QSV, Media Foundation, libx264, from the FFmpeg already shipped. Ctrl+PrintScreen opens the overlay in a video mode (a 240 x 180 minimum, kept to one screen) and stops the recording. Five phases, the first a command-line probe that answers the two-GPU question on the laptop.
This commit is contained in:
parent
7f8bcdd1ad
commit
5758f0a071
@ -29,7 +29,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
GIT_BRANCH :: "check";
|
||||
GIT_HASH :: "-";
|
||||
VERSION :: "0.1.1";
|
||||
VERSION :: "0.1.2";
|
||||
BUILD_DATE :: "-";
|
||||
DEBUG :: true;
|
||||
COMPILE_TESTS :: true;
|
||||
|
||||
@ -8,7 +8,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
|
||||
|
||||
VERSION :: "0.1.1";
|
||||
VERSION :: "0.1.2";
|
||||
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
|
||||
// builds never write over the ones you run by hand.
|
||||
build_directory := "bin";
|
||||
|
||||
353
video-capture-plan.html
Normal file
353
video-capture-plan.html
Normal file
@ -0,0 +1,353 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Video Capture Plan</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #fbfaf8; --surface: #ffffff; --text: #1d1d1f; --dim: #5d5d66; --line: #e3e1dc;
|
||||
--accent: #3a5bd9; --accent-soft: #e9edfb; --warn: #9a5b00; --warn-soft: #fbf1e1;
|
||||
--code-bg: #f2f0ec; --ok: #2e7d4f; --ok-soft: #e5f3ea;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root:not([data-theme="light"]) {
|
||||
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
|
||||
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
|
||||
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
|
||||
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
|
||||
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--text);
|
||||
font: 16px/1.6 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||
}
|
||||
main { max-width: 860px; margin: 0 auto; padding: 40px 16px 80px; }
|
||||
header p { color: var(--dim); margin-top: 4px; }
|
||||
h1 { font-size: 2rem; line-height: 1.2; margin: 0; }
|
||||
h2 { font-size: 1.35rem; margin: 48px 0 8px; padding-top: 16px; border-top: 1px solid var(--line); }
|
||||
h3 { font-size: 1.05rem; margin: 28px 0 6px; }
|
||||
p, li { max-width: 72ch; }
|
||||
a { color: var(--accent); }
|
||||
code, pre { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; font-size: 0.88em; }
|
||||
code { background: var(--code-bg); padding: 1px 5px; border-radius: 4px; }
|
||||
pre { background: var(--code-bg); padding: 12px 14px; border-radius: 8px; overflow-x: auto; line-height: 1.45; }
|
||||
pre code { background: none; padding: 0; }
|
||||
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 0.94rem; }
|
||||
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||
th { color: var(--dim); font-weight: 600; }
|
||||
.table-wrap { overflow-x: auto; }
|
||||
.summary { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 16px 20px; margin-top: 24px; }
|
||||
.summary ul { margin: 6px 0 0; padding-left: 20px; }
|
||||
.note { background: var(--accent-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
|
||||
.risk { background: var(--warn-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
|
||||
.risk strong { color: var(--warn); }
|
||||
.tag { display: inline-block; font-size: 0.78rem; font-weight: 600; padding: 1px 8px; border-radius: 99px; background: var(--accent-soft); color: var(--accent); }
|
||||
.tag.later { background: var(--code-bg); color: var(--dim); }
|
||||
.tag.decided { background: var(--ok-soft); color: var(--ok); }
|
||||
nav ol { columns: 2; column-gap: 32px; padding-left: 20px; margin: 8px 0 0; }
|
||||
@media (max-width: 600px) { nav ol { columns: 1; } h1 { font-size: 1.6rem; } }
|
||||
.phase { border-left: 3px solid var(--accent); padding: 2px 0 2px 14px; margin: 16px 0; }
|
||||
.phase h3 { margin-top: 0; }
|
||||
figure { margin: 18px 0; }
|
||||
figcaption { color: var(--dim); font-size: 0.88rem; margin-top: 6px; }
|
||||
svg text { fill: var(--text); font: 13px system-ui, sans-serif; }
|
||||
svg .dim { fill: var(--dim); }
|
||||
svg .box { fill: var(--surface); stroke: var(--line); }
|
||||
svg .hl { fill: var(--accent-soft); stroke: var(--accent); }
|
||||
svg .arrow { stroke: var(--dim); fill: none; marker-end: url(#a); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
|
||||
<header>
|
||||
<h1>Screen video capture in mcap</h1>
|
||||
<p>Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · <strong>not started</strong></p>
|
||||
</header>
|
||||
|
||||
<div class="summary">
|
||||
<strong>In short</strong>
|
||||
<ul>
|
||||
<li><span class="tag decided">Decided</span> <strong>DXGI Desktop Duplication first.</strong> If it can't run, for example on a laptop with two GPUs where it fails with <code>DXGI_ERROR_UNSUPPORTED</code>, recording falls back to <strong>Windows.Graphics.Capture</strong>. Both sit behind one interface, so everything after the capture is shared. It's more code than Windows.Graphics.Capture alone, but each method covers the cases where the other fails.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>Encoding uses the FFmpeg mcap already ships.</strong> It tries NVENC first, then AMD AMF, Intel QSV, Media Foundation and <code>libx264</code>. The output is H.264 in MP4, with frames written only when the screen changes. That FFmpeg build already includes every one of these encoders.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen starts a recording.</strong> It opens the same frozen-screen overlay as stills, in a video mode. The region must be at least <strong>240 × 180</strong> pixels, which Snagit doesn't require.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen again stops it</strong> and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>A recording becomes an <code>.mcap</code></strong> in <em>Screen Captures</em>, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.</li>
|
||||
<li>It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.</li>
|
||||
<li><span class="tag later">Later</span> Audio is phase 5. Recording one window, and regions that span two screens, come after.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<nav>
|
||||
<ol>
|
||||
<li><a href="#scope">Scope</a></li>
|
||||
<li><a href="#using">Using it</a></li>
|
||||
<li><a href="#sources">Grabbing frames</a></li>
|
||||
<li><a href="#pipeline">The frame pipeline</a></li>
|
||||
<li><a href="#encoding">Encoding</a></li>
|
||||
<li><a href="#library">Into the library</a></li>
|
||||
<li><a href="#hotkeys">Hotkeys</a></li>
|
||||
<li><a href="#audio">Audio</a></li>
|
||||
<li><a href="#phases">Phases</a></li>
|
||||
<li><a href="#risks">Risks</a></li>
|
||||
<li><a href="#questions">Open questions</a></li>
|
||||
</ol>
|
||||
</nav>
|
||||
|
||||
<h2 id="scope">1. Scope</h2>
|
||||
<p><strong>In:</strong></p>
|
||||
<ul>
|
||||
<li>Recording a rectangle on one screen, or a whole screen, to H.264 MP4 at up to 60 frames a second, with the mouse cursor.</li>
|
||||
<li>Pause and resume, stop, and discard.</li>
|
||||
<li>Saving into the library with the window and app it was recorded from.</li>
|
||||
<li>System audio and the microphone, in the last phase.</li>
|
||||
</ul>
|
||||
<p><strong>Out, for now:</strong></p>
|
||||
<ul>
|
||||
<li>Recording one window that follows it as it moves.</li>
|
||||
<li>Regions that span two screens.</li>
|
||||
<li>Webcam overlay.</li>
|
||||
<li>Trimming and editing.</li>
|
||||
<li>HDR output.</li>
|
||||
<li>macOS and Linux.</li>
|
||||
</ul>
|
||||
<p>Windows only, like the stills overlay. It needs Windows 10 version 2004 or later; the Windows.Graphics.Capture features used below need that version.</p>
|
||||
|
||||
<h2 id="using">2. Using it</h2>
|
||||
<ol>
|
||||
<li><strong>Ctrl+PrintScreen</strong> freezes the screens and shows the overlay from stills, in its video mode. The outline and the magnifier are red, not orange, so you can see it will record.</li>
|
||||
<li><strong>Drag a region.</strong> The magnifier's label shows its size.
|
||||
<ul>
|
||||
<li>The drag is kept on the screen where it started (see <a href="#sources">3.4</a>).</li>
|
||||
<li><strong>Below 240 × 180</strong>, the outline is drawn dashed and the label reads <code>320 × 120 · at least 240 × 180</code>. Letting go then starts nothing, and the overlay stays open to drag again.</li>
|
||||
<li>A click without dragging, or Enter, takes the whole screen, as for stills.</li>
|
||||
<li>Escape or a right click cancels, as for stills.</li>
|
||||
</ul></li>
|
||||
<li><strong>Recording starts</strong> as soon as you let go, with no countdown (see <a href="#questions">open question 4</a>).
|
||||
<ul>
|
||||
<li>A thin red border is drawn just <em>outside</em> the region, so it's never in the video.</li>
|
||||
<li>A small toolbar sits beside the region, outside it where there's room. It shows the time, the size and frames dropped, with <em>Pause</em>, <em>Stop</em> and <em>Discard</em>.</li>
|
||||
<li>Both the border and the toolbar are kept out of the capture with <code>SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE)</code>, which needs Windows 10 2004. Whether Desktop Duplication respects this is checked in phase 1. The border is outside the region either way, so only the toolbar depends on it.</li>
|
||||
</ul></li>
|
||||
<li><strong>Ctrl+PrintScreen</strong> again, or <em>Stop</em>, ends the recording at once:
|
||||
<ul>
|
||||
<li>The encoder is flushed and the file closed.</li>
|
||||
<li>The file becomes an <code>.mcap</code>.</li>
|
||||
<li>mcap comes to the front showing it in the Capture window, as it does after a still.</li>
|
||||
</ul></li>
|
||||
</ol>
|
||||
<p>Dimensions are rounded down to even numbers after the minimum is checked, since H.264's 4:2:0 colour needs that. 240 × 180 is already even, so a region that passes the check stays at least 240 × 180.</p>
|
||||
|
||||
<h2 id="sources">3. Grabbing frames</h2>
|
||||
<p>One interface, two implementations, chosen when the recording starts:</p>
|
||||
<pre><code>Capture_Source :: struct {
|
||||
start: (source: *Capture_Source, screen: Screen_Ref, region: Rect, cursor: bool) -> ok: bool, error: string, fall_back: bool;
|
||||
next_frame: (source: *Capture_Source, timeout_ms: u32) -> Captured_Frame, status: Frame_Status; // NEW, SAME, LOST, FAILED
|
||||
stop: (source: *Capture_Source);
|
||||
name: string; // "DXGI" or "WGC", for the log and the toolbar's tooltip
|
||||
}
|
||||
|
||||
Captured_Frame :: struct {
|
||||
texture: *ID3D11Texture2D; // the region only, region-sized, owned by the source until the next call
|
||||
time: s64; // QueryPerformanceCounter ticks: when it was on screen
|
||||
cursor: Cursor_State; // DXGI only: where, visible, and the shape when it changed
|
||||
}</code></pre>
|
||||
<p>Both run on a capture thread of their own, with <strong>their own D3D11 device</strong>, not the app's. The app's device starts on WARP and moves to the GPU later (<code>#WarpFirstStartup</code> in <code>graphics/DX11.jai</code>). Desktop Duplication also needs a device on the GPU the screen is connected to, which may not be the one the app draws with.</p>
|
||||
|
||||
<h3>3.1 DXGI Desktop Duplication (first choice)</h3>
|
||||
<ul>
|
||||
<li><strong>Finding the screen:</strong> <code>CreateDXGIFactory1</code>, then <code>EnumAdapters1</code> and <code>EnumOutputs</code>, to find the output whose desktop rectangle holds the region. The device is created on <em>that</em> output's adapter, with <code>D3D11CreateDevice(adapter, D3D_DRIVER_TYPE_UNKNOWN, ...)</code>. The jai <code>dxgi</code> module already has every type needed: <code>IDXGIOutput5</code>, <code>DuplicateOutput1</code>, <code>IDXGIOutputDuplication</code>, <code>DXGI_OUTDUPL_FRAME_INFO</code>.</li>
|
||||
<li><strong>Duplicating:</strong> <code>IDXGIOutput5::DuplicateOutput1</code> with <code>B8G8R8A8_UNORM</code>. mcap is per-monitor DPI aware (v2), and the plain <code>DuplicateOutput</code> is documented to fail for such processes. Phase 1 confirms this, and falls back to <code>IDXGIOutput1::DuplicateOutput</code> where <code>IDXGIOutput5</code> is missing.</li>
|
||||
<li><strong>Each frame:</strong>
|
||||
<ul>
|
||||
<li><code>AcquireNextFrame(timeout)</code> waits for one.</li>
|
||||
<li><code>LastPresentTime == 0</code> means only the mouse moved: the cursor is updated and no new image is copied.</li>
|
||||
<li><code>CopySubresourceRegion</code> copies the region into a region-sized texture.</li>
|
||||
<li><code>ReleaseFrame</code> is called straight after the copy. Holding a frame stalls the desktop.</li>
|
||||
</ul></li>
|
||||
<li><strong>The cursor isn't drawn into the image.</strong> Its position comes with each frame, and its shape from <code>GetFramePointerShape</code> when <code>PointerShapeBufferSize</code> is set. It is drawn onto the copy on the CPU, after it's read back (<a href="#pipeline">4</a>). There are three kinds of shape to handle: colour, masked colour, and monochrome. The monochrome kind (the text I-beam) inverts what's under it.</li>
|
||||
<li><strong>Errors while recording:</strong>
|
||||
<ul>
|
||||
<li><code>DXGI_ERROR_ACCESS_LOST</code> comes from a display mode change, the UAC or lock screen, or a game switching to fullscreen. The duplication is made again, retrying every 250 ms, and the recording goes on. The toolbar shows the gap.</li>
|
||||
<li><code>DXGI_ERROR_WAIT_TIMEOUT</code> means nothing changed.</li>
|
||||
<li>Anything else ends the recording, keeping what was captured.</li>
|
||||
</ul></li>
|
||||
</ul>
|
||||
|
||||
<h3>3.2 When to fall back</h3>
|
||||
<p>The choice is made once, at the start, and logged:</p>
|
||||
<div class="table-wrap"><table>
|
||||
<tr><th>Desktop Duplication says</th><th>Meaning</th><th>Then</th></tr>
|
||||
<tr><td><code>DXGI_ERROR_UNSUPPORTED</code> from <code>DuplicateOutput1</code></td><td>Two GPUs, and the process or device isn't on the one driving the screen. Also returned when no driver can duplicate.</td><td>Windows.Graphics.Capture</td></tr>
|
||||
<tr><td><code>DXGI_ERROR_NOT_CURRENTLY_AVAILABLE</code></td><td>Too many duplications of this screen at once (other recorders running)</td><td>Windows.Graphics.Capture</td></tr>
|
||||
<tr><td><code>E_ACCESSDENIED</code></td><td>The secure desktop (UAC, lock screen) is showing</td><td>Retry for 2 s, then tell the user</td></tr>
|
||||
<tr><td>Screen rotated (<code>DXGI_OUTDUPL_DESC.Rotation</code> isn't identity)</td><td>Frames would come sideways</td><td>Windows.Graphics.Capture, which hands them over upright</td></tr>
|
||||
<tr><td>No output holds the region</td><td>The screens changed since the freeze</td><td>Error, nothing recorded</td></tr>
|
||||
</table></div>
|
||||
<p>A setting forces one method, for testing and for machines where one misbehaves: <code>capture: auto</code> (the default), <code>dxgi</code> or <code>wgc</code>, under a new <code>[video]</code> section of <code>settings.config</code>.</p>
|
||||
|
||||
<h3>3.3 Windows.Graphics.Capture (fallback)</h3>
|
||||
<ul>
|
||||
<li><strong>Setup:</strong> the WinRT runtime is started on the capture thread (<code>RoInitialize</code>, multithreaded).
|
||||
<ul>
|
||||
<li><code>IGraphicsCaptureItemInterop::CreateForMonitor</code> makes the item for the screen.</li>
|
||||
<li><code>CreateDirect3D11DeviceFromDXGIDevice</code> wraps the capture thread's device.</li>
|
||||
<li><code>Direct3D11CaptureFramePool::CreateFreeThreaded</code> makes a pool of 2 BGRA frames.</li>
|
||||
</ul></li>
|
||||
<li><strong>Polling, not events:</strong> <code>TryGetNextFrame</code> is called at the frame rate, with the same timeout as DXGI. This avoids implementing WinRT event-handler objects in Jai.</li>
|
||||
<li><strong>Cursor and border:</strong> <code>IsCursorCaptureEnabled</code> draws the cursor into the image, so the cursor step in 3.1 is skipped. <code>IsBorderRequired = false</code> removes the yellow border on Windows 11; whether an unpackaged app may do that without asking is checked in phase 2. Windows 10 always shows the border.</li>
|
||||
<li><strong>Bindings:</strong> roughly six WinRT interfaces, written by hand as COM vtables, plus HSTRINGs (<code>WindowsCreateString</code>). This is the bulk of phase 2.</li>
|
||||
</ul>
|
||||
|
||||
<h3>3.4 One screen per recording</h3>
|
||||
<p>Desktop Duplication works one output at a time, and Windows.Graphics.Capture one monitor at a time. A region over two screens would need two captures stitched together, possibly at different DPI and refresh rates. In video mode the overlay keeps the drag on the screen where it started. Spanning screens is <span class="tag later">Later</span>.</p>
|
||||
|
||||
<h2 id="pipeline">4. The frame pipeline</h2>
|
||||
<pre><code>capture thread encoder thread
|
||||
next_frame ─► region texture ─► staging ring (3) ─► queue (≤ 8 frames) ─► AVFrame bgr0 ─► encoder ─► MP4
|
||||
Map the one from drops the oldest when full,
|
||||
2 frames back, copy counted as dropped
|
||||
out, draw the cursor</code></pre>
|
||||
<ul>
|
||||
<li><strong>Reading back without stalls:</strong> three staging textures are used in turn. The CPU reads the one copied two frames ago, so <code>Map</code> never waits on the GPU. At 4K and 60 frames a second, about 2 GB/s is read back, well within what the hardware can do.</li>
|
||||
<li><strong>Timestamps</strong> come from when each frame was on screen (<code>LastPresentTime</code>, or the system-relative time of a Windows.Graphics.Capture frame), in a 1/90000 timebase. Time paused is subtracted.</li>
|
||||
<li><strong>The frame rate</strong> is capped at the setting (30 by default, or 60). When nothing changes, no frame is written, apart from one each second so players can seek. A still screen costs almost nothing.</li>
|
||||
<li><strong>Buffers</strong> come from a fixed pool the size of the region, made at the start. Nothing is allocated per frame.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="encoding">5. Encoding</h2>
|
||||
<ul>
|
||||
<li><strong>The encoder</strong> is the first of <code>h264_nvenc</code>, <code>h264_amf</code>, <code>h264_qsv</code>, <code>h264_mf</code> and <code>libx264</code> that opens at the region's size. Each is opened once, so a driver that's present but broken falls through to the next. The one used is logged and stored with the recording.</li>
|
||||
<li><strong>The input</strong> is BGRA as <code>AV_PIX_FMT_BGR0</code>. NVENC converts it to its own colour format on the GPU. For the CPU encoders, <code>swscale</code> converts it to <code>yuv420p</code> first.</li>
|
||||
<li><strong>Quality</strong> is constant quality, about CQ 23 for NVENC (CRF 23 for <code>libx264</code>), with a keyframe every 2 s. NVENC uses preset <code>p4</code>. Screen content (text, flat colour) compresses well at this setting. Phase 1 measures the file sizes.</li>
|
||||
<li><strong>Crash-safe files:</strong> the MP4 is written fragmented (<code>movflags=frag_keyframe+empty_moov</code>) to <code>%TEMP%\mcap\recording-*.mp4</code>. If mcap or Windows crashes mid-recording, the file still plays up to the last fragment. On the next start, a leftover file is offered for import.</li>
|
||||
<li><strong>Bindings:</strong> <code>modules/ffmpeg</code> already binds everything encoding needs (<code>avcodec_find_encoder_by_name</code>, <code>avcodec_send_frame</code>, <code>avcodec_receive_packet</code>, <code>avformat_alloc_output_context2</code>, <code>avformat_write_header</code>, <code>av_interleaved_write_frame</code>, <code>av_write_trailer</code>, <code>avio_open</code>). <code>swresample</code> ships, but has no bindings yet. Audio needs them (phase 5).</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="library">6. Into the library</h2>
|
||||
<ul>
|
||||
<li><strong>Like an import:</strong> on stop, the MP4 goes through the path imported videos take (<code>import_video_mcap</code> and <code>mcap_write_streamed</code>, <code>video_poster</code> for the thumbnail frame) into <code>Screen Captures</code>. The temporary file is then deleted.</li>
|
||||
<li><strong>metadata.json</strong> is a still's: the title and app of the window under the region's middle when recording started, and when it started.</li>
|
||||
<li><strong>capture.json</strong> is a still's (region, position, app path) plus <code>Duration</code>, <code>FrameRate</code>, <code>VideoCodec</code>, <code>Encoder</code> (<code>h264_nvenc</code> ...), <code>Capture</code> (<code>DXGI</code> or <code>WGC</code>), <code>DroppedFrames</code> and <code>Paused</code> (seconds).</li>
|
||||
<li><strong>The catalog</strong> today marks a capture as a video only from <code>VideoCodec</code> in <code>import.json</code> (<code>app/catalog.jai</code>). It will read the same keys from <code>capture.json</code>, so recordings show as videos, play with Open, and are skipped by segmentation.</li>
|
||||
<li><strong>The thumbnail frame:</strong> <code>video_poster</code> picks a frame a tenth of the way in, at most 3 s, to skip a phone video's black start. A screen recording has no black start, but the first second is often the region being chosen. The same rule is kept unless it picks badly.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="hotkeys">7. Hotkeys</h2>
|
||||
<div class="table-wrap"><table>
|
||||
<tr><th>Action in <code>[global_hotkeys]</code></th><th>Default</th><th>What it does</th></tr>
|
||||
<tr><td><code>capture_region</code> (stills, built)</td><td><code>PrintScreen</code></td><td>Opens the overlay for a still.</td></tr>
|
||||
<tr><td><code>record_region</code></td><td><code>Ctrl-PrintScreen</code></td><td>Opens the overlay in video mode. While a recording runs, it stops and saves it.</td></tr>
|
||||
</table></div>
|
||||
<ul>
|
||||
<li><strong>One key for start and stop.</strong> It's registered all the time, as <code>capture_region</code> is, and only <code>global_hotkey_fired</code> needs to know whether a recording is running. No key is borrowed from the app being recorded: a plain Escape, or Windows' own Ctrl+Escape, would be.</li>
|
||||
<li><strong>Escape inside the overlay</strong>, before recording starts, cancels as it does for stills.</li>
|
||||
<li><strong>Saving and discarding:</strong> stopping always saves. Discarding is the toolbar's <em>Discard</em> button, which asks first.</li>
|
||||
<li><strong>PrintScreen during a recording</strong> still takes a still. The overlay would freeze the screen, and that freeze would end up in the video, so the still is taken without the overlay: the whole screen under the cursor, saved as usual. Open question 1 asks whether that's right.</li>
|
||||
<li><strong>Settings files:</strong> <code>record_region</code> is a new <code>windows_only</code> action. Existing settings files get it appended on the next launch, as <code>capture_region</code> was.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="audio">8. Audio <span class="tag later">Phase 5</span></h2>
|
||||
<ul>
|
||||
<li><strong>Sources:</strong> system audio comes from WASAPI loopback on the default output device, and the microphone from WASAPI capture. Each is optional, set in the toolbar and remembered under <code>[video]</code>.</li>
|
||||
<li><strong>Mixing:</strong> both are mixed to 48 kHz stereo float with <code>swresample</code> (bindings to generate), and encoded as AAC at 160 kb/s into the same MP4.</li>
|
||||
<li><strong>Sync:</strong> each audio packet's QPC time is lined up with the video's.</li>
|
||||
<li><strong>Silence:</strong> loopback sends nothing while nothing plays, so silence is written to fill the gaps.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="phases">9. Phases</h2>
|
||||
<p>Each phase ends with something to run and check, and is a commit (0.1.3 onwards).</p>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 1: <code>record-probe</code> with Desktop Duplication</h3>
|
||||
<p>A command-line tool, <code>tools/record_probe.jai</code>, built with <code>jai first.jai - record_probe</code>:</p>
|
||||
<ul>
|
||||
<li>It lists adapters and outputs, picks a screen or region, and duplicates it on the output's adapter.</li>
|
||||
<li>It records N seconds through the pipeline (section 4) and encoder (section 5) to an MP4.</li>
|
||||
<li>It prints which method and encoder it used, frames captured, written and dropped, time per stage, and the file size.</li>
|
||||
</ul>
|
||||
<p>It answers four questions on your laptop:</p>
|
||||
<ul>
|
||||
<li>Does <code>DuplicateOutput1</code> succeed on the internal screen, and on an external one?</li>
|
||||
<li>Does it succeed with CUDA loaded in the same process? mcap loads onnxruntime with CUDA, which may keep the process on the NVIDIA GPU.</li>
|
||||
<li>Does a window with <code>WDA_EXCLUDEFROMCAPTURE</code> stay out of the frames?</li>
|
||||
<li>What does 4K at 60 frames a second cost?</li>
|
||||
</ul>
|
||||
<p><strong>Done when:</strong> 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A <code>DXGI_ERROR_UNSUPPORTED</code> is also an answer: it means phase 2 is needed on this machine.</p>
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 2: Windows.Graphics.Capture fallback</h3>
|
||||
<p>The second source behind the same interface: WinRT bindings, polling <code>TryGetNextFrame</code>, the cursor and border settings. The fallback rules from 3.2, and <code>-capture dxgi|wgc|auto</code> on the probe.</p>
|
||||
<p><strong>Done when:</strong></p>
|
||||
<ul>
|
||||
<li>The probe records the same screens with <code>-capture wgc</code>.</li>
|
||||
<li><code>auto</code> picks Desktop Duplication where it works and Windows.Graphics.Capture where it doesn't. The latter is shown either on the two-GPU laptop or on a rotated screen.</li>
|
||||
<li>Whether the yellow border can be turned off is recorded.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 3: Recording in the app</h3>
|
||||
<p>The pieces:</p>
|
||||
<ul>
|
||||
<li>The <code>record_region</code> action, which starts a recording or, during one, stops it.</li>
|
||||
<li>The overlay's video mode: red outline, the 240 × 180 minimum, the drag kept to one screen.</li>
|
||||
<li>The capture and encoder threads in the app.</li>
|
||||
<li>Stopping, and the file into the library (section 6).</li>
|
||||
<li>The catalog reading <code>capture.json</code>.</li>
|
||||
<li>A shutdown step that finishes the file.</li>
|
||||
<li>Tests of the minimum-size and even-size rules, and of the <code>capture.json</code> round trip.</li>
|
||||
</ul>
|
||||
<p><strong>Done when:</strong></p>
|
||||
<ul>
|
||||
<li>Ctrl+PrintScreen, a drag, then Ctrl+PrintScreen again leaves a playable video in Screen Captures, shown as a video in the grid and opened in the Capture window.</li>
|
||||
<li>A 239 × 300 drag doesn't start a recording.</li>
|
||||
<li>Quitting mid-recording keeps it.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 4: Border, toolbar, pause and recovery</h3>
|
||||
<p>The pieces:</p>
|
||||
<ul>
|
||||
<li>The border and toolbar windows, kept out of the capture.</li>
|
||||
<li>Pause, resume, and Discard with a confirmation.</li>
|
||||
<li>The dropped-frame count.</li>
|
||||
<li>Recovery from <code>ACCESS_LOST</code>.</li>
|
||||
<li>Offering a leftover temporary file for import at startup.</li>
|
||||
<li>The <code>[video]</code> settings: frame rate, cursor, capture method.</li>
|
||||
</ul>
|
||||
<p><strong>Done when:</strong> neither the border nor the toolbar appears in a recording. Pausing for 10 s leaves no gap in the video. A UAC prompt or a resolution change mid-recording leaves a video that plays through.</p>
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 5: Audio</h3>
|
||||
<p>Section 8: <code>swresample</code> bindings, loopback and microphone, mixing, AAC.</p>
|
||||
<p><strong>Done when:</strong> a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="risks">10. Risks</h2>
|
||||
<div class="risk"><strong>Two GPUs.</strong> This is the reason for the fallback. One step beyond it: with the NVIDIA control panel set to the high-performance GPU for everything, both methods might be affected. Phase 1 tests your laptop with and without that setting.</div>
|
||||
<div class="risk"><strong>HDR screens.</strong> Asking for 8-bit BGRA from an HDR screen gives Windows' own tone mapping, which can look washed out. Recording HDR (16-bit float frames, HEVC 10-bit) is out of scope. The toolbar will say when the screen is HDR.</div>
|
||||
<div class="risk"><strong>Protected content</strong> (DRM video in browsers, some streaming apps) records as black with either method. Nothing can be done about it, so it's documented in the README.</div>
|
||||
<div class="risk"><strong>Laptop power.</strong> 60 frames a second at 4K keeps the GPU awake. The default is 30, and the probe measures the cost.</div>
|
||||
|
||||
<h2 id="questions">11. Open questions</h2>
|
||||
<ol>
|
||||
<li><strong>PrintScreen during a recording:</strong> take the whole screen without the overlay (this plan), ignore it, or freeze anyway and accept the frozen moment in the video?</li>
|
||||
<li><strong>The folder:</strong> recordings in <em>Screen Captures</em> beside stills, or a folder of their own (<em>Screen Recordings</em>)? This plan uses <em>Screen Captures</em>, so the grid shows both together.</li>
|
||||
<li><strong>A time limit:</strong> none by default, apart from stopping when the disk has less than 2 GB free?</li>
|
||||
<li><strong>A countdown</strong> (3, 2, 1) before recording starts, as Snagit has? This plan starts at once.</li>
|
||||
<li><strong>Mouse clicks:</strong> should they be shown in the video, for example as a ring at each click, as Snagit can?</li>
|
||||
</ol>
|
||||
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Reference in New Issue
Block a user