From 5758f0a071bf4ea05714ff1dc0bb658180c4ebd9 Mon Sep 17 00:00:00 2001 From: Musa Mahmood Date: Sat, 10 Oct 2026 06:31:37 +0100 Subject: [PATCH] [0.1.2] Video capture plan: Desktop Duplication with a Windows.Graphics.Capture fallback, NVENC, Ctrl+PrintScreen - video-capture-plan.html: recording a region of one screen to H.264 MP4 and into the library as a video .mcap. DXGI Desktop Duplication first, on a device of its own on the screen's adapter; Windows.Graphics.Capture when duplication cannot run (two GPUs, rotated screens, too many duplications). Encoders tried in turn: NVENC, AMF, QSV, Media Foundation, libx264, from the FFmpeg already shipped. Ctrl+PrintScreen opens the overlay in a video mode (a 240 x 180 minimum, kept to one screen) and stops the recording. Five phases, the first a command-line probe that answers the two-GPU question on the laptop. --- check_targets.jai | 2 +- first.jai | 2 +- video-capture-plan.html | 353 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 355 insertions(+), 2 deletions(-) create mode 100644 video-capture-plan.html diff --git a/check_targets.jai b/check_targets.jai index 7ba0497..09110b2 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.1.1"; +VERSION :: "0.1.2"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 27cccfc..51c376b 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.1.1"; +VERSION :: "0.1.2"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; diff --git a/video-capture-plan.html b/video-capture-plan.html new file mode 100644 index 0000000..7529890 --- /dev/null +++ b/video-capture-plan.html @@ -0,0 +1,353 @@ + + + + + +Video Capture Plan + + + +
+ +
+

Screen video capture in mcap

+

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · not started

+
+ +
+ In short +
    +
  • Decided DXGI Desktop Duplication first. If it can't run, for example on a laptop with two GPUs where it fails with DXGI_ERROR_UNSUPPORTED, recording falls back to Windows.Graphics.Capture. Both sit behind one interface, so everything after the capture is shared. It's more code than Windows.Graphics.Capture alone, but each method covers the cases where the other fails.
    • +
  • Decided Encoding uses the FFmpeg mcap already ships. It tries NVENC first, then AMD AMF, Intel QSV, Media Foundation and libx264. The output is H.264 in MP4, with frames written only when the screen changes. That FFmpeg build already includes every one of these encoders.
    • +
  • Decided Ctrl+PrintScreen starts a recording. It opens the same frozen-screen overlay as stills, in a video mode. The region must be at least 240 × 180 pixels, which Snagit doesn't require.
    • +
  • Decided Ctrl+PrintScreen again stops it and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.
    • +
  • Decided A recording becomes an .mcap in Screen Captures, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.
    • +
  • It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.
    • +
  • Later Audio is phase 5. Recording one window, and regions that span two screens, come after.
    • +
+
+ + + +

1. Scope

+

In:

+ +

Out, for now:

+ +

Windows only, like the stills overlay. It needs Windows 10 version 2004 or later; the Windows.Graphics.Capture features used below need that version.

+ +

2. Using it

+
    +
  1. Ctrl+PrintScreen freezes the screens and shows the overlay from stills, in its video mode. The outline and the magnifier are red, not orange, so you can see it will record.
    2. +
  2. Drag a region. The magnifier's label shows its size. +
      +
    • The drag is kept on the screen where it started (see 3.4).
      • +
    • Below 240 × 180, the outline is drawn dashed and the label reads 320 × 120 · at least 240 × 180. Letting go then starts nothing, and the overlay stays open to drag again.
      • +
    • A click without dragging, or Enter, takes the whole screen, as for stills.
      • +
    • Escape or a right click cancels, as for stills.
      • +
    3. +
  3. Recording starts as soon as you let go, with no countdown (see open question 4). +
      +
    • A thin red border is drawn just outside the region, so it's never in the video.
      • +
    • A small toolbar sits beside the region, outside it where there's room. It shows the time, the size and frames dropped, with Pause, Stop and Discard.
      • +
    • Both the border and the toolbar are kept out of the capture with SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE), which needs Windows 10 2004. Whether Desktop Duplication respects this is checked in phase 1. The border is outside the region either way, so only the toolbar depends on it.
      • +
    4. +
  4. Ctrl+PrintScreen again, or Stop, ends the recording at once: +
      +
    • The encoder is flushed and the file closed.
      • +
    • The file becomes an .mcap.
      • +
    • mcap comes to the front showing it in the Capture window, as it does after a still.
      • +
    5. +
+

Dimensions are rounded down to even numbers after the minimum is checked, since H.264's 4:2:0 colour needs that. 240 × 180 is already even, so a region that passes the check stays at least 240 × 180.

+ +

3. Grabbing frames

+

One interface, two implementations, chosen when the recording starts:

+
Capture_Source :: struct {
+  start:      (source: *Capture_Source, screen: Screen_Ref, region: Rect, cursor: bool) -> ok: bool, error: string, fall_back: bool;
+  next_frame: (source: *Capture_Source, timeout_ms: u32) -> Captured_Frame, status: Frame_Status;  // NEW, SAME, LOST, FAILED
+  stop:       (source: *Capture_Source);
+  name:       string;   // "DXGI" or "WGC", for the log and the toolbar's tooltip
+}
+
+Captured_Frame :: struct {
+  texture: *ID3D11Texture2D;  // the region only, region-sized, owned by the source until the next call
+  time:    s64;               // QueryPerformanceCounter ticks: when it was on screen
+  cursor:  Cursor_State;      // DXGI only: where, visible, and the shape when it changed
+}
+

Both run on a capture thread of their own, with their own D3D11 device, not the app's. The app's device starts on WARP and moves to the GPU later (#WarpFirstStartup in graphics/DX11.jai). Desktop Duplication also needs a device on the GPU the screen is connected to, which may not be the one the app draws with.

+ +

3.1 DXGI Desktop Duplication (first choice)

+ + +

3.2 When to fall back

+

The choice is made once, at the start, and logged:

+
+ + + + + + +
Desktop Duplication saysMeaningThen
DXGI_ERROR_UNSUPPORTED from DuplicateOutput1Two GPUs, and the process or device isn't on the one driving the screen. Also returned when no driver can duplicate.Windows.Graphics.Capture
DXGI_ERROR_NOT_CURRENTLY_AVAILABLEToo many duplications of this screen at once (other recorders running)Windows.Graphics.Capture
E_ACCESSDENIEDThe secure desktop (UAC, lock screen) is showingRetry for 2 s, then tell the user
Screen rotated (DXGI_OUTDUPL_DESC.Rotation isn't identity)Frames would come sidewaysWindows.Graphics.Capture, which hands them over upright
No output holds the regionThe screens changed since the freezeError, nothing recorded
+

A setting forces one method, for testing and for machines where one misbehaves: capture: auto (the default), dxgi or wgc, under a new [video] section of settings.config.

+ +

3.3 Windows.Graphics.Capture (fallback)

+ + +

3.4 One screen per recording

+

Desktop Duplication works one output at a time, and Windows.Graphics.Capture one monitor at a time. A region over two screens would need two captures stitched together, possibly at different DPI and refresh rates. In video mode the overlay keeps the drag on the screen where it started. Spanning screens is Later.

+ +

4. The frame pipeline

+
capture thread                                          encoder thread
+  next_frame ─► region texture ─► staging ring (3) ─►   queue (≤ 8 frames) ─► AVFrame bgr0 ─► encoder ─► MP4
+                                  Map the one from        drops the oldest when full,
+                                  2 frames back, copy     counted as dropped
+                                  out, draw the cursor
+ + +

5. Encoding

+ + +

6. Into the library

+ + +

7. Hotkeys

+
+ + + +
Action in [global_hotkeys]DefaultWhat it does
capture_region (stills, built)PrintScreenOpens the overlay for a still.
record_regionCtrl-PrintScreenOpens the overlay in video mode. While a recording runs, it stops and saves it.
+ + +

8. Audio Phase 5

+ + +

9. Phases

+

Each phase ends with something to run and check, and is a commit (0.1.3 onwards).

+ +
+

Phase 1: record-probe with Desktop Duplication

+

A command-line tool, tools/record_probe.jai, built with jai first.jai - record_probe:

+
    +
  • It lists adapters and outputs, picks a screen or region, and duplicates it on the output's adapter.
    • +
  • It records N seconds through the pipeline (section 4) and encoder (section 5) to an MP4.
    • +
  • It prints which method and encoder it used, frames captured, written and dropped, time per stage, and the file size.
    • +
+

It answers four questions on your laptop:

+
    +
  • Does DuplicateOutput1 succeed on the internal screen, and on an external one?
    • +
  • Does it succeed with CUDA loaded in the same process? mcap loads onnxruntime with CUDA, which may keep the process on the NVIDIA GPU.
    • +
  • Does a window with WDA_EXCLUDEFROMCAPTURE stay out of the frames?
    • +
  • What does 4K at 60 frames a second cost?
    • +
+

Done when: 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A DXGI_ERROR_UNSUPPORTED is also an answer: it means phase 2 is needed on this machine.

+
+ +
+

Phase 2: Windows.Graphics.Capture fallback

+

The second source behind the same interface: WinRT bindings, polling TryGetNextFrame, the cursor and border settings. The fallback rules from 3.2, and -capture dxgi|wgc|auto on the probe.

+

Done when:

+
    +
  • The probe records the same screens with -capture wgc.
    • +
  • auto picks Desktop Duplication where it works and Windows.Graphics.Capture where it doesn't. The latter is shown either on the two-GPU laptop or on a rotated screen.
    • +
  • Whether the yellow border can be turned off is recorded.
    • +
+
+ +
+

Phase 3: Recording in the app

+

The pieces:

+
    +
  • The record_region action, which starts a recording or, during one, stops it.
    • +
  • The overlay's video mode: red outline, the 240 × 180 minimum, the drag kept to one screen.
    • +
  • The capture and encoder threads in the app.
    • +
  • Stopping, and the file into the library (section 6).
    • +
  • The catalog reading capture.json.
    • +
  • A shutdown step that finishes the file.
    • +
  • Tests of the minimum-size and even-size rules, and of the capture.json round trip.
    • +
+

Done when:

+
    +
  • Ctrl+PrintScreen, a drag, then Ctrl+PrintScreen again leaves a playable video in Screen Captures, shown as a video in the grid and opened in the Capture window.
    • +
  • A 239 × 300 drag doesn't start a recording.
    • +
  • Quitting mid-recording keeps it.
    • +
+
+ +
+

Phase 4: Border, toolbar, pause and recovery

+

The pieces:

+
    +
  • The border and toolbar windows, kept out of the capture.
    • +
  • Pause, resume, and Discard with a confirmation.
    • +
  • The dropped-frame count.
    • +
  • Recovery from ACCESS_LOST.
    • +
  • Offering a leftover temporary file for import at startup.
    • +
  • The [video] settings: frame rate, cursor, capture method.
    • +
+

Done when: neither the border nor the toolbar appears in a recording. Pausing for 10 s leaves no gap in the video. A UAC prompt or a resolution change mid-recording leaves a video that plays through.

+
+ +
+

Phase 5: Audio

+

Section 8: swresample bindings, loopback and microphone, mixing, AAC.

+

Done when: a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.

+
+ +

10. Risks

+
Two GPUs. This is the reason for the fallback. One step beyond it: with the NVIDIA control panel set to the high-performance GPU for everything, both methods might be affected. Phase 1 tests your laptop with and without that setting.
+
HDR screens. Asking for 8-bit BGRA from an HDR screen gives Windows' own tone mapping, which can look washed out. Recording HDR (16-bit float frames, HEVC 10-bit) is out of scope. The toolbar will say when the screen is HDR.
+
Protected content (DRM video in browsers, some streaming apps) records as black with either method. Nothing can be done about it, so it's documented in the README.
+
Laptop power. 60 frames a second at 4K keeps the GPU awake. The default is 30, and the probe measures the cost.
+ +

11. Open questions

+
    +
  1. PrintScreen during a recording: take the whole screen without the overlay (this plan), ignore it, or freeze anyway and accept the frozen moment in the video?
    2. +
  2. The folder: recordings in Screen Captures beside stills, or a folder of their own (Screen Recordings)? This plan uses Screen Captures, so the grid shows both together.
    3. +
  3. A time limit: none by default, apart from stopping when the disk has less than 2 GB free?
    4. +
  4. A countdown (3, 2, 1) before recording starts, as Snagit has? This plan starts at once.
    5. +
  5. Mouse clicks: should they be shown in the video, for example as a ring at each click, as Snagit can?
    6. +
+ +
+ +