Screen video capture in mcap

1. Scope

In:

• Recording a rectangle on one screen, or a whole screen, to H.264 MP4 at up to 60 frames a second, with the mouse cursor.
• Pause and resume, stop, and discard.
• Saving into the library with the window and app it was recorded from.
• System audio and the microphone, in the last phase.

Out, for now:

• Recording one window that follows it as it moves.
• Regions that span two screens.
• Webcam overlay.
• Trimming and editing.
• HDR output.
• macOS and Linux.

Windows only, like the stills overlay. It needs Windows 10 version 2004 or later; the Windows.Graphics.Capture features used below need that version.

2. Using it

1. Ctrl+PrintScreen freezes the screens and shows the overlay from stills, in its video mode. The outline and the magnifier are red, not orange, so you can see it will record.
2. Drag a region. The magnifier's label shows its size.
   • The drag is kept on the screen where it started (see 3.4).
   • Below 240 × 180, the outline is drawn dashed and the label reads 320 × 120 · at least 240 × 180. Letting go then starts nothing, and the overlay stays open to drag again.
   • A click without dragging, or Enter, takes the whole screen, as for stills.
   • Escape or a right click cancels, as for stills.
3. Recording starts as soon as you let go, with no countdown (see open question 4).
   • A thin red border is drawn just outside the region, so it's never in the video.
   • A small toolbar sits beside the region, outside it where there's room. It shows the time, the size and frames dropped, with Pause, Stop and Discard.
   • Both the border and the toolbar are kept out of the capture with SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE), which needs Windows 10 2004. Whether Desktop Duplication respects this is checked in phase 1. The border is outside the region either way, so only the toolbar depends on it.
4. Ctrl+PrintScreen again, or Stop, ends the recording at once:
   • The encoder is flushed and the file closed.
   • The file becomes an .mcap.
   • mcap comes to the front showing it in the Capture window, as it does after a still.

Dimensions are rounded down to even numbers after the minimum is checked, since H.264's 4:2:0 colour needs that. 240 × 180 is already even, so a region that passes the check stays at least 240 × 180.

3. Grabbing frames

One interface, two implementations, chosen when the recording starts:

Capture_Source :: struct {
  start:      (source: *Capture_Source, screen: Screen_Ref, region: Rect, cursor: bool) -> ok: bool, error: string, fall_back: bool;
  next_frame: (source: *Capture_Source, timeout_ms: u32) -> Captured_Frame, status: Frame_Status;  // NEW, SAME, LOST, FAILED
  stop:       (source: *Capture_Source);
  name:       string;   // "DXGI" or "WGC", for the log and the toolbar's tooltip
}

Captured_Frame :: struct {
  texture: *ID3D11Texture2D;  // the region only, region-sized, owned by the source until the next call
  time:    s64;               // QueryPerformanceCounter ticks: when it was on screen
  cursor:  Cursor_State;      // DXGI only: where, visible, and the shape when it changed
}

Both run on a capture thread of their own, with their own D3D11 device, not the app's. The app's device starts on WARP and moves to the GPU later (#WarpFirstStartup in graphics/DX11.jai). Desktop Duplication also needs a device on the GPU the screen is connected to, which may not be the one the app draws with.

3.1 DXGI Desktop Duplication (first choice)

• Finding the screen: CreateDXGIFactory1, then EnumAdapters1 and EnumOutputs, to find the output whose desktop rectangle holds the region. The device is created on that output's adapter, with D3D11CreateDevice(adapter, D3D_DRIVER_TYPE_UNKNOWN, ...). The jai dxgi module already has every type needed: IDXGIOutput5, DuplicateOutput1, IDXGIOutputDuplication, DXGI_OUTDUPL_FRAME_INFO.
• Duplicating: IDXGIOutput5::DuplicateOutput1 with B8G8R8A8_UNORM. mcap is per-monitor DPI aware (v2), and the plain DuplicateOutput is documented to fail for such processes. Phase 1 confirms this, and falls back to IDXGIOutput1::DuplicateOutput where IDXGIOutput5 is missing.
• Each frame:
  • AcquireNextFrame(timeout) waits for one.
  • LastPresentTime == 0 means only the mouse moved: the cursor is updated and no new image is copied.
  • CopySubresourceRegion copies the region into a region-sized texture.
  • ReleaseFrame is called straight after the copy. Holding a frame stalls the desktop.
• The cursor isn't drawn into the image. Its position comes with each frame, and its shape from GetFramePointerShape when PointerShapeBufferSize is set. It is drawn onto the copy on the CPU, after it's read back (4). There are three kinds of shape to handle: colour, masked colour, and monochrome. The monochrome kind (the text I-beam) inverts what's under it.
• Errors while recording:
  • DXGI_ERROR_ACCESS_LOST comes from a display mode change, the UAC or lock screen, or a game switching to fullscreen. The duplication is made again, retrying every 250 ms, and the recording goes on. The toolbar shows the gap.
  • DXGI_ERROR_WAIT_TIMEOUT means nothing changed.
  • Anything else ends the recording, keeping what was captured.

3.2 When to fall back

The choice is made once, at the start, and logged:

A setting forces one method, for testing and for machines where one misbehaves: capture: auto (the default), dxgi or wgc, under a new [video] section of settings.config.

3.3 Windows.Graphics.Capture (fallback)

• Setup: the WinRT runtime is started on the capture thread (RoInitialize, multithreaded).
  • IGraphicsCaptureItemInterop::CreateForMonitor makes the item for the screen.
  • CreateDirect3D11DeviceFromDXGIDevice wraps the capture thread's device.
  • Direct3D11CaptureFramePool::CreateFreeThreaded makes a pool of 2 BGRA frames.
• Polling, not events: TryGetNextFrame is called at the frame rate, with the same timeout as DXGI. This avoids implementing WinRT event-handler objects in Jai.
• Cursor and border: IsCursorCaptureEnabled draws the cursor into the image, so the cursor step in 3.1 is skipped. IsBorderRequired = false removes the yellow border on Windows 11; whether an unpackaged app may do that without asking is checked in phase 2. Windows 10 always shows the border.
• Bindings: roughly six WinRT interfaces, written by hand as COM vtables, plus HSTRINGs (WindowsCreateString). This is the bulk of phase 2.

3.4 One screen per recording

Desktop Duplication works one output at a time, and Windows.Graphics.Capture one monitor at a time. A region over two screens would need two captures stitched together, possibly at different DPI and refresh rates. In video mode the overlay keeps the drag on the screen where it started. Spanning screens is Later.

4. The frame pipeline

capture thread                                          encoder thread
  next_frame ─► region texture ─► staging ring (3) ─►   queue (≤ 8 frames) ─► AVFrame bgr0 ─► encoder ─► MP4
                                  Map the one from        drops the oldest when full,
                                  2 frames back, copy     counted as dropped
                                  out, draw the cursor

• Reading back without stalls: three staging textures are used in turn. The CPU reads the one copied two frames ago, so Map never waits on the GPU. At 4K and 60 frames a second, about 2 GB/s is read back, well within what the hardware can do.
• Timestamps come from when each frame was on screen (LastPresentTime, or the system-relative time of a Windows.Graphics.Capture frame), in a 1/90000 timebase. Time paused is subtracted.
• The frame rate is capped at the setting (30 by default, or 60). When nothing changes, no frame is written, apart from one each second so players can seek. A still screen costs almost nothing.
• Buffers come from a fixed pool the size of the region, made at the start. Nothing is allocated per frame.

5. Encoding

• The encoder is the first of h264_nvenc, h264_amf, h264_qsv, h264_mf and libx264 that opens at the region's size. Each is opened once, so a driver that's present but broken falls through to the next. The one used is logged and stored with the recording.
• The input is BGRA as AV_PIX_FMT_BGR0. NVENC converts it to its own colour format on the GPU. For the CPU encoders, swscale converts it to yuv420p first.
• Quality is constant quality, about CQ 23 for NVENC (CRF 23 for libx264), with a keyframe every 2 s. NVENC uses preset p4. Screen content (text, flat colour) compresses well at this setting. Phase 1 measures the file sizes.
• Crash-safe files: the MP4 is written fragmented (movflags=frag_keyframe+empty_moov) to %TEMP%\mcap\recording-*.mp4. If mcap or Windows crashes mid-recording, the file still plays up to the last fragment. On the next start, a leftover file is offered for import.
• Bindings: modules/ffmpeg already binds everything encoding needs (avcodec_find_encoder_by_name, avcodec_send_frame, avcodec_receive_packet, avformat_alloc_output_context2, avformat_write_header, av_interleaved_write_frame, av_write_trailer, avio_open). swresample ships, but has no bindings yet. Audio needs them (phase 5).

6. Into the library

• Like an import: on stop, the MP4 goes through the path imported videos take (import_video_mcap and mcap_write_streamed, video_poster for the thumbnail frame) into Screen Captures. The temporary file is then deleted.
• metadata.json is a still's: the title and app of the window under the region's middle when recording started, and when it started.
• capture.json is a still's (region, position, app path) plus Duration, FrameRate, VideoCodec, Encoder (h264_nvenc ...), Capture (DXGI or WGC), DroppedFrames and Paused (seconds).
• The catalog today marks a capture as a video only from VideoCodec in import.json (app/catalog.jai). It will read the same keys from capture.json, so recordings show as videos, play with Open, and are skipped by segmentation.
• The thumbnail frame: video_poster picks a frame a tenth of the way in, at most 3 s, to skip a phone video's black start. A screen recording has no black start, but the first second is often the region being chosen. The same rule is kept unless it picks badly.

7. Hotkeys

• One key for start and stop. It's registered all the time, as capture_region is, and only global_hotkey_fired needs to know whether a recording is running. No key is borrowed from the app being recorded: a plain Escape, or Windows' own Ctrl+Escape, would be.
• Escape inside the overlay, before recording starts, cancels as it does for stills.
• Saving and discarding: stopping always saves. Discarding is the toolbar's Discard button, which asks first.
• PrintScreen during a recording still takes a still. The overlay would freeze the screen, and that freeze would end up in the video, so the still is taken without the overlay: the whole screen under the cursor, saved as usual. Open question 1 asks whether that's right.
• Settings files: record_region is a new windows_only action. Existing settings files get it appended on the next launch, as capture_region was.

8. Audio Phase 5

• Sources: system audio comes from WASAPI loopback on the default output device, and the microphone from WASAPI capture. Each is optional, set in the toolbar and remembered under [video].
• Mixing: both are mixed to 48 kHz stereo float with swresample (bindings to generate), and encoded as AAC at 160 kb/s into the same MP4.
• Sync: each audio packet's QPC time is lined up with the video's.
• Silence: loopback sends nothing while nothing plays, so silence is written to fill the gaps.

9. Phases

Each phase ends with something to run and check, and is a commit (0.1.3 onwards).

10. Risks

11. Open questions

1. PrintScreen during a recording: take the whole screen without the overlay (this plan), ignore it, or freeze anyway and accept the frozen moment in the video?
2. The folder: recordings in Screen Captures beside stills, or a folder of their own (Screen Recordings)? This plan uses Screen Captures, so the grid shows both together.
3. A time limit: none by default, apart from stopping when the disk has less than 2 GB free?
4. A countdown (3, 2, 1) before recording starts, as Snagit has? This plan starts at once.
5. Mouse clicks: should they be shown in the video, for example as a ring at each click, as Snagit can?