From 06a2aacc9c07dbff22a8d35d5e8ee32528d2e465 Mon Sep 17 00:00:00 2001 From: Musa Mahmood Date: Sat, 10 Oct 2026 06:48:43 +0100 Subject: [PATCH] [0.1.3] Video capture phase 1: modules/Screen_Record and record-probe, Desktop Duplication to NVENC - modules/Screen_Record: a region of one screen to H.264 in a fragmented MP4. A capture thread duplicates the screen (DuplicateOutput1, else DuplicateOutput) on a D3D11 device of its own on the screen's adapter, copies the region on the GPU, reads it back through three staging textures and draws the cursor in (colour, masked colour and monochrome shapes). An encoder thread feeds FFmpeg: h264_nvenc, else h264_amf, h264_qsv, h264_mf, libx264, each tried in turn; BGRA straight to NVENC, swscale to NV12 or YUV 4:2:0 (BT.709) for the rest. A frame is sent when the screen or cursor changed and its turn on an fps schedule is due, with a repeat each second of an unchanged screen; the encoder falling 8 frames behind drops the oldest. A lost duplication (mode change, secure desktop) is made again and recording goes on. record_start says fall_back when duplication cannot run (unsupported, too many, rotated screen). Regions under 240 x 180 are refused, then sizes are rounded down to even; a process whose view of the desktop is scaled is refused. - tools/record_probe.jai (jai first.jai - record_probe): lists the screens, records a screen or region and reports source, encoder, frames captured, sent, written and dropped, time per stage and file size; -legacy, -cuda, -exclude-test, -animate, -snapshot DIR, -check FILE. - On the RTX 5080 laptop: 60 s of 4K at 30 fps, 1802 frames, none dropped, 18 MB a minute; every change kept at 60 fps; plain DuplicateOutput works when DPI aware; a CUDA context changes nothing; WDA_EXCLUDEFROMCAPTURE is respected; QSV, Media Foundation and libx264 keep up too; a recording killed mid-way still plays. The two-GPU failure was not reproduced: the screens are on the NVIDIA GPU in this mode. The plan records the results. --- README.md | 4 + check_targets.jai | 2 +- first.jai | 4 +- modules/Screen_Record/cursor.jai | 48 +++ modules/Screen_Record/dxgi_source.jai | 238 +++++++++++++++ modules/Screen_Record/encoder.jai | 213 +++++++++++++ modules/Screen_Record/module.jai | 410 ++++++++++++++++++++++++++ tools/record_probe.jai | 301 +++++++++++++++++++ video-capture-plan.html | 42 ++- 9 files changed, 1258 insertions(+), 4 deletions(-) create mode 100644 modules/Screen_Record/cursor.jai create mode 100644 modules/Screen_Record/dxgi_source.jai create mode 100644 modules/Screen_Record/encoder.jai create mode 100644 modules/Screen_Record/module.jai create mode 100644 tools/record_probe.jai diff --git a/README.md b/README.md index 9e5a66d..7a86210 100644 --- a/README.md +++ b/README.md @@ -356,6 +356,10 @@ converted copy. HEIF and TIFF); previews; and videos' frames and what their containers say, with FFmpeg (`video.jai`), and how many frames a GIF has. `bin\image-probe FILE...` (`jai first.jai - image_probe`) shows what it makes of files. +- `modules/Screen_Record`: a region of one screen recorded to H.264 in an MP4, with DXGI Desktop + Duplication and FFmpeg (NVENC, else AMF, QSV, Media Foundation or libx264), for screen recording + (`video-capture-plan.html`). `bin ecord-probe` (`jai first.jai - record_probe`) records with it + and says how it went; `-list` shows the screens, `-check FILE` reads a recording back. - `modules/Snagx`: `.snagx` to `.mcap`, copied from SnagitConverter. - `modules/Capture_File`: reading and writing `.mcap` files, with a video's original left on disk. - `modules/Segment`: panoptic segmentation with Mask2Former, run with onnxruntime (see its README and diff --git a/check_targets.jai b/check_targets.jai index 09110b2..64c7e84 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.1.2"; +VERSION :: "0.1.3"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 51c376b..5c937d8 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.1.2"; +VERSION :: "0.1.3"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; @@ -67,6 +67,8 @@ build_project :: () { if array_find(args, "image_probe") array_add(*pending, start_tool_build("image-probe", "tools/image_probe.jai", optimized = build_release)); // jai first.jai - segment_probe: also bin/segment-probe, what modules/Segment makes of image files. if array_find(args, "segment_probe") array_add(*pending, start_tool_build("segment-probe", "tools/segment_probe.jai", optimized = build_release)); + // jai first.jai - record_probe: also bin/record-probe, a screen recorded with modules/Screen_Record (video-capture-plan.html, phase 1). + if array_find(args, "record_probe") array_add(*pending, start_tool_build("record-probe", "tools/record_probe.jai", optimized = build_release)); wait_for_builds(pending); diff --git a/modules/Screen_Record/cursor.jai b/modules/Screen_Record/cursor.jai new file mode 100644 index 0000000..3533547 --- /dev/null +++ b/modules/Screen_Record/cursor.jai @@ -0,0 +1,48 @@ +// The mouse cursor, drawn into frames read back from Desktop Duplication, which leaves it out of the +// image and hands over its shape and position separately. Three kinds of shape: +// +// COLOR BGRA, blended by its alpha. +// MASKED_COLOR BGRA, where alpha 0 replaces the pixel under it and 0xFF XORs it. +// MONOCHROME 1 bit a pixel, twice as tall: an AND mask over an XOR mask. AND 1 XOR 1 inverts the +// pixel under it, which is how the text I-beam shows on any background. + +Cursor_Shape :: struct { + type: DXGI_OUTDUPL_POINTER_SHAPE_TYPE; + width, height, pitch: s32; // height as handed over: twice the cursor's for MONOCHROME + data: [..] u8; +} + +// Draws shape with its top left at x, y (in the frame's pixels; it may reach past the edges) into +// BGRA pixels, rows packed. +cursor_draw :: (pixels: *u8, width: s32, height: s32, shape: *Cursor_Shape, x: s32, y: s32) { + if !shape.data.count return; + rows := ifx shape.type == .MONOCHROME then shape.height / 2 else shape.height; + for row: 0..rows - 1 { + dy := y + row; + if dy < 0 || dy >= height continue; + for column: 0..shape.width - 1 { + dx := x + column; + if dx < 0 || dx >= width continue; + d := pixels + (cast(s64) dy * width + dx) * 4; + if shape.type == { + case .COLOR; + s := shape.data.data + row * shape.pitch + column * 4; + a := cast(u32) s[3]; + if a == 0 continue; + for c: 0..2 d[c] = cast(u8) ((cast(u32) s[c] * a + cast(u32) d[c] * (255 - a) + 127) / 255); + case .MASKED_COLOR; + s := shape.data.data + row * shape.pitch + column * 4; + if s[3] == 0 for c: 0..2 d[c] = s[c]; + else for c: 0..2 d[c] ^= s[c]; + case .MONOCHROME; + bit := cast(u8) (0x80 >> (column % 8)); + and_mask := shape.data[row * shape.pitch + column / 8] & bit; + xor_mask := shape.data[(row + rows) * shape.pitch + column / 8] & bit; + if and_mask && !xor_mask continue; // transparent + if and_mask { for c: 0..2 d[c] = 255 - d[c]; continue; } // inverted + value: u8 = ifx xor_mask then cast(u8) 255 else 0; // white or black + for c: 0..2 d[c] = value; + } + } + } +} diff --git a/modules/Screen_Record/dxgi_source.jai b/modules/Screen_Record/dxgi_source.jai new file mode 100644 index 0000000..3584e09 --- /dev/null +++ b/modules/Screen_Record/dxgi_source.jai @@ -0,0 +1,238 @@ +// DXGI Desktop Duplication: the screens as the GPU composes them, a frame each time one changes. +// +// It runs on a D3D11 device of its own, made on the adapter the screen is attached to: duplicating +// with a device on another adapter fails. On a laptop with two GPUs, where the screen hangs off one +// and the process may run on the other, DuplicateOutput1 can fail with DXGI_ERROR_UNSUPPORTED all the +// same; that, too many duplications of the screen at once, and a rotated screen are what +// dxgi_open returns fall_back for. +// +// DuplicateOutput1 (IDXGIOutput5) is used where there is one: the plain DuplicateOutput is documented +// to fail for a process aware of each monitor's DPI, as mcap is. legacy forces the plain one, for +// record-probe to check that. + +Dxgi_Source :: struct { + adapter: *IDXGIAdapter1; + output: *IDXGIOutput1; + output5: *IDXGIOutput5; // null before Windows 10 1703 + device: *ID3D11Device; + device_context: *ID3D11DeviceContext; + duplication: *IDXGIOutputDuplication; + legacy: bool; + method: string; // which duplication call worked, for the log + + box: D3D11_BOX; // the region, in the output's pixels + latest: *ID3D11Texture2D; // the region as last seen + staging: [RECORD_STAGING] *ID3D11Texture2D; + + copy_ticks: s64; // spent taking images out of frames: QueryInterface and the copy's submission + pointer_position: POINT; // the cursor's top left, in the output's pixels + pointer_visible: bool; + shape: Cursor_Shape; +} + +RECORD_STAGING :: 3; // staging textures in turn: the one read is two frames behind the one copied + +// Every screen attached to the desktop, on every adapter, in DXGI's order. Temp. +record_list_screens :: () -> [] Record_Screen { + screens: [..] Record_Screen; + screens.allocator = temp; + factory: *IDXGIFactory1; + if FAILED(CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory)) return screens; + defer IUnknown_Release(factory); + + adapter_index: u32; + while true { + adapter: *IDXGIAdapter1; + if FAILED(IDXGIFactory1_EnumAdapters1(factory, adapter_index, *adapter)) break; + defer IUnknown_Release(adapter); + adapter_desc: DXGI_ADAPTER_DESC1; + IDXGIAdapter1_GetDesc1(adapter, *adapter_desc); + adapter_name := wide_text(adapter_desc.Description.data); + + output_index: u32; + while true { + output: *IDXGIOutput; + if FAILED(IDXGIAdapter_EnumOutputs(adapter, output_index, *output)) break; + desc: DXGI_OUTPUT_DESC; + IDXGIOutput_GetDesc(output, *desc); + IUnknown_Release(output); + if desc.AttachedToDesktop != .FALSE { + array_add(*screens, .{adapter_index = adapter_index, output_index = output_index, adapter = adapter_name, + name = wide_text(desc.DeviceName.data), desktop = desc.DesktopCoordinates, rotation = desc.Rotation}); + } + output_index += 1; + } + adapter_index += 1; + } + return screens; +} + +// Makes the device on the screen's adapter, and duplicates the screen. region is in the screen's +// pixels and lies on it. +dxgi_open :: (s: *Dxgi_Source, screen: Record_Screen, region: RECT, legacy: bool) -> ok: bool, error: string, fall_back: bool { + s.legacy = legacy; + if screen.rotation != .IDENTITY && screen.rotation != .UNSPECIFIED return false, "the screen is rotated", true; + + factory: *IDXGIFactory1; + hr := CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory); + if FAILED(hr) return false, hresult_text("CreateDXGIFactory1", hr), false; + defer IUnknown_Release(factory); + hr = IDXGIFactory1_EnumAdapters1(factory, screen.adapter_index, *s.adapter); + if FAILED(hr) return false, hresult_text("the screen's adapter is gone", hr), false; + output: *IDXGIOutput; + hr = IDXGIAdapter_EnumOutputs(s.adapter, screen.output_index, *output); + if FAILED(hr) return false, hresult_text("the screen is gone", hr), false; + defer IUnknown_Release(output); + hr = IUnknown_QueryInterface(output, *uid(IDXGIOutput1_UUID), xx *s.output); + if FAILED(hr) return false, hresult_text("the screen cannot be duplicated (no IDXGIOutput1)", hr), true; + IUnknown_QueryInterface(output, *uid(IDXGIOutput5_UUID), xx *s.output5); // optional + + levels := D3D_FEATURE_LEVEL.[._11_1, ._11_0, ._10_1, ._10_0]; + level: D3D_FEATURE_LEVEL; + hr = D3D11CreateDevice(s.adapter, .UNKNOWN, null, .BGRA_SUPPORT, levels.data, levels.count, D3D11_SDK_VERSION, *s.device, *level, *s.device_context); + if FAILED(hr) return false, hresult_text("D3D11CreateDevice on the screen's adapter", hr), true; + + s.box = .{left = xx region.left, top = xx region.top, right = xx region.right, bottom = xx region.bottom, front = 0, back = 1}; + width := cast(u32) (region.right - region.left); + height := cast(u32) (region.bottom - region.top); + desc: D3D11_TEXTURE2D_DESC; + desc.Width = width; + desc.Height = height; + desc.MipLevels = 1; + desc.ArraySize = 1; + desc.Format = .B8G8R8A8_UNORM; + desc.SampleDesc.Count = 1; + desc.Usage = .DEFAULT; + hr = ID3D11Device_CreateTexture2D(s.device, *desc, null, *s.latest); + if FAILED(hr) return false, hresult_text("a texture for the region", hr), false; + desc.Usage = .STAGING; + desc.CPUAccessFlags = .READ; + for * s.staging { + hr = ID3D11Device_CreateTexture2D(s.device, *desc, null, it); + if FAILED(hr) return false, hresult_text("a staging texture", hr), false; + } + + hr = dxgi_duplicate(s); + if FAILED(hr) { + // Unsupported: two GPUs, the device on the wrong one. Not currently available: too many duplications. + fall_back := hresult_is(hr, DXGI_ERROR_UNSUPPORTED) || hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE); + return false, hresult_text(tprint("Desktop Duplication (%)", s.method), hr), fall_back; + } + + // A process not aware of each monitor's DPI is told the desktop in scaled pixels, while the frames + // come in the screen's own: the region would be the wrong part of the screen. + duplication_desc: DXGI_OUTDUPL_DESC; + IDXGIOutputDuplication_GetDesc(s.duplication, *duplication_desc); + desktop_width := screen.desktop.right - screen.desktop.left; + desktop_height := screen.desktop.bottom - screen.desktop.top; + if duplication_desc.ModeDesc.Width != cast(u32) desktop_width || duplication_desc.ModeDesc.Height != cast(u32) desktop_height { + return false, tprint("the screen is % x %, but this process sees it as % x %: it must be aware of each monitor's DPI", + duplication_desc.ModeDesc.Width, duplication_desc.ModeDesc.Height, desktop_width, desktop_height), false; + } + return true, "", false; +} + +// Duplicates the screen, again after the duplication was lost. +dxgi_duplicate :: (s: *Dxgi_Source) -> HRESULT { + if s.duplication { IUnknown_Release(s.duplication); s.duplication = null; } + if s.output5 && !s.legacy { + s.method = "DuplicateOutput1"; + formats := DXGI_FORMAT.[.B8G8R8A8_UNORM]; + return IDXGIOutput5_DuplicateOutput1(s.output5, s.device, 0, formats.count, formats.data, *s.duplication); + } + s.method = "DuplicateOutput"; + return IDXGIOutput1_DuplicateOutput(s.output, s.device, *s.duplication); +} + +Dxgi_Next :: enum u8 { NONE; IMAGE; POINTER; IMAGE_AND_POINTER; LOST; FAILED; } + +// Waits up to timeout_ms for the screen to change. A new image is copied into s.latest; a moved or +// changed cursor updates s.pointer_*. On LOST the duplication is gone: call dxgi_duplicate. +dxgi_next :: (s: *Dxgi_Source, timeout_ms: u32) -> Dxgi_Next, error: string { + info: DXGI_OUTDUPL_FRAME_INFO; + resource: *IDXGIResource; + hr := IDXGIOutputDuplication_AcquireNextFrame(s.duplication, timeout_ms, *info, *resource); + if hresult_is(hr, DXGI_ERROR_WAIT_TIMEOUT) return .NONE, ""; + if hresult_is(hr, DXGI_ERROR_ACCESS_LOST) return .LOST, ""; + if FAILED(hr) return .FAILED, hresult_text("AcquireNextFrame", hr); + defer IDXGIOutputDuplication_ReleaseFrame(s.duplication); // held, it stalls the desktop + defer IUnknown_Release(resource); + + image := info.LastPresentTime.QuadPart != 0; + if image { + start: LARGE_INTEGER; + QueryPerformanceCounter(*start); + texture: *ID3D11Texture2D; + hr = IUnknown_QueryInterface(resource, *uid(ID3D11Texture2D_UUID), xx *texture); + if FAILED(hr) return .FAILED, hresult_text("the frame is not a texture", hr); + ID3D11DeviceContext_CopySubresourceRegion(s.device_context, s.latest, 0, 0, 0, 0, texture, 0, *s.box); + IUnknown_Release(texture); + end: LARGE_INTEGER; + QueryPerformanceCounter(*end); + s.copy_ticks += end.QuadPart - start.QuadPart; + } + + pointer := info.LastMouseUpdateTime.QuadPart != 0; + if pointer { + s.pointer_position = info.PointerPosition.Position; + s.pointer_visible = info.PointerPosition.Visible != .FALSE; + if info.PointerShapeBufferSize { + array_resize(*s.shape.data, info.PointerShapeBufferSize, initialize = false); + needed: u32; + shape_info: DXGI_OUTDUPL_POINTER_SHAPE_INFO; + hr = IDXGIOutputDuplication_GetFramePointerShape(s.duplication, cast(u32) s.shape.data.count, s.shape.data.data, *needed, *shape_info); + if SUCCEEDED(hr) { + s.shape.type = cast(DXGI_OUTDUPL_POINTER_SHAPE_TYPE) shape_info.Type; + s.shape.width = cast(s32) shape_info.Width; + s.shape.height = cast(s32) shape_info.Height; + s.shape.pitch = cast(s32) shape_info.Pitch; + } else { + array_reset_keeping_memory(*s.shape.data); + } + } + } + if image && pointer return .IMAGE_AND_POINTER, ""; + if image return .IMAGE, ""; + if pointer return .POINTER, ""; + return .NONE, ""; +} + +dxgi_close :: (s: *Dxgi_Source) { + release :: (p: *$T) { if p.* { IUnknown_Release(p.*); p.* = null; } } + release(*s.duplication); + for * s.staging release(it); + release(*s.latest); + release(*s.device_context); + release(*s.device); + release(*s.output5); + release(*s.output); + release(*s.adapter); + array_free(s.shape.data); + s.shape = .{}; +} + +#scope_module + +hresult_is :: (hr: HRESULT, code: u32) -> bool { + return cast,no_check(u32) hr == code; +} + +// "what failed: 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", temp. +hresult_text :: (what: string, hr: HRESULT) -> string { + code := cast,no_check(u32) hr; + name := ""; + if code == { + case DXGI_ERROR_UNSUPPORTED; name = " (DXGI_ERROR_UNSUPPORTED)"; + case DXGI_ERROR_NOT_CURRENTLY_AVAILABLE; name = " (DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)"; + case DXGI_ERROR_ACCESS_LOST; name = " (DXGI_ERROR_ACCESS_LOST)"; + case 0x80070005; name = " (E_ACCESSDENIED)"; + case 0x80004002; name = " (E_NOINTERFACE)"; + case 0x80070057; name = " (E_INVALIDARG)"; + } + return tprint("%: 0x%%", what, formatInt(code, base = 16, minimum_digits = 8), name); +} + +wide_text :: (wide: *s16) -> string { + text, ok := wide_to_utf8(cast(*u16) wide,, temp); + return ifx ok then text else ""; +} diff --git a/modules/Screen_Record/encoder.jai b/modules/Screen_Record/encoder.jai new file mode 100644 index 0000000..3d37dee --- /dev/null +++ b/modules/Screen_Record/encoder.jai @@ -0,0 +1,213 @@ +// The encoder thread's half: frames of BGRA pixels to H.264 in a fragmented MP4, with FFmpeg +// (modules/ffmpeg). The first encoder in RECORD_ENCODERS that opens at the size wanted is used: +// NVENC, AMD's AMF, Intel's QSV, Windows' Media Foundation, then libx264 on the CPU. NVENC takes the +// BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with +// swscale (BT.709, limited range, as tagged). +// +// The file is written fragmented (a fragment each keyframe, every 2 s), so one cut short by a crash +// still plays up to its last fragment. + +RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"]; + +Encoder :: struct { + format: *AVFormatContext; + codec: *AVCodecContext; + stream: *AVStream; + frame: *AVFrame; + packet: *AVPacket; + sws: *SwsContext; + name: string; // the encoder used + input: AVPixelFormat; // what it is given: BGR0 as captured, or converted to NV12 or YUV420P + width, height: s32; + last_pts: s64 = -1; + header_written: bool; + bytes: s64; // of packets written +} + +// Opens the file and the first encoder that will take width x height (wanted alone, when given). +// Failures to open an encoder are listed in tried (temp), for the log. +encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string) -> ok: bool, error: string, tried: string { + av_log_set_level(AV_LOG_ERROR); + e.width, e.height = width, height; + result := avformat_alloc_output_context2(*e.format, null, "mp4", temp_c_string(path)); + if result < 0 || !e.format return false, av_error("FFmpeg cannot make an MP4", result), ""; + + tried: [..] string; + tried.allocator = temp; + names: [] string = RECORD_ENCODERS; + if wanted names = .[wanted]; + for names { + ok, why := encoder_try(e, it, width, height, fps); + if ok break; + array_add(*tried, tprint("%: %", it, why)); + } + tried_text := join(..tried, "; ",, temp); + if !e.codec { + encoder_free(e); + return false, tprint("no H.264 encoder opened (%)", tried_text), tried_text; + } + + e.stream = avformat_new_stream(e.format, null); + if !e.stream { encoder_free(e); return false, "FFmpeg cannot add a stream", tried_text; } + avcodec_parameters_from_context(e.stream.codecpar, e.codec); + e.stream.time_base = e.codec.time_base; + + result = avio_open(*e.format.pb, temp_c_string(path), AVIO_FLAG_WRITE); + if result < 0 { encoder_free(e); return false, av_error(tprint("FFmpeg cannot write %", path), result), tried_text; } + options: *AVDictionary; + av_dict_set(*options, "movflags", "frag_keyframe+empty_moov+default_base_moof", 0); + result = avformat_write_header(e.format, *options); + av_dict_free(*options); + if result < 0 { encoder_free(e); return false, av_error("FFmpeg cannot start the MP4", result), tried_text; } + e.header_written = true; + + e.packet = av_packet_alloc(); + e.frame = av_frame_alloc(); + if !e.packet || !e.frame { encoder_free(e); return false, "FFmpeg is out of memory", tried_text; } + if e.input != .BGR0 { + e.sws = sws_getContext(width, height, .BGR0, width, height, e.input, SWS_POINT, null, null, null); + if !e.sws { encoder_free(e); return false, "swscale cannot convert to the encoder's format", tried_text; } + coefficients := cast(*[4] s32) sws_getCoefficients(SWS_CS_ITU709); + sws_setColorspaceDetails(e.sws, coefficients, 1, coefficients, 0, 0, 1 << 16, 1 << 16); + } + return true, "", tried_text; +} + +// Encodes one frame; pts in RECORD_TIME_BASE ticks, made later than the last one if it is not. +encoder_write :: (e: *Encoder, pixels: *u8, pts: s64) -> ok: bool, error: string { + pts_out := max(pts, e.last_pts + 1); + e.last_pts = pts_out; + + frame := e.frame; + av_frame_unref(frame); + frame.format = cast(s32) e.input; + frame.width = e.width; + frame.height = e.height; + if e.input == .BGR0 { + // Not reference-counted: avcodec_send_frame copies it, so the buffer goes back to the pool at once. + frame.data[0] = pixels; + frame.linesize[0] = e.width * 4; + } else { + result := av_frame_get_buffer(frame, 0); + if result < 0 return false, av_error("FFmpeg cannot make a frame", result); + source := pixels; + stride := e.width * 4; + sws_scale(e.sws, *source, *stride, 0, e.height, frame.data.data, frame.linesize.data); + } + frame.pts = pts_out; + + result := avcodec_send_frame(e.codec, frame); + av_frame_unref(frame); + if result < 0 return false, av_error(tprint("% cannot encode", e.name), result); + drained, error := encoder_drain(e); + return drained, error; +} + +// Flushes the encoder, finishes the file and closes it. +encoder_close :: (e: *Encoder) -> ok: bool, error: string { + ok, error := true, ""; + if e.codec && e.header_written { + avcodec_send_frame(e.codec, null); + ok, error = encoder_drain(e); + result := av_write_trailer(e.format); + if ok && result < 0 ok, error = false, av_error("FFmpeg cannot finish the MP4", result); + } + encoder_free(e); + return ok, error; +} + +#scope_file + +RECORD_TIME_BASE :: 90000; // pts ticks a second: MPEG's + +encoder_try :: (e: *Encoder, name: string, width: s32, height: s32, fps: s32) -> ok: bool, error: string { + codec := avcodec_find_encoder_by_name(temp_c_string(name)); + if !codec return false, "not in this FFmpeg"; + input := encoder_input_format(codec); + if input == .NONE return false, "takes no format this can give it"; + + c := avcodec_alloc_context3(codec); + if !c return false, "out of memory"; + c.width = width; + c.height = height; + c.time_base = .{1, RECORD_TIME_BASE}; + c.framerate = .{fps, 1}; + c.pix_fmt = input; + c.gop_size = fps * 2; + c.max_b_frames = 0; + c.sample_aspect_ratio = .{1, 1}; + if input != .BGR0 { // swscale's conversion; NVENC picks and tags its own for BGRA + c.colorspace = .BT709; + c.color_primaries = .BT709; + c.color_trc = .BT709; + c.color_range = .MPEG; + } + if e.format.oformat.flags & AVFMT_GLOBALHEADER c.flags |= AV_CODEC_FLAG_GLOBAL_HEADER; + + options: *AVDictionary; + set :: (key: string, value: string) #expand { av_dict_set(*`options, temp_c_string(key), temp_c_string(value), 0); } + if name == { + case "h264_nvenc"; set("preset", "p4"); set("rc", "vbr"); set("cq", "23"); c.bit_rate = 0; + case "h264_amf"; set("rc", "cqp"); set("qp_i", "22"); set("qp_p", "24"); set("quality", "balanced"); + case "h264_qsv"; set("global_quality", "23"); set("preset", "medium"); + case "h264_mf"; set("rate_control", "quality"); set("quality", "70"); set("scenario", "display_remoting"); + case "libx264"; set("preset", "veryfast"); set("crf", "23"); + } + result := avcodec_open2(c, codec, *options); + av_dict_free(*options); + if result < 0 { + avcodec_free_context(*c); + return false, av_error("cannot open", result); + } + e.codec = c; + e.input = input; + e.name = name; + return true, ""; +} + +// BGRA as captured when the encoder takes it, else what swscale converts it to. +encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat { + if !codec.pix_fmts return .NONE; + for wanted: AVPixelFormat.[.BGR0, .BGRA, .NV12, .YUV420P] { + format := codec.pix_fmts; + while format.* != .NONE { + if format.* == wanted return ifx wanted == .BGRA then .BGR0 else wanted; + format += 1; + } + } + return .NONE; +} + +encoder_drain :: (e: *Encoder) -> ok: bool, error: string { + AVERROR_EAGAIN :: -11; + while true { + result := avcodec_receive_packet(e.codec, e.packet); + if result == AVERROR_EAGAIN || result == cast,no_check(s32) AVERROR_EOF return true, ""; + if result < 0 return false, av_error(tprint("% failed", e.name), result); + av_packet_rescale_ts(e.packet, e.codec.time_base, e.stream.time_base); + e.packet.stream_index = e.stream.index; + e.bytes += e.packet.size; + result = av_interleaved_write_frame(e.format, e.packet); // takes the packet's data + if result < 0 return false, av_error("FFmpeg cannot write to the MP4", result); + } + return true, ""; +} + +encoder_free :: (e: *Encoder) { + if e.sws sws_freeContext(e.sws); + if e.frame av_frame_free(*e.frame); + if e.packet av_packet_free(*e.packet); + if e.codec avcodec_free_context(*e.codec); + if e.format { + if e.format.pb avio_closep(*e.format.pb); + avformat_free_context(e.format); + } + e.sws, e.format, e.stream = null, null, null; +} + +// What FFmpeg says an error code is, after what failed (temp). +av_error :: (what: string, code: s32) -> string { + buffer: [256] u8; + av_strerror(code, buffer.data, buffer.count); + return tprint("% (%)", what, to_string(buffer.data)); +} diff --git a/modules/Screen_Record/module.jai b/modules/Screen_Record/module.jai new file mode 100644 index 0000000..8e4883a --- /dev/null +++ b/modules/Screen_Record/module.jai @@ -0,0 +1,410 @@ +// Screen_Record: a region of one screen recorded to H.264 in an MP4, for the mcap app's screen +// recording and tools/record_probe.jai (video-capture-plan.html). Windows only. +// +// Two threads. The capture thread takes the screen with DXGI Desktop Duplication (dxgi_source.jai), +// on a D3D11 device of its own, copies the region on the GPU, reads it back through a ring of staging +// textures, so Map never waits on the GPU, and draws the cursor into it (cursor.jai). The encoder +// thread hands the frames to FFmpeg (encoder.jai). Between them is a pool of RECORD_BUFFERS frames; +// when the encoder falls RECORD_QUEUE frames behind, the oldest waiting is dropped and counted. +// +// A frame is taken at most fps times a second, and only when the screen or the cursor changed, with +// one a second while nothing does, so players can seek. Each carries the time it was taken. +// +// recorder: Recorder; +// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0}); +// ... +// ok, error = record_stop(*recorder); +// +// fall_back says Desktop Duplication cannot run here: on a laptop with two GPUs, on a rotated screen, +// or with too many duplications of the screen running. Windows.Graphics.Capture takes over then +// (phase 2 of the plan). + +RECORD_MIN_WIDTH :: 240; // a smaller region is refused +RECORD_MIN_HEIGHT :: 180; +RECORD_QUEUE :: 8; // frames waiting for the encoder at most +RECORD_BUFFERS :: RECORD_QUEUE + 2; // and the one being filled, and the one being encoded + +Record_Screen :: struct { + adapter_index, output_index: u32; + adapter: string; // "NVIDIA GeForce RTX 5080 Laptop GPU" + name: string; // "\\.\DISPLAY1" + desktop: RECT; // on the virtual desktop + rotation: DXGI_MODE_ROTATION; +} + +Record_Options :: struct { + path: string; // the .mp4 written + screen: s32 = -1; // into record_list_screens(); -1: the one holding region + region: RECT; // on the virtual desktop; empty: the whole screen + fps: s32 = 30; + cursor := true; + encoder: string; // "" tries RECORD_ENCODERS in turn + legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe +} + +Record_Stats :: struct { + source: string; // "DXGI DuplicateOutput1 on " + encoder: string; + width, height: s32; + captured: s64; // new images of the screen + pointer_moves: s64; // cursor changes with no new image + emitted: s64; // frames sent to be encoded + repeats: s64; // of them, the once-a-second repeat of an unchanged screen + written: s64; // encoded + dropped: s64; // the encoder was too far behind + lost: s64; // times the duplication was lost and made again + copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks + bytes: s64; // encoded + seconds: float64; // from start to stop +} + +Recorder :: struct { + options: Record_Options; + source: Dxgi_Source; + encoder: Encoder; + width, height: s32; + frequency: s64; // QueryPerformanceCounter's + start_ticks: s64; + + capture_thread, encoder_thread: Thread; + running: bool; + stopping: bool; // set by record_stop; the capture thread ends, then the encoder + capture_ended: bool; + + mutex: Mutex; + ready: Semaphore; // signalled for each frame queued, and once when capture ends + frames: [RECORD_BUFFERS] Record_Frame; + free: [RECORD_BUFFERS] s32; + free_count: s32; + queue: [RECORD_BUFFERS] s32; // oldest first + queue_count: s32; + last_frame: [] u8; // the last frame queued, BGRA, for record_last_frame + stats: Record_Stats; + error: string; // why it stopped by itself; heap +} + +Record_Frame :: struct { + pixels: [] u8; // BGRA, rows packed + ticks: s64; +} + +// Opens the screen, the encoder and the file, and starts recording. +record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool { + r.* = .{}; + r.options = options; + r.options.path = copy_string(options.path); + r.options.encoder = copy_string(options.encoder); + if r.options.fps <= 0 r.options.fps = 30; + + screens := record_list_screens(); + if !screens.count return false, "no screen to record", false; + screen_index := options.screen; + region := options.region; + has_region := region.right > region.left && region.bottom > region.top; + if screen_index < 0 { + if !has_region screen_index = 0; + else for screens if rect_inside(region, it.desktop) { screen_index = xx it_index; break; } + if screen_index < 0 return false, "the region is not on one screen", false; + } + if screen_index >= screens.count return false, tprint("there is no screen %", screen_index), false; + screen := screens[screen_index]; + if !has_region region = screen.desktop; + if !rect_inside(region, screen.desktop) return false, "the region is not on that screen", false; + + // The minimum first, then even both ways, in the screen's pixels: H.264's 4:2:0 colour needs it. + // RECORD_MIN_WIDTH and _HEIGHT are even, so what passes stays at least that size. + #assert RECORD_MIN_WIDTH % 2 == 0 && RECORD_MIN_HEIGHT % 2 == 0; + if region.right - region.left < RECORD_MIN_WIDTH || region.bottom - region.top < RECORD_MIN_HEIGHT + return false, tprint("% x % is smaller than % x %", region.right - region.left, region.bottom - region.top, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false; + local := RECT.{region.left - screen.desktop.left, region.top - screen.desktop.top, region.right - screen.desktop.left, region.bottom - screen.desktop.top}; + local.right = local.left + ((local.right - local.left) & cast(s32) ~1); + local.bottom = local.top + ((local.bottom - local.top) & cast(s32) ~1); + r.width = local.right - local.left; + r.height = local.bottom - local.top; + + ok, error, fall_back := dxgi_open(*r.source, screen, local, options.legacy_duplicate); + if !ok { dxgi_close(*r.source); return false, error, fall_back; } + + tried: string; + ok, error, tried = encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder); + if tried log("[record] Encoders that did not open: %", tried); + if !ok { dxgi_close(*r.source); return false, error, false; } + + r.stats.source = sprint("DXGI % on %", r.source.method, screen.adapter); + r.stats.encoder = copy_string(r.encoder.name); + r.stats.width, r.stats.height = r.width, r.height; + + init(*r.mutex, "record"); + init(*r.ready); + for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false); + for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it; + r.free_count = RECORD_BUFFERS; + r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8); + + frequency: LARGE_INTEGER; + QueryPerformanceFrequency(*frequency); + r.frequency = frequency.QuadPart; + r.start_ticks = ticks_now(); + thread_init(*r.capture_thread, record_capture_proc, name = "record capture"); + thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder"); + r.running = true; + thread_start(*r.capture_thread, r); + thread_start(*r.encoder_thread, r); + return true, "", false; +} + +// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an +// error, or the file could not be finished; what was recorded up to then is in the file all the same. +record_stop :: (r: *Recorder) -> ok: bool, error: string { + if !r.running return false, "not recording"; + atomic_swap(*r.stopping, true); + thread_is_done(*r.capture_thread, -1); + thread_is_done(*r.encoder_thread, -1); + thread_deinit(*r.capture_thread); + thread_deinit(*r.encoder_thread); + r.running = false; + + closed, close_error := encoder_close(*r.encoder); + r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency; + r.stats.bytes = r.encoder.bytes; + dxgi_close(*r.source); + for * r.frames array_free(it.pixels); + destroy(*r.ready); + destroy(*r.mutex); + + if r.error return false, r.error; + if !closed return false, close_error; + return true, ""; +} + +// As they are now; strings are the recorder's. +record_stats :: (r: *Recorder) -> Record_Stats { + if !r.running return r.stats; + lock(*r.mutex); + defer unlock(*r.mutex); + return r.stats; +} + +// The last frame queued, BGRA (temp), cursor and all. +record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 { + if r.running lock(*r.mutex); + pixels := array_copy(r.last_frame,, temp); + if r.running unlock(*r.mutex); + return pixels, r.width, r.height; +} + +// Whether recording stopped by itself (the screen went away, the encoder failed); why, then. +record_failed :: (r: *Recorder) -> bool, string { + lock(*r.mutex); + defer unlock(*r.mutex); + return r.error.count > 0, r.error; +} + +#scope_file + +Staged :: struct { pending: bool; ticks: s64; pointer: POINT; pointer_visible: bool; } + +record_capture_proc :: (thread: *Thread) -> s64 { + r := cast(*Recorder) thread.data; + s := *r.source; + // A change is sent as soon as it comes once its turn is due: turns are a frame's time apart on a + // schedule that moves on a frame at a time, so the rate never passes fps, and a tenth of a frame + // early counts as on time, so a 60 Hz screen's frames, which jitter, all make a 60 fps cap. + period := r.frequency / r.options.fps; + early := period / 10; + turn := ticks_now(); + last_emit: s64 = -r.frequency; + have_image, changed := false, false; + staged: [RECORD_STAGING] Staged; + emits: s64; + + while !atomic_read(*r.stopping) { + now := ticks_now(); + due := ifx changed then turn - early else last_emit + r.frequency; // the change, or the repeat + wait := clamp((due - now) * 1000 / r.frequency, 0, 100); + next, error := dxgi_next(s, cast(u32) wait); + if next == { + case .FAILED; + record_fail(r, error); + break; + case .LOST; + // A mode change, the secure desktop, a game going fullscreen: duplicated again, and on. + lock(*r.mutex); r.stats.lost += 1; unlock(*r.mutex); + while !atomic_read(*r.stopping) { + hr := dxgi_duplicate(s); + if SUCCEEDED(hr) break; + if !hresult_is(hr, DXGI_ERROR_ACCESS_LOST) && !hresult_is(hr, 0x80070005) && !hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE) { + record_fail(r, hresult_text("the screen could not be duplicated again", hr)); + break; + } + sleep_milliseconds(250); + } + if atomic_read(*r.stopping) || r.error break; + continue; + case .IMAGE; #through; + case .IMAGE_AND_POINTER; + have_image, changed = true, true; + lock(*r.mutex); + r.stats.captured += 1; + r.stats.copy_ticks = s.copy_ticks; + unlock(*r.mutex); + case .POINTER; + if r.options.cursor && have_image { + changed = true; + lock(*r.mutex); r.stats.pointer_moves += 1; unlock(*r.mutex); + } + } + + now = ticks_now(); + if !have_image continue; + repeat := !changed && now - last_emit >= r.frequency; + if !(changed && now >= turn - early) && !repeat continue; + turn = max(turn + period, now - period); // after a quiet spell, no burst to catch up + + // Copied to staging now; read back two frames on, when the GPU is done with it. + slot := emits % RECORD_STAGING; + if staged[slot].pending record_read_back(r, slot, staged[slot]); + ID3D11DeviceContext_CopyResource(s.device_context, s.staging[slot], s.latest); + staged[slot] = .{pending = true, ticks = now, pointer = s.pointer_position, pointer_visible = s.pointer_visible}; + emits += 1; + last_emit = now; + changed = false; + lock(*r.mutex); + r.stats.emitted += 1; + if repeat r.stats.repeats += 1; + unlock(*r.mutex); + } + + // What is still staged, oldest first. + for 0..RECORD_STAGING - 1 { + slot := (emits + it) % RECORD_STAGING; + if staged[slot].pending record_read_back(r, slot, staged[slot]); + } + atomic_swap(*r.capture_ended, true); + signal(*r.ready); + return 0; +} + +// Copies a staging texture into a free frame, draws the cursor and queues it. +record_read_back :: (r: *Recorder, slot: s64, staged: Staged) { + s := *r.source; + t := ticks_now(); + lock(*r.mutex); + if !r.free_count { // every frame is waiting: drop the oldest + r.free[r.free_count] = r.queue[0]; + r.free_count += 1; + for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it]; + r.queue_count -= 1; + r.stats.dropped += 1; + } + r.free_count -= 1; + index := r.free[r.free_count]; + unlock(*r.mutex); + frame := *r.frames[index]; + + mapped: D3D11_MAPPED_SUBRESOURCE; + hr := ID3D11DeviceContext_Map(s.device_context, s.staging[slot], 0, .READ, 0, *mapped); + if FAILED(hr) { + lock(*r.mutex); r.free[r.free_count] = index; r.free_count += 1; unlock(*r.mutex); + record_fail(r, hresult_text("reading the frame back", hr)); + return; + } + row := cast(s64) r.width * 4; + for y: 0..r.height - 1 memcpy(frame.pixels.data + y * row, cast(*u8) mapped.pData + cast(s64) y * mapped.RowPitch, row); + ID3D11DeviceContext_Unmap(s.device_context, s.staging[slot], 0); + if r.options.cursor && staged.pointer_visible { + cursor_draw(frame.pixels.data, r.width, r.height, *s.shape, + staged.pointer.x - cast(s32) s.box.left, staged.pointer.y - cast(s32) s.box.top); + } + frame.ticks = staged.ticks; + + lock(*r.mutex); + r.queue[r.queue_count] = index; + r.queue_count += 1; + memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count); + r.stats.readback_ticks += ticks_now() - t; + unlock(*r.mutex); + signal(*r.ready); +} + +record_encoder_proc :: (thread: *Thread) -> s64 { + r := cast(*Recorder) thread.data; + first: s64 = -1; + while true { + wait_for(*r.ready, 100); + lock(*r.mutex); + index: s32 = -1; + if r.queue_count { + index = r.queue[0]; + for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it]; + r.queue_count -= 1; + } + unlock(*r.mutex); + if index < 0 { + if atomic_read(*r.capture_ended) break; + continue; + } + + frame := *r.frames[index]; + if first < 0 first = frame.ticks; + pts := (frame.ticks - first) * 90000 / r.frequency; + t := ticks_now(); + ok, error := encoder_write(*r.encoder, frame.pixels.data, pts); + + lock(*r.mutex); + r.free[r.free_count] = index; + r.free_count += 1; + r.stats.encode_ticks += ticks_now() - t; + if ok r.stats.written += 1; + r.stats.bytes = r.encoder.bytes; + unlock(*r.mutex); + if !ok { + record_fail(r, error); + // Frames still come until the capture thread sees it; they are let go. + while !atomic_read(*r.capture_ended) { + wait_for(*r.ready, 100); + lock(*r.mutex); + for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; } + r.queue_count = 0; + unlock(*r.mutex); + } + break; + } + } + return 0; +} + +// Ends the recording from one of its threads; the first reason is kept. +record_fail :: (r: *Recorder, error: string) { + log_error("[record] %", error); + lock(*r.mutex); + if !r.error r.error = copy_string(error); + unlock(*r.mutex); + atomic_swap(*r.stopping, true); +} + +ticks_now :: () -> s64 { + t: LARGE_INTEGER; + QueryPerformanceCounter(*t); + return t.QuadPart; +} + +rect_inside :: (inner: RECT, outer: RECT) -> bool { + return inner.left >= outer.left && inner.top >= outer.top && inner.right <= outer.right && inner.bottom <= outer.bottom; +} + +#scope_export + +#load "dxgi_source.jai"; +#load "encoder.jai"; +#load "cursor.jai"; + +#import "Basic"; +#import "String"; +#import "Atomics"; +#import "Thread"; +#import "Windows"; +#import "Windows_Utf8"; +#import "dxgi"; +#import "d3d11"; +#import "ffmpeg"; diff --git a/tools/record_probe.jai b/tools/record_probe.jai new file mode 100644 index 0000000..ec7c30c --- /dev/null +++ b/tools/record_probe.jai @@ -0,0 +1,301 @@ +// record-probe: phase 1 of video-capture-plan.html. Records a screen, or a region of one, with +// modules/Screen_Record (DXGI Desktop Duplication and FFmpeg) and says how it went: which screens +// there are and on which adapter, which duplication call and encoder worked, frames captured, written +// and dropped, time per stage, and the file's size. It also answers what the plan leaves to phase 1: +// +// -legacy the plain DuplicateOutput, not DuplicateOutput1: does it fail for a process aware +// of each monitor's DPI, as mcap is (this probe is, unless -dpi-unaware)? +// -cuda makes a CUDA context first, as onnxruntime does in mcap: does duplication still work? +// -exclude-test puts two squares on the region, one kept out of captures with +// WDA_EXCLUDEFROMCAPTURE: is it out of the recording, and the other in? +// -animate moves a square across the region the whole time, so the screen changes every +// frame: for measuring throughput. +// -check FILE reads FILE back with FFmpeg and says what it holds, and nothing else: for a +// recording cut short by a crash. +// -snapshot DIR the last frame captured (captured.png) and a frame decoded back from the file +// (decoded.png), to compare by eye. +// +// Build: jai first.jai - record_probe +// Run: bin/record-probe [-list] [-screen N] [-region X Y W H] [-seconds S] [-fps F] [-encoder NAME] +// [-out FILE.mp4] [-no-cursor] [-legacy] [-cuda] [-exclude-test] [-snapshot DIR] +// [-animate] [-dpi-unaware] +// Exit code: 0 recorded, 1 failed, 3 Desktop Duplication cannot run here (the fallback's case). + +main :: () { + args := get_command_line_arguments(); + options: Record_Options; + options.path = tprint("%/record-probe.mp4", get_working_directory()); + seconds: float64 = 10; + list_only, cuda, exclude_test, dpi_unaware, animate := false, false, false, false, false; + snapshot_folder: string; + i := 1; + number :: (index: s64) -> s32 #expand { return cast(s32) string_to_int(`args[index]); } + while i < args.count { + a := args[i]; + more := args.count - i - 1; + if a == "-list" { list_only = true; i += 1; continue; } + if a == "-screen" && more >= 1 { options.screen = number(i + 1); i += 2; continue; } + if a == "-region" && more >= 4 { + x, y, w, h := number(i + 1), number(i + 2), number(i + 3), number(i + 4); + options.region = .{x, y, x + w, y + h}; + i += 5; continue; + } + if a == "-seconds" && more >= 1 { seconds = string_to_float64(args[i + 1]); i += 2; continue; } + if a == "-fps" && more >= 1 { options.fps = number(i + 1); i += 2; continue; } + if a == "-encoder" && more >= 1 { options.encoder = args[i + 1]; i += 2; continue; } + if a == "-out" && more >= 1 { options.path = args[i + 1]; i += 2; continue; } + if a == "-snapshot" && more >= 1 { snapshot_folder = args[i + 1]; i += 2; continue; } + if a == "-no-cursor" { options.cursor = false; i += 1; continue; } + if a == "-legacy" { options.legacy_duplicate = true; i += 1; continue; } + if a == "-cuda" { cuda = true; i += 1; continue; } + if a == "-exclude-test" { exclude_test = true; i += 1; continue; } + if a == "-dpi-unaware" { dpi_unaware = true; i += 1; continue; } + if a == "-animate" { animate = true; i += 1; continue; } + if a == "-check" && more >= 1 { + image, info, ok, error := video_poster(args[i + 1]); + if !ok { print("%: cannot be read: %\n", args[i + 1], error); exit(1); } + print("%: % x %, % s, %, % fps; a frame decoded at % s\n", args[i + 1], info.width, info.height, FF(info.duration, 2), info.codec, FF(info.frame_rate, 1), FF(info.poster_time, 2)); + exit(0); + } + print("Unknown argument %\n", a); + exit(2); + } + + if !dpi_unaware { + DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 :: cast(HANDLE) -4; + if !SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2) print("Could not become per-monitor DPI aware (v2).\n"); + } + print("DPI awareness: %\n", ifx dpi_unaware then "none" else "per monitor (v2), as mcap"); + + screens := record_list_screens(); + print("Screens:\n"); + for screens { + d := it.desktop; + print(" %: % on adapter % (%): % x % at %, %%\n", it_index, it.name, it.adapter_index, it.adapter, + d.right - d.left, d.bottom - d.top, d.left, d.top, + ifx it.rotation == .IDENTITY || it.rotation == .UNSPECIFIED then "" else tprint(", rotated %", it.rotation)); + } + if list_only return; + + if cuda { + ok, text := cuda_make_context(); + print("CUDA: %\n", text); + if !ok exit(1); + } + + recorder: Recorder; + ok, error, fall_back := record_start(*recorder, options); + if !ok { + print("Cannot record: %\n", error); + if fall_back print("Desktop Duplication cannot run here: this is the case for the Windows.Graphics.Capture fallback.\n"); + exit(ifx fall_back then cast(s32) 3 else 1); + } + stats := record_stats(*recorder); + print("Recording % x % for % s at up to % fps with % and %, to %\n", stats.width, stats.height, seconds, options.fps, stats.source, stats.encoder, options.path); + + region := options.region; + if region.right <= region.left region = screens[ifx options.screen >= 0 then options.screen else 0].desktop; + excluded, control: HWND; + if exclude_test excluded, control = exclusion_windows(region); + mover: HWND; + if animate mover = square_window("record-probe animate", 0x0000A5FF, region.left, region.top, 120); + + start := current_time_monotonic(); + last_print := start; + while to_float64_seconds(current_time_monotonic() - start) < seconds { + pump_messages(); + if mover { // back and forth across the region, about 240 times a second + elapsed := to_float64_seconds(current_time_monotonic() - start); + span := region.right - region.left - 120; + phase := fmod_cycling(elapsed * 0.5, 2.0); + x := region.left + cast(s32) (span * ifx phase < 1 then phase else 2 - phase); + SetWindowPos(mover, HWND_TOPMOST, x, region.top + (region.bottom - region.top) / 2 - 60, 0, 0, SWP_NOSIZE | SWP_NOACTIVATE); + sleep_milliseconds(4); + } else { + sleep_milliseconds(20); + } + if to_float64_seconds(current_time_monotonic() - last_print) >= 1 { + last_print = current_time_monotonic(); + s := record_stats(*recorder); + print(" % s: % captured, % sent (% repeats), % written, % dropped, % lost, % KB\n", + FF(to_float64_seconds(last_print - start), 0), s.captured, s.emitted, s.repeats, s.written, s.dropped, s.lost, s.bytes / 1024); + } + failed, _ := record_failed(*recorder); + if failed break; + } + + last, width, height := record_last_frame(*recorder); + if mover DestroyWindow(mover); + stopped, stop_error := record_stop(*recorder); + s := recorder.stats; + print("\n%\n", ifx stopped then "Recorded." else tprint("Recording failed: %", stop_error)); + ms_each :: (ticks: s64, count: s64) -> string #expand { + if !count return "-"; + return tprint("% ms", FF(cast(float64) ticks / `recorder.frequency * 1000 / count, 2)); + } + file_size := file_size_of(options.path); + print(" source: %\n", s.source); + print(" encoder: %\n", s.encoder); + print(" size: % x %\n", s.width, s.height); + print(" seconds: %\n", FF(s.seconds, 1)); + print(" captured: % new images of the screen (% a second), % cursor-only changes\n", s.captured, FF(s.captured / s.seconds, 1), s.pointer_moves); + print(" sent: % frames (% a second), % of them repeats of an unchanged screen\n", s.emitted, FF(s.emitted / s.seconds, 1), s.repeats); + print(" written: %, dropped %, duplication lost %\n", s.written, s.dropped, s.lost); + print(" each: copy % (per image), read back % , encode % (per frame)\n", + ms_each(s.copy_ticks, s.captured), ms_each(s.readback_ticks, s.emitted), ms_each(s.encode_ticks, s.written)); + print(" file: % KB, % MB a minute\n", file_size / 1024, FF(cast(float64) file_size / (1024 * 1024) / s.seconds * 60, 1)); + + if exclude_test { + excluded_rect := window_rect_in(excluded, region); + control_rect := window_rect_in(control, region); + excluded_seen := colour_fraction(last, width, height, excluded_rect, EXCLUDED_BGR); + control_seen := colour_fraction(last, width, height, control_rect, CONTROL_BGR); + print(" exclusion: the excluded square is % in the last frame, the control square %: %\n", + percent(excluded_seen), percent(control_seen), + ifx control_seen < 0.9 then "inconclusive (the control is not there)" + else ifx excluded_seen < 0.01 then "WDA_EXCLUDEFROMCAPTURE is respected" + else "WDA_EXCLUDEFROMCAPTURE is NOT respected"); + DestroyWindow(excluded); + DestroyWindow(control); + } + + if snapshot_folder { + make_directory_if_it_does_not_exist(snapshot_folder, recursive = true); + rgba := bgra_to_rgba(last); + captured_path := tprint("%/captured.png", snapshot_folder); + stbi_write_png(temp_c_string(captured_path), width, height, 4, rgba.data, width * 4); + image, info, decoded, decode_error := video_poster(options.path); + if decoded { + decoded_path := tprint("%/decoded.png", snapshot_folder); + stbi_write_png(temp_c_string(decoded_path), image.width, image.height, 4, image.pixels, image.width * 4); + print(" snapshot: % and % (decoded at % s of %: %, % fps)\n", captured_path, decoded_path, FF(info.poster_time, 2), FF(info.duration, 2), info.codec, FF(info.frame_rate, 1)); + image_free(*image); + } else { + print(" snapshot: %; the file could not be decoded back: %\n", captured_path, decode_error); + } + } + exit(ifx stopped then cast(s32) 0 else 1); +} + +#scope_file + +EXCLUDED_BGR :: u8.[255, 0, 255]; // magenta +CONTROL_BGR :: u8.[255, 255, 0]; // cyan + +// Two squares on the region, topmost: one kept out of captures, one not. +exclusion_windows :: (region: RECT) -> excluded: HWND, control: HWND { + width := region.right - region.left; + height := region.bottom - region.top; + side := min(cast(s32) 160, width / 4, height / 2); + y := region.top + (height - side) / 2; + excluded := square_window("record-probe excluded", 0x00FF00FF, region.left + width / 4 - side / 2, y, side); + control := square_window("record-probe control", 0x00FFFF00, region.left + 3 * width / 4 - side / 2, y, side); + WDA_EXCLUDEFROMCAPTURE :: 0x11; + if !SetWindowDisplayAffinity(excluded, WDA_EXCLUDEFROMCAPTURE) print("SetWindowDisplayAffinity failed; the test cannot say anything.\n"); + pump_messages(); + return excluded, control; +} + +// A square of one colour, topmost, that never takes the focus. +square_window :: (class_name: string, colour: u32, x: s32, y: s32, side: s32) -> HWND { + instance := GetModuleHandleW(null); + wc: WNDCLASSEXW; + wc.cbSize = size_of(WNDCLASSEXW); + wc.lpfnWndProc = xx DefWindowProcW; + wc.hInstance = instance; + wc.hbrBackground = CreateSolidBrush(colour); + wc.lpszClassName = utf8_to_wide(class_name,, temp); + RegisterClassExW(*wc); + return CreateWindowExW(WS_EX_TOPMOST | WS_EX_TOOLWINDOW | WS_EX_NOACTIVATE, utf8_to_wide(class_name,, temp), utf8_to_wide(class_name,, temp), + WS_POPUP | WS_VISIBLE, x, y, side, side, null, null, instance, null); +} + +// A window's rectangle, in the region's pixels, a few pixels in from its edges. +window_rect_in :: (window: HWND, region: RECT) -> RECT { + r: RECT; + GetWindowRect(window, *r); + return .{r.left - region.left + 4, r.top - region.top + 4, r.right - region.left - 4, r.bottom - region.top - 4}; +} + +colour_fraction :: (pixels: [] u8, width: s32, height: s32, r: RECT, bgr: [3] u8) -> float64 { + total, same: s64; + for y: max(r.top, 0)..min(r.bottom, height) - 1 for x: max(r.left, 0)..min(r.right, width) - 1 { + p := pixels.data + (cast(s64) y * width + x) * 4; + total += 1; + if abs(cast(s32) p[0] - bgr[0]) <= 8 && abs(cast(s32) p[1] - bgr[1]) <= 8 && abs(cast(s32) p[2] - bgr[2]) <= 8 same += 1; + } + return ifx total then cast(float64) same / total else 0; +} + +pump_messages :: () { + msg: MSG; + while PeekMessageW(*msg, null, 0, 0, PM_REMOVE) { + TranslateMessage(*msg); + DispatchMessageW(*msg); + } +} + +// A CUDA context on the first device, as onnxruntime makes for OCR and labels in mcap, and kept. +cuda_make_context :: () -> ok: bool, text: string { + library := LoadLibraryW(utf8_to_wide("nvcuda.dll",, temp)); + if !library return false, "nvcuda.dll is not installed"; + Cu_Init :: #type (flags: u32) -> s32 #c_call; + Cu_Device :: #type (device: *s32, ordinal: s32) -> s32 #c_call; + Cu_Context :: #type (cuda_context: **void, flags: u32, device: s32) -> s32 #c_call; + Cu_Name :: #type (name: *u8, length: s32, device: s32) -> s32 #c_call; + cu_init := cast(Cu_Init) GetProcAddress(library, "cuInit"); + cu_device_get := cast(Cu_Device) GetProcAddress(library, "cuDeviceGet"); + cu_ctx_create := cast(Cu_Context) GetProcAddress(library, "cuCtxCreate_v2"); + cu_name := cast(Cu_Name) GetProcAddress(library, "cuDeviceGetName"); + if !cu_init || !cu_device_get || !cu_ctx_create || !cu_name return false, "nvcuda.dll lacks the calls needed"; + result := cu_init(0); + if result return false, tprint("cuInit failed (%)", result); + device: s32; + result = cu_device_get(*device, 0); + if result return false, tprint("cuDeviceGet failed (%)", result); + name: [256] u8; + cu_name(name.data, name.count, device); + cuda_context: *void; + result = cu_ctx_create(*cuda_context, 0, device); + if result return false, tprint("cuCtxCreate failed (%)", result); + return true, tprint("a context on %, kept for the recording", to_string(name.data)); +} + +bgra_to_rgba :: (pixels: [] u8) -> [] u8 { + out := NewArray(pixels.count, u8,, temp); + for i: 0..pixels.count / 4 - 1 { + out[i * 4 + 0] = pixels[i * 4 + 2]; + out[i * 4 + 1] = pixels[i * 4 + 1]; + out[i * 4 + 2] = pixels[i * 4 + 0]; + out[i * 4 + 3] = 255; + } + return out; +} + +file_size_of :: (path: string) -> s64 { + _, size, ok := file_modtime_and_size(path); + return ifx ok then size else 0; +} + +percent :: (fraction: float64) -> string { + return join(tprint("%", FF(fraction * 100, 1)), "%",, temp); +} + +FF :: (value: float64, decimals: s64) -> FormatFloat #expand { + return formatFloat(value, trailing_width = decimals, zero_removal = .NO); +} + +SetProcessDpiAwarenessContext :: (value: HANDLE) -> BOOL #foreign user32_dll; +SetWindowDisplayAffinity :: (window: HWND, affinity: u32) -> BOOL #foreign user32_dll; +user32_dll :: #library,system "user32"; + +#import "Basic"; +#import "String"; +#import "File"; +#import "File_Utilities"; +#import "Math"; +#import "Windows"; +#import "Windows_Utf8"; +#import "Screen_Record"; +#import "Image_Decode"; +#import "stb_image_write"; diff --git a/video-capture-plan.html b/video-capture-plan.html index 7529890..0b3f3a5 100644 --- a/video-capture-plan.html +++ b/video-capture-plan.html @@ -68,7 +68,7 @@

Screen video capture in mcap

-

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · not started

+

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · phase 1 built, mcap 0.1.3

@@ -80,6 +80,7 @@
  • Decided Ctrl+PrintScreen again stops it and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.
  • Decided A recording becomes an .mcap in Screen Captures, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.
  • It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.
    • +
  • Done Phase 1 (2026-10-10, mcap 0.1.3): modules/Screen_Record and record-probe. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see phase 1's result).
  • Later Audio is phase 5. Recording one window, and regions that span two screens, come after.
    • @@ -265,7 +266,7 @@ Captured_Frame :: struct {

    Each phase ends with something to run and check, and is a commit (0.1.3 onwards).

    -

    Phase 1: record-probe with Desktop Duplication

    +

    Phase 1: record-probe with Desktop Duplication Done

    A command-line tool, tools/record_probe.jai, built with jai first.jai - record_probe:

    • It lists adapters and outputs, picks a screen or region, and duplicates it on the output's adapter.
      • @@ -280,6 +281,43 @@ Captured_Frame :: struct {
    • What does 4K at 60 frames a second cost?

    Done when: 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A DXGI_ERROR_UNSUPPORTED is also an answer: it means phase 2 is needed on this machine.

    +
    Result (2026-10-10). Release build, RTX 5080 laptop. The screens are a 3840 × 2160 external and the 2560 × 1600 internal; DXGI lists both on the NVIDIA adapter. +
      +
    • The done-when run: 60.2 s of the 4K screen at 30 fps, with a square moving across it the whole time (-animate): +
        +
      • 3,602 images taken from the screen and 1,802 frames sent, all of them encoded with NVENC and none dropped.
        • +
      • 18 MB a minute.
        • +
      • FFmpeg reads it back as 60.07 s at 30 fps.
        • +
      • Per frame: 4.4 ms to read back and 13.4 ms to encode, on separate threads.
        • +
      + The frame decoded back matches the one captured, cursor included, by eye. Playing it in VLC and Media Player is still to check.
      • +
    • 60 fps: all 59.3 changes a second are kept. On a fixed tick grid, a 60 Hz screen beat against the 60 fps cap and lost about 7% of frames. Frames are now sent as soon as their turn is due, on a schedule that advances by exactly one frame each time. Caps of 30, 60 and 24 give 29.8, 59.3 and 23.8 frames a second.
      • +
    • The questions: +
        +
      • Both screens duplicate with DuplicateOutput1.
        • +
      • The plain DuplicateOutput works too in a per-monitor DPI-aware process. The documented failure doesn't happen here; DuplicateOutput1 stays the first choice.
        • +
      • With a CUDA context in the process, duplication still works.
        • +
      • WDA_EXCLUDEFROMCAPTURE is respected: the excluded square is 0% present in the frame and the control square 100%.
        • +
      • The two-GPU failure wasn't reproduced. Task Manager shows the Intel GPU, but no screen is attached to it now: the laptop is in discrete-GPU mode. Testing it means switching the laptop to Optimus (hybrid) mode in the NVIDIA Control Panel or the BIOS, which needs a restart.
        • +
      • +
    • Encoders, 4K at 30 fps, 8 s of the same motion. All of them keep up: +
      + + + + + + +
      EncoderEncode per frameMB a minute
      h264_nvenc13.7 ms23.4
      h264_qsv (the Intel GPU, though no screen is attached to it)12.3 ms16.7
      h264_mf16.8 ms24.0
      libx264 (veryfast, CRF 23)14.9 ms11.4
      h264_amfNot available: no AMD driver (amfrt64.dll), so it falls through to the next
      + Most of NVENC's 13.7 ms is FFmpeg copying the 33 MB frame before the GPU sees it. Handing it the frame without the copy is a later optimisation, not needed at 4K 60.
      • +
    • A crash mid-recording: killed about 7 s in, the fragmented MP4 still reads back with 6 s of video. At most the last keyframe interval (2 s) is lost.
      • +
    • The minimum size: 239 × 300 is refused, and 241 × 181 records at 240 × 180. The minimum is checked before rounding to even.
      • +
    +

    Changes from the plan:

    +
      +
    • The DPI check: once, a process that isn't DPI aware saw the 4K screen as 1920 × 1080, and the region would have been the wrong part of it. This couldn't be reproduced afterwards. dxgi_open now compares the duplication's size with the desktop's and refuses when they differ.
      • +
    • The probe's extra options: -animate, which moves a square for measuring throughput, and -check FILE, which reads a recording back.
      • +