diff --git a/README.md b/README.md index 9e5a66d..7a86210 100644 --- a/README.md +++ b/README.md @@ -356,6 +356,10 @@ converted copy. HEIF and TIFF); previews; and videos' frames and what their containers say, with FFmpeg (`video.jai`), and how many frames a GIF has. `bin\image-probe FILE...` (`jai first.jai - image_probe`) shows what it makes of files. +- `modules/Screen_Record`: a region of one screen recorded to H.264 in an MP4, with DXGI Desktop + Duplication and FFmpeg (NVENC, else AMF, QSV, Media Foundation or libx264), for screen recording + (`video-capture-plan.html`). `bin ecord-probe` (`jai first.jai - record_probe`) records with it + and says how it went; `-list` shows the screens, `-check FILE` reads a recording back. - `modules/Snagx`: `.snagx` to `.mcap`, copied from SnagitConverter. - `modules/Capture_File`: reading and writing `.mcap` files, with a video's original left on disk. - `modules/Segment`: panoptic segmentation with Mask2Former, run with onnxruntime (see its README and diff --git a/check_targets.jai b/check_targets.jai index 09110b2..64c7e84 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.1.2"; +VERSION :: "0.1.3"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 51c376b..5c937d8 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.1.2"; +VERSION :: "0.1.3"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; @@ -67,6 +67,8 @@ build_project :: () { if array_find(args, "image_probe") array_add(*pending, start_tool_build("image-probe", "tools/image_probe.jai", optimized = build_release)); // jai first.jai - segment_probe: also bin/segment-probe, what modules/Segment makes of image files. if array_find(args, "segment_probe") array_add(*pending, start_tool_build("segment-probe", "tools/segment_probe.jai", optimized = build_release)); + // jai first.jai - record_probe: also bin/record-probe, a screen recorded with modules/Screen_Record (video-capture-plan.html, phase 1). + if array_find(args, "record_probe") array_add(*pending, start_tool_build("record-probe", "tools/record_probe.jai", optimized = build_release)); wait_for_builds(pending); diff --git a/modules/Screen_Record/cursor.jai b/modules/Screen_Record/cursor.jai new file mode 100644 index 0000000..3533547 --- /dev/null +++ b/modules/Screen_Record/cursor.jai @@ -0,0 +1,48 @@ +// The mouse cursor, drawn into frames read back from Desktop Duplication, which leaves it out of the +// image and hands over its shape and position separately. Three kinds of shape: +// +// COLOR BGRA, blended by its alpha. +// MASKED_COLOR BGRA, where alpha 0 replaces the pixel under it and 0xFF XORs it. +// MONOCHROME 1 bit a pixel, twice as tall: an AND mask over an XOR mask. AND 1 XOR 1 inverts the +// pixel under it, which is how the text I-beam shows on any background. + +Cursor_Shape :: struct { + type: DXGI_OUTDUPL_POINTER_SHAPE_TYPE; + width, height, pitch: s32; // height as handed over: twice the cursor's for MONOCHROME + data: [..] u8; +} + +// Draws shape with its top left at x, y (in the frame's pixels; it may reach past the edges) into +// BGRA pixels, rows packed. +cursor_draw :: (pixels: *u8, width: s32, height: s32, shape: *Cursor_Shape, x: s32, y: s32) { + if !shape.data.count return; + rows := ifx shape.type == .MONOCHROME then shape.height / 2 else shape.height; + for row: 0..rows - 1 { + dy := y + row; + if dy < 0 || dy >= height continue; + for column: 0..shape.width - 1 { + dx := x + column; + if dx < 0 || dx >= width continue; + d := pixels + (cast(s64) dy * width + dx) * 4; + if shape.type == { + case .COLOR; + s := shape.data.data + row * shape.pitch + column * 4; + a := cast(u32) s[3]; + if a == 0 continue; + for c: 0..2 d[c] = cast(u8) ((cast(u32) s[c] * a + cast(u32) d[c] * (255 - a) + 127) / 255); + case .MASKED_COLOR; + s := shape.data.data + row * shape.pitch + column * 4; + if s[3] == 0 for c: 0..2 d[c] = s[c]; + else for c: 0..2 d[c] ^= s[c]; + case .MONOCHROME; + bit := cast(u8) (0x80 >> (column % 8)); + and_mask := shape.data[row * shape.pitch + column / 8] & bit; + xor_mask := shape.data[(row + rows) * shape.pitch + column / 8] & bit; + if and_mask && !xor_mask continue; // transparent + if and_mask { for c: 0..2 d[c] = 255 - d[c]; continue; } // inverted + value: u8 = ifx xor_mask then cast(u8) 255 else 0; // white or black + for c: 0..2 d[c] = value; + } + } + } +} diff --git a/modules/Screen_Record/dxgi_source.jai b/modules/Screen_Record/dxgi_source.jai new file mode 100644 index 0000000..3584e09 --- /dev/null +++ b/modules/Screen_Record/dxgi_source.jai @@ -0,0 +1,238 @@ +// DXGI Desktop Duplication: the screens as the GPU composes them, a frame each time one changes. +// +// It runs on a D3D11 device of its own, made on the adapter the screen is attached to: duplicating +// with a device on another adapter fails. On a laptop with two GPUs, where the screen hangs off one +// and the process may run on the other, DuplicateOutput1 can fail with DXGI_ERROR_UNSUPPORTED all the +// same; that, too many duplications of the screen at once, and a rotated screen are what +// dxgi_open returns fall_back for. +// +// DuplicateOutput1 (IDXGIOutput5) is used where there is one: the plain DuplicateOutput is documented +// to fail for a process aware of each monitor's DPI, as mcap is. legacy forces the plain one, for +// record-probe to check that. + +Dxgi_Source :: struct { + adapter: *IDXGIAdapter1; + output: *IDXGIOutput1; + output5: *IDXGIOutput5; // null before Windows 10 1703 + device: *ID3D11Device; + device_context: *ID3D11DeviceContext; + duplication: *IDXGIOutputDuplication; + legacy: bool; + method: string; // which duplication call worked, for the log + + box: D3D11_BOX; // the region, in the output's pixels + latest: *ID3D11Texture2D; // the region as last seen + staging: [RECORD_STAGING] *ID3D11Texture2D; + + copy_ticks: s64; // spent taking images out of frames: QueryInterface and the copy's submission + pointer_position: POINT; // the cursor's top left, in the output's pixels + pointer_visible: bool; + shape: Cursor_Shape; +} + +RECORD_STAGING :: 3; // staging textures in turn: the one read is two frames behind the one copied + +// Every screen attached to the desktop, on every adapter, in DXGI's order. Temp. +record_list_screens :: () -> [] Record_Screen { + screens: [..] Record_Screen; + screens.allocator = temp; + factory: *IDXGIFactory1; + if FAILED(CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory)) return screens; + defer IUnknown_Release(factory); + + adapter_index: u32; + while true { + adapter: *IDXGIAdapter1; + if FAILED(IDXGIFactory1_EnumAdapters1(factory, adapter_index, *adapter)) break; + defer IUnknown_Release(adapter); + adapter_desc: DXGI_ADAPTER_DESC1; + IDXGIAdapter1_GetDesc1(adapter, *adapter_desc); + adapter_name := wide_text(adapter_desc.Description.data); + + output_index: u32; + while true { + output: *IDXGIOutput; + if FAILED(IDXGIAdapter_EnumOutputs(adapter, output_index, *output)) break; + desc: DXGI_OUTPUT_DESC; + IDXGIOutput_GetDesc(output, *desc); + IUnknown_Release(output); + if desc.AttachedToDesktop != .FALSE { + array_add(*screens, .{adapter_index = adapter_index, output_index = output_index, adapter = adapter_name, + name = wide_text(desc.DeviceName.data), desktop = desc.DesktopCoordinates, rotation = desc.Rotation}); + } + output_index += 1; + } + adapter_index += 1; + } + return screens; +} + +// Makes the device on the screen's adapter, and duplicates the screen. region is in the screen's +// pixels and lies on it. +dxgi_open :: (s: *Dxgi_Source, screen: Record_Screen, region: RECT, legacy: bool) -> ok: bool, error: string, fall_back: bool { + s.legacy = legacy; + if screen.rotation != .IDENTITY && screen.rotation != .UNSPECIFIED return false, "the screen is rotated", true; + + factory: *IDXGIFactory1; + hr := CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory); + if FAILED(hr) return false, hresult_text("CreateDXGIFactory1", hr), false; + defer IUnknown_Release(factory); + hr = IDXGIFactory1_EnumAdapters1(factory, screen.adapter_index, *s.adapter); + if FAILED(hr) return false, hresult_text("the screen's adapter is gone", hr), false; + output: *IDXGIOutput; + hr = IDXGIAdapter_EnumOutputs(s.adapter, screen.output_index, *output); + if FAILED(hr) return false, hresult_text("the screen is gone", hr), false; + defer IUnknown_Release(output); + hr = IUnknown_QueryInterface(output, *uid(IDXGIOutput1_UUID), xx *s.output); + if FAILED(hr) return false, hresult_text("the screen cannot be duplicated (no IDXGIOutput1)", hr), true; + IUnknown_QueryInterface(output, *uid(IDXGIOutput5_UUID), xx *s.output5); // optional + + levels := D3D_FEATURE_LEVEL.[._11_1, ._11_0, ._10_1, ._10_0]; + level: D3D_FEATURE_LEVEL; + hr = D3D11CreateDevice(s.adapter, .UNKNOWN, null, .BGRA_SUPPORT, levels.data, levels.count, D3D11_SDK_VERSION, *s.device, *level, *s.device_context); + if FAILED(hr) return false, hresult_text("D3D11CreateDevice on the screen's adapter", hr), true; + + s.box = .{left = xx region.left, top = xx region.top, right = xx region.right, bottom = xx region.bottom, front = 0, back = 1}; + width := cast(u32) (region.right - region.left); + height := cast(u32) (region.bottom - region.top); + desc: D3D11_TEXTURE2D_DESC; + desc.Width = width; + desc.Height = height; + desc.MipLevels = 1; + desc.ArraySize = 1; + desc.Format = .B8G8R8A8_UNORM; + desc.SampleDesc.Count = 1; + desc.Usage = .DEFAULT; + hr = ID3D11Device_CreateTexture2D(s.device, *desc, null, *s.latest); + if FAILED(hr) return false, hresult_text("a texture for the region", hr), false; + desc.Usage = .STAGING; + desc.CPUAccessFlags = .READ; + for * s.staging { + hr = ID3D11Device_CreateTexture2D(s.device, *desc, null, it); + if FAILED(hr) return false, hresult_text("a staging texture", hr), false; + } + + hr = dxgi_duplicate(s); + if FAILED(hr) { + // Unsupported: two GPUs, the device on the wrong one. Not currently available: too many duplications. + fall_back := hresult_is(hr, DXGI_ERROR_UNSUPPORTED) || hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE); + return false, hresult_text(tprint("Desktop Duplication (%)", s.method), hr), fall_back; + } + + // A process not aware of each monitor's DPI is told the desktop in scaled pixels, while the frames + // come in the screen's own: the region would be the wrong part of the screen. + duplication_desc: DXGI_OUTDUPL_DESC; + IDXGIOutputDuplication_GetDesc(s.duplication, *duplication_desc); + desktop_width := screen.desktop.right - screen.desktop.left; + desktop_height := screen.desktop.bottom - screen.desktop.top; + if duplication_desc.ModeDesc.Width != cast(u32) desktop_width || duplication_desc.ModeDesc.Height != cast(u32) desktop_height { + return false, tprint("the screen is % x %, but this process sees it as % x %: it must be aware of each monitor's DPI", + duplication_desc.ModeDesc.Width, duplication_desc.ModeDesc.Height, desktop_width, desktop_height), false; + } + return true, "", false; +} + +// Duplicates the screen, again after the duplication was lost. +dxgi_duplicate :: (s: *Dxgi_Source) -> HRESULT { + if s.duplication { IUnknown_Release(s.duplication); s.duplication = null; } + if s.output5 && !s.legacy { + s.method = "DuplicateOutput1"; + formats := DXGI_FORMAT.[.B8G8R8A8_UNORM]; + return IDXGIOutput5_DuplicateOutput1(s.output5, s.device, 0, formats.count, formats.data, *s.duplication); + } + s.method = "DuplicateOutput"; + return IDXGIOutput1_DuplicateOutput(s.output, s.device, *s.duplication); +} + +Dxgi_Next :: enum u8 { NONE; IMAGE; POINTER; IMAGE_AND_POINTER; LOST; FAILED; } + +// Waits up to timeout_ms for the screen to change. A new image is copied into s.latest; a moved or +// changed cursor updates s.pointer_*. On LOST the duplication is gone: call dxgi_duplicate. +dxgi_next :: (s: *Dxgi_Source, timeout_ms: u32) -> Dxgi_Next, error: string { + info: DXGI_OUTDUPL_FRAME_INFO; + resource: *IDXGIResource; + hr := IDXGIOutputDuplication_AcquireNextFrame(s.duplication, timeout_ms, *info, *resource); + if hresult_is(hr, DXGI_ERROR_WAIT_TIMEOUT) return .NONE, ""; + if hresult_is(hr, DXGI_ERROR_ACCESS_LOST) return .LOST, ""; + if FAILED(hr) return .FAILED, hresult_text("AcquireNextFrame", hr); + defer IDXGIOutputDuplication_ReleaseFrame(s.duplication); // held, it stalls the desktop + defer IUnknown_Release(resource); + + image := info.LastPresentTime.QuadPart != 0; + if image { + start: LARGE_INTEGER; + QueryPerformanceCounter(*start); + texture: *ID3D11Texture2D; + hr = IUnknown_QueryInterface(resource, *uid(ID3D11Texture2D_UUID), xx *texture); + if FAILED(hr) return .FAILED, hresult_text("the frame is not a texture", hr); + ID3D11DeviceContext_CopySubresourceRegion(s.device_context, s.latest, 0, 0, 0, 0, texture, 0, *s.box); + IUnknown_Release(texture); + end: LARGE_INTEGER; + QueryPerformanceCounter(*end); + s.copy_ticks += end.QuadPart - start.QuadPart; + } + + pointer := info.LastMouseUpdateTime.QuadPart != 0; + if pointer { + s.pointer_position = info.PointerPosition.Position; + s.pointer_visible = info.PointerPosition.Visible != .FALSE; + if info.PointerShapeBufferSize { + array_resize(*s.shape.data, info.PointerShapeBufferSize, initialize = false); + needed: u32; + shape_info: DXGI_OUTDUPL_POINTER_SHAPE_INFO; + hr = IDXGIOutputDuplication_GetFramePointerShape(s.duplication, cast(u32) s.shape.data.count, s.shape.data.data, *needed, *shape_info); + if SUCCEEDED(hr) { + s.shape.type = cast(DXGI_OUTDUPL_POINTER_SHAPE_TYPE) shape_info.Type; + s.shape.width = cast(s32) shape_info.Width; + s.shape.height = cast(s32) shape_info.Height; + s.shape.pitch = cast(s32) shape_info.Pitch; + } else { + array_reset_keeping_memory(*s.shape.data); + } + } + } + if image && pointer return .IMAGE_AND_POINTER, ""; + if image return .IMAGE, ""; + if pointer return .POINTER, ""; + return .NONE, ""; +} + +dxgi_close :: (s: *Dxgi_Source) { + release :: (p: *$T) { if p.* { IUnknown_Release(p.*); p.* = null; } } + release(*s.duplication); + for * s.staging release(it); + release(*s.latest); + release(*s.device_context); + release(*s.device); + release(*s.output5); + release(*s.output); + release(*s.adapter); + array_free(s.shape.data); + s.shape = .{}; +} + +#scope_module + +hresult_is :: (hr: HRESULT, code: u32) -> bool { + return cast,no_check(u32) hr == code; +} + +// "what failed: 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", temp. +hresult_text :: (what: string, hr: HRESULT) -> string { + code := cast,no_check(u32) hr; + name := ""; + if code == { + case DXGI_ERROR_UNSUPPORTED; name = " (DXGI_ERROR_UNSUPPORTED)"; + case DXGI_ERROR_NOT_CURRENTLY_AVAILABLE; name = " (DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)"; + case DXGI_ERROR_ACCESS_LOST; name = " (DXGI_ERROR_ACCESS_LOST)"; + case 0x80070005; name = " (E_ACCESSDENIED)"; + case 0x80004002; name = " (E_NOINTERFACE)"; + case 0x80070057; name = " (E_INVALIDARG)"; + } + return tprint("%: 0x%%", what, formatInt(code, base = 16, minimum_digits = 8), name); +} + +wide_text :: (wide: *s16) -> string { + text, ok := wide_to_utf8(cast(*u16) wide,, temp); + return ifx ok then text else ""; +} diff --git a/modules/Screen_Record/encoder.jai b/modules/Screen_Record/encoder.jai new file mode 100644 index 0000000..3d37dee --- /dev/null +++ b/modules/Screen_Record/encoder.jai @@ -0,0 +1,213 @@ +// The encoder thread's half: frames of BGRA pixels to H.264 in a fragmented MP4, with FFmpeg +// (modules/ffmpeg). The first encoder in RECORD_ENCODERS that opens at the size wanted is used: +// NVENC, AMD's AMF, Intel's QSV, Windows' Media Foundation, then libx264 on the CPU. NVENC takes the +// BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with +// swscale (BT.709, limited range, as tagged). +// +// The file is written fragmented (a fragment each keyframe, every 2 s), so one cut short by a crash +// still plays up to its last fragment. + +RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"]; + +Encoder :: struct { + format: *AVFormatContext; + codec: *AVCodecContext; + stream: *AVStream; + frame: *AVFrame; + packet: *AVPacket; + sws: *SwsContext; + name: string; // the encoder used + input: AVPixelFormat; // what it is given: BGR0 as captured, or converted to NV12 or YUV420P + width, height: s32; + last_pts: s64 = -1; + header_written: bool; + bytes: s64; // of packets written +} + +// Opens the file and the first encoder that will take width x height (wanted alone, when given). +// Failures to open an encoder are listed in tried (temp), for the log. +encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string) -> ok: bool, error: string, tried: string { + av_log_set_level(AV_LOG_ERROR); + e.width, e.height = width, height; + result := avformat_alloc_output_context2(*e.format, null, "mp4", temp_c_string(path)); + if result < 0 || !e.format return false, av_error("FFmpeg cannot make an MP4", result), ""; + + tried: [..] string; + tried.allocator = temp; + names: [] string = RECORD_ENCODERS; + if wanted names = .[wanted]; + for names { + ok, why := encoder_try(e, it, width, height, fps); + if ok break; + array_add(*tried, tprint("%: %", it, why)); + } + tried_text := join(..tried, "; ",, temp); + if !e.codec { + encoder_free(e); + return false, tprint("no H.264 encoder opened (%)", tried_text), tried_text; + } + + e.stream = avformat_new_stream(e.format, null); + if !e.stream { encoder_free(e); return false, "FFmpeg cannot add a stream", tried_text; } + avcodec_parameters_from_context(e.stream.codecpar, e.codec); + e.stream.time_base = e.codec.time_base; + + result = avio_open(*e.format.pb, temp_c_string(path), AVIO_FLAG_WRITE); + if result < 0 { encoder_free(e); return false, av_error(tprint("FFmpeg cannot write %", path), result), tried_text; } + options: *AVDictionary; + av_dict_set(*options, "movflags", "frag_keyframe+empty_moov+default_base_moof", 0); + result = avformat_write_header(e.format, *options); + av_dict_free(*options); + if result < 0 { encoder_free(e); return false, av_error("FFmpeg cannot start the MP4", result), tried_text; } + e.header_written = true; + + e.packet = av_packet_alloc(); + e.frame = av_frame_alloc(); + if !e.packet || !e.frame { encoder_free(e); return false, "FFmpeg is out of memory", tried_text; } + if e.input != .BGR0 { + e.sws = sws_getContext(width, height, .BGR0, width, height, e.input, SWS_POINT, null, null, null); + if !e.sws { encoder_free(e); return false, "swscale cannot convert to the encoder's format", tried_text; } + coefficients := cast(*[4] s32) sws_getCoefficients(SWS_CS_ITU709); + sws_setColorspaceDetails(e.sws, coefficients, 1, coefficients, 0, 0, 1 << 16, 1 << 16); + } + return true, "", tried_text; +} + +// Encodes one frame; pts in RECORD_TIME_BASE ticks, made later than the last one if it is not. +encoder_write :: (e: *Encoder, pixels: *u8, pts: s64) -> ok: bool, error: string { + pts_out := max(pts, e.last_pts + 1); + e.last_pts = pts_out; + + frame := e.frame; + av_frame_unref(frame); + frame.format = cast(s32) e.input; + frame.width = e.width; + frame.height = e.height; + if e.input == .BGR0 { + // Not reference-counted: avcodec_send_frame copies it, so the buffer goes back to the pool at once. + frame.data[0] = pixels; + frame.linesize[0] = e.width * 4; + } else { + result := av_frame_get_buffer(frame, 0); + if result < 0 return false, av_error("FFmpeg cannot make a frame", result); + source := pixels; + stride := e.width * 4; + sws_scale(e.sws, *source, *stride, 0, e.height, frame.data.data, frame.linesize.data); + } + frame.pts = pts_out; + + result := avcodec_send_frame(e.codec, frame); + av_frame_unref(frame); + if result < 0 return false, av_error(tprint("% cannot encode", e.name), result); + drained, error := encoder_drain(e); + return drained, error; +} + +// Flushes the encoder, finishes the file and closes it. +encoder_close :: (e: *Encoder) -> ok: bool, error: string { + ok, error := true, ""; + if e.codec && e.header_written { + avcodec_send_frame(e.codec, null); + ok, error = encoder_drain(e); + result := av_write_trailer(e.format); + if ok && result < 0 ok, error = false, av_error("FFmpeg cannot finish the MP4", result); + } + encoder_free(e); + return ok, error; +} + +#scope_file + +RECORD_TIME_BASE :: 90000; // pts ticks a second: MPEG's + +encoder_try :: (e: *Encoder, name: string, width: s32, height: s32, fps: s32) -> ok: bool, error: string { + codec := avcodec_find_encoder_by_name(temp_c_string(name)); + if !codec return false, "not in this FFmpeg"; + input := encoder_input_format(codec); + if input == .NONE return false, "takes no format this can give it"; + + c := avcodec_alloc_context3(codec); + if !c return false, "out of memory"; + c.width = width; + c.height = height; + c.time_base = .{1, RECORD_TIME_BASE}; + c.framerate = .{fps, 1}; + c.pix_fmt = input; + c.gop_size = fps * 2; + c.max_b_frames = 0; + c.sample_aspect_ratio = .{1, 1}; + if input != .BGR0 { // swscale's conversion; NVENC picks and tags its own for BGRA + c.colorspace = .BT709; + c.color_primaries = .BT709; + c.color_trc = .BT709; + c.color_range = .MPEG; + } + if e.format.oformat.flags & AVFMT_GLOBALHEADER c.flags |= AV_CODEC_FLAG_GLOBAL_HEADER; + + options: *AVDictionary; + set :: (key: string, value: string) #expand { av_dict_set(*`options, temp_c_string(key), temp_c_string(value), 0); } + if name == { + case "h264_nvenc"; set("preset", "p4"); set("rc", "vbr"); set("cq", "23"); c.bit_rate = 0; + case "h264_amf"; set("rc", "cqp"); set("qp_i", "22"); set("qp_p", "24"); set("quality", "balanced"); + case "h264_qsv"; set("global_quality", "23"); set("preset", "medium"); + case "h264_mf"; set("rate_control", "quality"); set("quality", "70"); set("scenario", "display_remoting"); + case "libx264"; set("preset", "veryfast"); set("crf", "23"); + } + result := avcodec_open2(c, codec, *options); + av_dict_free(*options); + if result < 0 { + avcodec_free_context(*c); + return false, av_error("cannot open", result); + } + e.codec = c; + e.input = input; + e.name = name; + return true, ""; +} + +// BGRA as captured when the encoder takes it, else what swscale converts it to. +encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat { + if !codec.pix_fmts return .NONE; + for wanted: AVPixelFormat.[.BGR0, .BGRA, .NV12, .YUV420P] { + format := codec.pix_fmts; + while format.* != .NONE { + if format.* == wanted return ifx wanted == .BGRA then .BGR0 else wanted; + format += 1; + } + } + return .NONE; +} + +encoder_drain :: (e: *Encoder) -> ok: bool, error: string { + AVERROR_EAGAIN :: -11; + while true { + result := avcodec_receive_packet(e.codec, e.packet); + if result == AVERROR_EAGAIN || result == cast,no_check(s32) AVERROR_EOF return true, ""; + if result < 0 return false, av_error(tprint("% failed", e.name), result); + av_packet_rescale_ts(e.packet, e.codec.time_base, e.stream.time_base); + e.packet.stream_index = e.stream.index; + e.bytes += e.packet.size; + result = av_interleaved_write_frame(e.format, e.packet); // takes the packet's data + if result < 0 return false, av_error("FFmpeg cannot write to the MP4", result); + } + return true, ""; +} + +encoder_free :: (e: *Encoder) { + if e.sws sws_freeContext(e.sws); + if e.frame av_frame_free(*e.frame); + if e.packet av_packet_free(*e.packet); + if e.codec avcodec_free_context(*e.codec); + if e.format { + if e.format.pb avio_closep(*e.format.pb); + avformat_free_context(e.format); + } + e.sws, e.format, e.stream = null, null, null; +} + +// What FFmpeg says an error code is, after what failed (temp). +av_error :: (what: string, code: s32) -> string { + buffer: [256] u8; + av_strerror(code, buffer.data, buffer.count); + return tprint("% (%)", what, to_string(buffer.data)); +} diff --git a/modules/Screen_Record/module.jai b/modules/Screen_Record/module.jai new file mode 100644 index 0000000..8e4883a --- /dev/null +++ b/modules/Screen_Record/module.jai @@ -0,0 +1,410 @@ +// Screen_Record: a region of one screen recorded to H.264 in an MP4, for the mcap app's screen +// recording and tools/record_probe.jai (video-capture-plan.html). Windows only. +// +// Two threads. The capture thread takes the screen with DXGI Desktop Duplication (dxgi_source.jai), +// on a D3D11 device of its own, copies the region on the GPU, reads it back through a ring of staging +// textures, so Map never waits on the GPU, and draws the cursor into it (cursor.jai). The encoder +// thread hands the frames to FFmpeg (encoder.jai). Between them is a pool of RECORD_BUFFERS frames; +// when the encoder falls RECORD_QUEUE frames behind, the oldest waiting is dropped and counted. +// +// A frame is taken at most fps times a second, and only when the screen or the cursor changed, with +// one a second while nothing does, so players can seek. Each carries the time it was taken. +// +// recorder: Recorder; +// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0}); +// ... +// ok, error = record_stop(*recorder); +// +// fall_back says Desktop Duplication cannot run here: on a laptop with two GPUs, on a rotated screen, +// or with too many duplications of the screen running. Windows.Graphics.Capture takes over then +// (phase 2 of the plan). + +RECORD_MIN_WIDTH :: 240; // a smaller region is refused +RECORD_MIN_HEIGHT :: 180; +RECORD_QUEUE :: 8; // frames waiting for the encoder at most +RECORD_BUFFERS :: RECORD_QUEUE + 2; // and the one being filled, and the one being encoded + +Record_Screen :: struct { + adapter_index, output_index: u32; + adapter: string; // "NVIDIA GeForce RTX 5080 Laptop GPU" + name: string; // "\\.\DISPLAY1" + desktop: RECT; // on the virtual desktop + rotation: DXGI_MODE_ROTATION; +} + +Record_Options :: struct { + path: string; // the .mp4 written + screen: s32 = -1; // into record_list_screens(); -1: the one holding region + region: RECT; // on the virtual desktop; empty: the whole screen + fps: s32 = 30; + cursor := true; + encoder: string; // "" tries RECORD_ENCODERS in turn + legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe +} + +Record_Stats :: struct { + source: string; // "DXGI DuplicateOutput1 on " + encoder: string; + width, height: s32; + captured: s64; // new images of the screen + pointer_moves: s64; // cursor changes with no new image + emitted: s64; // frames sent to be encoded + repeats: s64; // of them, the once-a-second repeat of an unchanged screen + written: s64; // encoded + dropped: s64; // the encoder was too far behind + lost: s64; // times the duplication was lost and made again + copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks + bytes: s64; // encoded + seconds: float64; // from start to stop +} + +Recorder :: struct { + options: Record_Options; + source: Dxgi_Source; + encoder: Encoder; + width, height: s32; + frequency: s64; // QueryPerformanceCounter's + start_ticks: s64; + + capture_thread, encoder_thread: Thread; + running: bool; + stopping: bool; // set by record_stop; the capture thread ends, then the encoder + capture_ended: bool; + + mutex: Mutex; + ready: Semaphore; // signalled for each frame queued, and once when capture ends + frames: [RECORD_BUFFERS] Record_Frame; + free: [RECORD_BUFFERS] s32; + free_count: s32; + queue: [RECORD_BUFFERS] s32; // oldest first + queue_count: s32; + last_frame: [] u8; // the last frame queued, BGRA, for record_last_frame + stats: Record_Stats; + error: string; // why it stopped by itself; heap +} + +Record_Frame :: struct { + pixels: [] u8; // BGRA, rows packed + ticks: s64; +} + +// Opens the screen, the encoder and the file, and starts recording. +record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool { + r.* = .{}; + r.options = options; + r.options.path = copy_string(options.path); + r.options.encoder = copy_string(options.encoder); + if r.options.fps <= 0 r.options.fps = 30; + + screens := record_list_screens(); + if !screens.count return false, "no screen to record", false; + screen_index := options.screen; + region := options.region; + has_region := region.right > region.left && region.bottom > region.top; + if screen_index < 0 { + if !has_region screen_index = 0; + else for screens if rect_inside(region, it.desktop) { screen_index = xx it_index; break; } + if screen_index < 0 return false, "the region is not on one screen", false; + } + if screen_index >= screens.count return false, tprint("there is no screen %", screen_index), false; + screen := screens[screen_index]; + if !has_region region = screen.desktop; + if !rect_inside(region, screen.desktop) return false, "the region is not on that screen", false; + + // The minimum first, then even both ways, in the screen's pixels: H.264's 4:2:0 colour needs it. + // RECORD_MIN_WIDTH and _HEIGHT are even, so what passes stays at least that size. + #assert RECORD_MIN_WIDTH % 2 == 0 && RECORD_MIN_HEIGHT % 2 == 0; + if region.right - region.left < RECORD_MIN_WIDTH || region.bottom - region.top < RECORD_MIN_HEIGHT + return false, tprint("% x % is smaller than % x %", region.right - region.left, region.bottom - region.top, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false; + local := RECT.{region.left - screen.desktop.left, region.top - screen.desktop.top, region.right - screen.desktop.left, region.bottom - screen.desktop.top}; + local.right = local.left + ((local.right - local.left) & cast(s32) ~1); + local.bottom = local.top + ((local.bottom - local.top) & cast(s32) ~1); + r.width = local.right - local.left; + r.height = local.bottom - local.top; + + ok, error, fall_back := dxgi_open(*r.source, screen, local, options.legacy_duplicate); + if !ok { dxgi_close(*r.source); return false, error, fall_back; } + + tried: string; + ok, error, tried = encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder); + if tried log("[record] Encoders that did not open: %", tried); + if !ok { dxgi_close(*r.source); return false, error, false; } + + r.stats.source = sprint("DXGI % on %", r.source.method, screen.adapter); + r.stats.encoder = copy_string(r.encoder.name); + r.stats.width, r.stats.height = r.width, r.height; + + init(*r.mutex, "record"); + init(*r.ready); + for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false); + for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it; + r.free_count = RECORD_BUFFERS; + r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8); + + frequency: LARGE_INTEGER; + QueryPerformanceFrequency(*frequency); + r.frequency = frequency.QuadPart; + r.start_ticks = ticks_now(); + thread_init(*r.capture_thread, record_capture_proc, name = "record capture"); + thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder"); + r.running = true; + thread_start(*r.capture_thread, r); + thread_start(*r.encoder_thread, r); + return true, "", false; +} + +// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an +// error, or the file could not be finished; what was recorded up to then is in the file all the same. +record_stop :: (r: *Recorder) -> ok: bool, error: string { + if !r.running return false, "not recording"; + atomic_swap(*r.stopping, true); + thread_is_done(*r.capture_thread, -1); + thread_is_done(*r.encoder_thread, -1); + thread_deinit(*r.capture_thread); + thread_deinit(*r.encoder_thread); + r.running = false; + + closed, close_error := encoder_close(*r.encoder); + r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency; + r.stats.bytes = r.encoder.bytes; + dxgi_close(*r.source); + for * r.frames array_free(it.pixels); + destroy(*r.ready); + destroy(*r.mutex); + + if r.error return false, r.error; + if !closed return false, close_error; + return true, ""; +} + +// As they are now; strings are the recorder's. +record_stats :: (r: *Recorder) -> Record_Stats { + if !r.running return r.stats; + lock(*r.mutex); + defer unlock(*r.mutex); + return r.stats; +} + +// The last frame queued, BGRA (temp), cursor and all. +record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 { + if r.running lock(*r.mutex); + pixels := array_copy(r.last_frame,, temp); + if r.running unlock(*r.mutex); + return pixels, r.width, r.height; +} + +// Whether recording stopped by itself (the screen went away, the encoder failed); why, then. +record_failed :: (r: *Recorder) -> bool, string { + lock(*r.mutex); + defer unlock(*r.mutex); + return r.error.count > 0, r.error; +} + +#scope_file + +Staged :: struct { pending: bool; ticks: s64; pointer: POINT; pointer_visible: bool; } + +record_capture_proc :: (thread: *Thread) -> s64 { + r := cast(*Recorder) thread.data; + s := *r.source; + // A change is sent as soon as it comes once its turn is due: turns are a frame's time apart on a + // schedule that moves on a frame at a time, so the rate never passes fps, and a tenth of a frame + // early counts as on time, so a 60 Hz screen's frames, which jitter, all make a 60 fps cap. + period := r.frequency / r.options.fps; + early := period / 10; + turn := ticks_now(); + last_emit: s64 = -r.frequency; + have_image, changed := false, false; + staged: [RECORD_STAGING] Staged; + emits: s64; + + while !atomic_read(*r.stopping) { + now := ticks_now(); + due := ifx changed then turn - early else last_emit + r.frequency; // the change, or the repeat + wait := clamp((due - now) * 1000 / r.frequency, 0, 100); + next, error := dxgi_next(s, cast(u32) wait); + if next == { + case .FAILED; + record_fail(r, error); + break; + case .LOST; + // A mode change, the secure desktop, a game going fullscreen: duplicated again, and on. + lock(*r.mutex); r.stats.lost += 1; unlock(*r.mutex); + while !atomic_read(*r.stopping) { + hr := dxgi_duplicate(s); + if SUCCEEDED(hr) break; + if !hresult_is(hr, DXGI_ERROR_ACCESS_LOST) && !hresult_is(hr, 0x80070005) && !hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE) { + record_fail(r, hresult_text("the screen could not be duplicated again", hr)); + break; + } + sleep_milliseconds(250); + } + if atomic_read(*r.stopping) || r.error break; + continue; + case .IMAGE; #through; + case .IMAGE_AND_POINTER; + have_image, changed = true, true; + lock(*r.mutex); + r.stats.captured += 1; + r.stats.copy_ticks = s.copy_ticks; + unlock(*r.mutex); + case .POINTER; + if r.options.cursor && have_image { + changed = true; + lock(*r.mutex); r.stats.pointer_moves += 1; unlock(*r.mutex); + } + } + + now = ticks_now(); + if !have_image continue; + repeat := !changed && now - last_emit >= r.frequency; + if !(changed && now >= turn - early) && !repeat continue; + turn = max(turn + period, now - period); // after a quiet spell, no burst to catch up + + // Copied to staging now; read back two frames on, when the GPU is done with it. + slot := emits % RECORD_STAGING; + if staged[slot].pending record_read_back(r, slot, staged[slot]); + ID3D11DeviceContext_CopyResource(s.device_context, s.staging[slot], s.latest); + staged[slot] = .{pending = true, ticks = now, pointer = s.pointer_position, pointer_visible = s.pointer_visible}; + emits += 1; + last_emit = now; + changed = false; + lock(*r.mutex); + r.stats.emitted += 1; + if repeat r.stats.repeats += 1; + unlock(*r.mutex); + } + + // What is still staged, oldest first. + for 0..RECORD_STAGING - 1 { + slot := (emits + it) % RECORD_STAGING; + if staged[slot].pending record_read_back(r, slot, staged[slot]); + } + atomic_swap(*r.capture_ended, true); + signal(*r.ready); + return 0; +} + +// Copies a staging texture into a free frame, draws the cursor and queues it. +record_read_back :: (r: *Recorder, slot: s64, staged: Staged) { + s := *r.source; + t := ticks_now(); + lock(*r.mutex); + if !r.free_count { // every frame is waiting: drop the oldest + r.free[r.free_count] = r.queue[0]; + r.free_count += 1; + for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it]; + r.queue_count -= 1; + r.stats.dropped += 1; + } + r.free_count -= 1; + index := r.free[r.free_count]; + unlock(*r.mutex); + frame := *r.frames[index]; + + mapped: D3D11_MAPPED_SUBRESOURCE; + hr := ID3D11DeviceContext_Map(s.device_context, s.staging[slot], 0, .READ, 0, *mapped); + if FAILED(hr) { + lock(*r.mutex); r.free[r.free_count] = index; r.free_count += 1; unlock(*r.mutex); + record_fail(r, hresult_text("reading the frame back", hr)); + return; + } + row := cast(s64) r.width * 4; + for y: 0..r.height - 1 memcpy(frame.pixels.data + y * row, cast(*u8) mapped.pData + cast(s64) y * mapped.RowPitch, row); + ID3D11DeviceContext_Unmap(s.device_context, s.staging[slot], 0); + if r.options.cursor && staged.pointer_visible { + cursor_draw(frame.pixels.data, r.width, r.height, *s.shape, + staged.pointer.x - cast(s32) s.box.left, staged.pointer.y - cast(s32) s.box.top); + } + frame.ticks = staged.ticks; + + lock(*r.mutex); + r.queue[r.queue_count] = index; + r.queue_count += 1; + memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count); + r.stats.readback_ticks += ticks_now() - t; + unlock(*r.mutex); + signal(*r.ready); +} + +record_encoder_proc :: (thread: *Thread) -> s64 { + r := cast(*Recorder) thread.data; + first: s64 = -1; + while true { + wait_for(*r.ready, 100); + lock(*r.mutex); + index: s32 = -1; + if r.queue_count { + index = r.queue[0]; + for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it]; + r.queue_count -= 1; + } + unlock(*r.mutex); + if index < 0 { + if atomic_read(*r.capture_ended) break; + continue; + } + + frame := *r.frames[index]; + if first < 0 first = frame.ticks; + pts := (frame.ticks - first) * 90000 / r.frequency; + t := ticks_now(); + ok, error := encoder_write(*r.encoder, frame.pixels.data, pts); + + lock(*r.mutex); + r.free[r.free_count] = index; + r.free_count += 1; + r.stats.encode_ticks += ticks_now() - t; + if ok r.stats.written += 1; + r.stats.bytes = r.encoder.bytes; + unlock(*r.mutex); + if !ok { + record_fail(r, error); + // Frames still come until the capture thread sees it; they are let go. + while !atomic_read(*r.capture_ended) { + wait_for(*r.ready, 100); + lock(*r.mutex); + for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; } + r.queue_count = 0; + unlock(*r.mutex); + } + break; + } + } + return 0; +} + +// Ends the recording from one of its threads; the first reason is kept. +record_fail :: (r: *Recorder, error: string) { + log_error("[record] %", error); + lock(*r.mutex); + if !r.error r.error = copy_string(error); + unlock(*r.mutex); + atomic_swap(*r.stopping, true); +} + +ticks_now :: () -> s64 { + t: LARGE_INTEGER; + QueryPerformanceCounter(*t); + return t.QuadPart; +} + +rect_inside :: (inner: RECT, outer: RECT) -> bool { + return inner.left >= outer.left && inner.top >= outer.top && inner.right <= outer.right && inner.bottom <= outer.bottom; +} + +#scope_export + +#load "dxgi_source.jai"; +#load "encoder.jai"; +#load "cursor.jai"; + +#import "Basic"; +#import "String"; +#import "Atomics"; +#import "Thread"; +#import "Windows"; +#import "Windows_Utf8"; +#import "dxgi"; +#import "d3d11"; +#import "ffmpeg"; diff --git a/tools/record_probe.jai b/tools/record_probe.jai new file mode 100644 index 0000000..ec7c30c --- /dev/null +++ b/tools/record_probe.jai @@ -0,0 +1,301 @@ +// record-probe: phase 1 of video-capture-plan.html. Records a screen, or a region of one, with +// modules/Screen_Record (DXGI Desktop Duplication and FFmpeg) and says how it went: which screens +// there are and on which adapter, which duplication call and encoder worked, frames captured, written +// and dropped, time per stage, and the file's size. It also answers what the plan leaves to phase 1: +// +// -legacy the plain DuplicateOutput, not DuplicateOutput1: does it fail for a process aware +// of each monitor's DPI, as mcap is (this probe is, unless -dpi-unaware)? +// -cuda makes a CUDA context first, as onnxruntime does in mcap: does duplication still work? +// -exclude-test puts two squares on the region, one kept out of captures with +// WDA_EXCLUDEFROMCAPTURE: is it out of the recording, and the other in? +// -animate moves a square across the region the whole time, so the screen changes every +// frame: for measuring throughput. +// -check FILE reads FILE back with FFmpeg and says what it holds, and nothing else: for a +// recording cut short by a crash. +// -snapshot DIR the last frame captured (captured.png) and a frame decoded back from the file +// (decoded.png), to compare by eye. +// +// Build: jai first.jai - record_probe +// Run: bin/record-probe [-list] [-screen N] [-region X Y W H] [-seconds S] [-fps F] [-encoder NAME] +// [-out FILE.mp4] [-no-cursor] [-legacy] [-cuda] [-exclude-test] [-snapshot DIR] +// [-animate] [-dpi-unaware] +// Exit code: 0 recorded, 1 failed, 3 Desktop Duplication cannot run here (the fallback's case). + +main :: () { + args := get_command_line_arguments(); + options: Record_Options; + options.path = tprint("%/record-probe.mp4", get_working_directory()); + seconds: float64 = 10; + list_only, cuda, exclude_test, dpi_unaware, animate := false, false, false, false, false; + snapshot_folder: string; + i := 1; + number :: (index: s64) -> s32 #expand { return cast(s32) string_to_int(`args[index]); } + while i < args.count { + a := args[i]; + more := args.count - i - 1; + if a == "-list" { list_only = true; i += 1; continue; } + if a == "-screen" && more >= 1 { options.screen = number(i + 1); i += 2; continue; } + if a == "-region" && more >= 4 { + x, y, w, h := number(i + 1), number(i + 2), number(i + 3), number(i + 4); + options.region = .{x, y, x + w, y + h}; + i += 5; continue; + } + if a == "-seconds" && more >= 1 { seconds = string_to_float64(args[i + 1]); i += 2; continue; } + if a == "-fps" && more >= 1 { options.fps = number(i + 1); i += 2; continue; } + if a == "-encoder" && more >= 1 { options.encoder = args[i + 1]; i += 2; continue; } + if a == "-out" && more >= 1 { options.path = args[i + 1]; i += 2; continue; } + if a == "-snapshot" && more >= 1 { snapshot_folder = args[i + 1]; i += 2; continue; } + if a == "-no-cursor" { options.cursor = false; i += 1; continue; } + if a == "-legacy" { options.legacy_duplicate = true; i += 1; continue; } + if a == "-cuda" { cuda = true; i += 1; continue; } + if a == "-exclude-test" { exclude_test = true; i += 1; continue; } + if a == "-dpi-unaware" { dpi_unaware = true; i += 1; continue; } + if a == "-animate" { animate = true; i += 1; continue; } + if a == "-check" && more >= 1 { + image, info, ok, error := video_poster(args[i + 1]); + if !ok { print("%: cannot be read: %\n", args[i + 1], error); exit(1); } + print("%: % x %, % s, %, % fps; a frame decoded at % s\n", args[i + 1], info.width, info.height, FF(info.duration, 2), info.codec, FF(info.frame_rate, 1), FF(info.poster_time, 2)); + exit(0); + } + print("Unknown argument %\n", a); + exit(2); + } + + if !dpi_unaware { + DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 :: cast(HANDLE) -4; + if !SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2) print("Could not become per-monitor DPI aware (v2).\n"); + } + print("DPI awareness: %\n", ifx dpi_unaware then "none" else "per monitor (v2), as mcap"); + + screens := record_list_screens(); + print("Screens:\n"); + for screens { + d := it.desktop; + print(" %: % on adapter % (%): % x % at %, %%\n", it_index, it.name, it.adapter_index, it.adapter, + d.right - d.left, d.bottom - d.top, d.left, d.top, + ifx it.rotation == .IDENTITY || it.rotation == .UNSPECIFIED then "" else tprint(", rotated %", it.rotation)); + } + if list_only return; + + if cuda { + ok, text := cuda_make_context(); + print("CUDA: %\n", text); + if !ok exit(1); + } + + recorder: Recorder; + ok, error, fall_back := record_start(*recorder, options); + if !ok { + print("Cannot record: %\n", error); + if fall_back print("Desktop Duplication cannot run here: this is the case for the Windows.Graphics.Capture fallback.\n"); + exit(ifx fall_back then cast(s32) 3 else 1); + } + stats := record_stats(*recorder); + print("Recording % x % for % s at up to % fps with % and %, to %\n", stats.width, stats.height, seconds, options.fps, stats.source, stats.encoder, options.path); + + region := options.region; + if region.right <= region.left region = screens[ifx options.screen >= 0 then options.screen else 0].desktop; + excluded, control: HWND; + if exclude_test excluded, control = exclusion_windows(region); + mover: HWND; + if animate mover = square_window("record-probe animate", 0x0000A5FF, region.left, region.top, 120); + + start := current_time_monotonic(); + last_print := start; + while to_float64_seconds(current_time_monotonic() - start) < seconds { + pump_messages(); + if mover { // back and forth across the region, about 240 times a second + elapsed := to_float64_seconds(current_time_monotonic() - start); + span := region.right - region.left - 120; + phase := fmod_cycling(elapsed * 0.5, 2.0); + x := region.left + cast(s32) (span * ifx phase < 1 then phase else 2 - phase); + SetWindowPos(mover, HWND_TOPMOST, x, region.top + (region.bottom - region.top) / 2 - 60, 0, 0, SWP_NOSIZE | SWP_NOACTIVATE); + sleep_milliseconds(4); + } else { + sleep_milliseconds(20); + } + if to_float64_seconds(current_time_monotonic() - last_print) >= 1 { + last_print = current_time_monotonic(); + s := record_stats(*recorder); + print(" % s: % captured, % sent (% repeats), % written, % dropped, % lost, % KB\n", + FF(to_float64_seconds(last_print - start), 0), s.captured, s.emitted, s.repeats, s.written, s.dropped, s.lost, s.bytes / 1024); + } + failed, _ := record_failed(*recorder); + if failed break; + } + + last, width, height := record_last_frame(*recorder); + if mover DestroyWindow(mover); + stopped, stop_error := record_stop(*recorder); + s := recorder.stats; + print("\n%\n", ifx stopped then "Recorded." else tprint("Recording failed: %", stop_error)); + ms_each :: (ticks: s64, count: s64) -> string #expand { + if !count return "-"; + return tprint("% ms", FF(cast(float64) ticks / `recorder.frequency * 1000 / count, 2)); + } + file_size := file_size_of(options.path); + print(" source: %\n", s.source); + print(" encoder: %\n", s.encoder); + print(" size: % x %\n", s.width, s.height); + print(" seconds: %\n", FF(s.seconds, 1)); + print(" captured: % new images of the screen (% a second), % cursor-only changes\n", s.captured, FF(s.captured / s.seconds, 1), s.pointer_moves); + print(" sent: % frames (% a second), % of them repeats of an unchanged screen\n", s.emitted, FF(s.emitted / s.seconds, 1), s.repeats); + print(" written: %, dropped %, duplication lost %\n", s.written, s.dropped, s.lost); + print(" each: copy % (per image), read back % , encode % (per frame)\n", + ms_each(s.copy_ticks, s.captured), ms_each(s.readback_ticks, s.emitted), ms_each(s.encode_ticks, s.written)); + print(" file: % KB, % MB a minute\n", file_size / 1024, FF(cast(float64) file_size / (1024 * 1024) / s.seconds * 60, 1)); + + if exclude_test { + excluded_rect := window_rect_in(excluded, region); + control_rect := window_rect_in(control, region); + excluded_seen := colour_fraction(last, width, height, excluded_rect, EXCLUDED_BGR); + control_seen := colour_fraction(last, width, height, control_rect, CONTROL_BGR); + print(" exclusion: the excluded square is % in the last frame, the control square %: %\n", + percent(excluded_seen), percent(control_seen), + ifx control_seen < 0.9 then "inconclusive (the control is not there)" + else ifx excluded_seen < 0.01 then "WDA_EXCLUDEFROMCAPTURE is respected" + else "WDA_EXCLUDEFROMCAPTURE is NOT respected"); + DestroyWindow(excluded); + DestroyWindow(control); + } + + if snapshot_folder { + make_directory_if_it_does_not_exist(snapshot_folder, recursive = true); + rgba := bgra_to_rgba(last); + captured_path := tprint("%/captured.png", snapshot_folder); + stbi_write_png(temp_c_string(captured_path), width, height, 4, rgba.data, width * 4); + image, info, decoded, decode_error := video_poster(options.path); + if decoded { + decoded_path := tprint("%/decoded.png", snapshot_folder); + stbi_write_png(temp_c_string(decoded_path), image.width, image.height, 4, image.pixels, image.width * 4); + print(" snapshot: % and % (decoded at % s of %: %, % fps)\n", captured_path, decoded_path, FF(info.poster_time, 2), FF(info.duration, 2), info.codec, FF(info.frame_rate, 1)); + image_free(*image); + } else { + print(" snapshot: %; the file could not be decoded back: %\n", captured_path, decode_error); + } + } + exit(ifx stopped then cast(s32) 0 else 1); +} + +#scope_file + +EXCLUDED_BGR :: u8.[255, 0, 255]; // magenta +CONTROL_BGR :: u8.[255, 255, 0]; // cyan + +// Two squares on the region, topmost: one kept out of captures, one not. +exclusion_windows :: (region: RECT) -> excluded: HWND, control: HWND { + width := region.right - region.left; + height := region.bottom - region.top; + side := min(cast(s32) 160, width / 4, height / 2); + y := region.top + (height - side) / 2; + excluded := square_window("record-probe excluded", 0x00FF00FF, region.left + width / 4 - side / 2, y, side); + control := square_window("record-probe control", 0x00FFFF00, region.left + 3 * width / 4 - side / 2, y, side); + WDA_EXCLUDEFROMCAPTURE :: 0x11; + if !SetWindowDisplayAffinity(excluded, WDA_EXCLUDEFROMCAPTURE) print("SetWindowDisplayAffinity failed; the test cannot say anything.\n"); + pump_messages(); + return excluded, control; +} + +// A square of one colour, topmost, that never takes the focus. +square_window :: (class_name: string, colour: u32, x: s32, y: s32, side: s32) -> HWND { + instance := GetModuleHandleW(null); + wc: WNDCLASSEXW; + wc.cbSize = size_of(WNDCLASSEXW); + wc.lpfnWndProc = xx DefWindowProcW; + wc.hInstance = instance; + wc.hbrBackground = CreateSolidBrush(colour); + wc.lpszClassName = utf8_to_wide(class_name,, temp); + RegisterClassExW(*wc); + return CreateWindowExW(WS_EX_TOPMOST | WS_EX_TOOLWINDOW | WS_EX_NOACTIVATE, utf8_to_wide(class_name,, temp), utf8_to_wide(class_name,, temp), + WS_POPUP | WS_VISIBLE, x, y, side, side, null, null, instance, null); +} + +// A window's rectangle, in the region's pixels, a few pixels in from its edges. +window_rect_in :: (window: HWND, region: RECT) -> RECT { + r: RECT; + GetWindowRect(window, *r); + return .{r.left - region.left + 4, r.top - region.top + 4, r.right - region.left - 4, r.bottom - region.top - 4}; +} + +colour_fraction :: (pixels: [] u8, width: s32, height: s32, r: RECT, bgr: [3] u8) -> float64 { + total, same: s64; + for y: max(r.top, 0)..min(r.bottom, height) - 1 for x: max(r.left, 0)..min(r.right, width) - 1 { + p := pixels.data + (cast(s64) y * width + x) * 4; + total += 1; + if abs(cast(s32) p[0] - bgr[0]) <= 8 && abs(cast(s32) p[1] - bgr[1]) <= 8 && abs(cast(s32) p[2] - bgr[2]) <= 8 same += 1; + } + return ifx total then cast(float64) same / total else 0; +} + +pump_messages :: () { + msg: MSG; + while PeekMessageW(*msg, null, 0, 0, PM_REMOVE) { + TranslateMessage(*msg); + DispatchMessageW(*msg); + } +} + +// A CUDA context on the first device, as onnxruntime makes for OCR and labels in mcap, and kept. +cuda_make_context :: () -> ok: bool, text: string { + library := LoadLibraryW(utf8_to_wide("nvcuda.dll",, temp)); + if !library return false, "nvcuda.dll is not installed"; + Cu_Init :: #type (flags: u32) -> s32 #c_call; + Cu_Device :: #type (device: *s32, ordinal: s32) -> s32 #c_call; + Cu_Context :: #type (cuda_context: **void, flags: u32, device: s32) -> s32 #c_call; + Cu_Name :: #type (name: *u8, length: s32, device: s32) -> s32 #c_call; + cu_init := cast(Cu_Init) GetProcAddress(library, "cuInit"); + cu_device_get := cast(Cu_Device) GetProcAddress(library, "cuDeviceGet"); + cu_ctx_create := cast(Cu_Context) GetProcAddress(library, "cuCtxCreate_v2"); + cu_name := cast(Cu_Name) GetProcAddress(library, "cuDeviceGetName"); + if !cu_init || !cu_device_get || !cu_ctx_create || !cu_name return false, "nvcuda.dll lacks the calls needed"; + result := cu_init(0); + if result return false, tprint("cuInit failed (%)", result); + device: s32; + result = cu_device_get(*device, 0); + if result return false, tprint("cuDeviceGet failed (%)", result); + name: [256] u8; + cu_name(name.data, name.count, device); + cuda_context: *void; + result = cu_ctx_create(*cuda_context, 0, device); + if result return false, tprint("cuCtxCreate failed (%)", result); + return true, tprint("a context on %, kept for the recording", to_string(name.data)); +} + +bgra_to_rgba :: (pixels: [] u8) -> [] u8 { + out := NewArray(pixels.count, u8,, temp); + for i: 0..pixels.count / 4 - 1 { + out[i * 4 + 0] = pixels[i * 4 + 2]; + out[i * 4 + 1] = pixels[i * 4 + 1]; + out[i * 4 + 2] = pixels[i * 4 + 0]; + out[i * 4 + 3] = 255; + } + return out; +} + +file_size_of :: (path: string) -> s64 { + _, size, ok := file_modtime_and_size(path); + return ifx ok then size else 0; +} + +percent :: (fraction: float64) -> string { + return join(tprint("%", FF(fraction * 100, 1)), "%",, temp); +} + +FF :: (value: float64, decimals: s64) -> FormatFloat #expand { + return formatFloat(value, trailing_width = decimals, zero_removal = .NO); +} + +SetProcessDpiAwarenessContext :: (value: HANDLE) -> BOOL #foreign user32_dll; +SetWindowDisplayAffinity :: (window: HWND, affinity: u32) -> BOOL #foreign user32_dll; +user32_dll :: #library,system "user32"; + +#import "Basic"; +#import "String"; +#import "File"; +#import "File_Utilities"; +#import "Math"; +#import "Windows"; +#import "Windows_Utf8"; +#import "Screen_Record"; +#import "Image_Decode"; +#import "stb_image_write"; diff --git a/video-capture-plan.html b/video-capture-plan.html index 7529890..0b3f3a5 100644 --- a/video-capture-plan.html +++ b/video-capture-plan.html @@ -68,7 +68,7 @@

Screen video capture in mcap

-

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · not started

+

Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · phase 1 built, mcap 0.1.3

@@ -80,6 +80,7 @@
  • Decided Ctrl+PrintScreen again stops it and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.
  • Decided A recording becomes an .mcap in Screen Captures, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.
  • It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.
    • +
  • Done Phase 1 (2026-10-10, mcap 0.1.3): modules/Screen_Record and record-probe. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see phase 1's result).
  • Later Audio is phase 5. Recording one window, and regions that span two screens, come after.
    • @@ -265,7 +266,7 @@ Captured_Frame :: struct {

    Each phase ends with something to run and check, and is a commit (0.1.3 onwards).

    -

    Phase 1: record-probe with Desktop Duplication

    +

    Phase 1: record-probe with Desktop Duplication Done

    A command-line tool, tools/record_probe.jai, built with jai first.jai - record_probe:

    Done when: 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A DXGI_ERROR_UNSUPPORTED is also an answer: it means phase 2 is needed on this machine.

    +
    Result (2026-10-10). Release build, RTX 5080 laptop. The screens are a 3840 × 2160 external and the 2560 × 1600 internal; DXGI lists both on the NVIDIA adapter. +
      +
    • The done-when run: 60.2 s of the 4K screen at 30 fps, with a square moving across it the whole time (-animate): +
        +
      • 3,602 images taken from the screen and 1,802 frames sent, all of them encoded with NVENC and none dropped.
        • +
      • 18 MB a minute.
        • +
      • FFmpeg reads it back as 60.07 s at 30 fps.
        • +
      • Per frame: 4.4 ms to read back and 13.4 ms to encode, on separate threads.
        • +
      + The frame decoded back matches the one captured, cursor included, by eye. Playing it in VLC and Media Player is still to check.
      • +
    • 60 fps: all 59.3 changes a second are kept. On a fixed tick grid, a 60 Hz screen beat against the 60 fps cap and lost about 7% of frames. Frames are now sent as soon as their turn is due, on a schedule that advances by exactly one frame each time. Caps of 30, 60 and 24 give 29.8, 59.3 and 23.8 frames a second.
      • +
    • The questions: +
        +
      • Both screens duplicate with DuplicateOutput1.
        • +
      • The plain DuplicateOutput works too in a per-monitor DPI-aware process. The documented failure doesn't happen here; DuplicateOutput1 stays the first choice.
        • +
      • With a CUDA context in the process, duplication still works.
        • +
      • WDA_EXCLUDEFROMCAPTURE is respected: the excluded square is 0% present in the frame and the control square 100%.
        • +
      • The two-GPU failure wasn't reproduced. Task Manager shows the Intel GPU, but no screen is attached to it now: the laptop is in discrete-GPU mode. Testing it means switching the laptop to Optimus (hybrid) mode in the NVIDIA Control Panel or the BIOS, which needs a restart.
        • +
      • +
    • Encoders, 4K at 30 fps, 8 s of the same motion. All of them keep up: +
      + + + + + + +
      EncoderEncode per frameMB a minute
      h264_nvenc13.7 ms23.4
      h264_qsv (the Intel GPU, though no screen is attached to it)12.3 ms16.7
      h264_mf16.8 ms24.0
      libx264 (veryfast, CRF 23)14.9 ms11.4
      h264_amfNot available: no AMD driver (amfrt64.dll), so it falls through to the next
      + Most of NVENC's 13.7 ms is FFmpeg copying the 33 MB frame before the GPU sees it. Handing it the frame without the copy is a later optimisation, not needed at 4K 60.
      • +
    • A crash mid-recording: killed about 7 s in, the fragmented MP4 still reads back with 6 s of video. At most the last keyframe interval (2 s) is lost.
      • +
    • The minimum size: 239 × 300 is refused, and 241 × 181 records at 240 × 180. The minimum is checked before rounding to even.
      • +
    +

    Changes from the plan:

    +
      +
    • The DPI check: once, a process that isn't DPI aware saw the 4K screen as 1920 × 1080, and the region would have been the wrong part of it. This couldn't be reproduced afterwards. dxgi_open now compares the duplication's size with the desktop's and refuses when they differ.
      • +
    • The probe's extra options: -animate, which moves a square for measuring throughput, and -check FILE, which reads a recording back.
      • +