MusaMahmood/mcap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: MusaMahmood:93274a2b04cb669c05f55613ea3b1deb0fceab9b
Branches Tags
MusaMahmood:master
MusaMahmood:gpu-bench
MusaMahmood:v0.1.0
..
compare: MusaMahmood:864ead49c99f6ac474a176e0d8faad5c470efb0d
Branches Tags
MusaMahmood:master
MusaMahmood:gpu-bench
MusaMahmood:v0.1.0

No commits in common. "93274a2b04cb669c05f55613ea3b1deb0fceab9b" and "864ead49c99f6ac474a176e0d8faad5c470efb0d" have entirely different histories.
93274a2b04 ... 864ead49c9

24 changed files with 10 additions and 3821 deletions
Show Stats Expand all files Collapse all files

63
README.md
View File

@ -9,9 +9,7 @@ and labels are tested on Windows with an NVIDIA GPU).
The `.mcap` files come from the **Import** window, which takes any folder of images (PNG, JPEG, HEIC,
AVIF, WebP, GIF, BMP, TIFF, camera RAW), videos (MP4, MOV, MKV, WebM, AVI, WMV, MPEG) and Snagit
`.snagx` captures, or from `snagit-converter`
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. On
Windows, **PrintScreen** captures a region of the screen straight into the library, as Snagit does;
see [Screen capture](#screen-capture).
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format.
## Getting started
@ -150,58 +148,6 @@ counted (click one to search for it), and the commonest words (click one to sear
of them are segmented, it also shows **What is in the images**: how many images hold each class
(person, sky, dog ...). Click one to search `has:` for it.
## Screen capture
On Windows, **PrintScreen** captures any part of the screen, from any app, even while mcap is hidden
in the tray:
1. **The screens freeze** the moment it is pressed: everything on every screen is copied and shown
darkened over them, so menus, tooltips and video hold still while you choose.
2. **Drag a rectangle** over what you want. It shows at full brightness, outlined. A magnifier beside
the cursor shows the pixels under it, with the cursor's position, or the selection's size while
dragging. The arrow keys move the cursor a pixel at a time, ten with Shift.
3. **Let go** to take it. A **click** without dragging, or **Enter**, takes the whole screen the
cursor is on. **Escape**, or a right click, cancels; a right click while dragging starts the
selection again.
What is taken is on the clipboard at once, ready to paste. It is also saved into the library, in
the folder **Screen Captures**, as a PNG in an `.mcap` with the window it was taken from (the window
under the middle of the selection): its title as the window title and its program as the app, so
searches find it as they find Snagit's. mcap then comes to the front showing it in the Capture
window, and OCR and labels read it once they are free (and segmentation, if the folder is set to
segment).
The key is the `capture_region` line of `[global_hotkeys]` in the settings file. Only one program can
have PrintScreen: if Snagit is running it has it, and so does Windows while Settings >
Accessibility > Keyboard > "Use the Print screen key to open screen capture" is on. When mcap cannot
register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S capture_region`.
### Recording the screen
**Ctrl+PrintScreen** records a region of one screen as a video:
1. **The screens freeze**, as for a still, with the outline and magnifier in red.
2. **Drag a region**, or click (or press Enter) for the whole screen under the cursor:
- The drag stays on the screen it began on: a recording is of one screen.
- A region smaller than **240 × 180** is outlined dashed, with the minimum in the label, and letting
go of it starts nothing; drag again.
- Escape or a right click cancels.
3. **Recording starts** at once, at up to 30 frames a second, cursor included. A frame is written
only when the screen changes, so a still screen costs almost nothing.
4. **Ctrl+PrintScreen again** stops it. The video goes into **Screen Captures** beside the stills,
as an H.264 MP4 in an `.mcap` with the window it was recorded from, and opens in the Capture window.
OCR and labels read a frame from it.
- **The screen** is taken with DXGI Desktop Duplication, or with Windows.Graphics.Capture where that
cannot run (a laptop with two GPUs, a rotated screen).
- **Encoding** uses NVENC where there is an NVIDIA GPU, else AMD's or Intel's encoder, Media
Foundation, or libx264.
- **Losing nothing:** a recording that stops by itself keeps what it has, and so does quitting while
recording.
- **PrintScreen while recording** takes the whole screen under the cursor at once, without freezing
the screens, which would show in the recording.
- **The key** is `record_region` in `[global_hotkeys]`.
## Importing
**Import** (above the grid) brings a folder of screenshots, photos and videos into the library:
@ -331,7 +277,6 @@ then in that order instead of by date.
| `-search QUERY` | Starts with QUERY in the search box. |
| `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. |
| `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. |
| `-record SECONDS` | Records the screen under the cursor that long (no overlay), puts it in the library, then exits: 0, or 1 if it failed. `-record-quit` quits while still recording instead, to check the recording is kept. |
| `-segment` | Segments the images not yet segmented in the library folders set to segment (see the Segmentation window), once the library has loaded, then exits: 0, or 1 if any image failed. |
| `-erase ocr`, `-erase labels`, `-erase segments`, `-erase all` | Erases that from every file in the library, then exits (for testing). |
| `-allow-multiple-instances` | Runs beside another copy instead of bringing that one forward. |
@ -383,12 +328,6 @@ converted copy.
HEIF and TIFF); previews; and videos' frames and what their containers say, with FFmpeg
(`video.jai`), and how many frames a GIF has. `bin\image-probe FILE...` (`jai first.jai -
image_probe`) shows what it makes of files.
- `modules/Screen_Record`: a region of one screen recorded to H.264 in an MP4, with DXGI Desktop
Duplication, or Windows.Graphics.Capture where duplication cannot run (two GPUs, a rotated screen),
and FFmpeg (NVENC, else AMF, QSV, Media Foundation or libx264), for screen recording
(`video-capture-plan.html`). `bin\record-probe` (`jai first.jai - record_probe`) records with it
and says how it went; `-list` shows the screens, `-capture auto|dxgi|wgc` picks the way, `-check
FILE` reads a recording back.
- `modules/Snagx`: `.snagx` to `.mcap`, copied from SnagitConverter.
- `modules/Capture_File`: reading and writing `.mcap` files, with a video's original left on disk.
- `modules/Segment`: panoptic segmentation with Mask2Former, run with onnxruntime (see its README and

14
assets/data/default_global_hotkeys.config
View File

@ -1,18 +1,8 @@
[global_hotkeys]
# Registered with the system, so they work from any application, even while the app is hidden in the tray.
# The format is that of [hotkeys], including "none" to turn an action off; keys are letters,
# digits, Space, F1-F24, Enter, Tab, Escape, the arrows, PageUp, PageDown, Home, End, Insert,
# Delete and PrintScreen. For example: Ctrl-Alt-Space bring_to_foreground
# digits, Space, F1-F24, Enter, Tab, Escape, the arrows, PageUp, PageDown, Home, End, Insert and
# Delete. For example: Ctrl-Alt-Space bring_to_foreground
# Changes apply as soon as the file is saved.
none bring_to_foreground
# Freezes every screen to drag a region over and capture it, into the clipboard and the library.
# PrintScreen is Snagit's key too: only one program can have it, so quit Snagit or choose another
# key, such as Ctrl-Shift-S. Windows takes it as well while Settings > Accessibility > Keyboard >
# "Use the Print screen key to open screen capture" is on.
PrintScreen capture_region
# Records a region of one screen (dragged in the same frozen view, at least 240 x 180, or a click
# for the whole screen) until pressed again; it goes into the library as a video.
Ctrl-PrintScreen record_region

2
assets/data/default_global_hotkeys_macos.config
View File

@ -5,5 +5,3 @@
# Changes apply as soon as the file is saved.
none bring_to_foreground
none capture_region
none record_region

2
check_targets.jai
View File

@ -29,7 +29,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
GIT_BRANCH :: "check";
GIT_HASH :: "-";
VERSION :: "0.1.5";
VERSION :: "0.1.0";
BUILD_DATE :: "-";
DEBUG :: true;
COMPILE_TESTS :: true;

4
first.jai
View File

@ -8,7 +8,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
VERSION :: "0.1.5";
VERSION :: "0.1.0";
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
// builds never write over the ones you run by hand.
build_directory := "bin";
@ -67,8 +67,6 @@ build_project :: () {
if array_find(args, "image_probe") array_add(*pending, start_tool_build("image-probe", "tools/image_probe.jai", optimized = build_release));
// jai first.jai - segment_probe: also bin/segment-probe, what modules/Segment makes of image files.
if array_find(args, "segment_probe") array_add(*pending, start_tool_build("segment-probe", "tools/segment_probe.jai", optimized = build_release));
// jai first.jai - record_probe: also bin/record-probe, a screen recorded with modules/Screen_Record (video-capture-plan.html, phase 1).
if array_find(args, "record_probe") array_add(*pending, start_tool_build("record-probe", "tools/record_probe.jai", optimized = build_release));
wait_for_builds(pending);

45
modules/Capture_File/README.md
View File

@ -6,8 +6,7 @@ then the original and rendered image, then any large chunks. There is no ZIP and
it; the images and videos keep their own compression.
A capture is a Snagit capture (from a `.snagx`), an imported image (a photo or screenshot in any image
format), an imported video (the mcap app's Import window), or a screen capture or recording (the mcap app's
PrintScreen and Ctrl+PrintScreen).
format), or an imported video (the mcap app's Import window).
The parts a library browser needs (header, text, OCR, front chunks and preview) come first, so it reads
the start of each file and never touches the images; a search, which needs no preview, reads less still.
@ -95,48 +94,6 @@ Orientation: 6 and, when the EXIF has them: Camer
So the parts a library reads from Snagit's `metadata.json` (date, title, app) work for photos too.
## Screen captures
A region of the screen taken with the mcap app's PrintScreen is stored as a PNG original with no
rendered image (kind 0), and its Capture Text holds two files:
```
[[file "metadata.json"]]
AppName: "Google Chrome" the program of the window under the region's middle:
its description, else its file's name; "" when unknown
CaptureDate: "2026-10-10 14:05:09" when it was taken, local time, as Snagit writes it
WindowName: "Assign Footprints - KiCad" that window's title
[[file "capture.json"]]
Region: "selection" or "screen": a click took the whole screen
Left: -1920 on the virtual desktop, whose origin is the primary
Top: 40 screen's top left, so it can be negative
Width: 700
Height: 300
AppPath: "C:/Program Files/Google/Chrome/Application/chrome.exe"
ContentHash: "5c1e9a0d77b2f413" 64-bit FNV-1a of the PNG's bytes
Format: "PNG"
```
## Screen recordings
A recording made with the mcap app's Ctrl+PrintScreen is a video: its original is the H.264 MP4 as
recorded, and its rendered image (kind 2) a frame from it as a PNG, as for an imported video. Its
metadata.json is a still's. Its capture.json has a still's keys, with Format "MP4", and these:
```
Duration: 12.48 seconds
FrameRate: 30 as the container says; frames are written only when the screen changes
VideoCodec: "h264"
PosterTime: 1.25 seconds into it of the rendered frame
Encoder: "h264_nvenc" FFmpeg's encoder: h264_nvenc, h264_amf, h264_qsv, h264_mf or libx264
Capture: "DXGI DuplicateOutput1 on NVIDIA GeForce RTX 5080 Laptop GPU"
or "Windows.Graphics.Capture on ..., border off"
DroppedFrames: 0 frames the encoder fell too far behind for
```
A reader tells a recording from a still by VideoCodec, as it tells an imported video by its import.json.
## Imported videos
A video imported from a folder (MP4, MOV, MKV, WebM, AVI, WMV, MPEG; read with FFmpeg, on Windows) is

115
modules/Screen_Record/capture_gpu.jai
View File

@ -1,115 +0,0 @@
// What both ways of taking the screen share (dxgi_source.jai, wgc_source.jai): a D3D11 device of
// their own on the adapter the screen is attached to, the region as last seen, copied out of each
// new image on the GPU, and the staging textures it is read back through.
Capture_Gpu :: struct {
adapter: *IDXGIAdapter1;
device: *ID3D11Device;
device_context: *ID3D11DeviceContext;
box: D3D11_BOX; // the region, in the screen's pixels
latest: *ID3D11Texture2D; // the region as last seen
staging: [RECORD_STAGING] *ID3D11Texture2D;
copy_ticks: s64; // spent taking the region out of images (QueryPerformanceCounter ticks)
}
RECORD_STAGING :: 3; // staging textures in turn: the one read is two frames behind the one copied
// What a source's next() found.
Capture_Next :: enum u8 { NONE; IMAGE; POINTER; IMAGE_AND_POINTER; LOST; FAILED; }
// The device on the screen's adapter, and the textures for a region (in the screen's pixels).
gpu_open :: (g: *Capture_Gpu, screen: Record_Screen, region: RECT) -> ok: bool, error: string {
factory: *IDXGIFactory1;
hr := CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory);
if FAILED(hr) return false, hresult_text("CreateDXGIFactory1", hr);
defer IUnknown_Release(factory);
hr = IDXGIFactory1_EnumAdapters1(factory, screen.adapter_index, *g.adapter);
if FAILED(hr) return false, hresult_text("the screen's adapter is gone", hr);
levels := D3D_FEATURE_LEVEL.[._11_1, ._11_0, ._10_1, ._10_0];
level: D3D_FEATURE_LEVEL;
hr = D3D11CreateDevice(g.adapter, .UNKNOWN, null, .BGRA_SUPPORT, levels.data, levels.count, D3D11_SDK_VERSION, *g.device, *level, *g.device_context);
if FAILED(hr) return false, hresult_text("D3D11CreateDevice on the screen's adapter", hr);
g.box = .{left = xx region.left, top = xx region.top, right = xx region.right, bottom = xx region.bottom, front = 0, back = 1};
desc: D3D11_TEXTURE2D_DESC;
desc.Width = cast(u32) (region.right - region.left);
desc.Height = cast(u32) (region.bottom - region.top);
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = .B8G8R8A8_UNORM;
desc.SampleDesc.Count = 1;
desc.Usage = .DEFAULT;
hr = ID3D11Device_CreateTexture2D(g.device, *desc, null, *g.latest);
if FAILED(hr) return false, hresult_text("a texture for the region", hr);
desc.Usage = .STAGING;
desc.CPUAccessFlags = .READ;
for * g.staging {
hr = ID3D11Device_CreateTexture2D(g.device, *desc, null, it);
if FAILED(hr) return false, hresult_text("a staging texture", hr);
}
return true, "";
}
// The region of a whole screen's image into g.latest.
gpu_copy_region :: (g: *Capture_Gpu, image: *ID3D11Texture2D) {
start := ticks_now();
ID3D11DeviceContext_CopySubresourceRegion(g.device_context, g.latest, 0, 0, 0, 0, image, 0, *g.box);
g.copy_ticks += ticks_now() - start;
}
gpu_close :: (g: *Capture_Gpu) {
for * g.staging com_release(it);
com_release(*g.latest);
com_release(*g.device_context);
com_release(*g.device);
com_release(*g.adapter);
}
// The calling thread made aware of each monitor's DPI (Windows 10 1703 on), so the desktop it is told
// of is in the screens' own pixels.
gpu_thread_dpi_aware :: () {
DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 :: cast(HANDLE) -4;
SetThreadDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2);
}
#scope_module
user32_dll :: #library,system "user32";
SetThreadDpiAwarenessContext :: (value: HANDLE) -> HANDLE #foreign user32_dll;
com_release :: (p: *$T) {
if p.* { IUnknown_Release(cast(*IUnknown) p.*); p.* = null; }
}
ticks_now :: () -> s64 {
t: LARGE_INTEGER;
QueryPerformanceCounter(*t);
return t.QuadPart;
}
hresult_is :: (hr: HRESULT, code: u32) -> bool {
return cast,no_check(u32) hr == code;
}
// "what failed: 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", temp.
hresult_text :: (what: string, hr: HRESULT) -> string {
code := cast,no_check(u32) hr;
name := "";
if code == {
case DXGI_ERROR_UNSUPPORTED; name = " (DXGI_ERROR_UNSUPPORTED)";
case DXGI_ERROR_NOT_CURRENTLY_AVAILABLE; name = " (DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)";
case DXGI_ERROR_ACCESS_LOST; name = " (DXGI_ERROR_ACCESS_LOST)";
case 0x80070005; name = " (E_ACCESSDENIED)";
case 0x80004002; name = " (E_NOINTERFACE)";
case 0x80070057; name = " (E_INVALIDARG)";
case 0x80040154; name = " (REGDB_E_CLASSNOTREG)";
case 0x80004001; name = " (E_NOTIMPL)";
}
return tprint("%: 0x%%", what, formatInt(code, base = 16, minimum_digits = 8), name);
}
wide_text :: (wide: *s16) -> string {
text, ok := wide_to_utf8(cast(*u16) wide,, temp);
return ifx ok then text else "";
}

48
modules/Screen_Record/cursor.jai
View File

@ -1,48 +0,0 @@
// The mouse cursor, drawn into frames read back from Desktop Duplication, which leaves it out of the
// image and hands over its shape and position separately. Three kinds of shape:
//
// COLOR BGRA, blended by its alpha.
// MASKED_COLOR BGRA, where alpha 0 replaces the pixel under it and 0xFF XORs it.
// MONOCHROME 1 bit a pixel, twice as tall: an AND mask over an XOR mask. AND 1 XOR 1 inverts the
// pixel under it, which is how the text I-beam shows on any background.
Cursor_Shape :: struct {
type: DXGI_OUTDUPL_POINTER_SHAPE_TYPE;
width, height, pitch: s32; // height as handed over: twice the cursor's for MONOCHROME
data: [..] u8;
}
// Draws shape with its top left at x, y (in the frame's pixels; it may reach past the edges) into
// BGRA pixels, rows packed.
cursor_draw :: (pixels: *u8, width: s32, height: s32, shape: *Cursor_Shape, x: s32, y: s32) {
if !shape.data.count return;
rows := ifx shape.type == .MONOCHROME then shape.height / 2 else shape.height;
for row: 0..rows - 1 {
dy := y + row;
if dy < 0 || dy >= height continue;
for column: 0..shape.width - 1 {
dx := x + column;
if dx < 0 || dx >= width continue;
d := pixels + (cast(s64) dy * width + dx) * 4;
if shape.type == {
case .COLOR;
s := shape.data.data + row * shape.pitch + column * 4;
a := cast(u32) s[3];
if a == 0 continue;
for c: 0..2 d[c] = cast(u8) ((cast(u32) s[c] * a + cast(u32) d[c] * (255 - a) + 127) / 255);
case .MASKED_COLOR;
s := shape.data.data + row * shape.pitch + column * 4;
if s[3] == 0 for c: 0..2 d[c] = s[c];
else for c: 0..2 d[c] ^= s[c];
case .MONOCHROME;
bit := cast(u8) (0x80 >> (column % 8));
and_mask := shape.data[row * shape.pitch + column / 8] & bit;
xor_mask := shape.data[(row + rows) * shape.pitch + column / 8] & bit;
if and_mask && !xor_mask continue; // transparent
if and_mask { for c: 0..2 d[c] = 255 - d[c]; continue; } // inverted
value: u8 = ifx xor_mask then cast(u8) 255 else 0; // white or black
for c: 0..2 d[c] = value;
}
}
}
}

164
modules/Screen_Record/dxgi_source.jai
View File

@ -1,164 +0,0 @@
// DXGI Desktop Duplication: the screens as the GPU composes them, a frame each time one changes,
// with the cursor apart (cursor.jai draws it in).
//
// It runs on a device made on the adapter the screen is attached to (capture_gpu.jai): duplicating
// with a device on another adapter fails. On a laptop with two GPUs, where the screen hangs off one
// and the process may run on the other, DuplicateOutput1 can fail with DXGI_ERROR_UNSUPPORTED all the
// same; that, too many duplications of the screen at once, and a rotated screen are what dxgi_open
// returns fall_back for, and Windows.Graphics.Capture (wgc_source.jai) takes over.
//
// DuplicateOutput1 (IDXGIOutput5) is used where there is one: the plain DuplicateOutput is documented
// to fail for a process aware of each monitor's DPI, as mcap is (it did not on an RTX 5080 laptop).
// legacy forces the plain one, for record-probe.
Dxgi_Source :: struct {
#as using gpu: Capture_Gpu;
output: *IDXGIOutput1;
output5: *IDXGIOutput5; // null before Windows 10 1703
duplication: *IDXGIOutputDuplication;
legacy: bool;
method: string; // which duplication call worked, for the log
pointer_position: POINT; // the cursor's top left, in the screen's pixels
pointer_visible: bool;
shape: Cursor_Shape;
}
// Every screen attached to the desktop, on every adapter, in DXGI's order. Temp.
record_list_screens :: () -> [] Record_Screen {
screens: [..] Record_Screen;
screens.allocator = temp;
factory: *IDXGIFactory1;
if FAILED(CreateDXGIFactory1(*uid(IDXGIFactory1_UUID), xx *factory)) return screens;
defer IUnknown_Release(factory);
adapter_index: u32;
while true {
adapter: *IDXGIAdapter1;
if FAILED(IDXGIFactory1_EnumAdapters1(factory, adapter_index, *adapter)) break;
defer IUnknown_Release(adapter);
adapter_desc: DXGI_ADAPTER_DESC1;
IDXGIAdapter1_GetDesc1(adapter, *adapter_desc);
adapter_name := wide_text(adapter_desc.Description.data);
output_index: u32;
while true {
output: *IDXGIOutput;
if FAILED(IDXGIAdapter_EnumOutputs(adapter, output_index, *output)) break;
desc: DXGI_OUTPUT_DESC;
IDXGIOutput_GetDesc(output, *desc);
IUnknown_Release(output);
if desc.AttachedToDesktop != .FALSE {
array_add(*screens, .{adapter_index = adapter_index, output_index = output_index, adapter = adapter_name,
name = wide_text(desc.DeviceName.data), desktop = desc.DesktopCoordinates,
rotation = desc.Rotation, monitor = desc.Monitor});
}
output_index += 1;
}
adapter_index += 1;
}
return screens;
}
// Duplicates the screen. region is in the screen's pixels and lies on it.
dxgi_open :: (s: *Dxgi_Source, screen: Record_Screen, region: RECT, legacy: bool) -> ok: bool, error: string, fall_back: bool {
s.legacy = legacy;
if screen.rotation != .IDENTITY && screen.rotation != .UNSPECIFIED return false, "the screen is rotated", true;
ok, error := gpu_open(s, screen, region);
if !ok return false, error, false;
output: *IDXGIOutput;
hr := IDXGIAdapter_EnumOutputs(s.adapter, screen.output_index, *output);
if FAILED(hr) return false, hresult_text("the screen is gone", hr), false;
defer IUnknown_Release(output);
hr = IUnknown_QueryInterface(output, *uid(IDXGIOutput1_UUID), xx *s.output);
if FAILED(hr) return false, hresult_text("the screen cannot be duplicated (no IDXGIOutput1)", hr), true;
IUnknown_QueryInterface(output, *uid(IDXGIOutput5_UUID), xx *s.output5); // optional
hr = dxgi_duplicate(s);
if FAILED(hr) {
// Unsupported: two GPUs, the device on the wrong one. Not currently available: too many duplications.
fall_back := hresult_is(hr, DXGI_ERROR_UNSUPPORTED) || hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE);
return false, hresult_text(tprint("Desktop Duplication (%)", s.method), hr), fall_back;
}
// A process not aware of each monitor's DPI is told the desktop in scaled pixels, while the frames
// come in the screen's own: the region would be the wrong part of the screen.
duplication_desc: DXGI_OUTDUPL_DESC;
IDXGIOutputDuplication_GetDesc(s.duplication, *duplication_desc);
desktop_width := screen.desktop.right - screen.desktop.left;
desktop_height := screen.desktop.bottom - screen.desktop.top;
if duplication_desc.ModeDesc.Width != cast(u32) desktop_width || duplication_desc.ModeDesc.Height != cast(u32) desktop_height {
return false, tprint("the screen is % x %, but this process sees it as % x %: it must be aware of each monitor's DPI",
duplication_desc.ModeDesc.Width, duplication_desc.ModeDesc.Height, desktop_width, desktop_height), false;
}
return true, "", false;
}
// Duplicates the screen, again after the duplication was lost.
dxgi_duplicate :: (s: *Dxgi_Source) -> HRESULT {
com_release(*s.duplication);
if s.output5 && !s.legacy {
s.method = "DuplicateOutput1";
formats := DXGI_FORMAT.[.B8G8R8A8_UNORM];
return IDXGIOutput5_DuplicateOutput1(s.output5, s.device, 0, formats.count, formats.data, *s.duplication);
}
s.method = "DuplicateOutput";
return IDXGIOutput1_DuplicateOutput(s.output, s.device, *s.duplication);
}
// Waits up to timeout_ms for the screen to change. A new image's region goes into s.latest; a moved
// or changed cursor updates s.pointer_* and s.shape. On LOST the duplication is gone: dxgi_duplicate.
dxgi_next :: (s: *Dxgi_Source, timeout_ms: u32) -> Capture_Next, error: string {
info: DXGI_OUTDUPL_FRAME_INFO;
resource: *IDXGIResource;
hr := IDXGIOutputDuplication_AcquireNextFrame(s.duplication, timeout_ms, *info, *resource);
if hresult_is(hr, DXGI_ERROR_WAIT_TIMEOUT) return .NONE, "";
if hresult_is(hr, DXGI_ERROR_ACCESS_LOST) return .LOST, "";
if FAILED(hr) return .FAILED, hresult_text("AcquireNextFrame", hr);
defer IDXGIOutputDuplication_ReleaseFrame(s.duplication); // held, it stalls the desktop
defer IUnknown_Release(resource);
image := info.LastPresentTime.QuadPart != 0;
if image {
texture: *ID3D11Texture2D;
hr = IUnknown_QueryInterface(resource, *uid(ID3D11Texture2D_UUID), xx *texture);
if FAILED(hr) return .FAILED, hresult_text("the frame is not a texture", hr);
gpu_copy_region(s, texture);
IUnknown_Release(texture);
}
pointer := info.LastMouseUpdateTime.QuadPart != 0;
if pointer {
s.pointer_position = info.PointerPosition.Position;
s.pointer_visible = info.PointerPosition.Visible != .FALSE;
if info.PointerShapeBufferSize {
array_resize(*s.shape.data, info.PointerShapeBufferSize, initialize = false);
needed: u32;
shape_info: DXGI_OUTDUPL_POINTER_SHAPE_INFO;
hr = IDXGIOutputDuplication_GetFramePointerShape(s.duplication, cast(u32) s.shape.data.count, s.shape.data.data, *needed, *shape_info);
if SUCCEEDED(hr) {
s.shape.type = cast(DXGI_OUTDUPL_POINTER_SHAPE_TYPE) shape_info.Type;
s.shape.width = cast(s32) shape_info.Width;
s.shape.height = cast(s32) shape_info.Height;
s.shape.pitch = cast(s32) shape_info.Pitch;
} else {
array_reset_keeping_memory(*s.shape.data);
}
}
}
if image && pointer return .IMAGE_AND_POINTER, "";
if image return .IMAGE, "";
if pointer return .POINTER, "";
return .NONE, "";
}
dxgi_close :: (s: *Dxgi_Source) {
com_release(*s.duplication);
com_release(*s.output5);
com_release(*s.output);
gpu_close(s);
array_free(s.shape.data);
s.shape = .{};
}

213
modules/Screen_Record/encoder.jai
View File

@ -1,213 +0,0 @@
// The encoder thread's half: frames of BGRA pixels to H.264 in a fragmented MP4, with FFmpeg
// (modules/ffmpeg). The first encoder in RECORD_ENCODERS that opens at the size wanted is used:
// NVENC, AMD's AMF, Intel's QSV, Windows' Media Foundation, then libx264 on the CPU. NVENC takes the
// BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with
// swscale (BT.709, limited range, as tagged).
//
// The file is written fragmented (a fragment each keyframe, every 2 s), so one cut short by a crash
// still plays up to its last fragment.
RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"];
Encoder :: struct {
format: *AVFormatContext;
codec: *AVCodecContext;
stream: *AVStream;
frame: *AVFrame;
packet: *AVPacket;
sws: *SwsContext;
name: string; // the encoder used
input: AVPixelFormat; // what it is given: BGR0 as captured, or converted to NV12 or YUV420P
width, height: s32;
last_pts: s64 = -1;
header_written: bool;
bytes: s64; // of packets written
}
// Opens the file and the first encoder that will take width x height (wanted alone, when given).
// Failures to open an encoder are listed in tried (temp), for the log.
encoder_open :: (e: *Encoder, path: string, width: s32, height: s32, fps: s32, wanted: string) -> ok: bool, error: string, tried: string {
av_log_set_level(AV_LOG_ERROR);
e.width, e.height = width, height;
result := avformat_alloc_output_context2(*e.format, null, "mp4", temp_c_string(path));
if result < 0 || !e.format return false, av_error("FFmpeg cannot make an MP4", result), "";
tried: [..] string;
tried.allocator = temp;
names: [] string = RECORD_ENCODERS;
if wanted names = .[wanted];
for names {
ok, why := encoder_try(e, it, width, height, fps);
if ok break;
array_add(*tried, tprint("%: %", it, why));
}
tried_text := join(..tried, "; ",, temp);
if !e.codec {
encoder_free(e);
return false, tprint("no H.264 encoder opened (%)", tried_text), tried_text;
}
e.stream = avformat_new_stream(e.format, null);
if !e.stream { encoder_free(e); return false, "FFmpeg cannot add a stream", tried_text; }
avcodec_parameters_from_context(e.stream.codecpar, e.codec);
e.stream.time_base = e.codec.time_base;
result = avio_open(*e.format.pb, temp_c_string(path), AVIO_FLAG_WRITE);
if result < 0 { encoder_free(e); return false, av_error(tprint("FFmpeg cannot write %", path), result), tried_text; }
options: *AVDictionary;
av_dict_set(*options, "movflags", "frag_keyframe+empty_moov+default_base_moof", 0);
result = avformat_write_header(e.format, *options);
av_dict_free(*options);
if result < 0 { encoder_free(e); return false, av_error("FFmpeg cannot start the MP4", result), tried_text; }
e.header_written = true;
e.packet = av_packet_alloc();
e.frame = av_frame_alloc();
if !e.packet || !e.frame { encoder_free(e); return false, "FFmpeg is out of memory", tried_text; }
if e.input != .BGR0 {
e.sws = sws_getContext(width, height, .BGR0, width, height, e.input, SWS_POINT, null, null, null);
if !e.sws { encoder_free(e); return false, "swscale cannot convert to the encoder's format", tried_text; }
coefficients := cast(*[4] s32) sws_getCoefficients(SWS_CS_ITU709);
sws_setColorspaceDetails(e.sws, coefficients, 1, coefficients, 0, 0, 1 << 16, 1 << 16);
}
return true, "", tried_text;
}
// Encodes one frame; pts in RECORD_TIME_BASE ticks, made later than the last one if it is not.
encoder_write :: (e: *Encoder, pixels: *u8, pts: s64) -> ok: bool, error: string {
pts_out := max(pts, e.last_pts + 1);
e.last_pts = pts_out;
frame := e.frame;
av_frame_unref(frame);
frame.format = cast(s32) e.input;
frame.width = e.width;
frame.height = e.height;
if e.input == .BGR0 {
// Not reference-counted: avcodec_send_frame copies it, so the buffer goes back to the pool at once.
frame.data[0] = pixels;
frame.linesize[0] = e.width * 4;
} else {
result := av_frame_get_buffer(frame, 0);
if result < 0 return false, av_error("FFmpeg cannot make a frame", result);
source := pixels;
stride := e.width * 4;
sws_scale(e.sws, *source, *stride, 0, e.height, frame.data.data, frame.linesize.data);
}
frame.pts = pts_out;
result := avcodec_send_frame(e.codec, frame);
av_frame_unref(frame);
if result < 0 return false, av_error(tprint("% cannot encode", e.name), result);
drained, error := encoder_drain(e);
return drained, error;
}
// Flushes the encoder, finishes the file and closes it.
encoder_close :: (e: *Encoder) -> ok: bool, error: string {
ok, error := true, "";
if e.codec && e.header_written {
avcodec_send_frame(e.codec, null);
ok, error = encoder_drain(e);
result := av_write_trailer(e.format);
if ok && result < 0 ok, error = false, av_error("FFmpeg cannot finish the MP4", result);
}
encoder_free(e);
return ok, error;
}
#scope_file
RECORD_TIME_BASE :: 90000; // pts ticks a second: MPEG's
encoder_try :: (e: *Encoder, name: string, width: s32, height: s32, fps: s32) -> ok: bool, error: string {
codec := avcodec_find_encoder_by_name(temp_c_string(name));
if !codec return false, "not in this FFmpeg";
input := encoder_input_format(codec);
if input == .NONE return false, "takes no format this can give it";
c := avcodec_alloc_context3(codec);
if !c return false, "out of memory";
c.width = width;
c.height = height;
c.time_base = .{1, RECORD_TIME_BASE};
c.framerate = .{fps, 1};
c.pix_fmt = input;
c.gop_size = fps * 2;
c.max_b_frames = 0;
c.sample_aspect_ratio = .{1, 1};
if input != .BGR0 { // swscale's conversion; NVENC picks and tags its own for BGRA
c.colorspace = .BT709;
c.color_primaries = .BT709;
c.color_trc = .BT709;
c.color_range = .MPEG;
}
if e.format.oformat.flags & AVFMT_GLOBALHEADER c.flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
options: *AVDictionary;
set :: (key: string, value: string) #expand { av_dict_set(*`options, temp_c_string(key), temp_c_string(value), 0); }
if name == {
case "h264_nvenc"; set("preset", "p4"); set("rc", "vbr"); set("cq", "23"); c.bit_rate = 0;
case "h264_amf"; set("rc", "cqp"); set("qp_i", "22"); set("qp_p", "24"); set("quality", "balanced");
case "h264_qsv"; set("global_quality", "23"); set("preset", "medium");
case "h264_mf"; set("rate_control", "quality"); set("quality", "70"); set("scenario", "display_remoting");
case "libx264"; set("preset", "veryfast"); set("crf", "23");
}
result := avcodec_open2(c, codec, *options);
av_dict_free(*options);
if result < 0 {
avcodec_free_context(*c);
return false, av_error("cannot open", result);
}
e.codec = c;
e.input = input;
e.name = name;
return true, "";
}
// BGRA as captured when the encoder takes it, else what swscale converts it to.
encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat {
if !codec.pix_fmts return .NONE;
for wanted: AVPixelFormat.[.BGR0, .BGRA, .NV12, .YUV420P] {
format := codec.pix_fmts;
while format.* != .NONE {
if format.* == wanted return ifx wanted == .BGRA then .BGR0 else wanted;
format += 1;
}
}
return .NONE;
}
encoder_drain :: (e: *Encoder) -> ok: bool, error: string {
AVERROR_EAGAIN :: -11;
while true {
result := avcodec_receive_packet(e.codec, e.packet);
if result == AVERROR_EAGAIN || result == cast,no_check(s32) AVERROR_EOF return true, "";
if result < 0 return false, av_error(tprint("% failed", e.name), result);
av_packet_rescale_ts(e.packet, e.codec.time_base, e.stream.time_base);
e.packet.stream_index = e.stream.index;
e.bytes += e.packet.size;
result = av_interleaved_write_frame(e.format, e.packet); // takes the packet's data
if result < 0 return false, av_error("FFmpeg cannot write to the MP4", result);
}
return true, "";
}
encoder_free :: (e: *Encoder) {
if e.sws sws_freeContext(e.sws);
if e.frame av_frame_free(*e.frame);
if e.packet av_packet_free(*e.packet);
if e.codec avcodec_free_context(*e.codec);
if e.format {
if e.format.pb avio_closep(*e.format.pb);
avformat_free_context(e.format);
}
e.sws, e.format, e.stream = null, null, null;
}
// What FFmpeg says an error code is, after what failed (temp).
av_error :: (what: string, code: s32) -> string {
buffer: [256] u8;
av_strerror(code, buffer.data, buffer.count);
return tprint("% (%)", what, to_string(buffer.data));
}

511
modules/Screen_Record/module.jai
View File

@ -1,511 +0,0 @@
// Screen_Record: a region of one screen recorded to H.264 in an MP4, for the mcap app's screen
// recording and tools/record_probe.jai (video-capture-plan.html). Windows only.
//
// Two threads. The capture thread takes the screen with DXGI Desktop Duplication (dxgi_source.jai),
// or with Windows.Graphics.Capture (wgc_source.jai) where duplication cannot run, each on a D3D11
// device of its own; copies the region on the GPU; reads it back through a ring of staging textures,
// so Map never waits on the GPU (capture_gpu.jai); and, for duplication, draws the cursor in
// (cursor.jai). The encoder thread hands the frames to FFmpeg (encoder.jai). Between them is a pool of
// RECORD_BUFFERS frames; when the encoder falls RECORD_QUEUE frames behind, the oldest waiting is
// dropped and counted.
//
// A frame is taken at most fps times a second, and only when the screen or the cursor changed, with
// one a second while nothing does, so players can seek. Each carries the time it was taken.
//
// recorder: Recorder;
// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0});
// ...
// ok, error = record_stop(*recorder);
//
// With capture .AUTO (the default), Desktop Duplication is tried first and Windows.Graphics.Capture
// takes over when it cannot run: on a laptop with two GPUs, on a rotated screen, or with too many
// duplications of the screen running. fall_back is only returned for .DXGI, which does not fall back.
RECORD_MIN_WIDTH :: 240; // a smaller region is refused
RECORD_MIN_HEIGHT :: 180;
RECORD_QUEUE :: 8; // frames waiting for the encoder at most
RECORD_BUFFERS :: RECORD_QUEUE + 2; // and the one being filled, and the one being encoded
// Whether a region this size may be recorded: at least RECORD_MIN_WIDTH x RECORD_MIN_HEIGHT. It is
// rounded down to even sizes after, which keeps it at least that, as both are even.
record_size_allowed :: (width: s64, height: s64) -> bool {
return width >= RECORD_MIN_WIDTH && height >= RECORD_MIN_HEIGHT;
}
Record_Screen :: struct {
adapter_index, output_index: u32;
adapter: string; // "NVIDIA GeForce RTX 5080 Laptop GPU"
name: string; // "\\.\DISPLAY1"
desktop: RECT; // on the virtual desktop
rotation: DXGI_MODE_ROTATION;
monitor: HMONITOR;
}
Record_Capture :: enum u8 {
AUTO; // Desktop Duplication, else Windows.Graphics.Capture
DXGI; // Desktop Duplication only
WGC; // Windows.Graphics.Capture only
}
// For record-probe: paths a machine may not take by itself.
Record_Testing :: enum_flags u8 {
DXGI_UNSUPPORTED; // Desktop Duplication fails as on a laptop with two GPUs (DXGI_ERROR_UNSUPPORTED)
WGC_POLL; // Windows.Graphics.Capture polled, as when it will not take the FrameArrived handler
KEEP_BORDER; // Windows.Graphics.Capture's yellow border left on
}
Record_Options :: struct {
path: string; // the .mp4 written
screen: s32 = -1; // into record_list_screens(); -1: the one holding region
region: RECT; // on the virtual desktop; empty: the whole screen
fps: s32 = 30;
cursor := true;
encoder: string; // "" tries RECORD_ENCODERS in turn
capture: Record_Capture;
legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe
testing: Record_Testing;
}
Record_Stats :: struct {
source: string; // "DXGI DuplicateOutput1 on <adapter>", "Windows.Graphics.Capture on <adapter>, border off"
fell_back_from: string; // why Desktop Duplication could not run, when it fell back
encoder: string;
width, height: s32;
captured: s64; // new images of the screen
pointer_moves: s64; // cursor changes with no new image (Desktop Duplication)
emitted: s64; // frames sent to be encoded
repeats: s64; // of them, the once-a-second repeat of an unchanged screen
written: s64; // encoded
dropped: s64; // the encoder was too far behind
lost: s64; // times the duplication was lost and made again
copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks
bytes: s64; // encoded
seconds: float64; // from start to stop
}
Recorder :: struct {
options: Record_Options;
screen: Record_Screen;
local: RECT; // the region, in the screen's pixels
dxgi: Dxgi_Source; // the capture thread's: one of these is open
wgc: Wgc_Source;
using_wgc: bool;
encoder: Encoder;
width, height: s32;
frequency: s64; // QueryPerformanceCounter's
start_ticks: s64;
capture_thread, encoder_thread: Thread;
capture_started, encoder_started: bool;
running: bool;
stopping: bool; // set by record_stop; the capture thread ends, then the encoder
capture_ended: bool;
opened: Semaphore; // the capture thread has opened its source, or failed to
open_ok, open_fall_back: bool;
open_error: string;
mutex: Mutex;
ready: Semaphore; // signalled for each frame queued, and once when capture ends
frames: [RECORD_BUFFERS] Record_Frame;
free: [RECORD_BUFFERS] s32;
free_count: s32;
queue: [RECORD_BUFFERS] s32; // oldest first
queue_count: s32;
last_frame: [] u8; // the last frame queued, BGRA, for record_last_frame
stats: Record_Stats;
error: string; // why it stopped by itself; heap
}
Record_Frame :: struct {
pixels: [] u8; // BGRA, rows packed
ticks: s64;
}
// Opens the screen, the encoder and the file, and starts recording.
record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool {
r.* = .{};
r.options = options;
r.options.path = copy_string(options.path);
r.options.encoder = copy_string(options.encoder);
if r.options.fps <= 0 r.options.fps = 30;
screens := record_list_screens();
if !screens.count return false, "no screen to record", false;
screen_index := options.screen;
region := options.region;
has_region := region.right > region.left && region.bottom > region.top;
if screen_index < 0 {
if !has_region screen_index = 0;
else for screens if rect_inside(region, it.desktop) { screen_index = xx it_index; break; }
if screen_index < 0 return false, "the region is not on one screen", false;
}
if screen_index >= screens.count return false, tprint("there is no screen %", screen_index), false;
r.screen = screens[screen_index];
r.screen.adapter = copy_string(r.screen.adapter);
r.screen.name = copy_string(r.screen.name);
if !has_region region = r.screen.desktop;
if !rect_inside(region, r.screen.desktop) return false, "the region is not on that screen", false;
// The minimum first, then even both ways, in the screen's pixels: H.264's 4:2:0 colour needs it.
// RECORD_MIN_WIDTH and _HEIGHT are even, so what passes stays at least that size.
#assert RECORD_MIN_WIDTH % 2 == 0 && RECORD_MIN_HEIGHT % 2 == 0;
if !record_size_allowed(region.right - region.left, region.bottom - region.top)
return false, tprint("% x % is smaller than % x %", region.right - region.left, region.bottom - region.top, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false;
d := r.screen.desktop;
r.local = .{region.left - d.left, region.top - d.top, region.right - d.left, region.bottom - d.top};
r.local.right = r.local.left + ((r.local.right - r.local.left) & cast(s32) ~1);
r.local.bottom = r.local.top + ((r.local.bottom - r.local.top) & cast(s32) ~1);
r.width = r.local.right - r.local.left;
r.height = r.local.bottom - r.local.top;
r.stats.width, r.stats.height = r.width, r.height;
init(*r.mutex, "record");
init(*r.ready);
init(*r.opened);
for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false);
for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it;
r.free_count = RECORD_BUFFERS;
r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8);
frequency: LARGE_INTEGER;
QueryPerformanceFrequency(*frequency);
r.frequency = frequency.QuadPart;
r.start_ticks = ticks_now();
// The source is opened on the capture thread, which uses it: WinRT's objects belong to the
// apartment they are made in.
thread_init(*r.capture_thread, record_capture_proc, name = "record capture");
r.capture_started = true;
thread_start(*r.capture_thread, r);
wait_for(*r.opened);
if !r.open_ok {
error, fall_back := copy_string(r.open_error,, temp), r.open_fall_back;
record_free(r);
return false, error, fall_back;
}
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder);
if tried log("[record] Encoders that did not open: %", tried);
if !ok {
error = copy_string(error,, temp);
record_free(r);
return false, error, false;
}
r.stats.encoder = copy_string(r.encoder.name);
thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder");
r.encoder_started = true;
r.running = true;
thread_start(*r.encoder_thread, r);
return true, "", false;
}
// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an
// error, or the file could not be finished; what was recorded up to then is in the file all the same.
record_stop :: (r: *Recorder) -> ok: bool, error: string {
if !r.running return false, "not recording";
atomic_swap(*r.stopping, true);
thread_is_done(*r.capture_thread, -1);
thread_is_done(*r.encoder_thread, -1);
r.running = false;
closed, close_error := encoder_close(*r.encoder);
r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency;
r.stats.bytes = r.encoder.bytes;
error := ifx r.error then copy_string(r.error,, temp) else "";
record_free(r);
if error return false, error;
if !closed return false, close_error;
return true, "";
}
// As they are now; strings are the recorder's.
record_stats :: (r: *Recorder) -> Record_Stats {
if !r.running return r.stats;
lock(*r.mutex);
defer unlock(*r.mutex);
return r.stats;
}
// The last frame queued, BGRA (temp), cursor and all.
record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 {
if r.running lock(*r.mutex);
pixels := array_copy(r.last_frame,, temp);
if r.running unlock(*r.mutex);
return pixels, r.width, r.height;
}
// Whether recording stopped by itself (the screen went away, the encoder failed); why, then.
record_failed :: (r: *Recorder) -> bool, string {
lock(*r.mutex);
defer unlock(*r.mutex);
return r.error.count > 0, r.error;
}
#scope_file
Staged :: struct { pending: bool; ticks: s64; pointer: POINT; pointer_visible: bool; }
// Stops the capture thread if it runs, and frees what record_start made. The stats' strings are kept.
record_free :: (r: *Recorder) {
atomic_swap(*r.stopping, true);
if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; }
if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; }
if r.encoder.format encoder_close(*r.encoder);
for * r.frames array_free(it.pixels);
array_free(r.last_frame);
r.last_frame = .[];
destroy(*r.opened);
destroy(*r.ready);
destroy(*r.mutex);
free(r.open_error);
free(r.error);
r.open_error, r.error = "", "";
}
// Desktop Duplication, unless asked for Windows.Graphics.Capture, which takes over when duplication
// cannot run here and the options allow.
record_open_source :: (r: *Recorder) -> bool {
if r.options.capture != .WGC {
ok, error, fall_back := dxgi_open(*r.dxgi, r.screen, r.local, r.options.legacy_duplicate);
if ok && r.options.testing & .DXGI_UNSUPPORTED {
ok, error, fall_back = false, "Desktop Duplication (testing): 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", true;
}
if ok {
r.stats.source = sprint("DXGI % on %", r.dxgi.method, r.screen.adapter);
return true;
}
dxgi_close(*r.dxgi);
if r.options.capture == .DXGI || !fall_back {
r.open_error, r.open_fall_back = copy_string(error), fall_back;
return false;
}
log("[record] Desktop Duplication cannot run here (%); Windows.Graphics.Capture instead.", error);
r.stats.fell_back_from = copy_string(error);
}
ok, error := wgc_open(*r.wgc, r.screen, r.local, r.options.cursor, r.options.testing);
if !ok {
wgc_close(*r.wgc);
r.open_error = copy_string(error);
return false;
}
r.using_wgc = true;
r.stats.source = sprint("Windows.Graphics.Capture on %, %%", r.screen.adapter, r.wgc.border, ifx r.wgc.polling then ", polled" else "");
return true;
}
record_capture_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
// Aware of each monitor's DPI, whatever the process is: Desktop Duplication refuses a thread that is
// not (DXGI_ERROR_UNSUPPORTED), and the mcap app's ImGui makes only its main thread aware.
gpu_thread_dpi_aware();
wgc_thread_start();
defer wgc_thread_end();
r.open_ok = record_open_source(r);
signal(*r.opened);
if !r.open_ok return 0;
gpu: *Capture_Gpu = ifx r.using_wgc then *r.wgc.gpu else *r.dxgi.gpu;
defer if r.using_wgc wgc_close(*r.wgc); else dxgi_close(*r.dxgi);
// A change is sent as soon as it comes once its turn is due: turns are a frame's time apart on a
// schedule that moves on a frame at a time, so the rate never passes fps, and a tenth of a frame
// early counts as on time, so a 60 Hz screen's frames, which jitter, all make a 60 fps cap.
period := r.frequency / r.options.fps;
early := period / 10;
turn := ticks_now();
last_emit: s64 = -r.frequency;
have_image, changed := false, false;
staged: [RECORD_STAGING] Staged;
emits: s64;
while !atomic_read(*r.stopping) {
now := ticks_now();
due := ifx changed then turn - early else last_emit + r.frequency; // the change, or the repeat
wait := cast(u32) clamp((due - now) * 1000 / r.frequency, 0, 100);
next: Capture_Next;
error: string;
if r.using_wgc next, error = wgc_next(*r.wgc, wait);
else next, error = dxgi_next(*r.dxgi, wait);
if next == {
case .FAILED;
record_fail(r, error);
break;
case .LOST;
// A mode change, the secure desktop, a game going fullscreen: duplicated again, and on.
lock(*r.mutex); r.stats.lost += 1; unlock(*r.mutex);
while !atomic_read(*r.stopping) {
hr := dxgi_duplicate(*r.dxgi);
if SUCCEEDED(hr) break;
if !hresult_is(hr, DXGI_ERROR_ACCESS_LOST) && !hresult_is(hr, 0x80070005) && !hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE) {
record_fail(r, hresult_text("the screen could not be duplicated again", hr));
break;
}
sleep_milliseconds(250);
}
if atomic_read(*r.stopping) || r.error break;
continue;
case .IMAGE; #through;
case .IMAGE_AND_POINTER;
have_image, changed = true, true;
lock(*r.mutex);
r.stats.captured += 1;
r.stats.copy_ticks = gpu.copy_ticks;
unlock(*r.mutex);
case .POINTER;
if r.options.cursor && have_image {
changed = true;
lock(*r.mutex); r.stats.pointer_moves += 1; unlock(*r.mutex);
}
}
now = ticks_now();
if !have_image continue;
repeat := !changed && now - last_emit >= r.frequency;
if !(changed && now >= turn - early) && !repeat continue;
turn = max(turn + period, now - period); // after a quiet spell, no burst to catch up
// Copied to staging now; read back two frames on, when the GPU is done with it.
slot := emits % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
ID3D11DeviceContext_CopyResource(gpu.device_context, gpu.staging[slot], gpu.latest);
staged[slot] = .{pending = true, ticks = now};
if !r.using_wgc staged[slot].pointer, staged[slot].pointer_visible = r.dxgi.pointer_position, r.dxgi.pointer_visible;
emits += 1;
last_emit = now;
changed = false;
lock(*r.mutex);
r.stats.emitted += 1;
if repeat r.stats.repeats += 1;
unlock(*r.mutex);
}
// What is still staged, oldest first.
for 0..RECORD_STAGING - 1 {
slot := (emits + it) % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
}
atomic_swap(*r.capture_ended, true);
signal(*r.ready);
return 0;
}
// Copies a staging texture into a free frame, draws the cursor (Desktop Duplication leaves it out)
// and queues it.
record_read_back :: (r: *Recorder, gpu: *Capture_Gpu, slot: s64, staged: Staged) {
t := ticks_now();
lock(*r.mutex);
if !r.free_count { // every frame is waiting: drop the oldest
r.free[r.free_count] = r.queue[0];
r.free_count += 1;
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
r.stats.dropped += 1;
}
r.free_count -= 1;
index := r.free[r.free_count];
unlock(*r.mutex);
frame := *r.frames[index];
mapped: D3D11_MAPPED_SUBRESOURCE;
hr := ID3D11DeviceContext_Map(gpu.device_context, gpu.staging[slot], 0, .READ, 0, *mapped);
if FAILED(hr) {
lock(*r.mutex); r.free[r.free_count] = index; r.free_count += 1; unlock(*r.mutex);
record_fail(r, hresult_text("reading the frame back", hr));
return;
}
row := cast(s64) r.width * 4;
for y: 0..r.height - 1 memcpy(frame.pixels.data + y * row, cast(*u8) mapped.pData + cast(s64) y * mapped.RowPitch, row);
ID3D11DeviceContext_Unmap(gpu.device_context, gpu.staging[slot], 0);
if r.options.cursor && staged.pointer_visible {
cursor_draw(frame.pixels.data, r.width, r.height, *r.dxgi.shape,
staged.pointer.x - cast(s32) gpu.box.left, staged.pointer.y - cast(s32) gpu.box.top);
}
frame.ticks = staged.ticks;
lock(*r.mutex);
r.queue[r.queue_count] = index;
r.queue_count += 1;
memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count);
r.stats.readback_ticks += ticks_now() - t;
unlock(*r.mutex);
signal(*r.ready);
}
record_encoder_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
first: s64 = -1;
while true {
wait_for(*r.ready, 100);
lock(*r.mutex);
index: s32 = -1;
if r.queue_count {
index = r.queue[0];
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
}
unlock(*r.mutex);
if index < 0 {
if atomic_read(*r.capture_ended) break;
continue;
}
frame := *r.frames[index];
if first < 0 first = frame.ticks;
pts := (frame.ticks - first) * 90000 / r.frequency;
t := ticks_now();
ok, error := encoder_write(*r.encoder, frame.pixels.data, pts);
lock(*r.mutex);
r.free[r.free_count] = index;
r.free_count += 1;
r.stats.encode_ticks += ticks_now() - t;
if ok r.stats.written += 1;
r.stats.bytes = r.encoder.bytes;
unlock(*r.mutex);
if !ok {
record_fail(r, error);
// Frames still come until the capture thread sees it; they are let go.
while !atomic_read(*r.capture_ended) {
wait_for(*r.ready, 100);
lock(*r.mutex);
for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; }
r.queue_count = 0;
unlock(*r.mutex);
}
break;
}
}
return 0;
}
// Ends the recording from one of its threads; the first reason is kept.
record_fail :: (r: *Recorder, error: string) {
log_error("[record] %", error);
lock(*r.mutex);
if !r.error r.error = copy_string(error);
unlock(*r.mutex);
atomic_swap(*r.stopping, true);
}
rect_inside :: (inner: RECT, outer: RECT) -> bool {
return inner.left >= outer.left && inner.top >= outer.top && inner.right <= outer.right && inner.bottom <= outer.bottom;
}
#scope_export
#load "capture_gpu.jai";
#load "dxgi_source.jai";
#load "wgc_source.jai";
#load "encoder.jai";
#load "cursor.jai";
#import "Basic";
#import "String";
#import "Atomics";
#import "Thread";
#import "Windows";
#import "Windows_Utf8";
#import "dxgi";
#import "d3d11";
#import "ffmpeg";

406
modules/Screen_Record/wgc_source.jai
View File

@ -1,406 +0,0 @@
// Windows.Graphics.Capture: the screen as Windows' own capture API hands it over, for when Desktop
// Duplication cannot run (dxgi_source.jai). It works across GPUs, hands over frames the right way up
// on a rotated screen, and draws the cursor in itself.
//
// It is WinRT, bound here by hand: the few interfaces it needs, as vtables after IInspectable's six
// methods. Everything is made and used on the capture thread, which is in the multithreaded apartment
// (RoInitialize), with a free-threaded frame pool, so no dispatcher queue or message loop is needed.
//
// A frame's arrival sets an event, through a small handler object (Frame_Handler); if Windows will not
// take the handler, the pool is polled every millisecond instead. Windows 11 lets the yellow border it
// draws around a screen being captured be turned off (IsBorderRequired); Windows 10 always draws it.
Wgc_Source :: struct {
#as using gpu: Capture_Gpu;
screen_size: Size_Int32; // the frames' size, as the pool was made for
item: *Capture_Item;
winrt_device: *IInspectable; // IDirect3DDevice, over gpu.device
pool: *Frame_Pool;
session: *Capture_Session;
handler: Frame_Handler;
handler_token: s64;
polling: bool; // Windows would not take the handler
border: string; // what came of turning the border off, for the log
}
// Starts capturing the screen. region is in the screen's pixels and lies on it.
wgc_open :: (s: *Wgc_Source, screen: Record_Screen, region: RECT, cursor: bool, testing: Record_Testing = 0) -> ok: bool, error: string {
ok, error := gpu_open(s, screen, region);
if !ok return false, error;
session_statics: *Session_Statics;
hr := activation_factory("Windows.Graphics.Capture.GraphicsCaptureSession", IID_SESSION_STATICS, xx *session_statics);
if FAILED(hr) return false, hresult_text("Windows.Graphics.Capture is not on this system", hr);
supported: u8;
session_statics.vtable.IsSupported(session_statics, *supported);
com_release(*session_statics);
if !supported return false, "Windows.Graphics.Capture is not supported here";
interop: *Capture_Item_Interop;
hr = activation_factory("Windows.Graphics.Capture.GraphicsCaptureItem", IID_ITEM_INTEROP, xx *interop);
if FAILED(hr) return false, hresult_text("IGraphicsCaptureItemInterop", hr);
hr = interop.vtable.CreateForMonitor(interop, screen.monitor, *IID_ITEM, xx *s.item);
com_release(*interop);
if FAILED(hr) return false, hresult_text("a capture item for the screen", hr);
s.item.vtable.get_Size(s.item, *s.screen_size);
desktop_width := screen.desktop.right - screen.desktop.left;
desktop_height := screen.desktop.bottom - screen.desktop.top;
if s.screen_size.width != desktop_width || s.screen_size.height != desktop_height {
return false, tprint("the screen is % x %, but this process sees it as % x %: it must be aware of each monitor's DPI",
s.screen_size.width, s.screen_size.height, desktop_width, desktop_height);
}
dxgi_device: *IDXGIDevice;
hr = IUnknown_QueryInterface(s.device, *uid(IDXGIDevice_UUID), xx *dxgi_device);
if FAILED(hr) return false, hresult_text("the device's IDXGIDevice", hr);
inspectable: *IInspectable;
hr = CreateDirect3D11DeviceFromDXGIDevice(dxgi_device, *inspectable);
com_release(*dxgi_device);
if FAILED(hr) return false, hresult_text("CreateDirect3D11DeviceFromDXGIDevice", hr);
hr = inspectable.vtable.QueryInterface(inspectable, *IID_DIRECT3D_DEVICE, xx *s.winrt_device);
com_release(*inspectable);
if FAILED(hr) return false, hresult_text("the device as IDirect3DDevice", hr);
pool_statics: *Frame_Pool_Statics2;
hr = activation_factory("Windows.Graphics.Capture.Direct3D11CaptureFramePool", IID_POOL_STATICS2, xx *pool_statics);
if FAILED(hr) return false, hresult_text("Direct3D11CaptureFramePool's statics", hr);
hr = pool_statics.vtable.CreateFreeThreaded(pool_statics, s.winrt_device, PIXEL_FORMAT_B8G8R8A8, POOL_BUFFERS, size_packed(s.screen_size), *s.pool);
com_release(*pool_statics);
if FAILED(hr) return false, hresult_text("a free-threaded frame pool", hr);
hr = s.pool.vtable.CreateCaptureSession(s.pool, s.item, *s.session);
if FAILED(hr) return false, hresult_text("a capture session", hr);
// The cursor (Windows 10 2004 on) and the border (Windows 11): optional, by what the system has.
session2: *Capture_Session2;
if SUCCEEDED(s.session.vtable.QueryInterface(s.session, *IID_SESSION2, xx *session2)) {
session2.vtable.put_IsCursorCaptureEnabled(session2, xx cursor);
com_release(*session2);
}
session3: *Capture_Session3;
if testing & .KEEP_BORDER {
s.border = "border left on";
} else if SUCCEEDED(s.session.vtable.QueryInterface(s.session, *IID_SESSION3, xx *session3)) {
hr = session3.vtable.put_IsBorderRequired(session3, 0);
s.border = ifx SUCCEEDED(hr) then "border off" else hresult_text("the border could not be turned off", hr);
com_release(*session3);
} else {
s.border = "border always drawn (no IGraphicsCaptureSession3: Windows 10)";
}
s.handler.vtable = *frame_handler_vtable;
s.handler.event = CreateEventW(null, 0, 0, null);
hr = ifx testing & .WGC_POLL then cast,no_check(HRESULT) 0x80004002 else s.pool.vtable.add_FrameArrived(s.pool, *s.handler, *s.handler_token);
if FAILED(hr) {
s.polling = true;
timeBeginPeriod(1); // so polling every millisecond is every millisecond
log("[record] FrameArrived would not take the handler (%); polling the frame pool.", hresult_text("add_FrameArrived", hr));
}
hr = s.session.vtable.StartCapture(s.session);
if FAILED(hr) return false, hresult_text("StartCapture", hr);
return true, "";
}
// Waits up to timeout_ms for a new frame; the newest's region goes into s.latest.
wgc_next :: (s: *Wgc_Source, timeout_ms: u32) -> Capture_Next, error: string {
frame := wgc_take_newest(s);
if !frame {
if s.polling {
start := ticks_now();
frequency: LARGE_INTEGER;
QueryPerformanceFrequency(*frequency);
while !frame && (ticks_now() - start) * 1000 / frequency.QuadPart < timeout_ms {
sleep_milliseconds(1);
frame = wgc_take_newest(s);
}
} else {
WaitForSingleObject(s.handler.event, timeout_ms);
frame = wgc_take_newest(s);
}
}
if !frame return .NONE, "";
defer close_and_release(frame);
// The screen changed size (a new resolution): the pool is made again; the region must still fit.
size: Size_Int32;
frame.vtable.get_ContentSize(frame, *size);
if size.width != s.screen_size.width || size.height != s.screen_size.height {
s.screen_size = size;
hr := s.pool.vtable.Recreate(s.pool, s.winrt_device, PIXEL_FORMAT_B8G8R8A8, POOL_BUFFERS, size_packed(size));
if FAILED(hr) return .FAILED, hresult_text("the frame pool could not be made again", hr);
if cast(s32) s.box.right > size.width || cast(s32) s.box.bottom > size.height
return .FAILED, tprint("the screen is now % x %, and the region is not on it", size.width, size.height);
return .NONE, "";
}
surface: *IInspectable;
hr := frame.vtable.get_Surface(frame, *surface);
if FAILED(hr) return .FAILED, hresult_text("the frame's surface", hr);
defer com_release(*surface);
access: *Dxgi_Interface_Access;
hr = surface.vtable.QueryInterface(surface, *IID_DXGI_INTERFACE_ACCESS, xx *access);
if FAILED(hr) return .FAILED, hresult_text("the surface's IDirect3DDxgiInterfaceAccess", hr);
defer com_release(*access);
texture: *ID3D11Texture2D;
hr = access.vtable.GetInterface(access, *uid(ID3D11Texture2D_UUID), xx *texture);
if FAILED(hr) return .FAILED, hresult_text("the surface's texture", hr);
gpu_copy_region(s, texture);
IUnknown_Release(texture);
return .IMAGE, "";
}
wgc_close :: (s: *Wgc_Source) {
if s.pool && s.handler_token s.pool.vtable.remove_FrameArrived(s.pool, s.handler_token);
if s.session close_and_release(s.session);
if s.pool close_and_release(s.pool);
s.session, s.pool = null, null;
com_release(*s.item);
com_release(*s.winrt_device);
if s.handler.event CloseHandle(s.handler.event);
if s.polling timeEndPeriod(1);
gpu_close(s);
s.* = .{};
}
// The thread's apartment: WinRT on this thread from here on.
wgc_thread_start :: () -> ok: bool {
if !winrt_load() return false;
return SUCCEEDED(winrt.RoInitialize(RO_INIT_MULTITHREADED));
}
wgc_thread_end :: () {
if winrt.RoUninitialize winrt.RoUninitialize();
}
#scope_file
POOL_BUFFERS :: 2;
PIXEL_FORMAT_B8G8R8A8 :: 87; // DirectXPixelFormat.B8G8R8A8UIntNormalized
RO_INIT_MULTITHREADED :: 1;
// The pool's newest frame, the older ones let go; null when there is none.
wgc_take_newest :: (s: *Wgc_Source) -> *Capture_Frame {
newest: *Capture_Frame;
while true {
frame: *Capture_Frame;
if FAILED(s.pool.vtable.TryGetNextFrame(s.pool, *frame)) || !frame break;
if newest close_and_release(newest);
newest = frame;
}
return newest;
}
// IClosable.Close, which hands a frame back to the pool at once, then Release.
close_and_release :: (object: *$T) {
closable: *Closable;
if SUCCEEDED(object.vtable.QueryInterface(object, *IID_CLOSABLE, xx *closable)) {
closable.vtable.Close(closable);
closable.vtable.Release(closable);
}
object.vtable.Release(object);
}
activation_factory :: (class_name: string, iid: GUID, factory: **void) -> HRESULT {
wide := utf8_to_wide(class_name,, temp);
length: u32;
while wide[length] length += 1;
name: HSTRING;
if !winrt.loaded return cast,no_check(HRESULT) 0x80004001; // E_NOTIMPL: no WinRT on this system
hr := winrt.WindowsCreateString(wide, length, *name);
if FAILED(hr) return hr;
defer winrt.WindowsDeleteString(name);
iid_copy := iid;
return winrt.RoGetActivationFactory(name, *iid_copy, factory);
}
size_packed :: (size: Size_Int32) -> u64 { // SizeInt32 by value: 8 bytes, in one register
width := cast(u64) cast,no_check(u32) size.width;
height := cast(u64) cast,no_check(u32) size.height;
return width | (height << 32);
}
// ---------------------------------------------------------------- the FrameArrived handler
// TypedEventHandler<Direct3D11CaptureFramePool, Object>: a COM object of our own whose Invoke sets an
// event. It lives in the Wgc_Source and is never freed by its count; it is unregistered first.
Frame_Handler :: struct {
vtable: *Frame_Handler_VTable;
event: HANDLE;
}
Frame_Handler_VTable :: struct {
QueryInterface: (this: *Frame_Handler, iid: *GUID, out: **void) -> HRESULT #c_call;
AddRef: (this: *Frame_Handler) -> u32 #c_call;
Release: (this: *Frame_Handler) -> u32 #c_call;
Invoke: (this: *Frame_Handler, sender: *void, args: *void) -> HRESULT #c_call;
}
frame_handler_vtable := Frame_Handler_VTable.{handler_query, handler_add_ref, handler_release, handler_invoke};
handler_query :: (this: *Frame_Handler, iid: *GUID, out: **void) -> HRESULT #c_call {
if guid_equal(iid, *IID_UNKNOWN) || guid_equal(iid, *IID_AGILE_OBJECT) || guid_equal(iid, *IID_FRAME_ARRIVED_HANDLER) {
out.* = this;
return 0;
}
out.* = null;
return cast,no_check(HRESULT) 0x80004002; // E_NOINTERFACE
}
handler_add_ref :: (this: *Frame_Handler) -> u32 #c_call { return 1; }
handler_release :: (this: *Frame_Handler) -> u32 #c_call { return 1; }
handler_invoke :: (this: *Frame_Handler, sender: *void, args: *void) -> HRESULT #c_call {
SetEvent(this.event);
return 0;
}
guid_equal :: (a: *GUID, b: *GUID) -> bool #c_call {
if a.Data1 != b.Data1 || a.Data2 != b.Data2 || a.Data3 != b.Data3 return false;
for 0..7 if a.Data4[it] != b.Data4[it] return false;
return true;
}
// ---------------------------------------------------------------- bindings
d3d11_dll :: #library,system "d3d11";
kernel32_dll :: #library,system "kernel32";
HSTRING :: *void;
// WinRT's own calls, from combase.dll at run time: its import library is runtimeobject.lib, which
// names no DLL of its own for #library to find.
winrt: struct {
loaded: bool;
RoInitialize: (init_type: s32) -> HRESULT #c_call;
RoUninitialize: () #c_call;
RoGetActivationFactory: (class_id: HSTRING, iid: *GUID, factory: **void) -> HRESULT #c_call;
WindowsCreateString: (source: *u16, length: u32, out: *HSTRING) -> HRESULT #c_call;
WindowsDeleteString: (text: HSTRING) -> HRESULT #c_call;
}
winrt_load :: () -> bool {
if winrt.loaded return true;
library := LoadLibraryW(utf8_to_wide("combase.dll",, temp));
if !library return false;
winrt.RoInitialize = xx GetProcAddress(library, "RoInitialize");
winrt.RoUninitialize = xx GetProcAddress(library, "RoUninitialize");
winrt.RoGetActivationFactory = xx GetProcAddress(library, "RoGetActivationFactory");
winrt.WindowsCreateString = xx GetProcAddress(library, "WindowsCreateString");
winrt.WindowsDeleteString = xx GetProcAddress(library, "WindowsDeleteString");
winrt.loaded = winrt.RoInitialize && winrt.RoUninitialize && winrt.RoGetActivationFactory && winrt.WindowsCreateString && winrt.WindowsDeleteString;
return winrt.loaded;
}
CreateDirect3D11DeviceFromDXGIDevice :: (device: *IDXGIDevice, out: **IInspectable) -> HRESULT #foreign d3d11_dll;
SetEvent :: (event: HANDLE) -> s32 #foreign kernel32_dll;
Size_Int32 :: struct { width, height: s32; }
IID_UNKNOWN := #run guid("00000000-0000-0000-C000-000000000046");
IID_AGILE_OBJECT := #run guid("94ea2b94-e9cc-49e0-c0ff-ee64ca8f5b90");
IID_FRAME_ARRIVED_HANDLER := #run guid("51a947f7-79cf-5a3e-a3a5-1289cfa6dfe8"); // TypedEventHandler<Direct3D11CaptureFramePool, Object>
IID_ITEM_INTEROP := #run guid("3628E81B-3CAC-4C60-B7F4-23CE0E0C3356");
IID_ITEM := #run guid("79C3F95B-31F7-4EC2-A464-632EF5D30760");
IID_SESSION_STATICS := #run guid("2224A540-5974-49AA-B232-0882536F4CB5");
IID_SESSION2 := #run guid("2C39AE40-7D2E-5044-804E-8B6799D4CF9E");
IID_SESSION3 := #run guid("F2CDD966-22AE-5EA1-9596-3A289344C3BE");
IID_POOL_STATICS2 := #run guid("589B103F-6BBC-5DF5-A991-02E28B3B66D5");
IID_DIRECT3D_DEVICE := #run guid("A37624AB-8D5F-4650-9D3E-9EAE3D9BC670");
IID_DXGI_INTERFACE_ACCESS := #run guid("A9B3D012-3DF2-4EE3-B8D1-8695F457D3C1");
IID_CLOSABLE := #run guid("30D5A829-7FA4-4026-83BB-D75BAE4EA99E");
guid :: (text: string) -> GUID {
hex :: (s: string) -> u64 {
value: u64;
for 0..s.count - 1 {
c := s[it];
digit: u64 = ifx c >= #char "a" then cast(u64) (c - #char "a" + 10) else ifx c >= #char "A" then cast(u64) (c - #char "A" + 10) else cast(u64) (c - #char "0");
value = value * 16 + digit;
}
return value;
}
g: GUID;
g.Data1 = cast(u32) hex(slice(text, 0, 8));
g.Data2 = cast(u16) hex(slice(text, 9, 4));
g.Data3 = cast(u16) hex(slice(text, 14, 4));
for 0..1 g.Data4[it] = cast(u8) hex(slice(text, 19 + it * 2, 2));
for 0..5 g.Data4[it + 2] = cast(u8) hex(slice(text, 24 + it * 2, 2));
return g;
}
// IUnknown, then IInspectable's three: every WinRT interface starts so.
Inspectable_Methods :: struct {
QueryInterface: (this: *void, iid: *GUID, out: **void) -> HRESULT #c_call;
AddRef: (this: *void) -> u32 #c_call;
Release: (this: *void) -> u32 #c_call;
GetIids: *void;
GetRuntimeClassName: *void;
GetTrustLevel: *void;
}
IInspectable :: struct { vtable: *Inspectable_Methods; }
Closable :: struct { vtable: *struct { using base: Inspectable_Methods; Close: (this: *Closable) -> HRESULT #c_call; }; }
Session_Statics :: struct { vtable: *struct { using base: Inspectable_Methods; IsSupported: (this: *Session_Statics, out: *u8) -> HRESULT #c_call; }; }
Capture_Item_Interop :: struct { vtable: *struct {
QueryInterface: (this: *void, iid: *GUID, out: **void) -> HRESULT #c_call;
AddRef: (this: *void) -> u32 #c_call;
Release: (this: *void) -> u32 #c_call;
CreateForWindow: (this: *Capture_Item_Interop, window: HWND, iid: *GUID, out: **void) -> HRESULT #c_call;
CreateForMonitor: (this: *Capture_Item_Interop, monitor: HMONITOR, iid: *GUID, out: **void) -> HRESULT #c_call;
}; }
Capture_Item :: struct { vtable: *struct {
using base: Inspectable_Methods;
get_DisplayName: *void;
get_Size: (this: *Capture_Item, out: *Size_Int32) -> HRESULT #c_call;
add_Closed: *void;
remove_Closed: *void;
}; }
Frame_Pool_Statics2 :: struct { vtable: *struct {
using base: Inspectable_Methods;
CreateFreeThreaded: (this: *Frame_Pool_Statics2, device: *IInspectable, format: s32, buffers: s32, size: u64, out: **Frame_Pool) -> HRESULT #c_call;
}; }
Frame_Pool :: struct { vtable: *struct {
using base: Inspectable_Methods;
Recreate: (this: *Frame_Pool, device: *IInspectable, format: s32, buffers: s32, size: u64) -> HRESULT #c_call;
TryGetNextFrame: (this: *Frame_Pool, out: **Capture_Frame) -> HRESULT #c_call;
add_FrameArrived: (this: *Frame_Pool, handler: *Frame_Handler, token: *s64) -> HRESULT #c_call;
remove_FrameArrived: (this: *Frame_Pool, token: s64) -> HRESULT #c_call;
CreateCaptureSession: (this: *Frame_Pool, item: *Capture_Item, out: **Capture_Session) -> HRESULT #c_call;
get_DispatcherQueue: *void;
}; }
Capture_Frame :: struct { vtable: *struct {
using base: Inspectable_Methods;
get_Surface: (this: *Capture_Frame, out: **IInspectable) -> HRESULT #c_call;
get_SystemRelativeTime: (this: *Capture_Frame, out: *s64) -> HRESULT #c_call;
get_ContentSize: (this: *Capture_Frame, out: *Size_Int32) -> HRESULT #c_call;
}; }
Capture_Session :: struct { vtable: *struct {
using base: Inspectable_Methods;
StartCapture: (this: *Capture_Session) -> HRESULT #c_call;
}; }
Capture_Session2 :: struct { vtable: *struct {
using base: Inspectable_Methods;
get_IsCursorCaptureEnabled: (this: *Capture_Session2, out: *u8) -> HRESULT #c_call;
put_IsCursorCaptureEnabled: (this: *Capture_Session2, value: u8) -> HRESULT #c_call;
}; }
Capture_Session3 :: struct { vtable: *struct {
using base: Inspectable_Methods;
get_IsBorderRequired: (this: *Capture_Session3, out: *u8) -> HRESULT #c_call;
put_IsBorderRequired: (this: *Capture_Session3, value: u8) -> HRESULT #c_call;
}; }
Dxgi_Interface_Access :: struct { vtable: *struct {
QueryInterface: (this: *void, iid: *GUID, out: **void) -> HRESULT #c_call;
AddRef: (this: *void) -> u32 #c_call;
Release: (this: *void) -> u32 #c_call;
GetInterface: (this: *Dxgi_Interface_Access, iid: *GUID, out: **void) -> HRESULT #c_call;
}; }

7
src/App.jai
View File

@ -27,9 +27,6 @@ USING_WINDOWS :: OS == .WINDOWS;
#load "app/segment_ui.jai";
#load "app/library_folders.jai";
#load "app/exit_shutdown.jai";
#load "app/screen_capture.jai";
#load "app/screen_record.jai";
#if OS == .WINDOWS #load "app/screen_capture_win32.jai";
#load "base/circle_buffer.jai";
#load "base/debug.jai";
@ -75,7 +72,6 @@ Shared_Channel :: #import "Shared_Memory_Channel";
#import "Capture_Text";
#import "Base64";
#import "onnxruntime";
#if OS == .WINDOWS #import "Screen_Record"; // screen recording: Desktop Duplication or Windows.Graphics.Capture, to NVENC
// Basic's arenas import Secure_Memory at its defaults; this import adds its test suite.
using Secure_Memory :: #import "Secure_Memory"()(LOAD_TESTS=COMPILE_TESTS);
@ -186,8 +182,6 @@ App :: struct {
show_style_editor: bool;
minimize_to_tray_this_frame: bool;
restore_window_this_frame: bool;
capture_region_this_frame: bool; // the capture_region global hotkey fired: app/screen_capture.jai
record_region_this_frame: bool; // the record_region global hotkey fired: app/screen_record.jai
is_master: bool; // Are we the first instance of this app launched?
allow_multiple_instances: bool; // Skips the single-instance check, so a dev build can run beside an installed one.
should_exit_early: bool;
@ -199,7 +193,6 @@ App :: struct {
stats_and_exit: bool; // -stats: log the Statistics window's numbers once the catalog has loaded, then exit
start_search: string; // -search QUERY: the search box's text at startup
erase_and_exit: string; // -erase ocr|labels|segments|all: erase that from the whole library once it has loaded, then exit
record_and_exit: float64; // -record SECONDS: record the screen under the cursor that long, put it in the library, then exit (0, or 1 if it failed)
segment_and_exit: bool; // -segment: segment the library folders set to segment once the catalog has loaded, then exit (0, or 1 if any image failed)
import_source, import_dest: string; // -import SOURCE DEST: import SOURCE into the library folder DEST (app/import.jai), then exit
last_window_restore_time: Apollo_Time;

6
src/app/app_ui.jai
View File

@ -15,15 +15,12 @@ app_ui_startup :: () {
// Every frame, before input is handed to ImGui. Poll background work here.
app_ui_frame_start :: () {
screen_capture_frame(); // also while hidden in the tray, where the frame ends before the UI
}
// Once, on the shutdown thread after the window is asked to close (exit_shutdown.jai): save and
// stop background work here. Must not touch ImGui, since the main thread is still drawing frames
// (the overlay; app_main_window_ui stops running the app's frames once shutdown starts).
app_ui_shutdown :: () {
app_shutdown_step(.CAPTURE);
screen_capture_shutdown();
app_shutdown_step(.IMPORT);
import_shutdown(); // before the viewer: it hands the viewer what it writes
app_shutdown_step(.VIEWER);
@ -39,7 +36,7 @@ app_ui_shutdown :: () {
}
// Which step of app_ui_shutdown is running, and since when, for the overlay a slow shutdown shows.
App_Shutdown_Step :: enum s32 { NONE; CAPTURE; IMPORT; VIEWER; OCR; SEGMENT; LABELS; STATS; }
App_Shutdown_Step :: enum s32 { NONE; IMPORT; VIEWER; OCR; SEGMENT; LABELS; STATS; }
app_shutdown: struct {
step: App_Shutdown_Step;
step_started_ns: s64; // monotonic
@ -57,7 +54,6 @@ app_ui_shutdown_status :: () -> string {
text: string;
if step == {
case .NONE; return "";
case .CAPTURE; text = "Saving a screen capture.";
case .IMPORT;
if importer.phase == .INDEXING text = "Stopping the import: it is indexing the library, and stops after the files it is on.";
else if importer.running text = "Stopping the import: finishing the files it is on (a large video takes a while to copy).";

2
src/app/catalog.jai
View File

@ -92,9 +92,7 @@ catalog_load :: (path: string) -> *Catalog_Entry, error: string {
entry.url = catalog_string(metadata.root, "WebURL");
entry.app = catalog_string(metadata.root, "AppName");
}
// An import's import.json, or a screen recording's capture.json, says what kind of media it is.
imported := capture_find_file(*doc, "import.json");
if !imported imported = capture_find_file(*doc, "capture.json");
if imported {
entry.video_codec = catalog_string(imported.root, "VideoCodec");
entry.duration = catalog_number(imported.root, "Duration");

14
src/app/main.jai
View File

@ -25,7 +25,6 @@ main :: () {
}
app.scan_and_exit = array_find(app.args, "-scan");
app.segment_and_exit = array_find(app.args, "-segment");
for app.args if it == "-record" && it_index + 1 < app.args.count app.record_and_exit = max(string_to_float64(app.args[it_index + 1]), 1.0);
app.stats_and_exit = array_find(app.args, "-stats");
for app.args if it == "-search" && it_index + 1 < app.args.count app.start_search = app.args[it_index + 1];
for app.args if it == "-erase" && it_index + 1 < app.args.count app.erase_and_exit = app.args[it_index + 1];
@ -467,19 +466,6 @@ app_frame :: (in_live_resize := false) -> bool {
platform_restore_window();
}
// Not from a live-resize frame: choosing the region runs a modal loop of its own.
if app.capture_region_this_frame && !in_live_resize {
app.capture_region_this_frame = false;
screen_capture_take();
app.capture_region_this_frame = false; // the hotkey pressed again while choosing
}
if app.record_region_this_frame && !in_live_resize {
app.record_region_this_frame = false;
screen_record_toggle();
app.record_region_this_frame = false;
app.capture_region_this_frame = false;
}
if app.should_exit {
app_exit_shutdown_start();
if app_exit_shutdown_poll() return false;

240
src/app/screen_capture.jai
View File

@ -1,240 +0,0 @@
// Screen capture, as Snagit does it: PrintScreen (the capture_region global hotkey, from any app,
// even with mcap hidden in the tray) freezes every screen, and a rectangle dragged over them is taken
// (app/screen_capture_win32.jai; Windows only so far). What is taken goes onto the clipboard at once,
// and into the library as an .mcap in the folder SCREEN_CAPTURE_FOLDER: the pixels as a PNG, with
// the window it was taken from in metadata.json, its title as WindowName and its program as AppName,
// as Snagit records them, so searches find it the same way. mcap then comes to the front showing
// it in the Capture window, and OCR and labels read it once they are free.
//
// The PNG is made and the file written on a thread of its own, as a large capture takes a second or
// so to compress. A screen recording (app/screen_record.jai) goes into the library through the same
// jobs.
SCREEN_CAPTURE_FOLDER :: "Screen Captures"; // in the library
Screen_Capture_Job :: struct {
shot: Screen_Shot;
date: string; // CaptureDate; heap
// A recording (app/screen_record.jai), not a still; its strings are heap.
video: bool;
recording: string; // the MP4 recorded, deleted once in the library
encoder: string; // "h264_nvenc"
source: string; // how the screen was taken: Record_Stats.source
dropped: s64; // frames the encoder fell too far behind for
thread: Thread;
busy: bool; // cleared by the thread as it ends
path: string; // what it wrote; heap
error: string; // or why it could not; heap
}
screen_capture: struct {
jobs: [..] *Screen_Capture_Job;
scan_pending: [..] s32; // captures OCR and labels have yet to read
segment_pending: [..] s32; // and then segmentation, when the folder is set to segment
}
// From app_frame when the hotkey fired, outside the ImGui frame: lets a region be chosen, copies it,
// and starts writing it into the library.
screen_capture_take :: () {
if app_exit_shutdown_started() return;
date := import_calendar_text(to_calendar(current_time_consensus(), .LOCAL));
start := current_time_monotonic();
// While recording, the overlay's frozen screen would be in the recording: the screen under the
// cursor is taken at once instead.
shot, ok, error := platform_screen_capture_select(immediate = screen_record.recording);
if !ok {
if error { log_error("[capture] %.", error); toast_show(.Error, "Screen capture: %", error); }
else log("[capture] Cancelled.");
return;
}
log("[capture] % x % at %, % (%) from \"%\" (%), chosen in % s.", shot.width, shot.height, shot.left, shot.top,
ifx shot.whole_screen then "the whole screen" else "a region", shot.window_title, shot.app_name,
FF(to_float64_seconds(current_time_monotonic() - start), 1));
if !platform_copy_image_to_clipboard(shot.pixels, shot.width, shot.height) log_error("[capture] Could not put the capture on the clipboard.");
job := New(Screen_Capture_Job,, context.default_allocator);
job.shot = shot;
job.date = copy_string(date,, context.default_allocator);
screen_capture_job_start(job);
}
// Writes the job into the library on a thread of its own; screen_capture_frame takes it from there.
// The job is freed when it cannot start.
screen_capture_job_start :: (job: *Screen_Capture_Job) -> bool {
job.busy = true;
if !thread_init(*job.thread, screen_capture_save_proc, temporary_storage_size = 1024 * 1024, name = "screen capture") {
what := ifx job.video then "recording" else "capture";
log_error("[capture] Could not start the thread to save the %.", what);
toast_show(.Error, "The screen % could not be saved", what);
screen_capture_job_free(job);
return false;
}
thread_start(*job.thread, job);
screen_capture.jobs.allocator = context.default_allocator;
array_add(*screen_capture.jobs, job);
return true;
}
// Every frame, even while hidden in the tray: the captures written, into the grid and the Capture
// window, and then read.
screen_capture_frame :: () {
if app_exit_shutdown_started() return;
screen_record_frame();
for job: screen_capture.jobs {
if atomic_read(*job.busy) continue;
thread_is_done(*job.thread, -1);
thread_deinit(*job.thread);
what := ifx job.video then "recording" else "capture";
if job.video screen_record_saved(job.path, job.error);
if job.error {
log_error("[capture] Could not save the %: %", what, job.error);
toast_show(.Error, "The screen % could not be saved: %", what, job.error);
} else {
log("[capture] Saved %.", job.path);
index := viewer_add_capture(job.path);
if index >= 0 {
viewer_request_full(index);
screen_capture.scan_pending.allocator = context.default_allocator;
screen_capture.segment_pending.allocator = context.default_allocator;
array_add(*screen_capture.scan_pending, index);
if !job.video && library_folder_segments(SCREEN_CAPTURE_FOLDER) array_add(*screen_capture.segment_pending, index);
}
app.restore_window_this_frame = true;
if job.video toast_show(.Success, "% x % recorded", job.shot.width, job.shot.height);
else toast_show(.Success, "% x % captured and copied to the clipboard", job.shot.width, job.shot.height);
}
screen_capture_job_free(job);
remove job;
}
// As an import's are, once the models are free: OCR and labels, then segmentation.
if screen_capture.scan_pending.count && !ocr_ui.thread_busy && !ocr_ui.scan.running && ocr_ui.engine && labels_ui_settled() {
ocr_scan_start(screen_capture.scan_pending);
array_reset_keeping_memory(*screen_capture.scan_pending);
}
// Once their catalog entries are in: the scan takes only images, and they say which.
if screen_capture.segment_pending.count && !screen_capture.scan_pending.count && !ocr_ui.thread_busy && !ocr_ui.scan.running {
catalogued := true;
for screen_capture.segment_pending if !viewer.captures[it].catalog && !viewer.captures[it].removed { catalogued = false; break; }
if catalogued {
segment_scan_start(screen_capture.segment_pending, quiet = true);
array_reset_keeping_memory(*screen_capture.segment_pending);
}
}
}
// On the shutdown thread: a capture being written is finished, not lost.
screen_capture_shutdown :: () {
screen_record_shutdown(); // a recording running is stopped and becomes a job, finished below
for screen_capture.jobs thread_is_done(*it.thread, -1);
}
screen_capture_job_free :: (job: *Screen_Capture_Job) {
push_allocator(context.default_allocator);
screen_shot_free(*job.shot);
free(job.date);
free(job.path);
free(job.error);
free(job.recording);
free(job.encoder);
free(job.source);
free(job);
}
screen_capture_save_proc :: (thread: *Thread) -> s64 {
job := cast(*Screen_Capture_Job) thread.data;
context.allocator = context.default_allocator;
defer atomic_swap(*job.busy, false);
path, error: string;
if job.video path, error = screen_record_write(job);
else path, error = screen_capture_write(job.shot, job.date);
if error job.error = copy_string(error);
else job.path = copy_string(path);
return 0;
}
// Writes the capture into the library: its path, or why it could not be (temp).
screen_capture_write :: (shot: Screen_Shot, date: string) -> path: string, error: string {
image := Image.{pixels = shot.pixels.data, width = shot.width, height = shot.height};
png, _, _, png_ok, png_error := image_preview_png(image, max(shot.width, shot.height)); // at its own size
if !png_ok return "", png_error;
preview, preview_width, preview_height, preview_ok, preview_error := image_preview_png(image, MCAP_PREVIEW_MAX_SIDE);
if !preview_ok return "", preview_error;
hash := import_hash(png);
doc := New(Capture_Document,, temp);
capture_document_init(doc, temp);
doc.comment = tprint("Captured by % %.", APP_NAME, VERSION);
metadata := capture_add_file(doc, "metadata.json");
import_add(doc, metadata, "AppName", import_text(shot.app_name));
import_add(doc, metadata, "CaptureDate", date);
import_add(doc, metadata, "WindowName", import_text(shot.window_title));
info := capture_add_file(doc, "capture.json");
import_add(doc, info, "Region", ifx shot.whole_screen then "screen" else "selection");
import_add_number(doc, info, "Left", xx shot.left);
import_add_number(doc, info, "Top", xx shot.top);
import_add_number(doc, info, "Width", xx shot.width);
import_add_number(doc, info, "Height", xx shot.height);
if shot.app_path import_add(doc, info, "AppPath", import_text(shot.app_path));
import_add(doc, info, "ContentHash", import_hash_text(hash));
import_add(doc, info, "Format", "PNG");
name := import_out_name(date, hash);
text, text_ok := import_document_text(doc, name);
if !text_ok return "", "its Capture Text does not read back cleanly (see the Event Log)";
m: Mcap;
m.text = text;
m.preview = .{width = cast(u16) preview_width, height = cast(u16) preview_height, png = preview};
m.original_name = tprint("%.png", path_strip_extension(name));
m.original = png;
m.rendered_kind = .NONE;
serializer, serialized, serialize_error := mcap_serialize(m,, temp);
if !serialized return "", serialize_error;
path, folder_error := screen_capture_out_path(name);
if folder_error return "", folder_error;
written, write_error := mcap_write(path, cast([] u8) string.{serializer.count, serializer.data});
if !written return "", write_error;
return path, "";
}
// Where a capture named name goes in SCREEN_CAPTURE_FOLDER, made if need be: name, or name_2 ... when
// another file has it. Temp.
screen_capture_out_path :: (name: string) -> path: string, error: string {
folder := tprint("%/%", viewer_folder(), SCREEN_CAPTURE_FOLDER);
if !make_directory_if_it_does_not_exist(folder, recursive = true) return "", tprint("% could not be made", folder);
path := tprint("%/%", folder, name);
number := 2;
while file_exists(path) {
path = tprint("%/%_%.%", folder, path_strip_extension(name), number, MCAP_EXTENSION);
number += 1;
}
return path, "";
}
#if OS != .WINDOWS {
Screen_Shot :: struct {
pixels: [] u8;
width, height: s32;
left, top: s32;
whole_screen: bool;
window_title, app_name, app_path: string;
}
screen_shot_free :: (shot: *Screen_Shot) {
array_free(shot.pixels,, context.default_allocator);
shot.* = .{};
}
platform_screen_capture_select :: (video := false, immediate := false) -> Screen_Shot, ok: bool, error: string {
return .{}, false, "not available on this system yet";
}
platform_copy_image_to_clipboard :: (pixels: [] u8, width: s32, height: s32) -> bool {
return false;
}
}

716
src/app/screen_capture_win32.jai
View File

@ -1,716 +0,0 @@
// The screen capture overlay on Windows (app/screen_capture.jai is the rest). The moment the hotkey
// fires, everything on every screen is copied, and shown frozen and darkened in a window over all of
// them. A rectangle dragged over it shows at full brightness, and letting go takes it; a click takes
// the whole screen the cursor is on. A magnifier beside the cursor shows the pixels under it, with the
// position or the selection's size, so a region can be chosen to the pixel; the arrow keys move the
// cursor a pixel at a time (ten with Shift). Escape or a right click cancels.
//
// For a recording (video), the outline and magnifier are red, the drag is kept on the screen it began
// on (a recording is of one screen), and a region smaller than RECORD_MIN_WIDTH x RECORD_MIN_HEIGHT is
// outlined dashed, with the minimum in the label, and letting go of it takes nothing: the overlay stays
// to drag again. No pixels are copied out for a recording.
//
// Drawn with GDI into a bitmap the size of the desktop, so it needs neither ImGui nor the D3D11 device
// and works while the main window is hidden in the tray. Only the parts that change are drawn again.
// The selection runs in a modal loop of its own: platform_screen_capture_select returns once a region
// is taken or the capture is cancelled.
// The region taken, and the window it was taken from.
Screen_Shot :: struct {
pixels: [] u8; // RGBA, rows packed; heap
width, height: s32;
left, top: s32; // on the virtual desktop, whose origin is the primary screen's top left
whole_screen: bool; // a click took the screen under the cursor
window_title: string; // the window under the region's center; heap, as are the two below
app_name: string; // its program's description ("Google Chrome"), or its file name
app_path: string; // its program's file
}
screen_shot_free :: (shot: *Screen_Shot) {
push_allocator(context.default_allocator);
array_free(shot.pixels);
free(shot.window_title);
free(shot.app_name);
free(shot.app_path);
shot.* = .{};
}
// Freezes the screens and lets a region be chosen: for a still, or with video for a recording. With
// immediate, no overlay: the whole screen under the cursor is taken at once. ok is false when it was
// cancelled (error "") or could not start (error says why).
platform_screen_capture_select :: (video := false, immediate := false) -> Screen_Shot, ok: bool, error: string {
shot: Screen_Shot;
fallback_window := GetForegroundWindow(); // before the overlay takes the focus
overlay.video = video;
// The window procedure runs in this context, during the modal loop below, and in no other.
overlay.app_context = *context;
defer overlay.app_context = null;
if !overlay_freeze() { overlay_release(); return shot, false, "the screen could not be copied"; }
defer overlay_release();
if immediate {
cursor: POINT;
GetCursorPos(*cursor);
overlay.result = overlay_monitor_rect(.{cursor.x - overlay.left, cursor.y - overlay.top});
overlay.whole_screen = true;
overlay.cancelled = false;
} else {
if !overlay_open() return shot, false, "the capture window could not be opened";
overlay_choose();
}
if overlay.cancelled return shot, false, "";
r := overlay.result;
shot.width = r.right - r.left;
shot.height = r.bottom - r.top;
if shot.width <= 0 || shot.height <= 0 return shot, false, "";
shot.left = overlay.left + r.left;
shot.top = overlay.top + r.top;
shot.whole_screen = overlay.whole_screen;
// The desktop's bitmap is BGRA (its alpha undefined), top-down.
if !video shot.pixels = NewArray(cast(s64) shot.width * shot.height * 4, u8, initialized = false,, context.default_allocator);
out := shot.pixels.data;
if !video for y: r.top..r.bottom - 1 {
row := overlay.screen_pixels + cast(s64) y * overlay.width;
for x: r.left..r.right - 1 {
p := row[x];
out[0] = cast,trunc(u8) (p >> 16);
out[1] = cast,trunc(u8) (p >> 8);
out[2] = cast,trunc(u8) p;
out[3] = 255;
out += 4;
}
}
// The window under the region's middle, now that the overlay is gone; else the one in front.
center := POINT.{shot.left + shot.width / 2, shot.top + shot.height / 2};
window := WindowFromPoint(center);
if window window = GetAncestor(window, GA_ROOT);
if !window window = fallback_window;
if window overlay_window_info(*shot, window);
return shot, true, "";
}
// Puts the RGBA pixels on the clipboard as a bitmap, which every program pastes.
platform_copy_image_to_clipboard :: (pixels: [] u8, width: s32, height: s32) -> bool {
row_bytes := cast(s64) width * 4;
size := size_of(BITMAPINFOHEADER) + row_bytes * height;
memory := GlobalAlloc(GMEM_MOVEABLE, cast(SIZE_T) size);
if !memory return false;
data := cast(*u8) GlobalLock(memory);
if !data { GlobalFree(memory); return false; }
header := cast(*BITMAPINFOHEADER) data;
header.* = .{};
header.biSize = size_of(BITMAPINFOHEADER);
header.biWidth = width;
header.biHeight = height; // bottom-up, as most programs expect of CF_DIB
header.biPlanes = 1;
header.biBitCount = 32;
header.biCompression = BI_RGB;
header.biSizeImage = cast(u32) (row_bytes * height);
out := data + size_of(BITMAPINFOHEADER);
for y: 0..height - 1 {
in := pixels.data + (height - 1 - y) * row_bytes;
for x: 0..width - 1 {
out[0] = in[2]; out[1] = in[1]; out[2] = in[0]; out[3] = 255;
out += 4; in += 4;
}
}
GlobalUnlock(memory);
if !OpenClipboard(platform.hwnd) { GlobalFree(memory); return false; }
defer CloseClipboard();
EmptyClipboard();
if !SetClipboardData(CF_DIB, memory) { GlobalFree(memory); return false; } // the clipboard owns it once set
return true;
}
#scope_file
overlay: struct {
hwnd: HWND;
class_registered: bool;
app_context: *#Context;
video: bool; // choosing a region to record
left, top: s32; // the virtual desktop's top left
width, height: s32; // and size, in pixels
scale: float; // the DPI scale of the screen the overlay's window counts as on
screen_dc, dim_dc, back_dc: HDC; // the screens as copied; the same darkened; what is drawn
screen_bitmap, dim_bitmap, back_bitmap: HBITMAP;
screen_old, dim_old, back_old: HGDIOBJ;
screen_pixels: *u32;
font, font_old: HFONT;
accent_brush, black_brush, label_brush, frame_brush: HBRUSH;
cursor: POINT; // in the overlay's pixels, as everything below
dragging: bool;
anchor: POINT;
drawn: Overlay_Layout; // what is on screen, to draw again where it was
active: bool; // has had the focus; losing it after cancels
done, cancelled: bool;
result: RECT;
whole_screen: bool;
}
Overlay_Layout :: struct {
valid: bool;
dragging: bool;
selection: RECT; // the pixels selected, while dragging
too_small: bool; // to record: smaller than RECORD_MIN_WIDTH x RECORD_MIN_HEIGHT
cursor: POINT;
magnifier: RECT; // its frame, and the label under it
label: RECT;
text: [80] u16;
text_count: s32;
}
MAGNIFIER_PIXELS :: 21; // the pixels it shows across: odd, so one is in the middle
MAGNIFIER_ZOOM :: 6; // each shown this many pixels wide, at 100%
MAGNIFIER_OFFSET :: 24; // from the cursor
BORDER :: 2; // the selection's outline, at 100%
CLICK_SLOP :: 3; // a drag smaller than this either way is a click
ACCENT_COLOR :: 0x0000A5FF; // COLORREF, 0x00BBGGRR: orange
RECORD_COLOR :: 0x002828E8; // red: choosing a region to record
DASH :: 6; // a too-small region's dashes and gaps, at 100%
GAP :: 4;
LABEL_COLOR :: 0x00202020;
FRAME_COLOR :: 0x00FFFFFF;
TEXT_COLOR :: 0x00FFFFFF;
OVERLAY_CLASS :: "mcapScreenCapture";
// Copies every screen into a bitmap, and a darkened copy of it.
overlay_freeze :: () -> bool {
overlay.left = GetSystemMetrics(SM_XVIRTUALSCREEN);
overlay.top = GetSystemMetrics(SM_YVIRTUALSCREEN);
overlay.width = GetSystemMetrics(SM_CXVIRTUALSCREEN);
overlay.height = GetSystemMetrics(SM_CYVIRTUALSCREEN);
if overlay.width <= 0 || overlay.height <= 0 return false;
desktop := GetDC(null);
if !desktop return false;
defer ReleaseDC(null, desktop);
dim_pixels: *u32;
ok: bool;
ok, overlay.screen_dc, overlay.screen_bitmap, overlay.screen_old, overlay.screen_pixels = overlay_bitmap(desktop);
if !ok return false;
ok, overlay.dim_dc, overlay.dim_bitmap, overlay.dim_old, dim_pixels = overlay_bitmap(desktop);
if !ok return false;
back_pixels: *u32;
ok, overlay.back_dc, overlay.back_bitmap, overlay.back_old, back_pixels = overlay_bitmap(desktop);
if !ok return false;
// CAPTUREBLT takes layered windows too, such as tooltips and menus with shadows.
if !BitBlt(overlay.screen_dc, 0, 0, overlay.width, overlay.height, desktop, overlay.left, overlay.top, cast(ROP) (cast(u32) ROP.SRCCOPY | CAPTUREBLT)) return false;
count := cast(s64) overlay.width * overlay.height;
for 0..count - 1 dim_pixels[it] = (overlay.screen_pixels[it] >> 1) & 0x7F7F7F7F; // half as bright
return true;
}
overlay_bitmap :: (desktop: HDC) -> ok: bool, dc: HDC, bitmap: HBITMAP, old: HGDIOBJ, pixels: *u32 {
info: BITMAPINFO;
info.bmiHeader.biSize = size_of(BITMAPINFOHEADER);
info.bmiHeader.biWidth = overlay.width;
info.bmiHeader.biHeight = -overlay.height; // top-down
info.bmiHeader.biPlanes = 1;
info.bmiHeader.biBitCount = 32;
info.bmiHeader.biCompression = BI_RGB;
dc := CreateCompatibleDC(desktop);
if !dc return false, null, null, null, null;
bits: *void;
bitmap := CreateDIBSection(dc, *info, DIB_RGB_COLORS, *bits, null, 0);
if !bitmap || !bits { DeleteDC(dc); return false, null, null, null, null; }
old := SelectObject(dc, bitmap);
return true, dc, bitmap, old, cast(*u32) bits;
}
overlay_open :: () -> bool {
instance := GetModuleHandleW(null);
if !overlay.class_registered {
wc: WNDCLASSEXW;
wc.cbSize = size_of(WNDCLASSEXW);
wc.lpfnWndProc = xx overlay_proc;
wc.hInstance = instance;
wc.hCursor = LoadCursorW(null, IDC_CROSS);
wc.lpszClassName = utf8_to_wide(OVERLAY_CLASS,, context.default_allocator);
if RegisterClassExW(*wc) == 0 return false;
overlay.class_registered = true;
}
overlay.cursor = .{};
overlay.dragging = false;
overlay.drawn = .{};
overlay.active = false;
overlay.done = false;
overlay.cancelled = false;
overlay.whole_screen = false;
// Topmost, and a tool window so it is not in the taskbar or Alt+Tab.
overlay.hwnd = CreateWindowExW(WS_EX_TOPMOST | WS_EX_TOOLWINDOW, utf8_to_wide(OVERLAY_CLASS,, temp), utf8_to_wide("Screen capture",, temp),
WS_POPUP, overlay.left, overlay.top, overlay.width, overlay.height, null, null, instance, null);
if !overlay.hwnd return false;
dpi := GetDpiForWindow(overlay.hwnd);
overlay.scale = ifx dpi then dpi / 96.0 else 1.0;
font_height := cast(s32) (15 * overlay.scale + 0.5);
overlay.font = CreateFontW(-font_height, 0, 0, 0, 600, 0, 0, 0, 1, 0, 0, CLEARTYPE_QUALITY, 0, utf8_to_wide("Segoe UI",, temp));
if overlay.font overlay.font_old = SelectObject(overlay.back_dc, overlay.font);
overlay.accent_brush = CreateSolidBrush(ifx overlay.video then cast(u32) RECORD_COLOR else ACCENT_COLOR);
overlay.black_brush = CreateSolidBrush(0);
overlay.label_brush = CreateSolidBrush(LABEL_COLOR);
overlay.frame_brush = CreateSolidBrush(FRAME_COLOR);
cursor: POINT;
GetCursorPos(*cursor);
overlay.cursor = .{cursor.x - overlay.left, cursor.y - overlay.top};
ShowWindow(overlay.hwnd, SW_SHOW);
SetForegroundWindow(overlay.hwnd); // allowed: the hotkey made this the app the user last used
overlay_update();
return true;
}
overlay_release :: () {
release :: (dc: *HDC, bitmap: *HBITMAP, old: HGDIOBJ) {
if dc.* {
SelectObject(dc.*, old);
DeleteDC(dc.*);
dc.* = null;
}
if bitmap.* { DeleteObject(bitmap.*); bitmap.* = null; }
}
if overlay.font {
if overlay.back_dc SelectObject(overlay.back_dc, overlay.font_old);
DeleteObject(overlay.font);
overlay.font = null;
}
release(*overlay.screen_dc, *overlay.screen_bitmap, overlay.screen_old);
release(*overlay.dim_dc, *overlay.dim_bitmap, overlay.dim_old);
release(*overlay.back_dc, *overlay.back_bitmap, overlay.back_old);
overlay.screen_pixels = null;
for brush: HBRUSH.[overlay.accent_brush, overlay.black_brush, overlay.label_brush, overlay.frame_brush] if brush DeleteObject(brush);
overlay.accent_brush, overlay.black_brush, overlay.label_brush, overlay.frame_brush = null, null, null, null;
}
overlay_proc :: (hwnd: HWND, message: u32, wParam: WPARAM, lParam: LPARAM) -> LRESULT #c_call {
if !overlay.app_context return DefWindowProcW(hwnd, message, wParam, lParam);
push_context overlay.app_context.* {
if message == {
case WM_SETCURSOR;
SetCursor(LoadCursorW(null, IDC_CROSS));
return 1;
case WM_ERASEBKGND;
return 1;
case WM_DPICHANGED;
return 0; // the window keeps covering the desktop, whatever DPI it counts as at
case WM_PAINT;
overlay_paint();
return 0;
case WM_ACTIVATE;
if (wParam & 0xFFFF) == WA_INACTIVE {
// Another window came in front: a stuck overlay would hide it. Not once done, as closing
// the overlay sends this too.
if overlay.active && !overlay.done overlay_cancel();
} else {
overlay.active = true;
}
return 0;
case WM_MOUSEMOVE;
overlay.cursor = overlay_point(lParam);
overlay_update();
return 0;
case WM_LBUTTONDOWN;
overlay.cursor = overlay_point(lParam);
overlay.anchor = overlay.cursor;
overlay.dragging = true;
SetCapture(hwnd);
overlay_update();
return 0;
case WM_LBUTTONUP;
if overlay.dragging {
ReleaseCapture();
overlay.cursor = overlay_point(lParam);
overlay_take();
}
return 0;
case WM_RBUTTONDOWN;
if overlay.dragging { // let go of the selection, and start again
overlay.dragging = false;
ReleaseCapture();
overlay_update();
} else {
overlay_cancel();
}
return 0;
case WM_KEYDOWN;
overlay_key(wParam);
return 0;
}
return DefWindowProcW(hwnd, message, wParam, lParam);
}
}
overlay_point :: (lParam: LPARAM) -> POINT {
return .{cast(s32) LOWORD(lParam), cast(s32) HIWORD(lParam)}; // signed: a capture reaches past the edges
}
overlay_key :: (key: WPARAM) {
step: s32 = ifx GetKeyState(VK_SHIFT) < 0 then cast(s32) 10 else 1;
dx, dy: s32;
if key == {
case VK_ESCAPE; overlay_cancel(); return;
case VK_RETURN;
if overlay.dragging ReleaseCapture();
overlay_take();
return;
case VK_LEFT; dx = -step;
case VK_RIGHT; dx = step;
case VK_UP; dy = -step;
case VK_DOWN; dy = step;
case; return;
}
// Moving the cursor sends WM_MOUSEMOVE, which does the rest.
x := clamp(overlay.cursor.x + dx, 0, overlay.width - 1);
y := clamp(overlay.cursor.y + dy, 0, overlay.height - 1);
capture_set_cursor_pos(overlay.left + x, overlay.top + y);
}
// The overlay's own message loop, until a region is taken or it is cancelled; then it is closed.
overlay_choose :: () {
while !overlay.done {
msg: MSG;
got := GetMessageW(*msg, null, 0, 0);
if got == -1 { overlay.cancelled = true; break; }
if got == 0 { // WM_QUIT: the app is asked to end; the main loop sees it again
PostQuitMessage(cast(s32) msg.wParam);
overlay.cancelled = true;
break;
}
TranslateMessage(*msg);
DispatchMessageW(*msg);
}
DestroyWindow(overlay.hwnd);
overlay.hwnd = null;
}
overlay_cancel :: () {
overlay.cancelled = true;
overlay.done = true;
}
// The selection, or on a click the screen under the cursor. A region too small to record is not
// taken: the overlay stays, to drag again.
overlay_take :: () {
selection := overlay_selection();
click := !overlay.dragging || overlay_is_click(selection);
if overlay.video && !click && overlay_too_small(selection) {
overlay.dragging = false;
overlay_update();
return;
}
if click {
selection = overlay_monitor_rect(overlay.cursor);
overlay.whole_screen = true;
}
overlay.dragging = false;
selection.left = clamp(selection.left, 0, overlay.width);
selection.right = clamp(selection.right, 0, overlay.width);
selection.top = clamp(selection.top, 0, overlay.height);
selection.bottom = clamp(selection.bottom, 0, overlay.height);
overlay.result = selection;
overlay.done = true;
}
// From the anchor to the cursor, both pixels in it. To record, on the screen the drag began on.
overlay_selection :: () -> RECT {
a, c := overlay.anchor, overlay.cursor;
bounds := ifx overlay.video then overlay_monitor_rect(a) else RECT.{0, 0, overlay.width, overlay.height};
c.x = clamp(c.x, bounds.left, bounds.right - 1);
c.y = clamp(c.y, bounds.top, bounds.bottom - 1);
return .{min(a.x, c.x), min(a.y, c.y), max(a.x, c.x) + 1, max(a.y, c.y) + 1};
}
overlay_is_click :: (selection: RECT) -> bool {
return selection.right - selection.left < CLICK_SLOP && selection.bottom - selection.top < CLICK_SLOP;
}
overlay_too_small :: (selection: RECT) -> bool {
return !record_size_allowed(selection.right - selection.left, selection.bottom - selection.top);
}
// The screen a point is on, in the overlay's pixels.
overlay_monitor_rect :: (point: POINT) -> RECT {
monitor := MonitorFromPoint(.{point.x + overlay.left, point.y + overlay.top}, MONITOR_DEFAULTTONEAREST);
info: MONITORINFO;
info.cbSize = size_of(MONITORINFO);
if !monitor || !GetMonitorInfoW(monitor, *info) return .{0, 0, overlay.width, overlay.height};
r := info.rcMonitor;
return .{r.left - overlay.left, r.top - overlay.top, r.right - overlay.left, r.bottom - overlay.top};
}
// Works out what to draw for the cursor and selection now, and has what changed drawn again: where
// each part was, and where it is.
overlay_update :: () {
layout: Overlay_Layout;
layout.valid = true;
layout.dragging = overlay.dragging;
layout.cursor = .{clamp(overlay.cursor.x, 0, overlay.width - 1), clamp(overlay.cursor.y, 0, overlay.height - 1)};
if layout.dragging {
layout.selection = overlay_selection();
layout.too_small = overlay.video && !overlay_is_click(layout.selection) && overlay_too_small(layout.selection);
}
text: string;
if layout.dragging text = tprint("% × %", layout.selection.right - layout.selection.left, layout.selection.bottom - layout.selection.top);
else text = tprint("%, %", layout.cursor.x + overlay.left, layout.cursor.y + overlay.top);
if layout.too_small text = tprint("% · at least % × %", text, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT);
wide := utf8_to_wide(text,, temp);
while wide && wide[layout.text_count] && layout.text_count < layout.text.count layout.text_count += 1;
memcpy(layout.text.data, wide, layout.text_count * size_of(u16));
// The magnifier below and right of the cursor, or on the other side where the screen ends.
zoom := overlay_zoom();
side := MAGNIFIER_PIXELS * zoom + 2; // and its frame
offset := cast(s32) (MAGNIFIER_OFFSET * overlay.scale);
padding := cast(s32) (4 * overlay.scale);
extent: SIZE;
GetTextExtentPoint32W(overlay.back_dc, layout.text.data, layout.text_count, *extent);
label_height := extent.cy + 2 * padding;
screen := overlay_monitor_rect(layout.cursor);
x := layout.cursor.x + offset;
y := layout.cursor.y + offset;
if x + side > screen.right x = layout.cursor.x - offset - side;
if y + side + label_height > screen.bottom y = layout.cursor.y - offset - side - label_height;
layout.magnifier = .{x, y, x + side, y + side};
label_width := max(side, extent.cx + 2 * padding);
layout.label = .{x, y + side, x + label_width, y + side + label_height};
previous := overlay.drawn;
overlay.drawn = layout;
if previous.valid overlay_invalidate(previous);
overlay_invalidate(layout);
}
overlay_zoom :: () -> s32 {
return max(cast(s32) (MAGNIFIER_ZOOM * overlay.scale + 0.5), 2);
}
overlay_invalidate :: (layout: Overlay_Layout) {
invalidate :: (r: RECT) {
rect := r;
InvalidateRect(overlay.hwnd, *rect, .FALSE);
}
border := overlay_border();
if layout.dragging {
s := layout.selection;
invalidate(.{s.left - border, s.top - border, s.right + border, s.bottom + border});
} else {
invalidate(.{0, layout.cursor.y, overlay.width, layout.cursor.y + 1});
invalidate(.{layout.cursor.x, 0, layout.cursor.x + 1, overlay.height});
}
invalidate(layout.magnifier);
invalidate(layout.label);
}
overlay_border :: () -> s32 {
return max(cast(s32) (BORDER * overlay.scale + 0.5), 1);
}
// Draws what has changed into the back bitmap, clipped to it, and puts it on screen.
overlay_paint :: () {
region := CreateRectRgn(0, 0, 0, 0);
if region GetUpdateRgn(overlay.hwnd, region, .FALSE);
paint: PAINTSTRUCT;
dc := BeginPaint(overlay.hwnd, *paint);
if region SelectClipRgn(overlay.back_dc, region);
overlay_draw(overlay.back_dc, overlay.drawn);
r := paint.rcPaint;
BitBlt(dc, r.left, r.top, r.right - r.left, r.bottom - r.top, overlay.back_dc, r.left, r.top, .SRCCOPY);
SelectClipRgn(overlay.back_dc, null);
if region DeleteObject(region);
EndPaint(overlay.hwnd, *paint);
}
overlay_draw :: (dc: HDC, layout: Overlay_Layout) {
fill :: (r: RECT, brush: HBRUSH) #expand {
rect := r;
FillRect(`dc, *rect, brush);
}
BitBlt(dc, 0, 0, overlay.width, overlay.height, overlay.dim_dc, 0, 0, .SRCCOPY);
if !layout.valid return;
border := overlay_border();
if layout.dragging {
s := layout.selection;
BitBlt(dc, s.left, s.top, s.right - s.left, s.bottom - s.top, overlay.screen_dc, s.left, s.top, .SRCCOPY);
// The outline just outside it, so every pixel taken shows as it is; dashed when too small to record.
outline := RECT.[
.{s.left - border, s.top - border, s.right + border, s.top},
.{s.left - border, s.bottom, s.right + border, s.bottom + border},
.{s.left - border, s.top, s.left, s.bottom},
.{s.right, s.top, s.right + border, s.bottom},
];
for outline {
if !layout.too_small { fill(it, overlay.accent_brush); continue; }
dash := max(cast(s32) (DASH * overlay.scale + 0.5), 2);
step := dash + max(cast(s32) (GAP * overlay.scale + 0.5), 2);
across := it.right - it.left > it.bottom - it.top;
start := ifx across then it.left else it.top;
end := ifx across then it.right else it.bottom;
position := start;
while position < end {
piece := it;
if across { piece.left = position; piece.right = min(position + dash, end); }
else { piece.top = position; piece.bottom = min(position + dash, end); }
fill(piece, overlay.accent_brush);
position += step;
}
}
} else {
fill(.{0, layout.cursor.y, overlay.width, layout.cursor.y + 1}, overlay.accent_brush);
fill(.{layout.cursor.x, 0, layout.cursor.x + 1, overlay.height}, overlay.accent_brush);
}
// The magnifier: the pixels around the cursor, as copied, each zoom pixels wide. Past the
// desktop's edges it is black.
m := layout.magnifier;
fill(m, overlay.frame_brush);
inner := RECT.{m.left + 1, m.top + 1, m.right - 1, m.bottom - 1};
fill(inner, overlay.black_brush);
zoom := overlay_zoom();
half :: MAGNIFIER_PIXELS / 2;
source := RECT.{layout.cursor.x - half, layout.cursor.y - half, layout.cursor.x + half + 1, layout.cursor.y + half + 1};
shown := RECT.{max(source.left, 0), max(source.top, 0), min(source.right, overlay.width), min(source.bottom, overlay.height)};
if shown.right > shown.left && shown.bottom > shown.top {
SetStretchBltMode(dc, .COLORONCOLOR);
StretchBlt(dc, inner.left + (shown.left - source.left) * zoom, inner.top + (shown.top - source.top) * zoom,
(shown.right - shown.left) * zoom, (shown.bottom - shown.top) * zoom,
overlay.screen_dc, shown.left, shown.top, shown.right - shown.left, shown.bottom - shown.top, .SRCCOPY);
}
// The pixel under the cursor, outlined.
c := RECT.{inner.left + half * zoom, inner.top + half * zoom, inner.left + (half + 1) * zoom, inner.top + (half + 1) * zoom};
fill(.{c.left - 1, c.top - 1, c.right + 1, c.top}, overlay.accent_brush);
fill(.{c.left - 1, c.bottom, c.right + 1, c.bottom + 1}, overlay.accent_brush);
fill(.{c.left - 1, c.top, c.left, c.bottom}, overlay.accent_brush);
fill(.{c.right, c.top, c.right + 1, c.bottom}, overlay.accent_brush);
// Under it, the position, or the selection's size.
fill(layout.label, overlay.label_brush);
SetBkMode(dc, TRANSPARENT);
SetTextColor(dc, TEXT_COLOR);
text := layout.text;
padding := cast(s32) (4 * overlay.scale);
TextOutW(dc, layout.label.left + padding, layout.label.top + padding, text.data, layout.text_count);
}
// The window's title and its program.
overlay_window_info :: (shot: *Screen_Shot, window: HWND) {
push_allocator(context.default_allocator);
title: [1024] u16;
count := GetWindowTextW(window, title.data, title.count);
if count > 0 {
text, ok := wide_to_utf8(title.data, count,, temp);
if ok shot.window_title = copy_string(text);
}
process_id: u32;
GetWindowThreadProcessId(window, *process_id);
if !process_id return;
process := OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, .FALSE, process_id);
if !process return;
defer CloseHandle(process);
path: [1024] u16;
length: u32 = path.count;
if !QueryFullProcessImageNameW(process, 0, path.data, *length) return;
text, ok := wide_to_utf8(path.data, cast(s32) length,, temp);
if !ok return;
shot.app_path = copy_string(text);
description := overlay_file_description(path.data);
shot.app_name = copy_string(ifx description then description else path_strip_extension(path_filename(text)));
}
// A program's description, as Task Manager shows it: "Google Chrome" for chrome.exe. "" when it has none. Temp.
overlay_file_description :: (path: *u16) -> string {
handle: u32;
size := GetFileVersionInfoSizeW(path, *handle);
if !size return "";
data := NewArray(size, u8, initialized = false,, temp);
if !GetFileVersionInfoW(path, 0, size, data.data) return "";
language := "040904b0"; // US English, Unicode: when it does not say
translation: *u16;
length: u32;
if VerQueryValueW(data.data, utf8_to_wide("\\VarFileInfo\\Translation",, temp), xx *translation, *length) != .FALSE && length >= 4 {
language = tprint("%1%2", formatInt(translation[0], base = 16, minimum_digits = 4), formatInt(translation[1], base = 16, minimum_digits = 4));
}
value: *u16;
key := utf8_to_wide(tprint("\\StringFileInfo\\%\\FileDescription", language),, temp);
if !VerQueryValueW(data.data, key, xx *value, *length) || length <= 1 return "";
text, ok := wide_to_utf8(value, cast(s32) (length - 1),, temp); // length counts the terminator
return ifx ok then trim(text) else "";
}
// ---------------------------------------------------------------- bindings
gdi32 :: #library,system "gdi32";
version_dll :: #library,system "version";
HFONT :: HANDLE;
HRGN :: HANDLE;
PAINTSTRUCT :: struct {
hdc: HDC;
fErase: BOOL;
rcPaint: RECT;
fRestore: BOOL;
fIncUpdate: BOOL;
rgbReserved: [32] u8;
}
CAPTUREBLT :: 0x40000000;
GA_ROOT :: 2;
WA_INACTIVE :: 0;
WM_ACTIVATE :: 0x0006;
TRANSPARENT :: 1;
CLEARTYPE_QUALITY :: 5;
PROCESS_QUERY_LIMITED_INFORMATION :: 0x1000;
GetMessageW :: (msg: *MSG, hwnd: HWND, filter_min: u32, filter_max: u32) -> s32 #foreign user32;
BeginPaint :: (hwnd: HWND, paint: *PAINTSTRUCT) -> HDC #foreign user32;
EndPaint :: (hwnd: HWND, paint: *PAINTSTRUCT) -> BOOL #foreign user32;
FillRect :: (dc: HDC, rect: *RECT, brush: HBRUSH) -> s32 #foreign user32;
GetUpdateRgn :: (hwnd: HWND, region: HRGN, erase: BOOL) -> s32 #foreign user32;
GetDpiForWindow :: (hwnd: HWND) -> u32 #foreign user32;
GetWindowTextW :: (hwnd: HWND, text: *u16, max_count: s32) -> s32 #foreign user32;
GetWindowThreadProcessId :: (hwnd: HWND, process_id: *u32) -> u32 #foreign user32;
WindowFromPoint :: (point: POINT) -> HWND #foreign user32;
GetAncestor :: (hwnd: HWND, flags: u32) -> HWND #foreign user32;
capture_set_cursor_pos :: (x: s32, y: s32) -> BOOL #foreign user32 "SetCursorPos";
CreateDIBSection :: (dc: HDC, info: *BITMAPINFO, usage: u32, bits: **void, section: HANDLE, offset: u32) -> HBITMAP #foreign gdi32;
CreateRectRgn :: (left: s32, top: s32, right: s32, bottom: s32) -> HRGN #foreign gdi32;
SelectClipRgn :: (dc: HDC, region: HRGN) -> s32 #foreign gdi32;
SetTextColor :: (dc: HDC, color: u32) -> u32 #foreign gdi32;
SetBkMode :: (dc: HDC, mode: s32) -> s32 #foreign gdi32;
TextOutW :: (dc: HDC, x: s32, y: s32, text: *u16, count: s32) -> BOOL #foreign gdi32;
GetTextExtentPoint32W :: (dc: HDC, text: *u16, count: s32, size: *SIZE) -> BOOL #foreign gdi32;
CreateFontW :: (height: s32, width: s32, escapement: s32, orientation: s32, weight: s32, italic: u32, underline: u32,
strike_out: u32, char_set: u32, out_precision: u32, clip_precision: u32, quality: u32,
pitch_and_family: u32, face: *u16) -> HFONT #foreign gdi32;
OpenProcess :: (access: u32, inherit: BOOL, process_id: u32) -> HANDLE #foreign kernel32;
QueryFullProcessImageNameW :: (process: HANDLE, flags: u32, name: *u16, size: *u32) -> BOOL #foreign kernel32;
GlobalFree :: (memory: HGLOBAL) -> HGLOBAL #foreign kernel32;
GetFileVersionInfoSizeW :: (path: *u16, handle: *u32) -> u32 #foreign version_dll;
GetFileVersionInfoW :: (path: *u16, handle: u32, length: u32, data: *void) -> BOOL #foreign version_dll;
VerQueryValueW :: (block: *void, sub_block: *u16, buffer: **void, length: *u32) -> BOOL #foreign version_dll;

238
src/app/screen_record.jai
View File

@ -1,238 +0,0 @@
// Screen recording (video-capture-plan.html): Ctrl+PrintScreen (the record_region global hotkey, from
// any app, even with mcap hidden in the tray) freezes the screens as PrintScreen does, in the
// overlay's video mode (app/screen_capture_win32.jai): a region of one screen, at least 240 x 180, or
// a click for the whole screen. It is recorded with modules/Screen_Record to an MP4 in the temporary
// folder until Ctrl+PrintScreen is pressed again. The MP4 then goes into the library as a video .mcap
// in SCREEN_CAPTURE_FOLDER, beside the stills, through app/screen_capture.jai's jobs: the MP4 as its
// original, a frame from it as its rendered image (which OCR and labels read), the window it was
// recorded from in metadata.json, as for a still, and how it was recorded in capture.json.
//
// A recording that stops by itself (the screen went away, the encoder failed) keeps what it has, and
// so does quitting while recording. PrintScreen while recording takes the screen under the cursor at
// once, without the overlay, whose frozen screen would be in the recording.
//
// Windows only, as modules/Screen_Record is.
SCREEN_RECORD_FPS :: 30;
screen_record: struct {
recording: bool;
flag_started: bool; // -record has started its recording
flag_start: Apollo_Time;
shot: Screen_Shot; // the region and the window it is of; no pixels
date: string; // when it started, as CaptureDate; heap
path: string; // the MP4 being written; heap
#if OS == .WINDOWS recorder: Recorder;
}
// From app_frame when the hotkey fired, outside the ImGui frame: starts recording, or stops.
screen_record_toggle :: () {
if screen_record.recording screen_record_stop();
else screen_record_start();
}
// Every frame (from screen_capture_frame): a recording that stopped by itself is finished and kept.
screen_record_frame :: () {
if app.record_and_exit > 0 screen_record_flag_frame();
#if OS == .WINDOWS {
if !screen_record.recording return;
failed, why := record_failed(*screen_record.recorder);
if failed {
toast_show(.Error, "The recording stopped: %", why);
screen_record_stop();
}
}
}
// -record SECONDS: the screen under the cursor, without the overlay, for that long; then
// screen_record_saved ends the app once it is in the library. With -record-quit, the app is quit while
// still recording instead, as quitting from the tray would: the shutdown keeps the recording.
screen_record_flag_frame :: () {
if !screen_record.flag_started {
if viewer.catalog_loaded < viewer.captures.count return; // the library is up, so the grid can take it
screen_record.flag_started = true;
screen_record.flag_start = current_time_monotonic();
screen_record_start(immediate = true);
if !screen_record.recording {
log_error("[record] -record could not start.");
app.exit_code = 1; app.should_exit = true; app.record_and_exit = 0;
}
return;
}
if screen_record.recording && to_float64_seconds(current_time_monotonic() - screen_record.flag_start) >= app.record_and_exit {
if array_find(app.args, "-record-quit") { log("[record] -record-quit: quitting while recording."); app.should_exit = true; app.record_and_exit = 0; }
else screen_record_stop();
}
}
// A recording is in the library (path) or could not be put there (error): -record's end.
screen_record_saved :: (path: string, error: string) {
if app.record_and_exit <= 0 return;
if error log_error("[record] -record failed: %", error);
else log("[record] -record is done: %", path);
app.exit_code = ifx error then cast(s32) 1 else 0;
app.should_exit = true;
app.record_and_exit = 0;
}
// On the shutdown thread: a recording running is stopped and kept.
screen_record_shutdown :: () {
if screen_record.recording screen_record_stop(quiet = true);
}
#if OS == .WINDOWS {
// immediate: the screen under the cursor, without the overlay (-record).
screen_record_start :: (immediate := false) {
if app_exit_shutdown_started() return;
date := import_calendar_text(to_calendar(current_time_consensus(), .LOCAL));
shot, ok, error := platform_screen_capture_select(video = true, immediate = immediate);
if !ok {
if error { log_error("[record] %.", error); toast_show(.Error, "Screen recording: %", error); }
else log("[record] Cancelled.");
return;
}
folder := tprint("%/%", windows_expand_environment("%TEMP%"), APP_FOLDER_NAME);
make_directory_if_it_does_not_exist(folder, recursive = true);
stamp := replace(replace(date, " ", "_",, temp), ":", "-",, temp);
path := tprint("%/recording-%.mp4", folder, stamp);
number := 2;
while file_exists(path) { path = tprint("%/recording-%_%.mp4", folder, stamp, number); number += 1; }
options: Record_Options;
options.path = path;
options.region = .{shot.left, shot.top, shot.left + shot.width, shot.top + shot.height};
options.fps = SCREEN_RECORD_FPS;
started, start_error := record_start(*screen_record.recorder, options);
if !started {
log_error("[record] Could not record % x % at %, %: %", shot.width, shot.height, shot.left, shot.top, start_error);
toast_show(.Error, "Could not record: %", start_error);
screen_shot_free(*shot);
return;
}
stats := record_stats(*screen_record.recorder);
screen_record.recording = true;
screen_record.shot = shot;
screen_record.date = copy_string(date,, context.default_allocator);
screen_record.path = copy_string(path,, context.default_allocator);
if stats.fell_back_from log("[record] Desktop Duplication could not run: %", stats.fell_back_from);
log("[record] Recording % x % at %, % (%) of \"%\" (%) with % and %, to %.", stats.width, stats.height, shot.left, shot.top,
ifx shot.whole_screen then "the whole screen" else "a region", shot.window_title, shot.app_name, stats.source, stats.encoder, path);
toast_show(.Success, "Recording % x %: % stops", stats.width, stats.height, screen_record_hotkey_text());
}
// Stops, and hands the MP4 to a job that puts it in the library. quiet: no toasts (shutdown).
screen_record_stop :: (quiet := false) {
if !screen_record.recording return;
screen_record.recording = false;
stopped, stop_error := record_stop(*screen_record.recorder);
stats := screen_record.recorder.stats;
if !stopped log_error("[record] %; keeping what was recorded.", stop_error);
log("[record] Stopped after % s: % frames written, % dropped, % KB.", FF(stats.seconds, 1), stats.written, stats.dropped, stats.bytes / 1024);
if !stats.written {
log_error("[record] Nothing was recorded; % is deleted.", screen_record.path);
screen_record_saved("", "nothing was recorded");
if !quiet toast_show(.Error, "Nothing was recorded");
file_delete(screen_record.path);
screen_shot_free(*screen_record.shot);
free(screen_record.date,, context.default_allocator);
free(screen_record.path,, context.default_allocator);
screen_record.date, screen_record.path = "", "";
return;
}
job := New(Screen_Capture_Job,, context.default_allocator);
job.video = true;
job.shot = screen_record.shot;
job.shot.width, job.shot.height = stats.width, stats.height; // as recorded: rounded down to even
job.date = screen_record.date;
job.recording = screen_record.path;
job.encoder = copy_string(stats.encoder,, context.default_allocator);
job.source = copy_string(stats.source,, context.default_allocator);
job.dropped = stats.dropped;
screen_record.shot = .{};
screen_record.date, screen_record.path = "", "";
screen_capture_job_start(job);
}
// The recording's .mcap in the library, from the job's MP4, which is deleted once it is in: its path,
// or why it could not be (temp). On the job's thread.
screen_record_write :: (job: *Screen_Capture_Job) -> path: string, error: string {
_, size, found := file_modtime_and_size(job.recording);
if !found || size <= 0 return "", tprint("% is missing", job.recording);
image, video, decoded, decode_error := video_poster(job.recording);
if !decoded return "", tprint("the recording cannot be read back: %", decode_error);
defer image_free(*image);
preview, preview_width, preview_height, preview_ok, preview_error := image_preview_png(image, MCAP_PREVIEW_MAX_SIDE);
if !preview_ok return "", preview_error;
poster, _, _, poster_ok, poster_error := image_preview_png(image, max(image.width, image.height)); // at its own size
if !poster_ok return "", poster_error;
hash, hashed := import_hash_file(.{path = job.recording, offset = 0, size = size});
if !hashed return "", tprint("% cannot be read", job.recording);
shot := job.shot;
doc := New(Capture_Document,, temp);
capture_document_init(doc, temp);
doc.comment = tprint("Recorded by % %.", APP_NAME, VERSION);
metadata := capture_add_file(doc, "metadata.json");
import_add(doc, metadata, "AppName", import_text(shot.app_name));
import_add(doc, metadata, "CaptureDate", job.date);
import_add(doc, metadata, "WindowName", import_text(shot.window_title));
info := capture_add_file(doc, "capture.json");
import_add(doc, info, "Region", ifx shot.whole_screen then "screen" else "selection");
import_add_number(doc, info, "Left", xx shot.left);
import_add_number(doc, info, "Top", xx shot.top);
import_add_number(doc, info, "Width", xx shot.width);
import_add_number(doc, info, "Height", xx shot.height);
if shot.app_path import_add(doc, info, "AppPath", import_text(shot.app_path));
import_add(doc, info, "ContentHash", import_hash_text(hash));
import_add(doc, info, "Format", "MP4");
import_add_number(doc, info, "Duration", import_milliseconds(video.duration));
if video.frame_rate > 0 import_add_number(doc, info, "FrameRate", video.frame_rate);
import_add(doc, info, "VideoCodec", import_text(video.codec));
import_add_number(doc, info, "PosterTime", import_milliseconds(video.poster_time));
import_add(doc, info, "Encoder", import_text(job.encoder));
import_add(doc, info, "Capture", import_text(job.source));
import_add_number(doc, info, "DroppedFrames", xx job.dropped);
name := import_out_name(job.date, hash);
text, text_ok := import_document_text(doc, name);
if !text_ok return "", "its Capture Text does not read back cleanly (see the Event Log)";
m: Mcap;
m.text = text;
m.preview = .{width = cast(u16) preview_width, height = cast(u16) preview_height, png = preview};
m.original_name = tprint("%.mp4", path_strip_extension(name));
m.rendered_kind = .OWN;
m.rendered = poster;
path, folder_error := screen_capture_out_path(name);
if folder_error return "", folder_error;
written, write_error := mcap_write_streamed(path, m, .{path = job.recording, offset = 0, size = size});
if !written return "", write_error;
file_delete(job.recording);
return path, "";
}
// The record_region hotkey as it is bound, for the toast: "Ctrl+PrintScreen". Temp.
screen_record_hotkey_text :: () -> string {
if !record_region_hotkey.binding_count return "the record_region hotkey";
return to_string(record_region_hotkey.bindings[0]);
}
} else {
screen_record_start :: (immediate := false) {
toast_show(.Error, "Screen recording is not available on this system yet");
if app.record_and_exit > 0 screen_record_saved("", "screen recording is Windows only");
}
screen_record_stop :: (quiet := false) {}
screen_record_write :: (job: *Screen_Capture_Job) -> path: string, error: string {
return "", "screen recording is Windows only";
}
}

2
src/base/platform/platform_windows.jai
View File

@ -298,7 +298,6 @@ platform_apply_global_hotkeys :: (actions: [] Hotkey_Action) { using platform;
id := global_hotkey_id(action_index, binding_index);
if !RegisterHotKey(hwnd, id, modifiers, vk) {
log_error("[hotkeys] Could not register the global hotkey % (%); another program may be using it.", to_string(combo), action.name);
if vk == VK_SNAPSHOT log_error("[hotkeys] PrintScreen is taken by Snagit when it runs, and by Windows when Settings > Accessibility > Keyboard > \"Use the Print screen key to open screen capture\" is on.");
array_add(*failed, to_string(combo));
continue;
}
@ -324,7 +323,6 @@ win32_virtual_key :: (key: ImGui.Key) -> u32 {
if k >= cast(s64) ImGui.Key._0 && k <= cast(s64) ImGui.Key._9 return cast(u32) (#char "0" + k - cast(s64) ImGui.Key._0);
if k >= cast(s64) ImGui.Key.F1 && k <= cast(s64) ImGui.Key.F24 return cast(u32) (VK_F1 + k - cast(s64) ImGui.Key.F1);
if key == {
case .PrintScreen; return VK_SNAPSHOT;
case .Space; return VK_SPACE;
case .Enter; return VK_RETURN;
case .Tab; return VK_TAB;

158
src/test/import_tests.jai
View File

@ -7,7 +7,7 @@
checks_run := 0;
checks_failed := 0;
MAX_GROUPS :: 16;
MAX_GROUPS :: 12;
group_names : [MAX_GROUPS] string;
group_checks : [MAX_GROUPS] int;
group_fails : [MAX_GROUPS] int;
@ -561,10 +561,7 @@ segment_overlay_tests :: () {
expect(same(segment_class_colour("person", 0), segment_class_default_colour("person")), "other classes keep their own");
segment_colour_set("dog", .[0, 0, 0], default = true, save = false);
expect(same(segment_class_colour("dog", 0), own), "back to its own colour");
// Only dog's choice is gone: the colours chosen in the app's own data folder are loaded too.
dog_chosen := false;
for segment_ui.overlay_state.colours if it.class == "dog" dog_chosen = true;
expect(!dog_chosen, "and its choice is gone");
expect(!segment_ui.overlay_state.colours.count, "and nothing chosen is left");
hidden := segment_overlay_pixels(*overlay, .["person"], 0.5);
expect(pixel(hidden, 0, 0)[3] == 0 && pixel(hidden, 2, 1)[3] == 0, "a hidden class is clear");
@ -603,155 +600,6 @@ segment_overlay_tests :: () {
file_delete(bare);
}
// A screen capture written into the library (app/screen_capture.jai): a PNG of its pixels, a preview,
// and the window it came from where search reads it.
screen_capture_tests :: () {
group("screen capture");
#if OS == .WINDOWS library := tprint("%/mcap-tests/library", windows_expand_environment("%TEMP%"));
else library := "/tmp/mcap-tests/library";
saved_library := app.library_folder;
app.library_folder = library;
defer app.library_folder = saved_library;
WIDTH :: 700;
HEIGHT :: 300;
shot: Screen_Shot;
shot.width, shot.height = WIDTH, HEIGHT;
shot.left, shot.top = -1920, 40; // on a screen left of the primary one
shot.pixels = NewArray(WIDTH * HEIGHT * 4, u8,, temp);
for y: 0..HEIGHT - 1 for x: 0..WIDTH - 1 {
p := shot.pixels.data + (y * WIDTH + x) * 4;
p[0], p[1], p[2], p[3] = cast,trunc(u8) x, cast,trunc(u8) y, 200, 255;
}
shot.window_title = "Assign Footprints \x01- KiCad"; // a control character, as titles can hold
shot.app_name = "KiCad";
shot.app_path = "C:/Program Files/KiCad/bin/kicad.exe";
path, error := screen_capture_write(shot, "2026-10-10 14:05:09");
expect(!error, "written: %", error);
expect(begins_with(path, tprint("%/%/2026-10-10_14-05-09_", library, SCREEN_CAPTURE_FOLDER)), "named by its date, in its folder: %", path);
again, again_error := screen_capture_write(shot, "2026-10-10 14:05:09");
expect(!again_error && again != path && ends_with(again, "_2.mcap"), "the same again gets a name of its own: %", again);
m, parsed := mcap_parse(cast([] u8) read_entire_file(path,, temp));
expect(parsed, "it reads back");
if parsed {
expect(m.rendered_kind == .NONE && ends_with(m.original_name, ".png"), "the original is a PNG: %", m.original_name);
image, decoded := image_decode(m.original);
expect(decoded && image.width == WIDTH && image.height == HEIGHT, "at its own size");
if decoded {
expect(memcmp(image.pixels, shot.pixels.data, shot.pixels.count) == 0, "every pixel as it was");
image_free(*image);
}
expect(max(m.preview.width, m.preview.height) == MCAP_PREVIEW_MAX_SIDE, "a preview: % x %", m.preview.width, m.preview.height);
text :: (object: Capture_Value, key: string) -> string {
value := capture_object_find(*object, key);
return ifx value && value.kind == .STRING then value.text else "";
}
number :: (object: Capture_Value, key: string) -> float64 {
value := capture_object_find(*object, key);
return ifx value && value.kind == .NUMBER then value.number else 0;
}
doc: Capture_Document;
capture_document_init(*doc, temp);
expect(capture_text_parse(*doc, path, m.text), "its Capture Text reads");
metadata := capture_find_file(*doc, "metadata.json");
capture := capture_find_file(*doc, "capture.json");
expect(metadata && capture, "metadata.json and capture.json");
if metadata {
expect(text(metadata.root, "WindowName") == "Assign Footprints - KiCad", "WindowName: %", text(metadata.root, "WindowName"));
expect(text(metadata.root, "AppName") == "KiCad", "AppName");
expect(text(metadata.root, "CaptureDate") == "2026-10-10 14:05:09", "CaptureDate");
}
if capture {
expect(number(capture.root, "Left") == -1920 && number(capture.root, "Top") == 40, "where on the desktop");
expect(text(capture.root, "Region") == "selection", "a selection");
}
}
file_delete(path);
file_delete(again);
}
// A screen recording written into the library (app/screen_record.jai): the MP4 as its original, a
// frame from it as its rendered image, and capture.json saying it is a video, as the catalog reads it.
// The MP4 is made here with modules/Screen_Record's encoder, from frames of a changing colour.
screen_record_tests :: () {
group("screen recording");
#if OS == .WINDOWS {
expect(record_size_allowed(240, 180), "240 x 180 may be recorded");
expect(!record_size_allowed(239, 180) && !record_size_allowed(240, 179), "smaller may not");
expect(record_size_allowed(3840, 2160), "a 4K screen may");
folder := tprint("%/mcap-tests", windows_expand_environment("%TEMP%"));
library := tprint("%/library", folder);
saved_library := app.library_folder;
app.library_folder = library;
defer app.library_folder = saved_library;
make_directory_if_it_does_not_exist(folder, recursive = true);
WIDTH :: 320;
HEIGHT :: 240;
FRAMES :: 30;
mp4 := tprint("%/recording-test.mp4", folder);
encoder: Encoder;
opened, open_error := encoder_open(*encoder, mp4, WIDTH, HEIGHT, 30, "");
expect(opened, "an encoder opened: %", open_error);
if !opened return;
pixels := NewArray(WIDTH * HEIGHT * 4, u8,, temp);
for frame: 0..FRAMES - 1 {
for i: 0..WIDTH * HEIGHT - 1 {
pixels[i * 4 + 0] = cast,trunc(u8) (frame * 8);
pixels[i * 4 + 1] = cast,trunc(u8) (i % WIDTH);
pixels[i * 4 + 2] = 128;
pixels[i * 4 + 3] = 255;
}
encoder_write(*encoder, pixels.data, frame * 3000); // 30 a second, in 90 kHz ticks
}
encoder_name := copy_string(encoder.name,, temp);
closed, close_error := encoder_close(*encoder);
expect(closed, "the MP4 finished: %", close_error);
_, mp4_size, _ := file_modtime_and_size(mp4);
job: Screen_Capture_Job;
job.video = true;
job.recording = mp4;
job.date = "2026-10-10 15:20:00";
job.encoder = encoder_name;
job.source = "DXGI DuplicateOutput1 on a test";
job.shot.width, job.shot.height = WIDTH, HEIGHT;
job.shot.left, job.shot.top = 100, 200;
job.shot.window_title = "Recording test - KiCad";
job.shot.app_name = "KiCad";
path, error := screen_record_write(*job);
expect(!error, "written: %", error);
if error return;
expect(begins_with(path, tprint("%/%/2026-10-10_15-20-00_", library, SCREEN_CAPTURE_FOLDER)), "named by its date, in its folder: %", path);
expect(!file_exists(mp4), "the MP4 is deleted once it is in the library");
m, parsed := mcap_parse(cast([] u8) read_entire_file(path,, temp));
expect(parsed, "it reads back");
if parsed {
expect(m.original.count == mp4_size && ends_with(m.original_name, ".mp4"), "the MP4 is its original: % of % bytes, %", m.original.count, mp4_size, m.original_name);
expect(m.rendered_kind == .OWN, "a frame of it is its rendered image");
poster, decoded := image_decode(m.rendered);
expect(decoded && poster.width == WIDTH && poster.height == HEIGHT, "at the recording's size");
if decoded image_free(*poster);
}
entry, entry_error := catalog_load(path);
expect(entry != null, "catalogued: %", entry_error);
if entry {
expect(entry.media == .VIDEO, "a video, from capture.json");
expect(entry.video_codec == "h264", "H.264: %", entry.video_codec);
expect(entry.duration > 0.9 && entry.duration < 1.1, "about a second: %", entry.duration);
expect(entry.title == "Recording test - KiCad" && entry.app == "KiCad", "the window it was recorded from");
catalog_entry_free(entry);
}
file_delete(path);
}
}
// What mcap_chunks_size counts, here where it cannot be seen.
mcap_chunks_size_for_test :: (chunks: [] Mcap_Chunk) -> s64 {
size: s64 = 4;
@ -775,7 +623,5 @@ run_import_tests :: (builder: *String_Builder = null) -> bool {
segment_chunk_tests();
library_folders_tests();
segment_overlay_tests();
screen_capture_tests();
screen_record_tests();
return report();
}

11
src/ui/imgui_keyboard.jai
View File

@ -308,18 +308,12 @@ hotkeys_missing_lines :: (actions: [] Hotkey_Action, bindings: [] Hotkey_Binding
// Registered with the OS, so they fire from any app: RegisterHotKey on Windows, Carbon's
// RegisterEventHotKey on macOS, none on Linux. They are never matched against ImGui's key state.
bring_to_foreground_hotkey: Hotkey;
capture_region_hotkey: Hotkey;
record_region_hotkey: Hotkey;
GLOBAL_HOTKEY_BRING_TO_FOREGROUND :: 0; // indices into global_hotkey_actions
GLOBAL_HOTKEY_CAPTURE_REGION :: 1;
GLOBAL_HOTKEY_RECORD_REGION :: 2;
GLOBAL_HOTKEY_ACTION_COUNT :: 3; // sizes the platforms' registration arrays
GLOBAL_HOTKEY_ACTION_COUNT :: 1; // sizes the platforms' registration arrays
global_hotkey_actions_build :: (out: *[..] Hotkey_Action) {
array_add(out, .{"bring_to_foreground", *bring_to_foreground_hotkey, false});
array_add(out, .{"capture_region", *capture_region_hotkey, true}); // app/screen_capture.jai
array_add(out, .{"record_region", *record_region_hotkey, true}); // app/screen_record.jai
}
global_hotkey_actions: [..] Hotkey_Action;
@ -375,7 +369,6 @@ global_hotkeys_install :: (bindings: [] Hotkey_Binding) {
for global_hotkey_actions it.hotkey.binding_count = 0;
for bindings {
if it.explicit_none continue;
if OS != .WINDOWS && global_hotkey_actions[it.action].windows_only continue;
hotkey := global_hotkey_actions[it.action].hotkey;
if hotkey.binding_count == MAX_HOTKEY_BINDINGS {
log("[settings] % already has % bindings; ignoring %.", global_hotkey_actions[it.action].name, MAX_HOTKEY_BINDINGS, to_string(it.combo));
@ -410,8 +403,6 @@ global_hotkey_fired :: (id: s64) #c_call {
if id < 1 return;
action := (id - 1) / MAX_HOTKEY_BINDINGS;
if action == GLOBAL_HOTKEY_BRING_TO_FOREGROUND app.restore_window_this_frame = true;
if action == GLOBAL_HOTKEY_CAPTURE_REGION app.capture_region_this_frame = true;
if action == GLOBAL_HOTKEY_RECORD_REGION app.record_region_this_frame = true;
}
// Parses the lines of section in data, or every line when data has no such header. Returns the

399
tools/record_probe.jai
View File

@ -1,399 +0,0 @@
// record-probe: phase 1 of video-capture-plan.html. Records a screen, or a region of one, with
// modules/Screen_Record (DXGI Desktop Duplication and FFmpeg) and says how it went: which screens
// there are and on which adapter, which duplication call and encoder worked, frames captured, written
// and dropped, time per stage, and the file's size. It also answers what the plan leaves to phase 1:
//
// -capture HOW auto (Desktop Duplication, else Windows.Graphics.Capture), dxgi or wgc.
// -border-test counts yellow pixels along the screen's edges, with GDI, before recording and a
// second in: does Windows draw its capture border (Windows.Graphics.Capture)?
// -keep-border leaves Windows.Graphics.Capture's border on: the border test's control.
// -simulate-unsupported Desktop Duplication fails as on a laptop with two GPUs: the fallback's path.
// -poll Windows.Graphics.Capture polled, as when it will not take its frame handler.
// -cursor-test puts the mouse in the middle of the region while recording, and back after;
// with -snapshot, cursor.png is the frame around it, four times as large.
// -legacy the plain DuplicateOutput, not DuplicateOutput1: does it fail for a process aware
// of each monitor's DPI, as mcap is (this probe is, unless -dpi-unaware)?
// -cuda makes a CUDA context first, as onnxruntime does in mcap: does duplication still work?
// -exclude-test puts two squares on the region, one kept out of captures with
// WDA_EXCLUDEFROMCAPTURE: is it out of the recording, and the other in?
// -animate moves a square across the region the whole time, so the screen changes every
// frame: for measuring throughput.
// -check FILE reads FILE back with FFmpeg and says what it holds, and nothing else: for a
// recording cut short by a crash.
// -snapshot DIR the last frame captured (captured.png) and a frame decoded back from the file
// (decoded.png), to compare by eye.
//
// Build: jai first.jai - record_probe
// Run: bin/record-probe [-list] [-screen N] [-region X Y W H] [-seconds S] [-fps F] [-encoder NAME]
// [-out FILE.mp4] [-no-cursor] [-legacy] [-cuda] [-exclude-test] [-snapshot DIR]
// [-animate] [-capture auto|dxgi|wgc] [-border-test] [-keep-border] [-cursor-test]
// [-simulate-unsupported] [-poll] [-dpi-unaware]
// Exit code: 0 recorded, 1 failed, 3 Desktop Duplication cannot run here (the fallback's case).
main :: () {
args := get_command_line_arguments();
options: Record_Options;
options.path = tprint("%/record-probe.mp4", get_working_directory());
seconds: float64 = 10;
list_only, cuda, exclude_test, dpi_unaware, animate, border_test, cursor_test := false, false, false, false, false, false, false;
snapshot_folder: string;
i := 1;
number :: (index: s64) -> s32 #expand { return cast(s32) string_to_int(`args[index]); }
while i < args.count {
a := args[i];
more := args.count - i - 1;
if a == "-list" { list_only = true; i += 1; continue; }
if a == "-screen" && more >= 1 { options.screen = number(i + 1); i += 2; continue; }
if a == "-region" && more >= 4 {
x, y, w, h := number(i + 1), number(i + 2), number(i + 3), number(i + 4);
options.region = .{x, y, x + w, y + h};
i += 5; continue;
}
if a == "-seconds" && more >= 1 { seconds = string_to_float64(args[i + 1]); i += 2; continue; }
if a == "-fps" && more >= 1 { options.fps = number(i + 1); i += 2; continue; }
if a == "-encoder" && more >= 1 { options.encoder = args[i + 1]; i += 2; continue; }
if a == "-out" && more >= 1 { options.path = args[i + 1]; i += 2; continue; }
if a == "-snapshot" && more >= 1 { snapshot_folder = args[i + 1]; i += 2; continue; }
if a == "-no-cursor" { options.cursor = false; i += 1; continue; }
if a == "-legacy" { options.legacy_duplicate = true; i += 1; continue; }
if a == "-cuda" { cuda = true; i += 1; continue; }
if a == "-exclude-test" { exclude_test = true; i += 1; continue; }
if a == "-dpi-unaware" { dpi_unaware = true; i += 1; continue; }
if a == "-animate" { animate = true; i += 1; continue; }
if a == "-border-test" { border_test = true; i += 1; continue; }
if a == "-keep-border" { options.testing |= .KEEP_BORDER; i += 1; continue; }
if a == "-simulate-unsupported" { options.testing |= .DXGI_UNSUPPORTED; i += 1; continue; }
if a == "-poll" { options.testing |= .WGC_POLL; i += 1; continue; }
if a == "-cursor-test" { cursor_test = true; i += 1; continue; }
if a == "-capture" && more >= 1 {
how := args[i + 1];
if how == "auto" options.capture = .AUTO;
else if how == "dxgi" options.capture = .DXGI;
else if how == "wgc" options.capture = .WGC;
else { print("-capture takes auto, dxgi or wgc\n"); exit(2); }
i += 2; continue;
}
if a == "-check" && more >= 1 {
image, info, ok, error := video_poster(args[i + 1]);
if !ok { print("%: cannot be read: %\n", args[i + 1], error); exit(1); }
print("%: % x %, % s, %, % fps; a frame decoded at % s\n", args[i + 1], info.width, info.height, FF(info.duration, 2), info.codec, FF(info.frame_rate, 1), FF(info.poster_time, 2));
exit(0);
}
print("Unknown argument %\n", a);
exit(2);
}
if !dpi_unaware {
DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 :: cast(HANDLE) -4;
if !SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2) {
error := GetLastError();
ERROR_ACCESS_DENIED :: 5;
if error != ERROR_ACCESS_DENIED print("Could not become per-monitor DPI aware (v2): error %\n", error);
// Access denied: the awareness was set already, as it can be once only. dxgi_open and wgc_open
// refuse when the desktop they are told of is scaled.
}
}
print("DPI awareness: %\n", ifx dpi_unaware then "none" else "per monitor (v2), as mcap");
screens := record_list_screens();
print("Screens:\n");
for screens {
d := it.desktop;
print(" %: % on adapter % (%): % x % at %, %%\n", it_index, it.name, it.adapter_index, it.adapter,
d.right - d.left, d.bottom - d.top, d.left, d.top,
ifx it.rotation == .IDENTITY || it.rotation == .UNSPECIFIED then "" else tprint(", rotated %", it.rotation));
}
if list_only return;
if cuda {
ok, text := cuda_make_context();
print("CUDA: %\n", text);
if !ok exit(1);
}
border_screen := screens[ifx options.screen >= 0 then options.screen else 0].desktop;
border_before: s64;
if border_test border_before = yellow_edge_pixels(border_screen);
recorder: Recorder;
ok, error, fall_back := record_start(*recorder, options);
if !ok {
print("Cannot record: %\n", error);
if fall_back print("Desktop Duplication cannot run here: this is the case for the Windows.Graphics.Capture fallback.\n");
exit(ifx fall_back then cast(s32) 3 else 1);
}
stats := record_stats(*recorder);
if stats.fell_back_from print("Desktop Duplication could not run (%), so:\n", stats.fell_back_from);
print("Recording % x % for % s at up to % fps with % and %, to %\n", stats.width, stats.height, seconds, options.fps, stats.source, stats.encoder, options.path);
region := options.region;
if region.right <= region.left region = screens[ifx options.screen >= 0 then options.screen else 0].desktop;
excluded, control: HWND;
if exclude_test excluded, control = exclusion_windows(region);
saved_cursor: POINT;
cursor_at: POINT;
if cursor_test {
GetCursorPos(*saved_cursor);
cursor_at = .{region.left + (region.right - region.left) / 2, region.top + (region.bottom - region.top) / 2};
SetCursorPos(cursor_at.x, cursor_at.y);
}
mover: HWND;
if animate mover = square_window("record-probe animate", 0x0000A5FF, region.left, region.top, 120);
start := current_time_monotonic();
last_print := start;
border_during: s64 = -1;
while to_float64_seconds(current_time_monotonic() - start) < seconds {
pump_messages();
if mover { // back and forth across the region, about 240 times a second
elapsed := to_float64_seconds(current_time_monotonic() - start);
span := region.right - region.left - 120;
phase := fmod_cycling(elapsed * 0.5, 2.0);
x := region.left + cast(s32) (span * ifx phase < 1 then phase else 2 - phase);
SetWindowPos(mover, HWND_TOPMOST, x, region.top + (region.bottom - region.top) / 2 - 60, 0, 0, SWP_NOSIZE | SWP_NOACTIVATE);
sleep_milliseconds(4);
} else {
sleep_milliseconds(20);
}
if to_float64_seconds(current_time_monotonic() - last_print) >= 1 {
last_print = current_time_monotonic();
s := record_stats(*recorder);
print(" % s: % captured, % sent (% repeats), % written, % dropped, % lost, % KB\n",
FF(to_float64_seconds(last_print - start), 0), s.captured, s.emitted, s.repeats, s.written, s.dropped, s.lost, s.bytes / 1024);
}
if border_test && border_during < 0 && to_float64_seconds(current_time_monotonic() - start) >= 1.5
border_during = yellow_edge_pixels(border_screen);
failed, _ := record_failed(*recorder);
if failed break;
}
last, width, height := record_last_frame(*recorder);
if mover DestroyWindow(mover);
if cursor_test SetCursorPos(saved_cursor.x, saved_cursor.y);
stopped, stop_error := record_stop(*recorder);
s := recorder.stats;
print("\n%\n", ifx stopped then "Recorded." else tprint("Recording failed: %", stop_error));
ms_each :: (ticks: s64, count: s64) -> string #expand {
if !count return "-";
return tprint("% ms", FF(cast(float64) ticks / `recorder.frequency * 1000 / count, 2));
}
file_size := file_size_of(options.path);
print(" source: %\n", s.source);
print(" encoder: %\n", s.encoder);
print(" size: % x %\n", s.width, s.height);
print(" seconds: %\n", FF(s.seconds, 1));
print(" captured: % new images of the screen (% a second), % cursor-only changes\n", s.captured, FF(s.captured / s.seconds, 1), s.pointer_moves);
print(" sent: % frames (% a second), % of them repeats of an unchanged screen\n", s.emitted, FF(s.emitted / s.seconds, 1), s.repeats);
print(" written: %, dropped %, duplication lost %\n", s.written, s.dropped, s.lost);
print(" each: copy % (per image), read back % , encode % (per frame)\n",
ms_each(s.copy_ticks, s.captured), ms_each(s.readback_ticks, s.emitted), ms_each(s.encode_ticks, s.written));
print(" file: % KB, % MB a minute\n", file_size / 1024, FF(cast(float64) file_size / (1024 * 1024) / s.seconds * 60, 1));
if exclude_test {
excluded_rect := window_rect_in(excluded, region);
control_rect := window_rect_in(control, region);
excluded_seen := colour_fraction(last, width, height, excluded_rect, EXCLUDED_BGR);
control_seen := colour_fraction(last, width, height, control_rect, CONTROL_BGR);
print(" exclusion: the excluded square is % in the last frame, the control square %: %\n",
percent(excluded_seen), percent(control_seen),
ifx control_seen < 0.9 then "inconclusive (the control is not there)"
else ifx excluded_seen < 0.01 then "WDA_EXCLUDEFROMCAPTURE is respected"
else "WDA_EXCLUDEFROMCAPTURE is NOT respected");
DestroyWindow(excluded);
DestroyWindow(control);
}
if border_test {
print(" border: % yellow pixels along the screen's edges before recording, % while: %\n", border_before, border_during,
ifx border_during < 0 then "not measured (too short)"
else ifx border_during > border_before + 200 then "a capture border is drawn"
else "no capture border");
}
if snapshot_folder {
make_directory_if_it_does_not_exist(snapshot_folder, recursive = true);
rgba := bgra_to_rgba(last);
if cursor_test { // 48 x 48 around where the mouse was put, four times as large
SIDE :: 48; SCALE :: 4;
crop := NewArray(SIDE * SCALE * SIDE * SCALE * 4, u8,, temp);
cx, cy := cursor_at.x - region.left - 8, cursor_at.y - region.top - 8;
for y: 0..SIDE * SCALE - 1 for x: 0..SIDE * SCALE - 1 {
sx, sy := clamp(cx + x / SCALE, 0, width - 1), clamp(cy + y / SCALE, 0, height - 1);
memcpy(crop.data + (y * SIDE * SCALE + x) * 4, rgba.data + (cast(s64) sy * width + sx) * 4, 4);
}
cursor_path := tprint("%/cursor.png", snapshot_folder);
stbi_write_png(temp_c_string(cursor_path), SIDE * SCALE, SIDE * SCALE, 4, crop.data, SIDE * SCALE * 4);
print(" cursor: %\n", cursor_path);
}
captured_path := tprint("%/captured.png", snapshot_folder);
stbi_write_png(temp_c_string(captured_path), width, height, 4, rgba.data, width * 4);
image, info, decoded, decode_error := video_poster(options.path);
if decoded {
decoded_path := tprint("%/decoded.png", snapshot_folder);
stbi_write_png(temp_c_string(decoded_path), image.width, image.height, 4, image.pixels, image.width * 4);
print(" snapshot: % and % (decoded at % s of %: %, % fps)\n", captured_path, decoded_path, FF(info.poster_time, 2), FF(info.duration, 2), info.codec, FF(info.frame_rate, 1));
image_free(*image);
} else {
print(" snapshot: %; the file could not be decoded back: %\n", captured_path, decode_error);
}
}
exit(ifx stopped then cast(s32) 0 else 1);
}
#scope_file
EXCLUDED_BGR :: u8.[255, 0, 255]; // magenta
CONTROL_BGR :: u8.[255, 255, 0]; // cyan
// Two squares on the region, topmost: one kept out of captures, one not.
exclusion_windows :: (region: RECT) -> excluded: HWND, control: HWND {
width := region.right - region.left;
height := region.bottom - region.top;
side := min(cast(s32) 160, width / 4, height / 2);
y := region.top + (height - side) / 2;
excluded := square_window("record-probe excluded", 0x00FF00FF, region.left + width / 4 - side / 2, y, side);
control := square_window("record-probe control", 0x00FFFF00, region.left + 3 * width / 4 - side / 2, y, side);
WDA_EXCLUDEFROMCAPTURE :: 0x11;
if !SetWindowDisplayAffinity(excluded, WDA_EXCLUDEFROMCAPTURE) print("SetWindowDisplayAffinity failed; the test cannot say anything.\n");
pump_messages();
return excluded, control;
}
// A square of one colour, topmost, that never takes the focus.
square_window :: (class_name: string, colour: u32, x: s32, y: s32, side: s32) -> HWND {
instance := GetModuleHandleW(null);
wc: WNDCLASSEXW;
wc.cbSize = size_of(WNDCLASSEXW);
wc.lpfnWndProc = xx DefWindowProcW;
wc.hInstance = instance;
wc.hbrBackground = CreateSolidBrush(colour);
wc.lpszClassName = utf8_to_wide(class_name,, temp);
RegisterClassExW(*wc);
return CreateWindowExW(WS_EX_TOPMOST | WS_EX_TOOLWINDOW | WS_EX_NOACTIVATE, utf8_to_wide(class_name,, temp), utf8_to_wide(class_name,, temp),
WS_POPUP | WS_VISIBLE, x, y, side, side, null, null, instance, null);
}
// Pixels along a screen's outer 6 pixels that look like Windows' yellow capture border, with GDI.
yellow_edge_pixels :: (screen: RECT) -> s64 {
width, height := screen.right - screen.left, screen.bottom - screen.top;
desktop := GetDC(null);
defer ReleaseDC(null, desktop);
dc := CreateCompatibleDC(desktop);
defer DeleteDC(dc);
bitmap := CreateCompatibleBitmap(desktop, width, height);
defer DeleteObject(bitmap);
old := SelectObject(dc, bitmap);
BitBlt(dc, 0, 0, width, height, desktop, screen.left, screen.top, cast(ROP) (cast(u32) ROP.SRCCOPY | 0x40000000)); // CAPTUREBLT
SelectObject(dc, old);
info: BITMAPINFO;
info.bmiHeader.biSize = size_of(BITMAPINFOHEADER);
info.bmiHeader.biWidth = width;
info.bmiHeader.biHeight = -height;
info.bmiHeader.biPlanes = 1;
info.bmiHeader.biBitCount = 32;
info.bmiHeader.biCompression = BI_RGB;
pixels := NewArray(cast(s64) width * height * 4, u8,, temp);
GetDIBits(dc, bitmap, 0, xx height, pixels.data, *info, DIB_RGB_COLORS);
EDGE :: 6;
count: s64;
for y: 0..height - 1 for x: 0..width - 1 {
if x >= EDGE && x < width - EDGE && y >= EDGE && y < height - EDGE continue;
p := pixels.data + (cast(s64) y * width + x) * 4;
b, g, r := cast(s32) p[0], cast(s32) p[1], cast(s32) p[2];
if r > 200 && g > 120 && b < 90 && r - b > 130 count += 1;
}
return count;
}
// A window's rectangle, in the region's pixels, a few pixels in from its edges.
window_rect_in :: (window: HWND, region: RECT) -> RECT {
r: RECT;
GetWindowRect(window, *r);
return .{r.left - region.left + 4, r.top - region.top + 4, r.right - region.left - 4, r.bottom - region.top - 4};
}
colour_fraction :: (pixels: [] u8, width: s32, height: s32, r: RECT, bgr: [3] u8) -> float64 {
total, same: s64;
for y: max(r.top, 0)..min(r.bottom, height) - 1 for x: max(r.left, 0)..min(r.right, width) - 1 {
p := pixels.data + (cast(s64) y * width + x) * 4;
total += 1;
if abs(cast(s32) p[0] - bgr[0]) <= 8 && abs(cast(s32) p[1] - bgr[1]) <= 8 && abs(cast(s32) p[2] - bgr[2]) <= 8 same += 1;
}
return ifx total then cast(float64) same / total else 0;
}
pump_messages :: () {
msg: MSG;
while PeekMessageW(*msg, null, 0, 0, PM_REMOVE) {
TranslateMessage(*msg);
DispatchMessageW(*msg);
}
}
// A CUDA context on the first device, as onnxruntime makes for OCR and labels in mcap, and kept.
cuda_make_context :: () -> ok: bool, text: string {
library := LoadLibraryW(utf8_to_wide("nvcuda.dll",, temp));
if !library return false, "nvcuda.dll is not installed";
Cu_Init :: #type (flags: u32) -> s32 #c_call;
Cu_Device :: #type (device: *s32, ordinal: s32) -> s32 #c_call;
Cu_Context :: #type (cuda_context: **void, flags: u32, device: s32) -> s32 #c_call;
Cu_Name :: #type (name: *u8, length: s32, device: s32) -> s32 #c_call;
cu_init := cast(Cu_Init) GetProcAddress(library, "cuInit");
cu_device_get := cast(Cu_Device) GetProcAddress(library, "cuDeviceGet");
cu_ctx_create := cast(Cu_Context) GetProcAddress(library, "cuCtxCreate_v2");
cu_name := cast(Cu_Name) GetProcAddress(library, "cuDeviceGetName");
if !cu_init || !cu_device_get || !cu_ctx_create || !cu_name return false, "nvcuda.dll lacks the calls needed";
result := cu_init(0);
if result return false, tprint("cuInit failed (%)", result);
device: s32;
result = cu_device_get(*device, 0);
if result return false, tprint("cuDeviceGet failed (%)", result);
name: [256] u8;
cu_name(name.data, name.count, device);
cuda_context: *void;
result = cu_ctx_create(*cuda_context, 0, device);
if result return false, tprint("cuCtxCreate failed (%)", result);
return true, tprint("a context on %, kept for the recording", to_string(name.data));
}
bgra_to_rgba :: (pixels: [] u8) -> [] u8 {
out := NewArray(pixels.count, u8,, temp);
for i: 0..pixels.count / 4 - 1 {
out[i * 4 + 0] = pixels[i * 4 + 2];
out[i * 4 + 1] = pixels[i * 4 + 1];
out[i * 4 + 2] = pixels[i * 4 + 0];
out[i * 4 + 3] = 255;
}
return out;
}
file_size_of :: (path: string) -> s64 {
_, size, ok := file_modtime_and_size(path);
return ifx ok then size else 0;
}
percent :: (fraction: float64) -> string {
return join(tprint("%", FF(fraction * 100, 1)), "%",, temp);
}
FF :: (value: float64, decimals: s64) -> FormatFloat #expand {
return formatFloat(value, trailing_width = decimals, zero_removal = .NO);
}
SetProcessDpiAwarenessContext :: (value: HANDLE) -> BOOL #foreign user32_dll;
SetWindowDisplayAffinity :: (window: HWND, affinity: u32) -> BOOL #foreign user32_dll;
SetCursorPos :: (x: s32, y: s32) -> BOOL #foreign user32_dll;
user32_dll :: #library,system "user32";
#import "Basic";
#import "String";
#import "File";
#import "File_Utilities";
#import "Math";
#import "Windows";
#import "Windows_Utf8";
#import "Screen_Record";
#import "Image_Decode";
#import "stb_image_write";

451
video-capture-plan.html
View File

@ -1,451 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Video Capture Plan</title>
<style>
:root {
--bg: #fbfaf8; --surface: #ffffff; --text: #1d1d1f; --dim: #5d5d66; --line: #e3e1dc;
--accent: #3a5bd9; --accent-soft: #e9edfb; --warn: #9a5b00; --warn-soft: #fbf1e1;
--code-bg: #f2f0ec; --ok: #2e7d4f; --ok-soft: #e5f3ea;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
}
:root[data-theme="dark"] {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
* { box-sizing: border-box; }
body {
margin: 0; background: var(--bg); color: var(--text);
font: 16px/1.6 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
}
main { max-width: 860px; margin: 0 auto; padding: 40px 16px 80px; }
header p { color: var(--dim); margin-top: 4px; }
h1 { font-size: 2rem; line-height: 1.2; margin: 0; }
h2 { font-size: 1.35rem; margin: 48px 0 8px; padding-top: 16px; border-top: 1px solid var(--line); }
h3 { font-size: 1.05rem; margin: 28px 0 6px; }
p, li { max-width: 72ch; }
a { color: var(--accent); }
code, pre { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; font-size: 0.88em; }
code { background: var(--code-bg); padding: 1px 5px; border-radius: 4px; }
pre { background: var(--code-bg); padding: 12px 14px; border-radius: 8px; overflow-x: auto; line-height: 1.45; }
pre code { background: none; padding: 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 0.94rem; }
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
th { color: var(--dim); font-weight: 600; }
.table-wrap { overflow-x: auto; }
.summary { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 16px 20px; margin-top: 24px; }
.summary ul { margin: 6px 0 0; padding-left: 20px; }
.note { background: var(--accent-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk { background: var(--warn-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk strong { color: var(--warn); }
.tag { display: inline-block; font-size: 0.78rem; font-weight: 600; padding: 1px 8px; border-radius: 99px; background: var(--accent-soft); color: var(--accent); }
.tag.later { background: var(--code-bg); color: var(--dim); }
.tag.decided { background: var(--ok-soft); color: var(--ok); }
nav ol { columns: 2; column-gap: 32px; padding-left: 20px; margin: 8px 0 0; }
@media (max-width: 600px) { nav ol { columns: 1; } h1 { font-size: 1.6rem; } }
.phase { border-left: 3px solid var(--accent); padding: 2px 0 2px 14px; margin: 16px 0; }
.phase h3 { margin-top: 0; }
figure { margin: 18px 0; }
figcaption { color: var(--dim); font-size: 0.88rem; margin-top: 6px; }
svg text { fill: var(--text); font: 13px system-ui, sans-serif; }
svg .dim { fill: var(--dim); }
svg .box { fill: var(--surface); stroke: var(--line); }
svg .hl { fill: var(--accent-soft); stroke: var(--accent); }
svg .arrow { stroke: var(--dim); fill: none; marker-end: url(#a); }
</style>
</head>
<body>
<main>
<header>
<h1>Screen video capture in mcap</h1>
<p>Planning document · 2026-10-10 · for mcap after 0.1.1 (still captures with PrintScreen) · <strong>phases 1 to 3 built, mcap 0.1.5</strong></p>
</header>
<div class="summary">
<strong>In short</strong>
<ul>
<li><span class="tag decided">Decided</span> <strong>DXGI Desktop Duplication first.</strong> If it can't run, for example on a laptop with two GPUs where it fails with <code>DXGI_ERROR_UNSUPPORTED</code>, recording falls back to <strong>Windows.Graphics.Capture</strong>. Both sit behind one interface, so everything after the capture is shared. It's more code than Windows.Graphics.Capture alone, but each method covers the cases where the other fails.</li>
<li><span class="tag decided">Decided</span> <strong>Encoding uses the FFmpeg mcap already ships.</strong> It tries NVENC first, then AMD AMF, Intel QSV, Media Foundation and <code>libx264</code>. The output is H.264 in MP4, with frames written only when the screen changes. That FFmpeg build already includes every one of these encoders.</li>
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen starts a recording.</strong> It opens the same frozen-screen overlay as stills, in a video mode. The region must be at least <strong>240 × 180</strong> pixels, which Snagit doesn't require.</li>
<li><span class="tag decided">Decided</span> <strong>Ctrl+PrintScreen again stops it</strong> and saves the recording. One key both starts and stops, and no other key is taken from the app being recorded.</li>
<li><span class="tag decided">Decided</span> <strong>A recording becomes an <code>.mcap</code></strong> in <em>Screen Captures</em>, the folder stills go to. It's stored the way imported videos are: the MP4 as the original, a frame from it for the thumbnail, OCR and labels.</li>
<li>It's built in five phases. Phase 1 is a command-line probe that answers the two-GPU question on your laptop before anything depends on it.</li>
<li><span class="tag decided">Done</span> <strong>Phase 1</strong> (2026-10-10, mcap 0.1.3): <code>modules/Screen_Record</code> and <code>record-probe</code>. 60 s of 4K at 30 fps with NVENC and no dropped frames. The two-GPU failure couldn't be reproduced, because this laptop's screens are currently driven by the NVIDIA GPU (see <a href="#phases">phase 1's result</a>).</li>
<li><span class="tag decided">Done</span> <strong>Phase 2</strong> (2026-10-10, mcap 0.1.4): the Windows.Graphics.Capture fallback. Automatic mode falls back to it on a simulated <code>DXGI_ERROR_UNSUPPORTED</code>. It records 4K at 60 fps with nothing dropped, draws the cursor exactly as Desktop Duplication's code does, and the yellow border can be turned off on Windows 11.</li>
<li><span class="tag decided">Done</span> <strong>Phase 3</strong> (2026-10-10, mcap 0.1.5): recording in the app. Ctrl+PrintScreen opens the overlay's video mode; Ctrl+PrintScreen again stops; the recording becomes a video <code>.mcap</code> in Screen Captures. In the app, Desktop Duplication first failed with <code>DXGI_ERROR_UNSUPPORTED</code>: only the main thread was DPI aware. The capture thread now makes itself aware.</li>
<li><span class="tag later">Later</span> Audio is phase 5. Recording one window, and regions that span two screens, come after.</li>
</ul>
</div>
<nav>
<ol>
<li><a href="#scope">Scope</a></li>
<li><a href="#using">Using it</a></li>
<li><a href="#sources">Grabbing frames</a></li>
<li><a href="#pipeline">The frame pipeline</a></li>
<li><a href="#encoding">Encoding</a></li>
<li><a href="#library">Into the library</a></li>
<li><a href="#hotkeys">Hotkeys</a></li>
<li><a href="#audio">Audio</a></li>
<li><a href="#phases">Phases</a></li>
<li><a href="#risks">Risks</a></li>
<li><a href="#questions">Open questions</a></li>
</ol>
</nav>
<h2 id="scope">1. Scope</h2>
<p><strong>In:</strong></p>
<ul>
<li>Recording a rectangle on one screen, or a whole screen, to H.264 MP4 at up to 60 frames a second, with the mouse cursor.</li>
<li>Pause and resume, stop, and discard.</li>
<li>Saving into the library with the window and app it was recorded from.</li>
<li>System audio and the microphone, in the last phase.</li>
</ul>
<p><strong>Out, for now:</strong></p>
<ul>
<li>Recording one window that follows it as it moves.</li>
<li>Regions that span two screens.</li>
<li>Webcam overlay.</li>
<li>Trimming and editing.</li>
<li>HDR output.</li>
<li>macOS and Linux.</li>
</ul>
<p>Windows only, like the stills overlay. It needs Windows 10 version 2004 or later; the Windows.Graphics.Capture features used below need that version.</p>
<h2 id="using">2. Using it</h2>
<ol>
<li><strong>Ctrl+PrintScreen</strong> freezes the screens and shows the overlay from stills, in its video mode. The outline and the magnifier are red, not orange, so you can see it will record.</li>
<li><strong>Drag a region.</strong> The magnifier's label shows its size.
<ul>
<li>The drag is kept on the screen where it started (see <a href="#sources">3.4</a>).</li>
<li><strong>Below 240 × 180</strong>, the outline is drawn dashed and the label reads <code>320 × 120 · at least 240 × 180</code>. Letting go then starts nothing, and the overlay stays open to drag again.</li>
<li>A click without dragging, or Enter, takes the whole screen, as for stills.</li>
<li>Escape or a right click cancels, as for stills.</li>
</ul></li>
<li><strong>Recording starts</strong> as soon as you let go, with no countdown (see <a href="#questions">open question 4</a>).
<ul>
<li>A thin red border is drawn just <em>outside</em> the region, so it's never in the video.</li>
<li>A small toolbar sits beside the region, outside it where there's room. It shows the time, the size and frames dropped, with <em>Pause</em>, <em>Stop</em> and <em>Discard</em>.</li>
<li>Both the border and the toolbar are kept out of the capture with <code>SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE)</code>, which needs Windows 10 2004. Whether Desktop Duplication respects this is checked in phase 1. The border is outside the region either way, so only the toolbar depends on it.</li>
</ul></li>
<li><strong>Ctrl+PrintScreen</strong> again, or <em>Stop</em>, ends the recording at once:
<ul>
<li>The encoder is flushed and the file closed.</li>
<li>The file becomes an <code>.mcap</code>.</li>
<li>mcap comes to the front showing it in the Capture window, as it does after a still.</li>
</ul></li>
</ol>
<p>Dimensions are rounded down to even numbers after the minimum is checked, since H.264's 4:2:0 colour needs that. 240 × 180 is already even, so a region that passes the check stays at least 240 × 180.</p>
<h2 id="sources">3. Grabbing frames</h2>
<p>One interface, two implementations, chosen when the recording starts:</p>
<pre><code>Capture_Source :: struct {
start: (source: *Capture_Source, screen: Screen_Ref, region: Rect, cursor: bool) -> ok: bool, error: string, fall_back: bool;
next_frame: (source: *Capture_Source, timeout_ms: u32) -> Captured_Frame, status: Frame_Status; // NEW, SAME, LOST, FAILED
stop: (source: *Capture_Source);
name: string; // "DXGI" or "WGC", for the log and the toolbar's tooltip
}
Captured_Frame :: struct {
texture: *ID3D11Texture2D; // the region only, region-sized, owned by the source until the next call
time: s64; // QueryPerformanceCounter ticks: when it was on screen
cursor: Cursor_State; // DXGI only: where, visible, and the shape when it changed
}</code></pre>
<p>Both run on a capture thread of their own, with <strong>their own D3D11 device</strong>, not the app's. The app's device starts on WARP and moves to the GPU later (<code>#WarpFirstStartup</code> in <code>graphics/DX11.jai</code>). Desktop Duplication also needs a device on the GPU the screen is connected to, which may not be the one the app draws with.</p>
<h3>3.1 DXGI Desktop Duplication (first choice)</h3>
<ul>
<li><strong>Finding the screen:</strong> <code>CreateDXGIFactory1</code>, then <code>EnumAdapters1</code> and <code>EnumOutputs</code>, to find the output whose desktop rectangle holds the region. The device is created on <em>that</em> output's adapter, with <code>D3D11CreateDevice(adapter, D3D_DRIVER_TYPE_UNKNOWN, ...)</code>. The jai <code>dxgi</code> module already has every type needed: <code>IDXGIOutput5</code>, <code>DuplicateOutput1</code>, <code>IDXGIOutputDuplication</code>, <code>DXGI_OUTDUPL_FRAME_INFO</code>.</li>
<li><strong>Duplicating:</strong> <code>IDXGIOutput5::DuplicateOutput1</code> with <code>B8G8R8A8_UNORM</code>. mcap is per-monitor DPI aware (v2), and the plain <code>DuplicateOutput</code> is documented to fail for such processes. Phase 1 confirms this, and falls back to <code>IDXGIOutput1::DuplicateOutput</code> where <code>IDXGIOutput5</code> is missing.</li>
<li><strong>Each frame:</strong>
<ul>
<li><code>AcquireNextFrame(timeout)</code> waits for one.</li>
<li><code>LastPresentTime == 0</code> means only the mouse moved: the cursor is updated and no new image is copied.</li>
<li><code>CopySubresourceRegion</code> copies the region into a region-sized texture.</li>
<li><code>ReleaseFrame</code> is called straight after the copy. Holding a frame stalls the desktop.</li>
</ul></li>
<li><strong>The cursor isn't drawn into the image.</strong> Its position comes with each frame, and its shape from <code>GetFramePointerShape</code> when <code>PointerShapeBufferSize</code> is set. It is drawn onto the copy on the CPU, after it's read back (<a href="#pipeline">4</a>). There are three kinds of shape to handle: colour, masked colour, and monochrome. The monochrome kind (the text I-beam) inverts what's under it.</li>
<li><strong>Errors while recording:</strong>
<ul>
<li><code>DXGI_ERROR_ACCESS_LOST</code> comes from a display mode change, the UAC or lock screen, or a game switching to fullscreen. The duplication is made again, retrying every 250 ms, and the recording goes on. The toolbar shows the gap.</li>
<li><code>DXGI_ERROR_WAIT_TIMEOUT</code> means nothing changed.</li>
<li>Anything else ends the recording, keeping what was captured.</li>
</ul></li>
</ul>
<h3>3.2 When to fall back</h3>
<p>The choice is made once, at the start, and logged:</p>
<div class="table-wrap"><table>
<tr><th>Desktop Duplication says</th><th>Meaning</th><th>Then</th></tr>
<tr><td><code>DXGI_ERROR_UNSUPPORTED</code> from <code>DuplicateOutput1</code></td><td>Two GPUs, and the process or device isn't on the one driving the screen. Also returned when no driver can duplicate.</td><td>Windows.Graphics.Capture</td></tr>
<tr><td><code>DXGI_ERROR_NOT_CURRENTLY_AVAILABLE</code></td><td>Too many duplications of this screen at once (other recorders running)</td><td>Windows.Graphics.Capture</td></tr>
<tr><td><code>E_ACCESSDENIED</code></td><td>The secure desktop (UAC, lock screen) is showing</td><td>Retry for 2 s, then tell the user</td></tr>
<tr><td>Screen rotated (<code>DXGI_OUTDUPL_DESC.Rotation</code> isn't identity)</td><td>Frames would come sideways</td><td>Windows.Graphics.Capture, which hands them over upright</td></tr>
<tr><td>No output holds the region</td><td>The screens changed since the freeze</td><td>Error, nothing recorded</td></tr>
</table></div>
<p>A setting forces one method, for testing and for machines where one misbehaves: <code>capture: auto</code> (the default), <code>dxgi</code> or <code>wgc</code>, under a new <code>[video]</code> section of <code>settings.config</code>.</p>
<h3>3.3 Windows.Graphics.Capture (fallback)</h3>
<ul>
<li><strong>Setup:</strong> the WinRT runtime is started on the capture thread (<code>RoInitialize</code>, multithreaded).
<ul>
<li><code>IGraphicsCaptureItemInterop::CreateForMonitor</code> makes the item for the screen.</li>
<li><code>CreateDirect3D11DeviceFromDXGIDevice</code> wraps the capture thread's device.</li>
<li><code>Direct3D11CaptureFramePool::CreateFreeThreaded</code> makes a pool of 2 BGRA frames.</li>
</ul></li>
<li><strong>Polling, not events:</strong> <code>TryGetNextFrame</code> is called at the frame rate, with the same timeout as DXGI. This avoids implementing WinRT event-handler objects in Jai.</li>
<li><strong>Cursor and border:</strong> <code>IsCursorCaptureEnabled</code> draws the cursor into the image, so the cursor step in 3.1 is skipped. <code>IsBorderRequired = false</code> removes the yellow border on Windows 11; whether an unpackaged app may do that without asking is checked in phase 2. Windows 10 always shows the border.</li>
<li><strong>Bindings:</strong> roughly six WinRT interfaces, written by hand as COM vtables, plus HSTRINGs (<code>WindowsCreateString</code>). This is the bulk of phase 2.</li>
</ul>
<h3>3.4 One screen per recording</h3>
<p>Desktop Duplication works one output at a time, and Windows.Graphics.Capture one monitor at a time. A region over two screens would need two captures stitched together, possibly at different DPI and refresh rates. In video mode the overlay keeps the drag on the screen where it started. Spanning screens is <span class="tag later">Later</span>.</p>
<h2 id="pipeline">4. The frame pipeline</h2>
<pre><code>capture thread encoder thread
next_frame ─► region texture ─► staging ring (3) ─► queue (≤ 8 frames) ─► AVFrame bgr0 ─► encoder ─► MP4
Map the one from drops the oldest when full,
2 frames back, copy counted as dropped
out, draw the cursor</code></pre>
<ul>
<li><strong>Reading back without stalls:</strong> three staging textures are used in turn. The CPU reads the one copied two frames ago, so <code>Map</code> never waits on the GPU. At 4K and 60 frames a second, about 2 GB/s is read back, well within what the hardware can do.</li>
<li><strong>Timestamps</strong> come from when each frame was on screen (<code>LastPresentTime</code>, or the system-relative time of a Windows.Graphics.Capture frame), in a 1/90000 timebase. Time paused is subtracted.</li>
<li><strong>The frame rate</strong> is capped at the setting (30 by default, or 60). When nothing changes, no frame is written, apart from one each second so players can seek. A still screen costs almost nothing.</li>
<li><strong>Buffers</strong> come from a fixed pool the size of the region, made at the start. Nothing is allocated per frame.</li>
</ul>
<h2 id="encoding">5. Encoding</h2>
<ul>
<li><strong>The encoder</strong> is the first of <code>h264_nvenc</code>, <code>h264_amf</code>, <code>h264_qsv</code>, <code>h264_mf</code> and <code>libx264</code> that opens at the region's size. Each is opened once, so a driver that's present but broken falls through to the next. The one used is logged and stored with the recording.</li>
<li><strong>The input</strong> is BGRA as <code>AV_PIX_FMT_BGR0</code>. NVENC converts it to its own colour format on the GPU. For the CPU encoders, <code>swscale</code> converts it to <code>yuv420p</code> first.</li>
<li><strong>Quality</strong> is constant quality, about CQ 23 for NVENC (CRF 23 for <code>libx264</code>), with a keyframe every 2 s. NVENC uses preset <code>p4</code>. Screen content (text, flat colour) compresses well at this setting. Phase 1 measures the file sizes.</li>
<li><strong>Crash-safe files:</strong> the MP4 is written fragmented (<code>movflags=frag_keyframe+empty_moov</code>) to <code>%TEMP%\mcap\recording-*.mp4</code>. If mcap or Windows crashes mid-recording, the file still plays up to the last fragment. On the next start, a leftover file is offered for import.</li>
<li><strong>Bindings:</strong> <code>modules/ffmpeg</code> already binds everything encoding needs (<code>avcodec_find_encoder_by_name</code>, <code>avcodec_send_frame</code>, <code>avcodec_receive_packet</code>, <code>avformat_alloc_output_context2</code>, <code>avformat_write_header</code>, <code>av_interleaved_write_frame</code>, <code>av_write_trailer</code>, <code>avio_open</code>). <code>swresample</code> ships, but has no bindings yet. Audio needs them (phase 5).</li>
</ul>
<h2 id="library">6. Into the library</h2>
<ul>
<li><strong>Like an import:</strong> on stop, the MP4 goes through the path imported videos take (<code>import_video_mcap</code> and <code>mcap_write_streamed</code>, <code>video_poster</code> for the thumbnail frame) into <code>Screen Captures</code>. The temporary file is then deleted.</li>
<li><strong>metadata.json</strong> is a still's: the title and app of the window under the region's middle when recording started, and when it started.</li>
<li><strong>capture.json</strong> is a still's (region, position, app path) plus <code>Duration</code>, <code>FrameRate</code>, <code>VideoCodec</code>, <code>Encoder</code> (<code>h264_nvenc</code> ...), <code>Capture</code> (<code>DXGI</code> or <code>WGC</code>), <code>DroppedFrames</code> and <code>Paused</code> (seconds).</li>
<li><strong>The catalog</strong> today marks a capture as a video only from <code>VideoCodec</code> in <code>import.json</code> (<code>app/catalog.jai</code>). It will read the same keys from <code>capture.json</code>, so recordings show as videos, play with Open, and are skipped by segmentation.</li>
<li><strong>The thumbnail frame:</strong> <code>video_poster</code> picks a frame a tenth of the way in, at most 3 s, to skip a phone video's black start. A screen recording has no black start, but the first second is often the region being chosen. The same rule is kept unless it picks badly.</li>
</ul>
<h2 id="hotkeys">7. Hotkeys</h2>
<div class="table-wrap"><table>
<tr><th>Action in <code>[global_hotkeys]</code></th><th>Default</th><th>What it does</th></tr>
<tr><td><code>capture_region</code> (stills, built)</td><td><code>PrintScreen</code></td><td>Opens the overlay for a still.</td></tr>
<tr><td><code>record_region</code></td><td><code>Ctrl-PrintScreen</code></td><td>Opens the overlay in video mode. While a recording runs, it stops and saves it.</td></tr>
</table></div>
<ul>
<li><strong>One key for start and stop.</strong> It's registered all the time, as <code>capture_region</code> is, and only <code>global_hotkey_fired</code> needs to know whether a recording is running. No key is borrowed from the app being recorded: a plain Escape, or Windows' own Ctrl+Escape, would be.</li>
<li><strong>Escape inside the overlay</strong>, before recording starts, cancels as it does for stills.</li>
<li><strong>Saving and discarding:</strong> stopping always saves. Discarding is the toolbar's <em>Discard</em> button, which asks first.</li>
<li><strong>PrintScreen during a recording</strong> still takes a still. The overlay would freeze the screen, and that freeze would end up in the video, so the still is taken without the overlay: the whole screen under the cursor, saved as usual. Open question 1 asks whether that's right.</li>
<li><strong>Settings files:</strong> <code>record_region</code> is a new <code>windows_only</code> action. Existing settings files get it appended on the next launch, as <code>capture_region</code> was.</li>
</ul>
<h2 id="audio">8. Audio <span class="tag later">Phase 5</span></h2>
<ul>
<li><strong>Sources:</strong> system audio comes from WASAPI loopback on the default output device, and the microphone from WASAPI capture. Each is optional, set in the toolbar and remembered under <code>[video]</code>.</li>
<li><strong>Mixing:</strong> both are mixed to 48 kHz stereo float with <code>swresample</code> (bindings to generate), and encoded as AAC at 160 kb/s into the same MP4.</li>
<li><strong>Sync:</strong> each audio packet's QPC time is lined up with the video's.</li>
<li><strong>Silence:</strong> loopback sends nothing while nothing plays, so silence is written to fill the gaps.</li>
</ul>
<h2 id="phases">9. Phases</h2>
<p>Each phase ends with something to run and check, and is a commit (0.1.3 onwards).</p>
<div class="phase">
<h3>Phase 1: <code>record-probe</code> with Desktop Duplication <span class="tag decided">Done</span></h3>
<p>A command-line tool, <code>tools/record_probe.jai</code>, built with <code>jai first.jai - record_probe</code>:</p>
<ul>
<li>It lists adapters and outputs, picks a screen or region, and duplicates it on the output's adapter.</li>
<li>It records N seconds through the pipeline (section 4) and encoder (section 5) to an MP4.</li>
<li>It prints which method and encoder it used, frames captured, written and dropped, time per stage, and the file size.</li>
</ul>
<p>It answers four questions on your laptop:</p>
<ul>
<li>Does <code>DuplicateOutput1</code> succeed on the internal screen, and on an external one?</li>
<li>Does it succeed with CUDA loaded in the same process? mcap loads onnxruntime with CUDA, which may keep the process on the NVIDIA GPU.</li>
<li>Does a window with <code>WDA_EXCLUDEFROMCAPTURE</code> stay out of the frames?</li>
<li>What does 4K at 60 frames a second cost?</li>
</ul>
<p><strong>Done when:</strong> 60 s of a 4K screen at 30 frames a second plays in VLC and Windows' Media Player with the cursor drawn, frames dropped are under 1%, and each of the questions above has a recorded answer. A <code>DXGI_ERROR_UNSUPPORTED</code> is also an answer: it means phase 2 is needed on this machine.</p>
<div class="note"><strong>Result (2026-10-10).</strong> Release build, RTX 5080 laptop. The screens are a 3840 × 2160 external and the 2560 × 1600 internal; DXGI lists both on the NVIDIA adapter.
<ul>
<li><strong>The done-when run:</strong> 60.2 s of the 4K screen at 30 fps, with a square moving across it the whole time (<code>-animate</code>):
<ul>
<li>3,602 images taken from the screen and 1,802 frames sent, all of them encoded with NVENC and none dropped.</li>
<li>18 MB a minute.</li>
<li>FFmpeg reads it back as 60.07 s at 30 fps.</li>
<li>Per frame: 4.4 ms to read back and 13.4 ms to encode, on separate threads.</li>
</ul>
The frame decoded back matches the one captured, cursor included, by eye. Playing it in VLC and Media Player is still to check.</li>
<li><strong>60 fps:</strong> all 59.3 changes a second are kept. On a fixed tick grid, a 60 Hz screen beat against the 60 fps cap and lost about 7% of frames. Frames are now sent as soon as their turn is due, on a schedule that advances by exactly one frame each time. Caps of 30, 60 and 24 give 29.8, 59.3 and 23.8 frames a second.</li>
<li><strong>The questions:</strong>
<ul>
<li><strong>Both screens</strong> duplicate with <code>DuplicateOutput1</code>.</li>
<li><strong>The plain <code>DuplicateOutput</code></strong> works too in a per-monitor DPI-aware process. The documented failure doesn't happen here; <code>DuplicateOutput1</code> stays the first choice.</li>
<li><strong>With a CUDA context in the process</strong>, duplication still works.</li>
<li><strong><code>WDA_EXCLUDEFROMCAPTURE</code> is respected:</strong> the excluded square is 0% present in the frame and the control square 100%.</li>
<li><strong>The two-GPU failure</strong> wasn't reproduced. Task Manager shows the Intel GPU, but no screen is attached to it now: the laptop is in discrete-GPU mode. Testing it means switching the laptop to Optimus (hybrid) mode in the NVIDIA Control Panel or the BIOS, which needs a restart.</li>
</ul></li>
<li><strong>Encoders</strong>, 4K at 30 fps, 8 s of the same motion. All of them keep up:
<div class="table-wrap"><table>
<tr><th>Encoder</th><th>Encode per frame</th><th>MB a minute</th></tr>
<tr><td><code>h264_nvenc</code></td><td>13.7 ms</td><td>23.4</td></tr>
<tr><td><code>h264_qsv</code> (the Intel GPU, though no screen is attached to it)</td><td>12.3 ms</td><td>16.7</td></tr>
<tr><td><code>h264_mf</code></td><td>16.8 ms</td><td>24.0</td></tr>
<tr><td><code>libx264</code> (veryfast, CRF 23)</td><td>14.9 ms</td><td>11.4</td></tr>
<tr><td><code>h264_amf</code></td><td colspan="2">Not available: no AMD driver (<code>amfrt64.dll</code>), so it falls through to the next</td></tr>
</table></div>
Most of NVENC's 13.7 ms is FFmpeg copying the 33 MB frame before the GPU sees it. Handing it the frame without the copy is a later optimisation, not needed at 4K 60.</li>
<li><strong>A crash mid-recording:</strong> killed about 7 s in, the fragmented MP4 still reads back with 6 s of video. At most the last keyframe interval (2 s) is lost.</li>
<li><strong>The minimum size:</strong> 239 × 300 is refused, and 241 × 181 records at 240 × 180. The minimum is checked before rounding to even.</li>
</ul>
<p><strong>Changes from the plan:</strong></p>
<ul>
<li><strong>The DPI check:</strong> once, a process that isn't DPI aware saw the 4K screen as 1920 × 1080, and the region would have been the wrong part of it. This couldn't be reproduced afterwards. <code>dxgi_open</code> now compares the duplication's size with the desktop's and refuses when they differ.</li>
<li><strong>The probe's extra options:</strong> <code>-animate</code>, which moves a square for measuring throughput, and <code>-check FILE</code>, which reads a recording back.</li>
</ul></div>
</div>
<div class="phase">
<h3>Phase 2: Windows.Graphics.Capture fallback <span class="tag decided">Done</span></h3>
<p>The second source behind the same interface: WinRT bindings, polling <code>TryGetNextFrame</code>, the cursor and border settings. The fallback rules from 3.2, and <code>-capture dxgi|wgc|auto</code> on the probe.</p>
<p><strong>Done when:</strong></p>
<ul>
<li>The probe records the same screens with <code>-capture wgc</code>.</li>
<li><code>auto</code> picks Desktop Duplication where it works and Windows.Graphics.Capture where it doesn't. The latter is shown either on the two-GPU laptop or on a rotated screen.</li>
<li>Whether the yellow border can be turned off is recorded.</li>
</ul>
<div class="note"><strong>Result (2026-10-10).</strong> Release build, the same laptop and screens as phase 1.
<ul>
<li><strong>Built:</strong>
<ul>
<li><code>wgc_source.jai</code>: hand-written bindings for eight WinRT interfaces.</li>
<li><code>capture_gpu.jai</code>: what both sources share, namely the device on the screen's adapter, the region copy and the staging ring. Each source is now only "the next image's region into a texture".</li>
<li><code>Record_Options.capture</code>: <code>.AUTO</code>, <code>.DXGI</code> or <code>.WGC</code>.</li>
<li>The probe's <code>-capture auto|dxgi|wgc</code>.</li>
</ul></li>
<li><strong>The fallback:</strong>
<ul>
<li><code>auto</code> picks Desktop Duplication where it works.</li>
<li>With <code>-simulate-unsupported</code>, which makes duplication fail with <code>DXGI_ERROR_UNSUPPORTED</code> as on a two-GPU laptop, <code>auto</code> logs why and records with Windows.Graphics.Capture.</li>
<li><code>-capture dxgi</code> with the same simulated failure refuses, with exit code 3.</li>
<li>The real two-GPU case is shelved: this laptop's GPU is pinned in the BIOS.</li>
</ul></li>
<li><strong>Recording:</strong> 4K at 60 fps with motion: 56.6 frames a second taken and every one encoded, none dropped. Windows.Graphics.Capture delivers a little under the screen's 60 Hz; Desktop Duplication delivered 59.3. A 641 × 481 region on the internal screen gives the same frame as Desktop Duplication's in phase 1.</li>
<li><strong>The border:</strong> <code>IsBorderRequired = false</code> works for mcap as an unpackaged app on Windows 11, with no access request. A GDI sample of the screen's outer 6 pixels found no yellow pixels with the border off, and 14,214 with it left on (<code>-keep-border</code>, the control).</li>
<li><strong>The cursor:</strong> with the mouse put in the region (<code>-cursor-test</code>), Windows.Graphics.Capture's own cursor and the one <code>cursor.jai</code> draws into Desktop Duplication's frames are the same, pixel for pixel, by eye at 4×.</li>
<li><strong><code>WDA_EXCLUDEFROMCAPTURE</code></strong> is respected by Windows.Graphics.Capture too: the excluded square is 0% present and the control square 100%.</li>
<li><strong>Waiting for frames:</strong> each new frame sets an event, through a small COM handler for <code>FrameArrived</code>, which Windows took. The polling fallback, for when Windows refuses the handler (<code>-poll</code>), takes 41 frames a second of a 60 Hz screen, enough for a 30 fps recording (29.0 sent), at a 1 ms timer resolution.</li>
</ul>
<p><strong>Changes from the plan:</strong></p>
<ul>
<li><strong>Events, not polling.</strong> The plan said to poll <code>TryGetNextFrame</code>, to avoid writing a WinRT event handler in Jai. The handler took about 30 lines, so it's the default, and polling is the fallback.</li>
<li><strong>The source opens on the capture thread</strong>, for both methods. WinRT objects belong to the apartment that made them, so <code>record_start</code> starts the capture thread and waits for it to say whether its source opened.</li>
<li><strong>The WinRT calls</strong> (<code>RoInitialize</code>, <code>RoGetActivationFactory</code>, HSTRINGs) are loaded from <code>combase.dll</code> at run time. Their import library, <code>runtimeobject.lib</code>, names no DLL of its own, which Jai's <code>#library</code> needs.</li>
<li><strong>Test switches:</strong> <code>Record_Options.testing</code> holds three, for the probe: simulate the two-GPU failure, force polling, keep the border.</li>
</ul></div>
</div>
<div class="phase">
<h3>Phase 3: Recording in the app <span class="tag decided">Done</span></h3>
<p>The pieces:</p>
<ul>
<li>The <code>record_region</code> action, which starts a recording or, during one, stops it.</li>
<li>The overlay's video mode: red outline, the 240 × 180 minimum, the drag kept to one screen.</li>
<li>The capture and encoder threads in the app.</li>
<li>Stopping, and the file into the library (section 6).</li>
<li>The catalog reading <code>capture.json</code>.</li>
<li>A shutdown step that finishes the file.</li>
<li>Tests of the minimum-size and even-size rules, and of the <code>capture.json</code> round trip.</li>
</ul>
<p><strong>Done when:</strong></p>
<ul>
<li>Ctrl+PrintScreen, a drag, then Ctrl+PrintScreen again leaves a playable video in Screen Captures, shown as a video in the grid and opened in the Capture window.</li>
<li>A 239 × 300 drag doesn't start a recording.</li>
<li>Quitting mid-recording keeps it.</li>
</ul>
<div class="note"><strong>Result (2026-10-10).</strong>
<ul>
<li><strong>Built:</strong>
<ul>
<li><code>src/app/screen_record.jai</code>: start, stop, and writing the recording's <code>.mcap</code>.</li>
<li>The overlay's video mode, in <code>screen_capture_win32.jai</code>.</li>
<li>The <code>record_region</code> hotkey, <code>Ctrl-PrintScreen</code> by default. Existing settings files get it appended, as yours did.</li>
<li>The catalog reading a video's keys from <code>capture.json</code> as well as <code>import.json</code>.</li>
<li>Saving reuses the stills' jobs: a thread copies the MP4 into the <code>.mcap</code> with <code>mcap_write_streamed</code>, then the grid, the Capture window and the OCR and label scan take it as they take a still.</li>
</ul></li>
<li><strong>A real bug, which the fallback hid:</strong> the first run in the app recorded with Windows.Graphics.Capture. Desktop Duplication had failed with <code>DXGI_ERROR_UNSUPPORTED</code>, which <code>record-probe</code> never saw. The cause is that ImGui's <code>EnableDpiAwareness</code> sets per-monitor DPI awareness for the main thread only, and Desktop Duplication refuses a thread that isn't aware. The capture thread now makes itself aware (<code>gpu_thread_dpi_aware</code>), and the app records with <code>DuplicateOutput1</code>. This also answers phase 1's question: the plain call isn't what fails for an unaware thread; both do.</li>
<li><strong>End to end</strong>, with <code>-record 5</code> on a scratch library: the internal screen (2560 × 1600), Desktop Duplication, NVENC, 5 s. Its <code>.mcap</code> lands in Screen Captures, and the temporary MP4 is deleted.</li>
<li><strong>Quitting while recording</strong> (<code>-record 4 -record-quit</code>): the shutdown stopped the recording, saved it and finished in 0.6 s.</li>
<li><strong>Tests:</strong> a "screen recording" group of 17 checks in the import suite:
<ul>
<li>The size rule: 240 × 180 allowed, 239 × 180 and 240 × 179 not.</li>
<li>An MP4 made with the module's encoder, written as a recording, reads back with the MP4 as its original and a frame of it, at its size, as its rendered image.</li>
<li>The catalog takes it as an H.264 video of about a second, with its window's title and app.</li>
</ul>
Every suite passes.</li>
<li><strong>Not checked by me: the overlay's video mode by hand.</strong> That means the red outline, the dashed outline under 240 × 180 and letting go of it, the drag held to one screen, and Ctrl+PrintScreen to start and stop. It needs someone at the keyboard and mouse.</li>
</ul>
<p><strong>Changes from the plan:</strong></p>
<ul>
<li><strong>PrintScreen while recording</strong> takes the whole screen under the cursor at once, without the overlay: open question 1's answer, for now.</li>
<li><strong><code>-record SECONDS</code></strong> (and <code>-record-quit</code>) on the app's command line, for end-to-end checks like the existing <code>-scan</code>.</li>
<li><strong>No indicator yet:</strong> a toast says recording started and which key stops it. The border and toolbar are phase 4.</li>
<li><strong>A settings-dependent test fixed:</strong> the segment overlay test expected no segment colours chosen, but it loads the app's real <code>segment-colours.config</code>, where you had chosen one. It now checks only the colour it set.</li>
</ul></div>
</div>
<div class="phase">
<h3>Phase 4: Border, toolbar, pause and recovery</h3>
<p>The pieces:</p>
<ul>
<li>The border and toolbar windows, kept out of the capture.</li>
<li>Pause, resume, and Discard with a confirmation.</li>
<li>The dropped-frame count.</li>
<li>Recovery from <code>ACCESS_LOST</code>.</li>
<li>Offering a leftover temporary file for import at startup.</li>
<li>The <code>[video]</code> settings: frame rate, cursor, capture method.</li>
</ul>
<p><strong>Done when:</strong> neither the border nor the toolbar appears in a recording. Pausing for 10 s leaves no gap in the video. A UAC prompt or a resolution change mid-recording leaves a video that plays through.</p>
</div>
<div class="phase">
<h3>Phase 5: Audio</h3>
<p>Section 8: <code>swresample</code> bindings, loopback and microphone, mixing, AAC.</p>
<p><strong>Done when:</strong> a 10-minute recording with a video playing and someone talking stays in sync to within 50 ms at the end, and silence doesn't shift it.</p>
</div>
<h2 id="risks">10. Risks</h2>
<div class="risk"><strong>Two GPUs.</strong> This is the reason for the fallback. One step beyond it: with the NVIDIA control panel set to the high-performance GPU for everything, both methods might be affected. Phase 1 tests your laptop with and without that setting.</div>
<div class="risk"><strong>HDR screens.</strong> Asking for 8-bit BGRA from an HDR screen gives Windows' own tone mapping, which can look washed out. Recording HDR (16-bit float frames, HEVC 10-bit) is out of scope. The toolbar will say when the screen is HDR.</div>
<div class="risk"><strong>Protected content</strong> (DRM video in browsers, some streaming apps) records as black with either method. Nothing can be done about it, so it's documented in the README.</div>
<div class="risk"><strong>Laptop power.</strong> 60 frames a second at 4K keeps the GPU awake. The default is 30, and the probe measures the cost.</div>
<h2 id="questions">11. Open questions</h2>
<ol>
<li><strong>PrintScreen during a recording:</strong> take the whole screen without the overlay (this plan), ignore it, or freeze anyway and accept the frozen moment in the video?</li>
<li><strong>The folder:</strong> recordings in <em>Screen Captures</em> beside stills, or a folder of their own (<em>Screen Recordings</em>)? This plan uses <em>Screen Captures</em>, so the grid shows both together.</li>
<li><strong>A time limit:</strong> none by default, apart from stopping when the disk has less than 2 GB free?</li>
<li><strong>A countdown</strong> (3, 2, 1) before recording starts, as Snagit has? This plan starts at once.</li>
<li><strong>Mouse clicks:</strong> should they be shown in the video, for example as a ring at each click, as Snagit can?</li>
</ol>
</main>
</body>
</html>