MusaMahmood/mcap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: MusaMahmood:8c8b2c6e38093e560e577cec26991de9dbe1934e
Branches Tags
MusaMahmood:master
MusaMahmood:gpu-bench
MusaMahmood:v0.1.0
..
compare: MusaMahmood:74b273f0e0ce63502e249f2dd48ced2686738cd3
Branches Tags
MusaMahmood:master
MusaMahmood:gpu-bench
MusaMahmood:v0.1.0

10 Commits
8c8b2c6e38 ... 74b273f0e0

Author SHA1 Message Date
Musa Mahmood
74b273f0e0 [0.1.34] macOS phase 7: screen recording with Opt+Cmd+Shift+2, through ScreenCaptureKit and VideoToolbox 2026-10-17 21:17:00 -04:00
Musa Mahmood
63a0cdb64e [0.1.33] macOS phase 6: screen capture stills with Cmd+Shift+2, through ScreenCaptureKit 2026-10-17 21:17:00 -04:00
Musa Mahmood
91f6cf768f [0.1.32] macOS phase 5: Mask2Former segmentation on the GPU through native Core ML 2026-10-17 21:17:00 -04:00
Musa Mahmood
a1ab9f3a96 [0.1.31] macOS phase 5: PP-OCRv5 server OCR on the GPU through native Core ML 2026-10-17 21:17:00 -04:00
Musa Mahmood
30cd00b6f2 [0.1.30] macOS phase 5: PP-OCRv5 mobile OCR on the GPU through native Core ML 2026-10-17 21:16:59 -04:00
Musa Mahmood
2d0fb855b1 [0.1.29] macOS phase 5: SigLIP 2 labels on the GPU through native Core ML 2026-10-17 21:16:31 -04:00
Musa Mahmood
3e353c77d8 [0.1.28] macOS phase 4: videos through FFmpeg 6.0 for macOS, an LGPL build made by build_macos.jai 2026-10-17 21:16:30 -04:00
Musa Mahmood
a66928b223 [0.1.27] macOS phase 3: HEIC, AVIF, WebP, TIFF and RAW through ImageIO 2026-10-17 21:16:30 -04:00
Musa Mahmood
5571c21d0b [0.1.26] macOS phase 2: folder picker, the Trash, the temporary folder, copying images 2026-10-17 21:16:30 -04:00
Musa Mahmood
12d0f30fc4 [0.1.25] macOS phase 1: builds and runs on a Mac, the library folder as a setting 2026-10-17 21:16:29 -04:00
72 changed files with 18062 additions and 1312 deletions
Show Stats Expand all files Collapse all files

5
.gitignore vendored
View File

@ -27,5 +27,10 @@ Thumbs.db
/modules/Labels/models/
# Segmentation: exported by tools/export_mask2former.py, not committed
/modules/Segment/models/
# FFmpeg for macOS: the source, build and headers modules/ffmpeg/build_macos.jai makes (its libraries in macos/ are kept)
/modules/ffmpeg/macos-build/
# The ScreenCaptureKit shim, as first.jai compiles it
/modules/Screen_Record/macos/*.o
/modules/Screen_Record/macos/*.a
# Releases: made by release.jai
/release/

78
README.md
View File

@ -9,9 +9,9 @@ and labels are tested on Windows with an NVIDIA GPU).
The `.mcap` files come from the **Import** window, which takes any folder of images (PNG, JPEG, HEIC,
AVIF, WebP, GIF, BMP, TIFF, camera RAW), videos (MP4, MOV, MKV, WebM, AVI, WMV, MPEG) and Snagit
`.snagx` captures, or from `snagit-converter`
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. On
Windows, **PrintScreen** captures a region of the screen straight into the library, as Snagit does;
see [Screen capture](#screen-capture).
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. **PrintScreen**
on Windows, **Cmd+Shift+2** on macOS, captures a region of the screen straight into the library, as
Snagit does; see [Screen capture](#screen-capture).
## Getting started
@ -28,11 +28,22 @@ see [Screen capture](#screen-capture).
place (`C:\Program Files\NVIDIA\CUDNN`) is found without changing PATH. The OCR window says
what, if anything, is missing.
**On macOS**, labelling, OCR and segmentation run on the GPU through Core ML, from Core ML copies
of the models made once with `tools/export_siglip_coreml.py` (SigLIP 2's image model, about ten
times as fast as the CPU), `tools/export_ppocr_coreml.py` (PP-OCRv5's detector and recognizer,
about twice as fast; run it with `--model mobile` and `--model server`) and
`tools/export_mask2former_coreml.py` (Mask2Former, about four times as fast; after
`tools/export_mask2former.py`). Their first lines say how to set up the Python they need. Without them, each runs on the
CPU. The first scan after making them is slower: Core ML prepares each image size the first time it
meets it, and macOS keeps that for later runs.
3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime
and the models beside the program.
4. **The library** is `D:\McapIndex\mcap`, a folder for each place imported from
(`SnagitLibraryPersonal`, `SnagitLibraryWork`, ...); `-library FOLDER` shows another folder.
4. **The library** is `D:\McapIndex\mcap` on Windows and `~/Pictures/mcap` on macOS and Linux, a
folder for each place imported from (`SnagitLibraryPersonal`, `SnagitLibraryWork`, ...).
`folder:` under `[library]` in `settings.config` names another, read when mcap starts, and
`-library FOLDER` overrides both.
**Import** (button above the grid) brings in more; see [Importing](#importing).
5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**,
@ -55,14 +66,15 @@ see [Screen capture](#screen-capture).
what search needs from every file, is still loading: a line beside the count says so). **Size**
sets the thumbnails' size. Click a thumbnail to open it in the Capture window.
- **Delete:** hover a thumbnail and click the trash button in its corner. You are asked whether to
move the file to the Recycle Bin or to delete it permanently, overwriting it with random bytes
move the file to the Recycle Bin (the Trash on macOS, a `deleted` folder in the library on Linux)
or to delete it permanently, overwriting it with random bytes
first; "Don't ask again" keeps the choice until the app restarts. Only the `.mcap` goes: the
`.snagx` it was converted from is untouched.
- **Copy:** click a thumbnail to select it (it is outlined, and shown in the Capture window), then
**Ctrl-C** copies its image, as the Capture window shows it (annotations and all; a video's
frame), and **Ctrl-Shift-C** its full path. Neither acts while you type in the search box. They
are `copy_image` and `copy_path` in `[hotkeys]` in `settings.config`. Copying images is Windows
only so far.
are `copy_image` and `copy_path` in `[hotkeys]` in `settings.config` (**Cmd** for Ctrl on macOS).
Copying images works on Windows and macOS, not yet on Linux.
- **Ctrl-F** puts the keyboard in the search box, its text selected (`focus_search` in `[hotkeys]`).
- **Files added or deleted outside the app** (in Explorer, by another program) appear in the grid
or leave it on their own, folders added, renamed or deleted too: the library is watched.
@ -170,8 +182,8 @@ of them are segmented, it also shows **What is in the images**: how many images
## Screen capture
On Windows, **PrintScreen** captures any part of the screen, from any app, even while mcap is hidden
in the tray:
**PrintScreen** on Windows, **Cmd+Shift+2** on macOS, captures any part of the screen, from any app,
even while mcap is hidden in the tray or the menu bar:
1. **The screens freeze** the moment it is pressed: everything on every screen is copied and shown
darkened over them, so menus, tooltips and video hold still while you choose.
@ -194,9 +206,21 @@ have PrintScreen: if Snagit is running it has it, and so does Windows while Sett
Accessibility > Keyboard > "Use the Print screen key to open screen capture" is on. When mcap cannot
register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S capture_region`.
On macOS (Cmd+Shift+3, 4 and 5 are the system's own screenshot keys):
- **Screen Recording** must be allowed for mcap (System Settings > Privacy & Security > Screen
Recording). The first capture asks for it and opens that pane; macOS may want mcap restarted after.
The permission belongs to the signed `mcap.app`, so start that, not the bare executable, which
would have it asked for on behalf of the terminal.
- Each screen freezes in a window of its own, and a selection stays on the screen it began on. What
is taken is that screen's own pixels, two to a point on a Retina screen, and the magnifier, the
position and the size are in them.
- The screens are copied with ScreenCaptureKit (`modules/Screen_Record/macos/capture.m`, which the
build compiles with clang).
### Recording the screen
**Ctrl+PrintScreen** records a region of one screen as a video:
**Ctrl+PrintScreen** (**Opt+Cmd+Shift+2** on macOS) records a region of one screen as a video:
1. **The screens freeze**, as for a still, with the outline and magnifier in red.
2. **Drag a region**, or click (or press Enter) for the whole screen under the cursor:
@ -214,14 +238,15 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S
and **Mic** (each on or off), **Pause** (the time paused is left out of the video), **Stop**, and **Discard**, which asks
first.
- **Neither is in the recording.** Neither takes the focus from the window being recorded.
4. **Ctrl+PrintScreen again**, or Stop, stops it. The video goes into **Screen Captures** beside the stills,
4. **The same key again**, or Stop, stops it. The video goes into **Screen Captures** beside the stills,
as an H.264 MP4 in an `.mcap` with the window it was recorded from, and opens in the Capture window.
OCR and labels read a frame from it.
- **The screen** is taken with DXGI Desktop Duplication, or with Windows.Graphics.Capture where that
cannot run (a laptop with two GPUs, a rotated screen).
cannot run (a laptop with two GPUs, a rotated screen). On macOS, with a ScreenCaptureKit stream
(`modules/Screen_Record/macos/capture.m`), which also brings the sound.
- **Encoding** uses NVENC where there is an NVIDIA GPU, else AMD's or Intel's encoder, Media
Foundation, or libx264.
Foundation, or libx264. On macOS, VideoToolbox.
- **Losing nothing:**
- A recording that stops by itself keeps what it has, and so does quitting while recording.
- When the screen is lost for a moment (a UAC prompt, the lock screen, a resolution change), the
@ -230,11 +255,13 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S
keeps all but its last two seconds or so.
- **Settings:** `[video]` in the settings file:
- `fps` (30) and `cursor` (true).
- `capture`: `auto` (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`.
- `capture` (Windows): `auto` (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`.
- `system_audio` (true): what the computer plays, from the default output device. The toolbar's
System sound button sets it.
- `microphone` (false): the default input device. The toolbar's Mic button sets it.
- Sound is AAC at 160 kb/s. A device unplugged mid-recording is picked up again from the new default.
- On macOS the microphone needs macOS 15 or later, and its permission (asked for the first time the
Mic button turns it on).
- **PrintScreen while recording** takes the whole screen under the cursor at once, without freezing
the screens, which would show in the recording.
- **The key** is `record_region` in `[global_hotkeys]`.
@ -243,7 +270,7 @@ register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S
**Import** (above the grid) brings a folder of screenshots, photos and videos into the library:
1. **Import from:** the folder, typed or chosen with **Browse...**. Everything under it is looked
1. **Import from:** the folder, typed or chosen with **Browse...** (Windows and macOS). Everything under it is looked
at, its folders too (not hidden ones, such as `.thumbnails`).
2. **Into library folder:** the folder in the library it goes into, such as `SnagitLibraryWork` or
`Phone`. It starts as the name of the folder imported from, or the library folder that folder went
@ -259,13 +286,14 @@ What becomes of each file:
way up (by its EXIF orientation), and where it came from: its file name (searched by `title:`), the
camera or phone (`app:`), and the date it was taken, from its EXIF, else a date in its name
(`IMG_20240512_101530`, `Screenshot 2024-05-12 at 10.15.30`), else the file's time. It is named
`2024-05-12_10-15-30_1a2b3c4d.mcap`, the date and the start of its hash. HEIC, AVIF, WebP and RAW
are read by Windows' own codecs: the HEIF, AV1, WebP and Raw Image Extensions from the Microsoft
Store.
`2024-05-12_10-15-30_1a2b3c4d.mcap`, the date and the start of its hash. HEIC, AVIF, WebP, TIFF and RAW
are read by the system's own codecs: on Windows the HEIF, AV1, WebP and Raw Image Extensions from
the Microsoft Store, and on macOS ImageIO, which has them all built in. Linux reads only PNG, JPEG,
GIF, BMP, PSD and TGA.
- **Animated GIFs** are images; their thumbnail is marked **GIF**, and the Capture window says how
long they run and opens them in the default viewer.
- **Videos** (MP4, MOV, M4V, 3GP, MKV, WebM, AVI, WMV, MPEG, MTS and M2TS, FLV), read with FFmpeg
(`modules/ffmpeg`, Windows only): an `.mcap` holding the file exactly as it was, of any size (it is
(`modules/ffmpeg`; Windows and macOS, not Linux): an `.mcap` holding the file exactly as it was, of any size (it is
copied a few MB at a time, never read whole), and a frame from it a tenth of the way in (at most 3
seconds), turned the right way up, which is what the grid, the Capture window, OCR and labels see.
The date is the phone's (Apple's creation date), else one in its name, else the video's own
@ -394,7 +422,7 @@ then in that order instead of by date.
| --- | --- |
| `FILE` (first) | Opens FILE: an `.mcap`, image or video; in view mode when mcap is already running. See [Opening a file](#opening-a-file). |
| `-open-import DEST` | With FILE: imports it into the library folder DEST as its Import button does, then exits: 0 once the library has it (imported, or there already), else 1. |
| `-library FOLDER` | Shows the `.mcap` files in FOLDER (and its folders) instead of `D:\McapIndex\mcap`. |
| `-library FOLDER` | Shows the `.mcap` files in FOLDER (and its folders) instead of the library folder in `settings.config` (`[library] folder`). |
| `-import SOURCE DEST` | Imports SOURCE, a folder or a single file, into the library folder DEST, as the Import window does, then exits: 0, or 1 if any file failed. |
| `-search QUERY` | Starts with QUERY in the search box. |
| `-cold` | Loads the library as on a cold start: its files and the catalog cache are dropped from the system's file cache first (Windows and Linux; not macOS, which allows it only with `sudo purge`). The `[startup]` lines in the log time each step. With `-stats`, a startup benchmark: `bin\mcap.exe -cold -stats`. |
@ -452,7 +480,8 @@ converted copy.
library index up to date (hashing the image in any `.mcap` it has not seen), then imports on
several threads; captures written are handed to the grid each frame.
- `modules/Image_Decode`: any image file's bytes to pixels, the right way up: stb_image for PNG,
JPEG, GIF, BMP, PSD and TGA, Windows Imaging Component for the rest; EXIF (from JPEG, PNG, WebP,
JPEG, GIF, BMP, PSD and TGA, and for the rest Windows Imaging Component (`wic.jai`) or macOS's
ImageIO (`imageio.jai`); EXIF (from JPEG, PNG, WebP,
HEIF and TIFF); previews; and videos' frames and what their containers say, with FFmpeg
(`video.jai`), and how many frames a GIF has. `bin\image-probe FILE...` (`jai first.jai -
image_probe`) shows what it makes of files.
@ -517,10 +546,11 @@ Binaries are written to `bin/`, with `assets/` copied beside them.
| Platform | Output | Notes |
| --- | --- | --- |
| Windows | `bin/mcap-debug.exe`, `bin/mcap.exe` | The release build gets the icon and a manifest, and no console. |
| macOS | `bin/mcap-macos-debug`, `bin/mcap.app` | Apple silicon (arm64) only, with a native `jai`. The release build is an ad hoc signed `.app`. |
| macOS | `bin/mcap-debug.app`, `bin/mcap.app` (and the bare `bin/mcap-macos-debug`, `bin/mcap-macos`) | Apple silicon (arm64) only, with a native `jai`; macOS 14 or later. Each `.app` holds onnxruntime and the models, and is signed with the self-signed "Ex4 Dev" identity when the keychain has it (made by Ex4's `scripts/macos_dev_identity.sh`), so macOS keeps its permissions across builds; otherwise ad hoc. |
| Linux | `bin/mcap-linux-debug`, `bin/mcap-linux` | Needs SDL3 installed (`libSDL3.so`). |
`jai check_targets.jai - linux` type-checks the Linux build from any OS without linking it.
`jai check_targets.jai - linux` type-checks the Linux build from any OS without linking it (on the
host's CPU: from a Mac, Linux on arm64).
The macOS check only works on a Mac, since the Objective-C module runs code at compile time.
Run any build with `run_tests` to run the test suites (debug builds include them),

4
assets/data/default_global_hotkeys_macos.config
View File

@ -5,5 +5,5 @@
# Changes apply as soon as the file is saved.
none bring_to_foreground
none capture_region
none record_region
Cmd-Shift-2 capture_region
Opt-Cmd-Shift-2 record_region

34
check_targets.jai
View File

@ -8,8 +8,10 @@
#run,stallable {
set_build_options_dc(.{do_output = false});
args := get_build_options().compile_time_command_line;
// Linux on the host's CPU: the ARM compiler cannot run x64 code at compile time (String's #asm
// in the hotkey checks' #run), so from a Mac it checks Linux on arm64.
target := Operating_System_Tag.LINUX;
cpu := CPU_Tag.X64;
cpu := CPU;
if array_find(args, "macos") { target = .MACOS; cpu = .ARM64; }
options := get_build_options();
@ -38,6 +40,9 @@ SHIM, jai_modules));
options.import_path = paths;
}
// The macOS check links nothing, but the ScreenCaptureKit shim is built as first.jai builds it.
#if OS == .MACOS if target == .MACOS build_macos_shim();
w := compiler_create_workspace("check");
set_build_options(options, w);
compiler_begin_intercept(w);
@ -47,7 +52,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
GIT_BRANCH :: "check";
GIT_HASH :: "-";
VERSION :: "0.1.24";
VERSION :: "0.1.34";
BUILD_DATE :: "-";
DEBUG :: true;
COMPILE_TESTS :: true;
@ -65,6 +70,31 @@ S, w);
compiler_end_intercept(w);
}
#if OS == .MACOS {
// macOS: modules/Screen_Record/macos/capture.m, mcap's ScreenCaptureKit shim, compiled with clang
// into capture.a beside it whenever the source is newer, for the app to link (screen_capture_macos.jai).
// check_targets.jai has the same; keep the two in step.
build_macos_shim :: () -> bool {
FOLDER :: "modules/Screen_Record/macos";
source := tprint("%/capture.m", FOLDER);
object := tprint("%/capture.o", FOLDER);
library := tprint("%/capture.a", FOLDER);
source_time, _, source_ok := file_modtime_and_size(source);
header_time := file_modtime_and_size(tprint("%/capture.h", FOLDER));
library_time, _, library_ok := file_modtime_and_size(library);
if source_ok && library_ok && library_time >= source_time && library_time >= header_time return true;
result := run_command("clang", "-c", "-fobjc-arc", "-O2", "-Wall", "-mmacosx-version-min=14.0", source, "-o", object);
if result.exit_code != 0 { log_error("ERROR: clang could not compile %.", source); return false; }
file_delete(library);
result = run_command("ar", "rcs", library, object);
if result.exit_code != 0 { log_error("ERROR: ar could not make %.", library); return false; }
log("Built %", library);
return true;
}
#import "Process";
#import "File_Utilities";
}
#import "Basic";
#import "Compiler";
#import "File";

164
first.jai
View File

@ -8,7 +8,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
VERSION :: "0.1.24";
VERSION :: "0.1.34";
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
// builds never write over the ones you run by hand.
build_directory := "bin";
@ -54,6 +54,7 @@ build_project :: () {
set_working_directory(#filepath); // so that we load fonts correctly.
make_directory_if_it_does_not_exist(build_directory);
#if OS == .MACOS if !build_macos_shim() return;
// Workspaces compile concurrently once source is added, so every build is
// started before any is waited on.
@ -125,6 +126,7 @@ start_main_build :: (optimized: bool) -> Pending_Build {
#if CPU == .X64 options.backend = .X64; // the fastest to compile; LLVM elsewhere
options.output_executable_name = tprint("%-debug", output_executable_name_temp);
}
#if OS == .MACOS options.minimum_os_version = MACOS_MINIMUM_VERSION;
#if OS == .WINDOWS {
file_delete(tprint("%/%.pdb", build_directory, options.output_executable_name));
@ -166,6 +168,7 @@ start_tool_build :: (name: string, file: string, optimized: bool) -> Pending_Bui
options.output_executable_name = tprint("%-debug", name);
}
options.output_path = build_directory;
#if OS == .MACOS options.minimum_os_version = MACOS_MINIMUM_VERSION;
w := compiler_create_workspace(options.output_executable_name);
set_build_options(options, w);
@ -194,52 +197,163 @@ finish_main_build :: (build: Pending_Build) {
}
}
} else #if OS == .MACOS {
if build.optimized create_macos_app_bundle(build.executable_name);
create_macos_app_bundle(build.executable_name, build.optimized);
}
log("Built %/%", build_directory, build.executable_name);
}
#if OS == .MACOS {
MACOS_APP_BUNDLE :: #run tprint("%.app", APP_NAME);
// macOS: modules/Screen_Record/macos/capture.m, mcap's ScreenCaptureKit shim, compiled with clang
// into capture.a beside it whenever the source is newer, for the app to link (screen_capture_macos.jai).
// check_targets.jai has the same; keep the two in step.
build_macos_shim :: () -> bool {
FOLDER :: "modules/Screen_Record/macos";
source := tprint("%/capture.m", FOLDER);
object := tprint("%/capture.o", FOLDER);
library := tprint("%/capture.a", FOLDER);
source_time, _, source_ok := file_modtime_and_size(source);
header_time := file_modtime_and_size(tprint("%/capture.h", FOLDER));
library_time, _, library_ok := file_modtime_and_size(library);
if source_ok && library_ok && library_time >= source_time && library_time >= header_time return true;
result := run_command("clang", "-c", "-fobjc-arc", "-O2", "-Wall", "-mmacosx-version-min=14.0", source, "-o", object);
if result.exit_code != 0 { log_error("ERROR: clang could not compile %.", source); return false; }
file_delete(library);
result = run_command("ar", "rcs", library, object);
if result.exit_code != 0 { log_error("ERROR: ar could not make %.", library); return false; }
log("Built %", library);
return true;
}
// The oldest macOS it runs on: ScreenCaptureKit's screenshots need 14, and so does onnxruntime 1.30.
MACOS_MINIMUM_VERSION :: Build_Options.Os_Version.{ major = 14, minor = 0 };
// A self-signed code-signing identity, made once per Mac by Ex4's scripts/macos_dev_identity.sh
// and shared with it. macOS ties Screen Recording and the other privacy permissions to an app's
// signature, and an ad hoc one changes with every build, so they would be asked for again each
// time. Without it in the keychain, the bundle is signed ad hoc.
MACOS_SIGNING_IDENTITY :: "Ex4 Dev";
MACOS_BUNDLE_DYLIBS :: string.["libSDL3.0.dylib"];
// The .app holds only the executable, the dylibs it loads from beside itself, the icon and
// Info.plist. The app finds data/, logs/ and images/ in the folder holding the bundle; see
// app_install_directory.
create_macos_app_bundle :: (executable_name: string) {
app := tprint("%/%", build_directory, MACOS_APP_BUNDLE);
run_command("rm", "-rf", app);
// mcap.app for the release build, mcap-debug.app for the debug one, with an identifier of its own
// so that its permissions are granted apart. Each holds the executable, SDL3 and FFmpeg beside it,
// onnxruntime in Contents/Frameworks, and the icon, Info.plist and models in Contents/Resources.
// The app finds data/, logs/ and images/ in the folder holding the bundle (see
// app_install_directory), and the models in Contents/Resources (app_resource_directory).
//
// The bundle is updated in place, not made afresh: the models are some 900 MB, so they are copied
// only when changed, as APFS clones, which take no space of their own.
//
// Started with `open`, the bundle asks for permissions as itself; the bare executable started
// from a terminal would have them asked for, and granted, to the terminal.
create_macos_app_bundle :: (executable_name: string, optimized: bool) {
bundle_name := ifx optimized then tprint("%.app", APP_NAME) else tprint("%-debug.app", APP_NAME);
identifier := ifx optimized then APP_BUNDLE_IDENTIFIER else tprint("%.debug", APP_BUNDLE_IDENTIFIER);
display := ifx optimized then APP_NAME else tprint("% Debug", APP_NAME);
app := tprint("%/%", build_directory, bundle_name);
executable_folder := tprint("%/Contents/MacOS", app);
frameworks_folder := tprint("%/Contents/Frameworks", app);
resources_folder := tprint("%/Contents/Resources", app);
// The executable folder is made afresh, so a renamed executable leaves nothing behind.
run_command("rm", "-rf", executable_folder);
make_directory_if_it_does_not_exist(executable_folder, recursive = true);
make_directory_if_it_does_not_exist(frameworks_folder, recursive = true);
make_directory_if_it_does_not_exist(resources_folder, recursive = true);
copied := copy_file(tprint("%/%", build_directory, executable_name), tprint("%/%", executable_folder, executable_name));
for MACOS_BUNDLE_DYLIBS copied &&= copy_file(tprint("assets/%", it), tprint("%/%", executable_folder, it));
// FFmpeg, linked against, so beside the executable as SDL3 is (@loader_path).
for file_list("modules/ffmpeg/macos") if ends_with(it, ".dylib") copied &&= copy_file(it, tprint("%/%", executable_folder, path_filename(it)));
if !copied {
log_error("ERROR: Couldn't copy the executable and its dylibs into %\n", app);
return;
}
run_command("chmod", "+x", tprint("%/%", executable_folder, executable_name));
// sips makes an .icns only from a size the format holds, and icon.png may be any square size.
resized := tprint("%/icon_1024.png", build_directory);
result := run_command("sips", "-z", "1024", "1024", "assets/images/icon.png", "--out", resized);
if result.exit_code == 0 result = run_command("sips", "-s", "format", "icns", resized, "--out", tprint("%/%.icns", resources_folder, APP_FOLDER_NAME));
file_delete(resized);
if result.exit_code != 0 log_error("ERROR: Couldn't make the icon for %; the Dock will show the generic one.\n", app);
// onnxruntime, and the models, as copy_assets puts them beside the executable.
ORT_LIBRARY :: "modules/onnxruntime/macos/libonnxruntime.dylib";
if file_exists(ORT_LIBRARY) clone_if_changed(ORT_LIBRARY, tprint("%/libonnxruntime.dylib", frameworks_folder));
Model_Folder :: struct { module: string; folder: string; }
for Model_Folder.[.{"OCR", "ocr-models"}, .{"Labels", "label-models"}, .{"Segment", "segment-models"}] {
folder := tprint("%/%", resources_folder, it.folder);
make_directory_if_it_does_not_exist(folder);
clone_if_changed(tprint("modules/%/models.config", it.module), tprint("%/models.config", folder));
models := tprint("modules/%/models", it.module);
if file_exists(models) for file_list(models) clone_if_changed(it, tprint("%/%", folder, path_filename(it)));
if file_exists(models) clone_packages(models, folder);
}
if !write_entire_file(tprint("%/Contents/Info.plist", app), tprint(MACOS_INFO_PLIST, APP_NAME, APP_NAME, executable_name, APP_BUNDLE_IDENTIFIER, APP_FOLDER_NAME, VERSION, VERSION)) {
// An .iconset of every size, made with sips, then iconutil: sips -s format icns fails on
// macOS 27 (error 13), and icon.png may be any square size.
icon := tprint("%/%.icns", resources_folder, APP_FOLDER_NAME);
if !file_exists(icon) {
iconset := tprint("%/%.iconset", build_directory, APP_FOLDER_NAME);
run_command("rm", "-rf", iconset);
make_directory_if_it_does_not_exist(iconset);
result: Process_Result;
for size: s32.[16, 32, 128, 256, 512] {
for scale: 1..2 {
pixels := tprint("%", size * scale);
name := ifx scale == 1 then tprint("icon_%x%.png", size, size) else tprint("icon_%x%@2x.png", size, size);
result = run_command("sips", "-z", pixels, pixels, "assets/images/icon.png", "--out", tprint("%/%", iconset, name));
if result.exit_code != 0 break size;
}
}
if result.exit_code == 0 result = run_command("iconutil", "-c", "icns", iconset, "-o", icon);
run_command("rm", "-rf", iconset);
if result.exit_code != 0 log_error("ERROR: Couldn't make the icon for %; the Dock will show the generic one.\n", app);
}
minimum := tprint("%.%", MACOS_MINIMUM_VERSION.major, MACOS_MINIMUM_VERSION.minor);
if !write_entire_file(tprint("%/Contents/Info.plist", app), tprint(MACOS_INFO_PLIST, display, display, executable_name, identifier, APP_FOLDER_NAME, VERSION, VERSION, minimum)) {
log_error("ERROR: Couldn't write %/Contents/Info.plist\n", app);
return;
}
// Ad hoc: a signature over the whole bundle, which macOS expects, not a Developer ID.
result = run_command("codesign", "--force", "--deep", "--sign", "-", app);
if result.exit_code != 0 log_error("WARNING: Couldn't sign %; it may still run.\n", app);
log("Built % (signed %)", app, sign_macos_app_bundle(app));
}
log("Built %", app);
// With MACOS_SIGNING_IDENTITY when the keychain has it, and otherwise ad hoc, which is still
// a signature over the whole bundle, as macOS expects. Returns what it was signed with.
sign_macos_app_bundle :: (app: string) -> string {
result, identities := run_command("security", "find-identity", "-p", "codesigning", capture_and_return_output = true);
if result.exit_code == 0 && contains(identities, tprint("\"%\"", MACOS_SIGNING_IDENTITY)) {
result = run_command("codesign", "--force", "--deep", "--sign", MACOS_SIGNING_IDENTITY, app);
if result.exit_code == 0 return MACOS_SIGNING_IDENTITY;
log_error("WARNING: Couldn't sign % as %; signing it ad hoc instead.\n", app, MACOS_SIGNING_IDENTITY);
} else {
log("No \"%\" identity in the keychain, so % is signed ad hoc, and macOS will ask for its permissions again after every build. Ex4's scripts/macos_dev_identity.sh makes one.", MACOS_SIGNING_IDENTITY, app);
}
result = run_command("codesign", "--force", "--deep", "--sign", "-", app);
if result.exit_code != 0 {
log_error("WARNING: Couldn't sign %; it may still run.\n", app);
return "not at all";
}
return "ad hoc";
}
// Core ML packages (.mlpackage, folders) in from, cloned whole into to: made afresh each build, which
// as clones costs no time or space.
clone_packages :: (from: string, to: string) {
visit_files(from, false, to, (info: *File_Visit_Info, to: string) {
if !ends_with(info.short_name, ".mlpackage") return;
target := tprint("%/%", to, info.short_name);
run_command("rm", "-rf", target);
result := run_command("cp", "-c", "-R", info.full_name, target);
if result.exit_code != 0 log_error("Could not copy % to %.", info.full_name, target);
}, visit_files = false, visit_directories = true);
}
// As copy_if_changed, but an APFS clone (cp -c), which shares the original's blocks.
clone_if_changed :: (from: string, to: string) {
from_time, from_size, from_ok := file_modtime_and_size(from);
to_time, to_size, to_ok := file_modtime_and_size(to);
if to_ok && from_ok && to_size == from_size && to_time >= from_time return;
file_delete(to);
result := run_command("cp", "-c", "-p", from, to);
if result.exit_code != 0 && !copy_file(from, to) log_error("Could not copy % to %.", from, to);
}
MACOS_INFO_PLIST :: #string PLIST
@ -257,7 +371,9 @@ finish_main_build :: (build: Pending_Build) {
<key>CFBundlePackageType</key> <string>APPL</string>
<key>CFBundleShortVersionString</key> <string>%</string>
<key>CFBundleVersion</key> <string>%</string>
<key>LSMinimumSystemVersion</key> <string>%</string>
<key>NSHighResolutionCapable</key> <true/>
<key>NSMicrophoneUsageDescription</key> <string>mcap records the microphone into a screen recording when its Mic button is on.</string>
</dict>
</plist>
PLIST
@ -286,13 +402,17 @@ copy_assets :: () {
copy_if_changed("modules/OCR/models.config", tprint("%/ocr-models/models.config", build_directory));
if file_exists("modules/OCR/models") {
for file_list("modules/OCR/models") copy_if_changed(it, tprint("%/ocr-models/%", build_directory, path_filename(it)));
#if OS == .MACOS clone_packages("modules/OCR/models", tprint("%/ocr-models", build_directory));
} else {
log("OCR: the models have not been fetched; run jai fetch_models.jai in modules/OCR for OCR.");
}
// Videos: FFmpeg's DLLs beside the executable (modules/ffmpeg/windows, in git), the same way.
// Videos: FFmpeg's libraries beside the executable (modules/ffmpeg/windows and macos, in git), the
// same way. The executable finds the dylibs there through its @loader_path rpath.
#if OS == .WINDOWS {
for file_list("modules/ffmpeg/windows") if ends_with(it, ".dll") copy_if_changed(it, tprint("%/%", build_directory, path_filename(it)));
} else #if OS == .MACOS {
for file_list("modules/ffmpeg/macos") if ends_with(it, ".dylib") copy_if_changed(it, tprint("%/%", build_directory, path_filename(it)));
}
// Labels: its models in bin/label-models, the same way.
@ -300,6 +420,7 @@ copy_assets :: () {
copy_if_changed("modules/Labels/models.config", tprint("%/label-models/models.config", build_directory));
if file_exists("modules/Labels/models") {
for file_list("modules/Labels/models") copy_if_changed(it, tprint("%/label-models/%", build_directory, path_filename(it)));
#if OS == .MACOS clone_packages("modules/Labels/models", tprint("%/label-models", build_directory));
} else {
log("Labels: the models have not been fetched; run jai fetch_models.jai in modules/Labels for labels.");
}
@ -309,6 +430,7 @@ copy_assets :: () {
copy_if_changed("modules/Segment/models.config", tprint("%/segment-models/models.config", build_directory));
if file_exists("modules/Segment/models") {
for file_list("modules/Segment/models") copy_if_changed(it, tprint("%/segment-models/%", build_directory, path_filename(it)));
#if OS == .MACOS clone_packages("modules/Segment/models", tprint("%/segment-models", build_directory));
} else {
log("Segmentation: the model has not been made; see modules/Segment/README.md.");
}

335
macos-plan.html
View File

@ -1,335 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>macOS Port Plan</title>
<style>
:root {
--bg: #fbfaf8; --surface: #ffffff; --text: #1d1d1f; --dim: #5d5d66; --line: #e3e1dc;
--accent: #3a5bd9; --accent-soft: #e9edfb; --warn: #9a5b00; --warn-soft: #fbf1e1;
--code-bg: #f2f0ec; --ok: #2e7d4f; --ok-soft: #e5f3ea;
}
@media (prefers-color-scheme: dark) {
:root:not([data-theme="light"]) {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
}
:root[data-theme="dark"] {
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
}
* { box-sizing: border-box; }
body {
margin: 0; background: var(--bg); color: var(--text);
font: 16px/1.6 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
}
main { max-width: 860px; margin: 0 auto; padding: 40px 16px 80px; }
header p { color: var(--dim); margin-top: 4px; }
h1 { font-size: 2rem; line-height: 1.2; margin: 0; }
h2 { font-size: 1.35rem; margin: 48px 0 8px; padding-top: 16px; border-top: 1px solid var(--line); }
h3 { font-size: 1.05rem; margin: 28px 0 6px; }
p, li { max-width: 72ch; }
a { color: var(--accent); }
code, pre { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; font-size: 0.88em; }
code { background: var(--code-bg); padding: 1px 5px; border-radius: 4px; }
pre { background: var(--code-bg); padding: 12px 14px; border-radius: 8px; overflow-x: auto; line-height: 1.45; }
pre code { background: none; padding: 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 0.94rem; }
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
th { color: var(--dim); font-weight: 600; }
.table-wrap { overflow-x: auto; }
.summary { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 16px 20px; margin-top: 24px; }
.summary ul { margin: 6px 0 0; padding-left: 20px; }
.note { background: var(--accent-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk { background: var(--warn-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
.risk strong { color: var(--warn); }
.tag { display: inline-block; font-size: 0.78rem; font-weight: 600; padding: 1px 8px; border-radius: 99px; background: var(--accent-soft); color: var(--accent); }
.tag.later { background: var(--code-bg); color: var(--dim); }
.tag.decided { background: var(--ok-soft); color: var(--ok); }
nav ol { columns: 2; column-gap: 32px; padding-left: 20px; margin: 8px 0 0; }
@media (max-width: 600px) { nav ol { columns: 1; } h1 { font-size: 1.6rem; } }
.phase { border-left: 3px solid var(--accent); padding: 2px 0 2px 14px; margin: 16px 0; }
.phase h3 { margin-top: 0; }
figure { margin: 18px 0; }
figcaption { color: var(--dim); font-size: 0.88rem; margin-top: 6px; }
svg text { fill: var(--text); font: 13px system-ui, sans-serif; }
svg .dim { fill: var(--dim); }
svg .box { fill: var(--surface); stroke: var(--line); }
svg .hl { fill: var(--accent-soft); stroke: var(--accent); }
svg .arrow { stroke: var(--dim); fill: none; marker-end: url(#a); }
</style>
</head>
<body>
<main>
<header>
<h1>mcap on macOS: building again, then feature parity</h1>
<p>Planning document · 2026-10-10 · for mcap after 0.1.11 · <strong>not started</strong></p>
</header>
<div class="summary">
<strong>In short</strong>
<ul>
<li><strong>Where it stands:</strong> mcap was built to compile on macOS (Apple silicon, SDL3 and Metal), but it hasn't been built on a Mac since at least 0.0.8. Two build breaks were found and fixed in 0.1.11 without a Mac. Everything added since the template is Windows first, and much of it Windows only.</li>
<li><span class="tag decided">Proposed</span> <strong>Eight phases</strong>, smallest risk first:
<ol>
<li>Build and run on a Mac.</li>
<li>Fill the small platform gaps: choosing a folder, the Trash, the clipboard, the temporary folder.</li>
<li>Images through ImageIO.</li>
<li>Video through FFmpeg for macOS.</li>
<li>The models on the GPU through CoreML.</li>
<li>Screen capture stills.</li>
<li>Screen recording.</li>
<li>A signed release.</li>
</ol></li>
<li><span class="tag decided">Proposed</span> <strong>One codebase, the same file formats.</strong> An <code>.mcap</code> made on either system opens on the other. Where Windows uses WIC, DXGI and WASAPI, macOS uses ImageIO and ScreenCaptureKit. FFmpeg, onnxruntime, the encoder, the remux and the mixer stay shared.</li>
<li><span class="tag decided">Proposed</span> <strong>ScreenCaptureKit through a small Objective-C shim.</strong> It is asynchronous: completion blocks, delegates, dispatch queues. A few hundred lines of Objective-C, compiled with clang into a static library with a plain C interface, is far safer than doing that from Jai. Open question 6.</li>
<li><strong>The big unknowns</strong> need a Mac to settle. Which macOS to require (14 Sonoma is proposed). Signing: macOS remembers the Screen Recording permission per signature, and an ad hoc signature changes with every build. What the capture keys should be: Mac keyboards have no PrintScreen. See <a href="#questions">open questions</a>.</li>
</ul>
</div>
<nav>
<ol>
<li><a href="#today">Where it stands</a></li>
<li><a href="#gaps">What is Windows only</a></li>
<li><a href="#approach">Approach</a></li>
<li><a href="#capture">Screen capture on macOS</a></li>
<li><a href="#permissions">Permissions and signing</a></li>
<li><a href="#phases">Phases</a></li>
<li><a href="#risks">Risks</a></li>
<li><a href="#questions">Open questions</a></li>
</ol>
</nav>
<h2 id="today">1. Where it stands</h2>
<p>From the code; none of this has been run on a Mac in this project's history.</p>
<div class="table-wrap"><table>
<tr><th>Part</th><th>macOS today</th></tr>
<tr><td>Window, ImGui, themes, fonts</td><td>SDL3 with SDL_GPU on Metal (<code>src/backends/sdl3_gpu3.jai</code>). The menu-bar tray is SDL's. Fonts and CJK fallbacks have macOS paths (<code>platform_macos.jai</code>).</td></tr>
<tr><td>Build</td><td><code>jai first.jai</code> makes <code>bin/mcap-macos-debug</code>, and with <code>- release</code> <code>bin/mcap.app</code>, signed ad hoc. Apple silicon only, with a native <code>jai</code>. The <code>.app</code> holds only the executable and <code>libSDL3.0.dylib</code>. onnxruntime and the models are copied beside it, not into it.</td></tr>
<tr><td>Type check from Windows</td><td>Not possible: <code>jai check_targets.jai - macos</code> runs only on a Mac, because the Objective-C module runs code at compile time.</td></tr>
<tr><td>Known breaks, fixed in 0.1.11</td><td>
<ul>
<li><code>unix_sdl3_gpu3.jai</code> pulled all of ImGui into the app's scope, so ImGui's <code>Image</code> clashed with Image_Decode's (broken since 0.0.8).</li>
<li>A Windows <code>RECT</code> was in the shared <code>App</code> struct (since 0.1.6).</li>
</ul>
The Linux check passes after both, and macOS shares the same files.</td></tr>
<tr><td>Library location</td><td>Hard-coded as <code>D:/McapIndex/mcap</code> (<code>VIEWER_FOLDER</code>). On a Mac only <code>-library</code> works.</td></tr>
<tr><td>Global hotkeys</td><td>Carbon <code>RegisterEventHotKey</code> (<code>macos_global_hotkeys.jai</code>). <code>capture_region</code> and <code>record_region</code> default to <code>none</code>.</td></tr>
<tr><td>OCR, labels, segmentation</td><td>onnxruntime 1.30.0 for macOS arm64 is in <code>fetch.jai</code>. Sessions use CUDA or the CPU (<code>session.jai</code>), so on a Mac it's CPU only, several times slower than a GPU.</td></tr>
</table></div>
<h2 id="gaps">2. What is Windows only</h2>
<div class="table-wrap"><table>
<tr><th>Feature</th><th>Windows</th><th>macOS today</th><th>macOS plan</th></tr>
<tr><td>Images other than PNG, JPEG, GIF, BMP, PSD, TGA</td><td>WIC (<code>Image_Decode/wic.jai</code>)</td><td>Refused</td><td>ImageIO, <code>CGImageSource</code> (phase 3)</td></tr>
<tr><td>Videos: import, frame for the thumbnail</td><td>FFmpeg (<code>modules/ffmpeg</code>, Windows bindings only)</td><td>Skipped</td><td>FFmpeg for macOS (phase 4)</td></tr>
<tr><td>Import: choosing a folder</td><td>Explorer's picker (<code>windows_choose_folder</code>)</td><td>Typed path only</td><td><code>NSOpenPanel</code> (phase 2)</td></tr>
<tr><td>Delete to the bin</td><td>Recycle Bin (<code>SHFileOperationW</code>)</td><td>A "deleted" folder beside the file</td><td><code>NSFileManager trashItemAtURL</code> (phase 2)</td></tr>
<tr><td>Temporary folder (open original, recordings)</td><td><code>%TEMP%</code></td><td><code>/tmp</code></td><td><code>NSTemporaryDirectory()</code> (phase 2)</td></tr>
<tr><td>Image to the clipboard</td><td><code>CF_DIB</code></td><td>None</td><td><code>NSPasteboard</code>, PNG (phase 2)</td></tr>
<tr><td>Models on the GPU</td><td>CUDA</td><td>CPU</td><td>CoreML execution provider (phase 5)</td></tr>
<tr><td>Screen capture stills</td><td>GDI overlay, <code>BitBlt</code></td><td>None</td><td>Cocoa overlay, ScreenCaptureKit screenshots (phase 6)</td></tr>
<tr><td>Screen recording, border, toolbar</td><td>DXGI or Windows.Graphics.Capture, GDI windows</td><td>None</td><td>ScreenCaptureKit stream, Cocoa panels (phase 7)</td></tr>
<tr><td>Sound</td><td>WASAPI</td><td>None</td><td>ScreenCaptureKit audio, plus its microphone or AVAudioEngine (phase 7)</td></tr>
<tr><td>Release</td><td><code>release.jai</code>: a zip with CUDA</td><td>None</td><td>A signed <code>.app</code> in a <code>.dmg</code> (phase 8)</td></tr>
</table></div>
<p>Already shared and fine as they are:</p>
<ul>
<li>The <code>.mcap</code> format, catalog, search, Stats.</li>
<li><code>.snagx</code> import, EXIF (<code>exif.jai</code> reads it itself).</li>
<li>The settings and hotkey files.</li>
<li>The encoder, remux and mixer in <code>modules/Screen_Record</code>. Their capture sources aren't shared.</li>
</ul>
<h2 id="approach">3. Approach</h2>
<ul>
<li><strong>Same interfaces, a second implementation.</strong> Each Windows-only piece already sits behind a function the app calls. The macOS version goes beside it, in its own file:
<ul>
<li><code>platform_screen_capture_select</code>, <code>platform_copy_image_to_clipboard</code>.</li>
<li><code>image_decode</code>'s WIC branch.</li>
<li><code>video_poster</code>.</li>
<li><code>Screen_Record</code>'s source: a third source beside <code>dxgi_source</code> and <code>wgc_source</code>.</li>
</ul>
The app's own files (<code>screen_capture.jai</code>, <code>screen_record.jai</code>, <code>import.jai</code>) shouldn't need to know which system they're on beyond what they do now.</li>
<li><strong>FFmpeg on macOS too</strong>, rather than AVFoundation. Video import, the thumbnail frame, the recording encoder and the remux are FFmpeg code already. On a Mac the encoder would be <code>h264_videotoolbox</code>, Apple's hardware encoder, put first in <code>RECORD_ENCODERS</code>. The cost is shipping FFmpeg's dylibs in the <code>.app</code>, and bindings for their macOS names.</li>
<li><strong>The Objective-C shim</strong> (<code>modules/Screen_Record/macos/capture.m</code>) holds ScreenCaptureKit and nothing else. It has a C interface:
<ul>
<li><code>mcap_sck_screenshot</code></li>
<li><code>mcap_sck_stream_start</code></li>
<li><code>mcap_sck_next_frame</code></li>
<li><code>mcap_sck_next_audio</code></li>
<li><code>mcap_sck_stream_stop</code></li>
</ul>
It is compiled by <code>first.jai</code> with clang when building on a Mac, and linked statically. AppKit work that is plain message sends (<code>NSOpenPanel</code>, <code>NSPasteboard</code>, the overlay and toolbar windows) stays in Jai through the Objective-C module the app already uses.</li>
<li><strong>The build is checked on every phase</strong> with <code>jai check_targets.jai - macos</code> and the test suites on the Mac. <code>jai check_targets.jai - linux</code> on Windows catches the shared files between Mac sessions.</li>
</ul>
<h2 id="capture">4. Screen capture on macOS</h2>
<ul>
<li><strong>Stills:</strong> the moment the key is pressed, each screen is copied:
<ul>
<li>Copied with <code>SCScreenshotManager captureImageWithFilter</code>, on macOS 14 and later. <code>CGWindowListCreateImage</code> is deprecated in macOS 15.</li>
<li>Shown frozen in a borderless <code>NSWindow</code> per screen, at the screen-saver level. A window can't span screens on macOS as the one Windows overlay does.</li>
<li>The drag, magnifier and keys work as on Windows. The overlay's drawing (darkened copy, bright selection, outline, magnifier) becomes Core Graphics in each window's <code>drawRect</code>, or an <code>NSImage</code> drawn by the shim.</li>
</ul></li>
<li><strong>Points and pixels:</strong> AppKit works in points. A Retina screen has 2 pixels a point, and screens can differ. The region is chosen in points and converted to each screen's pixels for the capture. The 240 × 180 minimum is in pixels, as on Windows.</li>
<li><strong>Recording:</strong> an <code>SCStream</code> for the region's display (<code>sourceRect</code>) delivers BGRA <code>CVPixelBuffer</code>s with timestamps on the host clock.
<ul>
<li>The shim hands them over as the pipeline's frames, and the encoder and the rest are unchanged.</li>
<li>The cursor: <code>showsCursor</code>.</li>
<li>Keeping mcap's own border and toolbar out: an <code>SCContentFilter</code> that excludes the app's windows. That's cleaner than Windows' window flag, and doesn't depend on it.</li>
</ul></li>
<li><strong>Sound:</strong>
<ul>
<li>System sound comes from the same stream (<code>capturesAudio</code>, macOS 13 on), as 48 kHz float.</li>
<li>The microphone comes from the stream too on macOS 15 (<code>captureMicrophone</code>), else from <code>AVAudioEngine</code>'s input.</li>
<li>Both go into the mixer as packets with host-clock times, as WASAPI's do.</li>
</ul></li>
<li><strong>The border and toolbar:</strong> non-activating <code>NSPanel</code>s at the floating level. The toolbar's buttons ask, and the app acts on its next frame, as on Windows.</li>
<li><strong>The window it was taken from:</strong> the frontmost app comes from <code>NSWorkspace</code>, and window titles from <code>CGWindowListCopyWindowInfo</code>. The titles need the Screen Recording permission, which capture needs anyway.</li>
<li><strong>Hotkeys:</strong> no PrintScreen on Mac keyboards. ⌘⇧3, 4 and 5 are the system's screenshot keys. See open question 3.</li>
</ul>
<h2 id="permissions">5. Permissions and signing</h2>
<ul>
<li><strong>Screen Recording</strong> (Privacy &amp; Security) must be granted before any capture. The first capture asks; until it is granted, the overlay gets only the desktop, without other apps' windows. mcap will check (<code>CGPreflightScreenCaptureAccess</code>) and say what to do instead of showing an empty capture.</li>
<li><strong>Microphone</strong> permission is asked the first time the Mic button is turned on. <code>Info.plist</code> gains <code>NSMicrophoneUsageDescription</code>.</li>
<li><strong>macOS remembers these per code signature.</strong> The release <code>.app</code> is signed ad hoc today, and an ad hoc signature changes with every build, so each build would ask again, or silently stay refused. A stable signing identity solves it: an Apple Development certificate for builds, and a Developer ID for releases, notarized so Gatekeeper opens it. See open question 4.</li>
</ul>
<h2 id="phases">6. Phases</h2>
<p>Each phase ends with something to run and check on the Mac, and is a commit. They are ordered so a usable mcap comes first: by the end of phase 4 the library browser, search, imports and OCR all work.</p>
<div class="phase">
<h3>Phase 1: Build and run on a Mac</h3>
<ul>
<li><strong>The type check:</strong> <code>jai check_targets.jai - macos</code>, fixing what it finds.</li>
<li><strong>The builds:</strong> <code>jai first.jai</code> and <code>- release</code>.</li>
<li><strong>A library folder for macOS:</strong> a default in place of <code>D:/McapIndex/mcap</code>, open question 5.</li>
<li><strong>The <code>.app</code> made whole:</strong> <code>libonnxruntime.dylib</code> in <code>Contents/Frameworks</code>, and the OCR, label and segmentation models in <code>Contents/Resources</code>, found from there.</li>
<li><strong>Tests:</strong> <code>run_tests -headless</code> on the Mac.</li>
</ul>
<p><strong>Done when:</strong></p>
<ul>
<li>Both builds succeed and the test suites pass on the Mac.</li>
<li><code>mcap.app</code>, started from Finder, shows a library copied from Windows: thumbnails, the Capture window, search, Stats.</li>
<li>An OCR scan reads captures on the CPU, and its time per capture is recorded.</li>
</ul>
</div>
<div class="phase">
<h3>Phase 2: The small platform gaps</h3>
<ul>
<li><strong>Choosing a folder:</strong> <code>NSOpenPanel</code> for Import's Browse button.</li>
<li><strong>Delete:</strong> into the Trash with <code>NSFileManager trashItemAtURL</code>, with "Move to Trash" in the delete dialog.</li>
<li><strong>Temporary files:</strong> <code>NSTemporaryDirectory()</code> for opening an original and for recordings.</li>
<li><strong>The clipboard:</strong> <code>platform_copy_image_to_clipboard</code> through <code>NSPasteboard</code>, as PNG and TIFF.</li>
</ul>
<p><strong>Done when:</strong> Browse picks a folder; a deleted capture is in the Trash and can be put back from there; Open plays a video's original; an image pasted from the clipboard (phase 6 is its first user, so a test fills it) pastes into Preview.</p>
</div>
<div class="phase">
<h3>Phase 3: Images through ImageIO</h3>
<p><code>Image_Decode/imageio.jai</code>: <code>CGImageSourceCreateWithData</code> and <code>CGImageSourceCreateImageAtIndex</code>, drawn into an RGBA bitmap context. It takes the formats WIC takes on Windows:</p>
<ul>
<li>HEIC and AVIF (AVIF from macOS 13).</li>
<li>WebP (from macOS 11).</li>
<li>TIFF.</li>
<li>The camera RAW formats.</li>
</ul>
<p>EXIF orientation stays <code>exif.jai</code>'s. ImageIO is asked for the image as stored, so it isn't turned twice.</p>
<p><strong>Done when:</strong> <code>image-probe</code> on the Mac decodes the same test images as on Windows, the same way up and the same size, and importing an iPhone photo folder makes the same <code>.mcap</code> files, except for the import date.</p>
</div>
<div class="phase">
<h3>Phase 4: Video through FFmpeg for macOS</h3>
<ul>
<li><strong>FFmpeg itself:</strong> a shared FFmpeg 6 build for macOS arm64, with VideoToolbox and the native AAC encoder. Its dylibs and their SHA-256 sums go in <code>modules/ffmpeg/fetch.jai</code>, as onnxruntime's are.</li>
<li><strong>Bindings:</strong> <code>modules/ffmpeg/macos.jai</code>, from the same headers with the dylib names, and <code>#load</code>ed on macOS.</li>
<li><strong>The video code:</strong> <code>video_ffmpeg.jai</code> and the import's video path, no longer Windows only.</li>
<li><strong>The <code>.app</code>:</strong> the dylibs go in it.</li>
</ul>
<p><strong>Done when:</strong> importing a folder with MP4, MOV and HEVC iPhone videos gives the same <code>.mcap</code> files as on Windows, with thumbnails, and they play with Open.</p>
</div>
<div class="phase">
<h3>Phase 5: The models on the GPU, with CoreML</h3>
<ul>
<li><strong>CoreML first:</strong> <code>session.jai</code> asks onnxruntime for the CoreML execution provider on macOS (<code>SessionOptionsAppendExecutionProvider</code> with <code>"CoreML"</code>, the ML Program format), and the CPU for what it can't take.</li>
<li><strong>Measured per model:</strong> PP-OCR detection and recognition, SigLIP 2, Mask2Former. Each is timed against the CPU and checked to give the same output as on Windows, within the tolerances the Windows probes already use.</li>
<li><strong>The status line</strong> says CoreML, as it says CUDA on Windows.</li>
</ul>
<p><strong>Done when:</strong> OCR, labels and segmentation give the same results as on Windows on the probes' test images, and each is faster than the CPU, or is left on the CPU with the reason recorded.</p>
</div>
<div class="phase">
<h3>Phase 6: Screen capture stills</h3>
<ul>
<li><strong>The shim's screenshot</strong> (<code>mcap_sck_screenshot</code>, per display).</li>
<li><strong>The overlay:</strong> a window per screen, with the drag, magnifier, keys, and a click for the whole screen.</li>
<li><strong>The window it came from:</strong> its title and app.</li>
<li><strong>The clipboard</strong> (phase 2).</li>
<li><strong>The capture hotkey</strong> (open question 3).</li>
<li><strong>Permission:</strong> the check, and what mcap says when Screen Recording isn't granted.</li>
</ul>
<p><strong>Done when:</strong> on a Mac with a Retina screen and an external non-Retina one:</p>
<ul>
<li>The hotkey freezes both screens.</li>
<li>A region dragged on either is taken at the screen's own pixels, and is on the clipboard and in Screen Captures with its window's title and app.</li>
<li>With the permission refused, mcap says how to grant it.</li>
</ul>
</div>
<div class="phase">
<h3>Phase 7: Screen recording</h3>
<ul>
<li><strong>The source:</strong> <code>modules/Screen_Record/sck_source.jai</code> over the shim's stream, as the third source. The encoder tries <code>h264_videotoolbox</code> first.</li>
<li><strong>Sound:</strong> system sound and the microphone, through the existing mixer.</li>
<li><strong>The overlay's video mode</strong>, with the 240 × 180 minimum.</li>
<li><strong>The border and toolbar</strong> as panels, kept out of the stream by the content filter. Pause, Discard, the sound buttons, and leftovers after a crash.</li>
</ul>
<p><strong>Done when</strong> the same checks pass as on Windows:</p>
<ul>
<li><code>record-probe</code> runs on the Mac, with <code>-sync-test</code>, <code>-pause</code> and <code>-mic-toggle</code>.</li>
<li>Neither border nor toolbar is in a recording.</li>
<li>A 10 s pause leaves no gap.</li>
<li>Beeps stay within about 25 ms of their flashes with no drift over 3 minutes.</li>
<li>A killed recording is offered at the next start.</li>
</ul>
</div>
<div class="phase">
<h3>Phase 8: A signed release</h3>
<ul>
<li><strong><code>release.jai</code> for macOS:</strong> the <code>.app</code> with everything in it (SDL3, onnxruntime, FFmpeg, the models), signed with a Developer ID with the hardened runtime and the entitlements it needs.</li>
<li><strong>Notarized and stapled.</strong></li>
<li><strong>Packaged</strong> in a <code>.dmg</code>, with its SHA-256 printed, as the Windows zip's is.</li>
</ul>
<p><strong>Done when:</strong> the <code>.dmg</code>, downloaded on a second Mac, opens without a Gatekeeper warning, asks once for Screen Recording, and keeps that permission across an update to the next release.</p>
</div>
<h2 id="risks">7. Risks</h2>
<div class="risk"><strong>No Mac in this loop yet.</strong> Everything here can be written on Windows, but only built, run and checked on a Mac. Each phase needs a session there, yours or a remote one (open question 1). The Linux type check catches the shared files in between, but not the Objective-C or the macOS-only files.</div>
<div class="risk"><strong>CoreML and the models.</strong> CoreML takes some operators and not others. Mask2Former's attention and the panoptic post-processing may fall back to the CPU in parts, which can be slower than the CPU alone, as the data moves back and forth. Phase 5 measures each model, and keeps the CPU where CoreML doesn't win.</div>
<div class="risk"><strong>Permissions and signing.</strong> Without a stable signature, Screen Recording has to be granted again after every build, and a refused permission gives a capture of the empty desktop rather than an error. Development needs a certificate early, by phase 6.</div>
<div class="risk"><strong>FFmpeg licensing.</strong> The Windows FFmpeg is a GPL build (with libx264). On macOS an LGPL build is enough: VideoToolbox does H.264, and FFmpeg's own AAC encoder does sound. That avoids GPL obligations for the Mac release; the Windows release should be checked against the same question.</div>
<div class="risk"><strong>Multi-screen overlays.</strong> One window per screen, with Spaces, full-screen apps and screens at different scales, is where macOS differs most from Windows. The overlay must sit above full-screen apps (<code>NSWindowCollectionBehaviorCanJoinAllSpaces</code> and <code>FullScreenAuxiliary</code>).</div>
<h2 id="questions">8. Open questions</h2>
<ol>
<li><strong>The Mac:</strong> which Mac and macOS version will the work be checked on? Can I build and run there (SSH, or a remote session), or will you run the builds and send back the output?</li>
<li><strong>The oldest macOS:</strong> 14 Sonoma, for ScreenCaptureKit's screenshots, is proposed. 13 would need the deprecated <code>CGWindowListCreateImage</code> for stills; 15 would give the microphone in the stream itself.</li>
<li><strong>The keys:</strong> Mac keyboards have no PrintScreen, and ⌘⇧3/4/5 are the system's. Proposed: <strong>⌘⇧2</strong> for a still and <strong>⌃⌘⇧2</strong> to record, or F13 (PrintScreen on Apple's full keyboards) and ⌃F13, or your own.</li>
<li><strong>Signing:</strong> do you have, or want, an Apple Developer account (99 USD a year) for a stable development signature now and Developer ID notarization for releases? Without it, phases 6 and 7 work but ask for permission after every build.</li>
<li><strong>The library on a Mac:</strong> where should it default to? Proposed: <code>~/Pictures/mcap</code>. Should the library folder become a setting on both systems, in place of the hard-coded <code>D:/McapIndex/mcap</code>?</li>
<li><strong>The shim:</strong> is a small Objective-C file built with clang acceptable in the repo, or should ScreenCaptureKit be done from Jai alone, at more cost and risk?</li>
</ol>
</main>
</body>
</html>

264
modules/Core_ML/module.jai Normal file
View File

@ -0,0 +1,264 @@
// Core_ML: a Core ML model (an .mlpackage made by coremltools) run on one float tensor, on macOS.
// The models that run faster here than through onnxruntime on the CPU (tools/export_siglip_coreml.py
// says how much) use it; the rest stay with modules/onnxruntime.
//
// model, ok, error := coreml_load(path, "pixel_values", "pooler_output");
// ok, error = coreml_run(model, input, s64.[1, 3, 384, 384], output); // output: the floats it gives
// coreml_free(model);
//
// Loading compiles the package (Core ML's compileModelAtURL, a fraction of a second) into a temporary
// .mlmodelc, removed by coreml_free. What Core ML makes of a model for the GPU, the first time each
// size runs, is kept by macOS across runs (the very first run of the OCR models took 5 s more, later
// ones none). The model runs on the CPU and GPU, never the Neural Engine
// unless asked: with float16 weights it can be badly wrong there (SigLIP 2: cosine 0.72 to the CPU).
// A model runs one prediction at a time (a mutex), from any thread; outputs are read outside it.
Coreml_Units :: enum s64 { // MLComputeUnits
CPU_ONLY :: 0;
CPU_AND_GPU :: 1;
ALL :: 2;
CPU_AND_ENGINE :: 3;
}
Coreml_Model :: struct {
model: *NSObject; // MLModel, retained
compiled: *NSObject; // the .mlmodelc's NSURL, retained; removed on free
input_name, output_name: string;
mutex: Mutex;
}
coreml_load :: (path: string, input_name: string, output_name: string, units := Coreml_Units.CPU_AND_GPU) -> *Coreml_Model, ok: bool, error: string {
init_objective_c(); // before anything Objective-C, the pool included; once per process does it all
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
if !file_exists(path) return null, false, tprint("% is not there.", path);
package := NSURL.fileURLWithPath(NSString.getTempString(path));
error: *NSError;
func_compile :: #type (*void, Selector, *NSURL, **NSError) -> *NSURL #c_call;
compiled := (cast(func_compile) objc_msgSend)(objc_getClass("MLModel"), selector("compileModelAtURL:error:"), package, *error);
if !compiled return null, false, tprint("Core ML could not compile %: %", path, error_text(error));
configuration := objc_init(objc_alloc(objc_getClass("MLModelConfiguration")));
defer release(configuration);
func_units :: #type (*void, Selector, s64) -> void #c_call;
(cast(func_units) objc_msgSend)(configuration, selector("setComputeUnits:"), cast(s64) units);
func_load :: #type (*void, Selector, *NSURL, *NSObject, **NSError) -> *NSObject #c_call;
model := (cast(func_load) objc_msgSend)(objc_getClass("MLModel"), selector("modelWithContentsOfURL:configuration:error:"), compiled, configuration, *error);
if !model {
remove_compiled(compiled);
return null, false, tprint("Core ML could not load %: %", path, error_text(error));
}
result := New(Coreml_Model,, context.default_allocator);
result.model = xx retain(model);
result.compiled = xx retain(compiled);
result.input_name = copy_string(input_name,, context.default_allocator);
result.output_name = copy_string(output_name,, context.default_allocator);
init(*result.mutex);
return result, true, "";
}
coreml_free :: (model: *Coreml_Model) {
if !model return;
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
release(model.model);
remove_compiled(cast(*NSURL) model.compiled);
release(model.compiled);
destroy(*model.mutex);
free(model.input_name,, context.default_allocator);
free(model.output_name,, context.default_allocator);
free(model,, context.default_allocator);
}
// A run's output, as float32 values one after another (C order): a view of Core ML's own array when it
// is float32 and packed, else a packed copy. Valid until coreml_release_output.
Coreml_Output :: struct {
data: *float;
dims: [8] s64;
dim_count: s64;
count: s64;
array: *NSObject; // the MLMultiArray, retained
copy: *float; // when data is a copy: the default allocator's
}
// Runs the model on input, a float32 tensor of shape: its output (model.output_name). ok is false,
// with why, when it fails.
coreml_predict :: (model: *Coreml_Model, input: [] float, shape: [] s64) -> Coreml_Output, ok: bool, error: string {
outs: [1] Coreml_Output;
ok, error := coreml_predict_outputs(model, input, shape, .[model.output_name], outs);
return outs[0], ok, error;
}
// As coreml_predict, for a model with several outputs: those named, into outs (one each, in order),
// each released with coreml_release_output. On failure none is set.
coreml_predict_outputs :: (model: *Coreml_Model, input: [] float, shape: [] s64, names: [] string, outs: [] Coreml_Output) -> ok: bool, error: string {
assert(names.count == outs.count);
init_objective_c();
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
// The input, over the caller's floats: packed, so each stride is the product of the sizes after it.
strides := NewArray(shape.count, s64,, temp);
step: s64 = 1;
for < shape { strides[it_index] = step; step *= it; }
if step != input.count return false, tprint("the input has % values, not % for its shape %", input.count, step, shape);
error: *NSError;
func_array_init :: #type (*void, Selector, *void, *NSObject, s64, *NSObject, *void, **NSError) -> *NSObject #c_call;
array := (cast(func_array_init) objc_msgSend)(objc_alloc(objc_getClass("MLMultiArray")), selector("initWithDataPointer:shape:dataType:strides:deallocator:error:"),
input.data, numbers(shape), MULTIARRAY_FLOAT32, numbers(strides), null, *error);
if !array return false, tprint("Core ML could not take the input: %", error_text(error));
autorelease(array);
func_value :: #type (*void, Selector, *NSObject) -> *NSObject #c_call;
value := (cast(func_value) objc_msgSend)(objc_getClass("MLFeatureValue"), selector("featureValueWithMultiArray:"), array);
key := NSString.getTempString(model.input_name);
func_dictionary :: #type (*void, Selector, *NSObject, *NSObject) -> *NSObject #c_call;
dictionary := (cast(func_dictionary) objc_msgSend)(objc_getClass("NSDictionary"), selector("dictionaryWithObject:forKey:"), value, key);
func_provider :: #type (*void, Selector, *NSObject, **NSError) -> *NSObject #c_call;
features := (cast(func_provider) objc_msgSend)(objc_alloc(objc_getClass("MLDictionaryFeatureProvider")), selector("initWithDictionary:error:"), dictionary, *error);
if !features return false, tprint("Core ML could not take the input: %", error_text(error));
autorelease(features);
func_predict :: #type (*void, Selector, *NSObject, **NSError) -> *NSObject #c_call;
lock(*model.mutex);
prediction := (cast(func_predict) objc_msgSend)(model.model, selector("predictionFromFeatures:error:"), features, *error);
unlock(*model.mutex);
if !prediction return false, tprint("Core ML could not run the model: %", error_text(error));
for names {
out, ok, read_error := read_output(prediction, it);
if !ok {
for 0..it_index - 1 coreml_release_output(*outs[it]);
return false, read_error;
}
outs[it_index] = out;
}
return true, "";
}
coreml_release_output :: (out: *Coreml_Output) {
if out.array release(out.array);
if out.copy free(out.copy,, context.default_allocator);
out.* = .{};
}
// coreml_predict, its output copied into output (which must hold exactly as many values).
coreml_run :: (model: *Coreml_Model, input: [] float, shape: [] s64, output: [] float) -> ok: bool, error: string {
out, ok, error := coreml_predict(model, input, shape);
if !ok return false, error;
defer coreml_release_output(*out);
if out.count != output.count return false, tprint("the model gives % values, not %", out.count, output.count);
memcpy(output.data, out.data, out.count * size_of(float));
return true, "";
}
#scope_file
// The output named name of a prediction (an MLFeatureProvider), as float32 values in C order.
read_output :: (prediction: *NSObject, name: string) -> Coreml_Output, ok: bool, error: string {
result: Coreml_Output;
func_value :: #type (*void, Selector, *NSObject) -> *NSObject #c_call;
out_value := (cast(func_value) objc_msgSend)(prediction, selector("featureValueForName:"), NSString.getTempString(name));
out_array := ifx out_value then cast(*NSObject) objc_msgSend_typed(out_value, selector("multiArrayValue")) else null;
if !out_array return result, false, tprint("the model gives no %", name);
out_shape := integers(cast(*NSObject) objc_msgSend_typed(out_array, selector("shape")));
out_strides := integers(cast(*NSObject) objc_msgSend_typed(out_array, selector("strides")));
if out_shape.count > result.dims.count || out_shape.count != out_strides.count return result, false, tprint("the model's output has an unexpected shape %", out_shape);
result.dim_count = out_shape.count;
result.count = 1;
for out_shape { result.dims[it_index] = it; result.count *= it; }
func_type :: #type (*void, Selector) -> s64 #c_call;
type := (cast(func_type) objc_msgSend)(out_array, selector("dataType"));
if type != MULTIARRAY_FLOAT32 && type != MULTIARRAY_FLOAT16 return result, false, tprint("the model's output is of type 0x%, not float32 or float16", formatInt(type, base = 16));
packed := true;
expected: s64 = 1;
for < out_shape { if out_strides[it_index] != expected packed = false; expected *= it; }
data := objc_msgSend_typed(out_array, selector("dataPointer"));
result.array = xx retain(out_array); // outlives the pool, and with it the data
if type == MULTIARRAY_FLOAT32 && packed {
result.data = cast(*float) data;
return result, true, "";
}
// A copy, packed: float16 widened, and rows Core ML padded (GPU outputs may be) closed up.
result.copy = cast(*float) alloc(result.count * size_of(float),, context.default_allocator);
result.data = result.copy;
index: [8] s64;
for i: 0..result.count - 1 {
offset: s64 = 0;
for d: 0..result.dim_count - 1 offset += index[d] * out_strides[d];
result.copy[i] = ifx type == MULTIARRAY_FLOAT32 then (cast(*float) data)[offset] else float16_to_float32((cast(*u16) data)[offset]);
d := result.dim_count - 1;
while d >= 0 { index[d] += 1; if index[d] < out_shape[d] break; index[d] = 0; d -= 1; }
}
return result, true, "";
}
MULTIARRAY_FLOAT32 :: 0x10000 | 32; // MLMultiArrayDataTypeFloat32
MULTIARRAY_FLOAT16 :: 0x10000 | 16; // MLMultiArrayDataTypeFloat16
numbers :: (values: [] s64) -> *NSObject {
objects := NewArray(values.count, *NSObject,, temp);
func_number :: #type (*void, Selector, s64) -> *NSObject #c_call;
for values objects[it_index] = (cast(func_number) objc_msgSend)(objc_getClass("NSNumber"), selector("numberWithLongLong:"), it);
func_array :: #type (*void, Selector, **NSObject, u64) -> *NSObject #c_call;
return (cast(func_array) objc_msgSend)(objc_getClass("NSArray"), selector("arrayWithObjects:count:"), objects.data, xx objects.count);
}
// An NSArray of NSNumbers, as integers. Temp.
integers :: (array: *NSObject) -> [] s64 {
if !array return .[];
count := cast(s64) objc_msgSend_typed(array, selector("count"));
result := NewArray(count, s64,, temp);
func_at :: #type (*void, Selector, u64) -> *NSObject #c_call;
func_value :: #type (*void, Selector) -> s64 #c_call;
for 0..count - 1 {
number := (cast(func_at) objc_msgSend)(array, selector("objectAtIndex:"), xx it);
result[it] = (cast(func_value) objc_msgSend)(number, selector("longLongValue"));
}
return result;
}
error_text :: (error: *NSError) -> string {
if !error return "no reason given";
return copy_string(to_string(NSError.localizedDescription(error)),, temp);
}
remove_compiled :: (url: *NSURL) {
func :: #type (*void, Selector, *NSURL, **NSError) -> BOOL #c_call;
(cast(func) objc_msgSend)(NSFileManager.defaultManager(), selector("removeItemAtURL:error:"), url, null);
}
float16_to_float32 :: (h: u16) -> float {
sign := (cast(u32) h & 0x8000) << 16;
exponent := (cast(u32) h >> 10) & 0x1F;
mantissa := cast(u32) h & 0x3FF;
bits: u32;
if exponent == 0 {
if mantissa == 0 bits = sign;
else { // subnormal: normalised
e: u32 = 113;
while !(mantissa & 0x400) { mantissa <<= 1; e -= 1; }
bits = sign | (e << 23) | ((mantissa & 0x3FF) << 13);
}
} else if exponent == 31 bits = sign | 0x7F80_0000 | (mantissa << 13);
else bits = sign | ((exponent + 112) << 23) | (mantissa << 13);
return (cast(*float) *bits).*;
}
CoreML_Framework :: #library,system,link_always "CoreML";
#import "Basic";
#import "File";
#import "File_Utilities";
#import "Thread";
#import "Objective_C";
#import "Objective_C/AppKit";

2
modules/Image_Decode/exif.jai
View File

@ -258,7 +258,7 @@ heif_info :: (bytes: [] u8) -> Heif_Info {
box := properties[index - 1];
type := cast(string) slice_bytes(box, 4, 4);
if type == "ispe" && box.count >= 20 { info.width = be32(box, 12); info.height = be32(box, 16); }
if type == "irot" && box.count >= 9 info.rotation = (box[8] & 3) * 90;
if type == "irot" && box.count >= 9 info.rotation = cast(s64) (box[8] & 3) * 90; // as a u8, 270 wrapped to 14
if type == "imir" && box.count >= 9 info.mirror = box[8] & 1;
}
}

149
modules/Image_Decode/imageio.jai Normal file
View File

@ -0,0 +1,149 @@
// ImageIO, macOS's own codecs: HEIC and AVIF, WebP, TIFF and the camera RAW formats, as WIC reads them
// on Windows. Only the few Core Foundation, ImageIO and Core Graphics calls used are declared.
// The primary image in bytes, as non-premultiplied RGBA (default allocator), as stored: not turned.
// orientation is what ImageIO says the image's orientation is (1-8, from a HEIF's irot and imir, or
// its EXIF), for image_decode to apply where exif.jai cannot read it.
//
// The pixels keep the image's own RGB color space, as WIC and stb_image leave them; only gray, CMYK
// and the like are converted, to sRGB.
imageio_decode :: (bytes: [] u8) -> Image, ok: bool, error: string, orientation: s64 {
image: Image;
// Not copied: the source is let go of before this returns.
data := CFDataCreateWithBytesNoCopy(null, bytes.data, bytes.count, kCFAllocatorNull);
if !data return image, false, "ImageIO could not take the bytes", 1;
defer CFRelease(data);
source := CGImageSourceCreateWithData(data, null);
if !source return image, false, "ImageIO cannot read it", 1;
defer CFRelease(source);
if CGImageSourceGetCount(source) < 1 return image, false, "ImageIO finds no image in it", 1;
index := CGImageSourceGetPrimaryImageIndex(source);
cg := CGImageSourceCreateImageAtIndex(source, index, null);
if !cg return image, false, "ImageIO cannot decode it", 1;
defer CGImageRelease(cg);
orientation: s32 = 1;
properties := CGImageSourceCopyPropertiesAtIndex(source, index, null);
if properties {
number := CFDictionaryGetValue(properties, kCGImagePropertyOrientation);
if number CFNumberGetValue(number, kCFNumberSInt32Type, *orientation);
CFRelease(properties);
}
w := CGImageGetWidth(cg);
h := CGImageGetHeight(cg);
if !w || !h return image, false, "ImageIO gives it no size", orientation;
if w * h * 4 > U32_MAX return image, false, tprint("it is % x %, too large to decode in one piece", w, h), orientation;
// Opaque images are drawn as 8-bit RGB with the fourth byte left out, so their pixels are exact.
// Others are drawn premultiplied, the only RGBA a bitmap context takes, at 16 bits a channel so
// that dividing back gives the 8-bit colors again, even where alpha is small.
alpha_info := CGImageGetAlphaInfo(cg);
opaque := alpha_info == kCGImageAlphaNone || alpha_info == kCGImageAlphaNoneSkipLast || alpha_info == kCGImageAlphaNoneSkipFirst;
bits: u64 = 16;
bitmap_info: u32 = kCGBitmapByteOrder16Little | kCGImageAlphaPremultipliedLast;
if opaque { bits = 8; bitmap_info = kCGBitmapByteOrder32Big | kCGImageAlphaNoneSkipLast; }
row_bytes := w * bits / 2;
count := cast(s64) (w * h);
pixels := cast(*u8) alloc(count * 4,, context.default_allocator);
drawn := pixels;
if !opaque drawn = cast(*u8) alloc(count * 8,, context.default_allocator);
defer if drawn != pixels free(drawn,, context.default_allocator);
memset(drawn, 0, cast(s64) (row_bytes * h));
space := CGImageGetColorSpace(cg);
own_space := space && CGColorSpaceGetModel(space) == kCGColorSpaceModelRGB;
bitmap: CGContextRef;
if own_space bitmap = CGBitmapContextCreate(drawn, w, h, bits, row_bytes, space, bitmap_info);
if !bitmap {
// Not RGB, or a space a bitmap context cannot draw in (an extended range one): converted to sRGB.
srgb := CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
defer CGColorSpaceRelease(srgb);
bitmap = CGBitmapContextCreate(drawn, w, h, bits, row_bytes, srgb, bitmap_info);
}
if !bitmap {
free(pixels,, context.default_allocator);
return image, false, "Core Graphics could not make a bitmap to draw it in", orientation;
}
CGContextSetBlendMode(bitmap, kCGBlendModeCopy);
CGContextDrawImage(bitmap, .{.{0, 0}, .{cast(float64) w, cast(float64) h}}, cg);
CGContextRelease(bitmap);
if opaque {
for 0..count - 1 pixels[it * 4 + 3] = 255;
} else {
wide := cast(*u16) drawn;
for 0..count - 1 {
in := wide + it * 4;
out := pixels + it * 4;
a := cast(u32) in[3];
out[3] = cast(u8) ((a * 255 + 32767) / 65535);
if a == 0 { out[0], out[1], out[2] = 0; continue; }
for c: 0..2 out[c] = cast(u8) min(255, (cast(u32) in[c] * 255 + a / 2) / a);
}
}
image = .{pixels = pixels, width = cast(s32) w, height = cast(s32) h};
return image, true, "", orientation;
}
#scope_file
CFTypeRef :: *void;
CFDataRef :: *void;
CFDictionaryRef :: *void;
CFStringRef :: *void;
CFAllocatorRef :: *void;
CGImageSourceRef :: *void;
CGImageRef :: *void;
CGColorSpaceRef :: *void;
CGContextRef :: *void;
CGPoint :: struct { x, y: float64; }
CGSize :: struct { width, height: float64; }
CGRect :: struct { origin: CGPoint; size: CGSize; }
kCFNumberSInt32Type :: 3;
kCGColorSpaceModelRGB :: 1;
kCGBlendModeCopy :: 17;
kCGImageAlphaNone :: 0;
kCGImageAlphaPremultipliedLast :: 1;
kCGImageAlphaNoneSkipLast :: 5;
kCGImageAlphaNoneSkipFirst :: 6;
kCGBitmapByteOrder16Little :: 1 << 12;
kCGBitmapByteOrder32Big :: 4 << 12;
CoreFoundation :: #library,system "CoreFoundation";
ImageIO :: #library,system "ImageIO";
CoreGraphics :: #library,system "CoreGraphics";
kCFAllocatorNull: CFAllocatorRef #elsewhere CoreFoundation;
kCGImagePropertyOrientation: CFStringRef #elsewhere ImageIO;
kCGColorSpaceSRGB: CFStringRef #elsewhere CoreGraphics;
CFDataCreateWithBytesNoCopy :: (allocator: CFAllocatorRef, bytes: *u8, length: s64, deallocator: CFAllocatorRef) -> CFDataRef #foreign CoreFoundation;
CFRelease :: (object: CFTypeRef) #foreign CoreFoundation;
CFDictionaryGetValue :: (dictionary: CFDictionaryRef, key: *void) -> *void #foreign CoreFoundation;
CFNumberGetValue :: (number: *void, type: s64, value: *void) -> u8 #foreign CoreFoundation;
CGImageSourceCreateWithData :: (data: CFDataRef, options: CFDictionaryRef) -> CGImageSourceRef #foreign ImageIO;
CGImageSourceGetCount :: (source: CGImageSourceRef) -> u64 #foreign ImageIO;
CGImageSourceGetPrimaryImageIndex :: (source: CGImageSourceRef) -> u64 #foreign ImageIO;
CGImageSourceCreateImageAtIndex :: (source: CGImageSourceRef, index: u64, options: CFDictionaryRef) -> CGImageRef #foreign ImageIO;
CGImageSourceCopyPropertiesAtIndex :: (source: CGImageSourceRef, index: u64, options: CFDictionaryRef) -> CFDictionaryRef #foreign ImageIO;
CGImageGetWidth :: (image: CGImageRef) -> u64 #foreign CoreGraphics;
CGImageGetHeight :: (image: CGImageRef) -> u64 #foreign CoreGraphics;
CGImageGetAlphaInfo :: (image: CGImageRef) -> u32 #foreign CoreGraphics;
CGImageGetColorSpace :: (image: CGImageRef) -> CGColorSpaceRef #foreign CoreGraphics;
CGImageRelease :: (image: CGImageRef) #foreign CoreGraphics;
CGColorSpaceGetModel :: (space: CGColorSpaceRef) -> s32 #foreign CoreGraphics;
CGColorSpaceCreateWithName :: (name: CFStringRef) -> CGColorSpaceRef #foreign CoreGraphics;
CGColorSpaceRelease :: (space: CGColorSpaceRef) #foreign CoreGraphics;
CGBitmapContextCreate :: (data: *void, width: u64, height: u64, bits_per_component: u64, bytes_per_row: u64, space: CGColorSpaceRef, bitmap_info: u32) -> CGContextRef #foreign CoreGraphics;
CGContextSetBlendMode :: (context_ref: CGContextRef, mode: s32) #foreign CoreGraphics;
CGContextDrawImage :: (context_ref: CGContextRef, rect: CGRect, image: CGImageRef) #foreign CoreGraphics;
CGContextRelease :: (context_ref: CGContextRef) #foreign CoreGraphics;

31
modules/Image_Decode/module.jai
View File

@ -1,21 +1,23 @@
// Image_Decode: any image file's bytes to RGBA pixels, the right way up, and the PNG previews .mcap
// files hold. stb_image reads PNG, JPEG, GIF, BMP, PSD and TGA; everything else (HEIC, AVIF, WebP,
// TIFF, camera RAW) goes to Windows' own codecs through WIC, which read what the installed Store
// extensions (HEIF, WebP, Raw Image) can. A photo's EXIF orientation is applied, so a phone's portrait
// TIFF, camera RAW) goes to the system's own codecs: on Windows WIC, which reads what the installed
// Store extensions (HEIF, WebP, Raw Image) can, and on macOS ImageIO (imageio.jai). Linux reads only
// what stb_image does. A photo's EXIF orientation is applied, so a phone's portrait
// photo comes out standing up.
//
// What the bytes are comes from their first bytes (image_kind), never from a file name, except for
// the camera RAW formats that are TIFF inside: image_kind_from_name tells those apart.
//
// Videos (video.jai) are told apart here too, by their first bytes or their extension; video_poster
// reads a frame from one with FFmpeg, on Windows, and gif_info counts an animated GIF's frames.
// reads a frame from one with FFmpeg, on Windows and macOS, and gif_info counts an animated GIF's frames.
//
// Thread-safe: every call stands alone, and WIC is set up and released on the calling thread.
// Thread-safe: every call stands alone, and WIC is set up and released on the calling thread; ImageIO
// needs no setting up.
Image_Kind :: enum u8 {
UNKNOWN;
PNG; JPEG; GIF; BMP; PSD; TGA; // stb_image
WEBP; HEIF; AVIF; TIFF; RAW; // WIC
WEBP; HEIF; AVIF; TIFF; RAW; // WIC on Windows, ImageIO on macOS
VIDEO; // MP4, QuickTime, Matroska and WebM, AVI, WMV, MPEG: see video_poster
}
@ -154,6 +156,7 @@ image_decode :: (bytes: [] u8, kind := Image_Kind.UNKNOWN, orient := true) -> Im
error = to_string(stbi_failure_reason());
}
}
container_orientation := 0; // ImageIO's, for HEIF and AVIF on macOS; 0 when not known
if !decoded {
#if OS == .WINDOWS {
wic_ok: bool;
@ -163,14 +166,25 @@ image_decode :: (bytes: [] u8, kind := Image_Kind.UNKNOWN, orient := true) -> Im
if error return image, false, tprint("% (stb_image: %)", wic_error, error);
return image, false, wic_error;
}
} else #if OS == .MACOS {
imageio_ok: bool;
imageio_error: string;
image, imageio_ok, imageio_error, container_orientation = imageio_decode(bytes);
if !imageio_ok {
if error return image, false, tprint("% (stb_image: %)", imageio_error, error);
return image, false, imageio_error;
}
} else {
return image, false, ifx error then tprint("stb_image: %", error) else tprint("% images are only read on Windows", image_kind_name(kind));
return image, false, ifx error then tprint("stb_image: %", error) else tprint("% images are only read on Windows and macOS", image_kind_name(kind));
}
}
if orient {
orientation := 1;
if kind == .HEIF || kind == .AVIF {
if (kind == .HEIF || kind == .AVIF) && container_orientation {
// ImageIO gives the image as stored, and the container's irot and imir as an orientation.
orientation = container_orientation;
} else if kind == .HEIF || kind == .AVIF {
// The container's irot and imir are the orientation; the EXIF tag only repeats it. WIC applies
// them (HEIF requires a reader to); a quarter turn it left out shows as the sides not swapped.
heif := heif_info(bytes);
@ -251,7 +265,8 @@ png_size :: (png: [] u8) -> width: s64, height: s64, ok: bool {
#load "exif.jai";
#load "video.jai";
#if OS == .WINDOWS #load "wic.jai";
#if OS == .WINDOWS #load "video_ffmpeg.jai";
#if OS == .MACOS #load "imageio.jai";
#if OS == .WINDOWS || OS == .MACOS #load "video_ffmpeg.jai";
#scope_module

4
modules/Image_Decode/video.jai
View File

@ -24,9 +24,9 @@ Video_Info :: struct {
// start with; the first frame of a clip shorter than a second.
VIDEO_POSTER_MAX_SECONDS :: 3.0;
#if OS != .WINDOWS {
#if OS == .LINUX {
video_poster :: (path: string) -> Image, Video_Info, ok: bool, error: string {
return .{}, .{}, false, "videos are only read on Windows";
return .{}, .{}, false, "videos are only read on Windows and macOS";
}
}

2
modules/Image_Decode/video_ffmpeg.jai
View File

@ -1,4 +1,4 @@
// video_poster on Windows, with FFmpeg (modules/ffmpeg). See video.jai.
// video_poster on Windows and macOS, with FFmpeg (modules/ffmpeg). See video.jai.
video_poster :: (path: string) -> Image, Video_Info, ok: bool, error: string {
image: Image;

3
modules/Labels/fetch_models.jai
View File

@ -52,7 +52,8 @@ sha256_of :: (path: string) -> string {
if lines.count < 2 return "";
return to_lower_copy(replace(trim(lines[1]), " ", "")); // line 2 is the hash
} else {
command := ifx OS == .MACOS then string.["shasum", "-a", "256", path] else string.["sha256sum", path];
#if OS == .MACOS command := string.["shasum", "-a", "256", path];
else command := string.["sha256sum", path];
result, output := run_command(..command, capture_and_return_output = true);
if result.exit_code != 0 return "";
return to_lower_copy(trim(split(output, " ")[0]));

3
modules/Labels/models.config
View File

@ -10,6 +10,8 @@
# logit_scale, logit_bias a phrase's score for an image is sigmoid(e^logit_scale * cosine + logit_bias),
# from the trained model (they are not in the ONNX encoders)
# image_batch images run through the vision model together
# vision_coreml on macOS, the vision model as a Core ML package, quoted, run on the GPU
# (tools/export_siglip_coreml.py makes it); without it, onnxruntime runs it
[model siglip2-base-patch16-384]
title: "SigLIP 2 base, 384 px"
@ -21,3 +23,4 @@ max_tokens: 64
logit_scale: 4.726024
logit_bias: -16.772243
image_batch: 8
vision_coreml: "siglip2-base-patch16-384-vision.mlpackage"

73
modules/Labels/module.jai
View File

@ -8,9 +8,11 @@
//
// The models are listed in models_folder/models.config (fetch_models.jai downloads them) and run
// with onnxruntime (modules/onnxruntime): the vision model on an NVIDIA GPU through CUDA when it
// can, else on the CPU; the text model, small inputs and only a few at a time, on the CPU.
// can, else on the CPU; the text model, small inputs and only a few at a time, on the CPU. On macOS
// the vision model runs on the GPU through Core ML (modules/Core_ML) when its vision_coreml package
// is there (tools/export_siglip_coreml.py makes it): about ten times the CPU's speed.
Labels_Provider :: enum { CPU; CUDA; }
Labels_Provider :: enum { CPU; CUDA; COREML; }
// Tightly packed or not: stride is the bytes from one row to the next.
Labels_Image :: struct {
@ -22,6 +24,7 @@ Labels_Model_Info :: struct {
name: string;
title: string;
vision, text, tokenizer: string; // file names in the models folder
vision_coreml: string; // the vision model as a Core ML package, for macOS; "" for none
image_size: s32 = 384;
max_tokens: s32 = 64;
logit_scale: float = 0;
@ -36,6 +39,7 @@ Labels_Engine :: struct {
env: *OrtEnv;
memory_info: *OrtMemoryInfo;
vision, text: Ort_Session;
#if OS == .MACOS coreml: *Coreml_Model; // the vision model, when it runs through Core ML
tokenizer: *Gemma_Tokenizer;
dimensions: s32; // of every vector
}
@ -75,6 +79,7 @@ labels_list_models :: (models_folder: string) -> [] Labels_Model_Info, error: st
case "vision"; model.vision = copy_string(value);
case "text"; model.text = copy_string(value);
case "tokenizer"; model.tokenizer = copy_string(value);
case "vision_coreml"; model.vision_coreml = copy_string(value);
case "image_size"; if number_ok model.image_size = cast(s32) number;
case "max_tokens"; if number_ok model.max_tokens = cast(s32) number;
case "logit_scale"; if number_ok model.logit_scale = number;
@ -113,11 +118,23 @@ labels_create :: (models_folder: string, model_name: string, prefer_gpu := true)
cores := cast(s32) max(get_number_of_processors(.ALL_LOGICAL), 1);
vision_path := tprint("%/%", models_folder, model.vision);
use_gpu := false;
if prefer_gpu {
#if OS == .MACOS if prefer_gpu && model.vision_coreml && !file_exists(tprint("%/%", models_folder, model.vision_coreml)) {
engine.gpu_explanation = sprint("Labelling runs on the CPU: % is not there; tools/export_siglip_coreml.py makes it.", model.vision_coreml);
} else if prefer_gpu && model.vision_coreml {
// Core ML on the GPU; on any failure, onnxruntime on the CPU below, with the reason.
coreml_path := tprint("%/%", models_folder, model.vision_coreml);
engine.coreml, ok, error = coreml_load(coreml_path, "pixel_values", "pooler_output");
if ok engine.provider = .COREML; // its first run is below, once the vectors' size is known
else engine.gpu_explanation = sprint("Labelling runs on the CPU: Core ML could not load % on the GPU:\n%", model.vision_coreml, error);
}
#if OS == .MACOS if prefer_gpu && !model.vision_coreml {
engine.gpu_explanation = sprint("Labelling runs on the CPU: there is no Core ML model for it; tools/export_siglip_coreml.py makes one.");
}
if prefer_gpu && OS != .MACOS {
available, explanation := ort_cuda_check("Labelling");
use_gpu = available;
if !available engine.gpu_explanation = copy_string(explanation);
} else {
} else if !prefer_gpu {
engine.gpu_explanation = copy_string("Labelling runs on the CPU, as it was asked to.");
}
if use_gpu {
@ -133,7 +150,7 @@ labels_create :: (models_folder: string, model_name: string, prefer_gpu := true)
use_gpu = false;
}
}
if !use_gpu {
if !use_gpu && engine.provider != .COREML {
engine.vision, ok, error = ort_create_session(engine.env, vision_path, false, cores, output = "pooler_output", missing_hint = HINT);
if !ok { labels_destroy(engine); return null, false, error; }
engine.provider = .CPU;
@ -144,6 +161,19 @@ labels_create :: (models_folder: string, model_name: string, prefer_gpu := true)
// The vectors' size, from a run of the text model on an empty text.
probe := labels_embed_texts(engine, .[""],, temp);
if !engine.dimensions { labels_destroy(engine); return null, false, "The text model gave no vector."; }
#if OS == .MACOS if engine.provider == .COREML {
// Core ML's first run; when it fails, onnxruntime on the CPU after all.
ok, error = labels_warm_up(engine);
if !ok {
coreml_free(engine.coreml);
engine.coreml = null;
engine.provider = .CPU;
engine.gpu_explanation = sprint("Labelling runs on the CPU: Core ML could not run % on the GPU:\n%", model.vision_coreml, error);
engine.vision, ok, error = ort_create_session(engine.env, vision_path, false, cores, output = "pooler_output", missing_hint = HINT);
if !ok { labels_destroy(engine); return null, false, error; }
}
}
return engine, true, "";
}
@ -152,6 +182,7 @@ labels_destroy :: (engine: *Labels_Engine) {
if engine.provider == .CUDA ort_gpu_lock();
ort_release_session(*engine.vision);
if engine.provider == .CUDA ort_gpu_unlock();
#if OS == .MACOS coreml_free(engine.coreml);
ort_release_session(*engine.text);
tokenizer_free(engine.tokenizer,, context.default_allocator);
if engine.memory_info ort.api.ReleaseMemoryInfo(engine.memory_info);
@ -161,7 +192,16 @@ labels_destroy :: (engine: *Labels_Engine) {
}
labels_gpu_status :: (engine: *Labels_Engine) -> in_use: bool, explanation: string {
return engine.provider == .CUDA, engine.gpu_explanation;
return engine.provider == .CUDA || engine.provider == .COREML, engine.gpu_explanation;
}
// What runs the vision model, for people and the files it labels: "CUDA", "Core ML" or "CPU".
labels_provider_name :: (engine: *Labels_Engine) -> string {
if engine.provider == {
case .CUDA; return "CUDA";
case .COREML; return "Core ML";
}
return "CPU";
}
// One unit vector per image, engine.dimensions long, one after another; from allocator. An image
@ -176,6 +216,18 @@ labels_embed_images :: (engine: *Labels_Engine, images: [] Labels_Image, allocat
input := NewArray(plane * 3 * batch, float,, context.default_allocator);
defer array_free(input,, context.default_allocator);
#if OS == .MACOS if engine.coreml {
// One image a run: the Core ML model takes [1, 3, size, size].
out := NewArray(dims, float,, temp);
for images {
labels_prepare_image(it, size, input.data);
ok, run_error := coreml_run(engine.coreml, array_view(input, 0, plane * 3), s64.[1, 3, size, size], out);
if !ok { error = tprint("%", run_error); continue; }
labels_normalize_into(out.data, dims, result.data + it_index * dims);
}
return result, error;
}
first := 0;
while first < images.count {
count := min(batch, images.count - first);
@ -279,6 +331,13 @@ labels_warm_up :: (engine: *Labels_Engine) -> ok: bool, error: string {
size := engine.model.image_size;
input := NewArray(3 * size * size, float, initialized = true,, context.default_allocator);
defer array_free(input,, context.default_allocator);
#if OS == .MACOS if engine.coreml {
out := NewArray(engine.dimensions, float,, context.default_allocator);
defer array_free(out,, context.default_allocator);
ok, error := coreml_run(engine.coreml, input, s64.[1, 3, size, size], out);
if !ok return false, tprint("The first run on the GPU failed: %", error);
return true, "";
}
out, ok, error := ort_run_float(*engine.vision, engine.memory_info, input, s64.[1, 3, size, size]);
if !ok return false, tprint("The first run on the GPU failed: %", error);
ort_release_output(*out);
@ -374,8 +433,10 @@ labels_strip_comment :: (line: string) -> string {
#import "Basic";
#import "String";
#import "File";
#import "File_Utilities";
#import "Math";
#import "System";
#import "Hash_Table";
#import "Float16";
#import "onnxruntime";
#if OS == .MACOS #import "Core_ML";

41
modules/OCR/detect.jai
View File

@ -63,8 +63,14 @@ ocr_detect_tile :: (engine: *Ocr_Engine, image: Ocr_Image, scale: float, scaled_
valid_h := tile_bottom - tile_top;
w := align_up_32(valid_w);
h := align_up_32(valid_h);
#if OS == .MACOS if engine.det_coreml {
// Up to a size the Core ML package takes: its steps, which det_max_side and det_tile_height stay within.
w = align_up(valid_w, model.coreml_det_width_step);
h = align_up(valid_h, model.coreml_det_height_step);
}
// Planar BGR, normalized; the padding to a multiple of 32 is the normalized zero (the mean colour).
// Planar BGR, normalized; the padding to a multiple of 32 (or the Core ML step) is the normalized
// zero (the mean colour).
plane := cast(s64) w * h;
input := NewArray(plane * 3, float, initialized = true,, context.default_allocator);
defer array_free(input,, context.default_allocator);
@ -83,14 +89,31 @@ ocr_detect_tile :: (engine: *Ocr_Engine, image: Ocr_Image, scale: float, scaled_
}
shape := s64.[1, 3, h, w];
out, ok, error := ocr_run(*engine.det, engine.memory_info, input, shape);
if !ok return false, error;
probabilities: *float;
dims: [] s64;
out: Ocr_Output;
defer ocr_release_output(*out);
if out.dim_count != 4 || out.dims[2] != h || out.dims[3] != w {
return false, tprint("The detector gave an output of shape %, not [1, 1, %, %].", array_view(out.dims, 0, out.dim_count), h, w);
#if OS == .MACOS {
coreml_out: Coreml_Output;
defer coreml_release_output(*coreml_out);
}
#if OS == .MACOS if engine.det_coreml {
ok: bool; error: string;
coreml_out, ok, error = coreml_predict(engine.det_coreml, input, shape);
if !ok return false, error;
probabilities, dims = coreml_out.data, array_view(coreml_out.dims, 0, coreml_out.dim_count);
}
if !probabilities {
ok: bool; error: string;
out, ok, error = ocr_run(*engine.det, engine.memory_info, input, shape);
if !ok return false, error;
probabilities, dims = out.data, array_view(out.dims, 0, out.dim_count);
}
if dims.count != 4 || dims[2] != h || dims[3] != w {
return false, tprint("The detector gave an output of shape %, not [1, 1, %, %].", dims, h, w);
}
regions := ocr_find_regions(out.data, w, valid_w, valid_h, model.det_threshold);
regions := ocr_find_regions(probabilities, w, valid_w, valid_h, model.det_threshold);
defer {
for regions array_free(it.outline);
array_free(regions,, context.default_allocator);
@ -254,6 +277,12 @@ align_up_32 :: (n: s32) -> s32 {
return (n + 31) & cast(s32) ~31;
}
// n rounded up to a multiple of step (at least one step).
align_up :: (n: s32, step: s32) -> s32 {
step = max(step, 1);
return max((n + step - 1) / step, 1) * step;
}
Region :: struct {
pixels: s64;
probability_sum: float64;

3
modules/OCR/fetch_models.jai
View File

@ -59,7 +59,8 @@ sha256_of :: (path: string) -> string {
if lines.count < 2 return "";
return to_lower_copy(replace(trim(lines[1]), " ", "")); // line 2 is the hash
} else {
command := ifx OS == .MACOS then string.["shasum", "-a", "256", path] else string.["sha256sum", path];
#if OS == .MACOS command := string.["shasum", "-a", "256", path];
else command := string.["sha256sum", path];
result, output := run_command(..command, capture_and_return_output = true);
if result.exit_code != 0 return "";
return to_lower_copy(trim(split(output, " ")[0]));

10
modules/OCR/models.config
View File

@ -16,6 +16,12 @@
# rec_batch_gpu text lines recognized together in one run, on the GPU
# rec_batch_cpu the same on the CPU, where one at a time is fastest: a batch is padded to its
# widest line, and the padding costs time and changes what is read a little
# detector_coreml, recognizer_coreml
# macOS: the detector and recognizer as Core ML packages, quoted, run on the GPU
# (tools/export_ppocr_coreml.py makes them); without both, onnxruntime runs OCR
# coreml_det_height_step, coreml_det_width_step, coreml_rec_batch, coreml_rec_width_step
# the sizes those packages take, as the script made them (the defaults: 128, 256,
# 16, 64): detection strips are padded up to the steps, lines to 16 at the step
[model ppocr-v5-mobile]
title: "PP-OCRv5 mobile (light)"
@ -31,6 +37,8 @@ unclip_ratio: 1.5
rec_height: 48
rec_batch_gpu: 16
rec_batch_cpu: 1
detector_coreml: "ch_PP-OCRv5_det_mobile.mlpackage"
recognizer_coreml: "ch_PP-OCRv5_rec_mobile.mlpackage"
[model ppocr-v5-server]
title: "PP-OCRv5 server (heavy)"
@ -46,3 +54,5 @@ unclip_ratio: 1.5
rec_height: 48
rec_batch_gpu: 8
rec_batch_cpu: 1
detector_coreml: "ch_PP-OCRv5_det_server.mlpackage"
recognizer_coreml: "ch_PP-OCRv5_rec_server.mlpackage"

76
modules/OCR/models.jai
View File

@ -14,6 +14,15 @@ Ocr_Model_Info :: struct {
rec_height: s32 = 48;
rec_batch_gpu: s32 = 16;
rec_batch_cpu: s32 = 1;
// macOS: the detector and recognizer as Core ML packages (tools/export_ppocr_coreml.py), "" for none,
// and the sizes they take, which inputs are padded up to: the detector's height and width multiples
// of these steps, and the recognizer coreml_rec_batch lines at a multiple of coreml_rec_width_step.
detector_coreml, recognizer_coreml: string;
coreml_det_height_step: s32 = 128;
coreml_det_width_step: s32 = 256;
coreml_rec_batch: s32 = 16;
coreml_rec_width_step: s32 = 64;
}
OCR_MODELS_CONFIG :: "models.config";
@ -64,6 +73,12 @@ ocr_list_models :: (models_folder: string) -> [] Ocr_Model_Info, error: string {
case "rec_height"; if number_ok model.rec_height = cast(s32) number;
case "rec_batch_gpu"; if number_ok model.rec_batch_gpu = cast(s32) number;
case "rec_batch_cpu"; if number_ok model.rec_batch_cpu = cast(s32) number;
case "detector_coreml"; model.detector_coreml = copy_string(value);
case "recognizer_coreml"; model.recognizer_coreml = copy_string(value);
case "coreml_det_height_step"; if number_ok model.coreml_det_height_step = cast(s32) number;
case "coreml_det_width_step"; if number_ok model.coreml_det_width_step = cast(s32) number;
case "coreml_rec_batch"; if number_ok model.coreml_rec_batch = cast(s32) number;
case "coreml_rec_width_step"; if number_ok model.coreml_rec_width_step = cast(s32) number;
case; log_error("[OCR] %:%: unknown key %. Ignored.", path, line_number, key);
}
}
@ -98,14 +113,67 @@ ocr_create_sessions :: (engine: *Ocr_Engine, models_folder: string, gpu: bool, i
if model.textline_ori {
engine.orientation, ok, error = ocr_create_session(engine.env, tprint("%/%", models_folder, model.textline_ori), gpu, intra_op_threads, engine.options);
if !ok return false, error;
// PP-LCNet text line orientation takes 160 by 80; a model that says otherwise is taken at its word.
shape := engine.orientation.input_shape;
engine.orientation_height = ifx engine.orientation.input_rank == 4 && shape[2] > 0 then cast(s32) shape[2] else 80;
engine.orientation_width = ifx engine.orientation.input_rank == 4 && shape[3] > 0 then cast(s32) shape[3] else 160;
ocr_orientation_size(engine);
}
return true, "";
}
// PP-LCNet text line orientation takes 160 by 80; a model that says otherwise is taken at its word.
ocr_orientation_size :: (engine: *Ocr_Engine) {
shape := engine.orientation.input_shape;
engine.orientation_height = ifx engine.orientation.input_rank == 4 && shape[2] > 0 then cast(s32) shape[2] else 80;
engine.orientation_width = ifx engine.orientation.input_rank == 4 && shape[3] > 0 then cast(s32) shape[3] else 160;
}
// macOS: the detector and recognizer through Core ML (modules/Core_ML) on the GPU, each run once on
// its smallest size, and the orientation classifier, if any, through onnxruntime on the CPU. On
// failure, why; what was made is left for ocr_release_coreml and ocr_release_sessions.
ocr_create_coreml :: (engine: *Ocr_Engine, models_folder: string, cores: s32) -> ok: bool, error: string {
#if OS != .MACOS {
return false, "Core ML is macOS's.";
} else {
model := *engine.model;
if !model.detector_coreml || !model.recognizer_coreml return false, tprint("there are no Core ML models for %; tools/export_ppocr_coreml.py makes them.", model.name);
for string.[model.detector_coreml, model.recognizer_coreml] {
if !file_exists(tprint("%/%", models_folder, it)) return false, tprint("% is not there; tools/export_ppocr_coreml.py makes it.", it);
}
ok: bool;
error: string;
engine.det_coreml, ok, error = coreml_load(tprint("%/%", models_folder, model.detector_coreml), "x", "y");
if !ok return false, error;
engine.rec_coreml, ok, error = coreml_load(tprint("%/%", models_folder, model.recognizer_coreml), "x", "y");
if !ok return false, error;
if model.textline_ori {
engine.orientation, ok, error = ocr_create_session(engine.env, tprint("%/%", models_folder, model.textline_ori), false, max(cores / 4, 1), engine.options);
if !ok return false, error;
ocr_orientation_size(engine);
}
// The first runs, at the smallest sizes the packages take.
Warm_Up :: struct { model: *Coreml_Model; shape: [4] s64; }
runs := Warm_Up.[
.{engine.det_coreml, .[1, 3, model.coreml_det_height_step, model.coreml_det_width_step]},
.{engine.rec_coreml, .[model.coreml_rec_batch, 3, model.rec_height, model.coreml_rec_width_step]},
];
for runs {
input := NewArray(it.shape[0] * it.shape[1] * it.shape[2] * it.shape[3], float, initialized = true,, context.default_allocator);
defer array_free(input,, context.default_allocator);
out, run_ok, run_error := coreml_predict(it.model, input, it.shape);
if !run_ok return false, tprint("Core ML could not run the % on the GPU: %", ifx it_index == 0 then "detector" else "recognizer", run_error);
coreml_release_output(*out);
}
return true, "";
}
}
ocr_release_coreml :: (engine: *Ocr_Engine) {
#if OS == .MACOS {
coreml_free(engine.det_coreml);
coreml_free(engine.rec_coreml);
engine.det_coreml, engine.rec_coreml = null;
}
}
ocr_release_sessions :: (engine: *Ocr_Engine) {
ocr_release_session(*engine.det);
ocr_release_session(*engine.rec);

33
modules/OCR/module.jai
View File

@ -12,7 +12,7 @@
// then recognition (recognize.jai) reads it level. When the model lists an orientation classifier,
// lines that look upside down are read both ways; lines read downwards or upwards are read too.
Ocr_Provider :: enum { CPU; CUDA; }
Ocr_Provider :: enum { CPU; CUDA; COREML; }
// Tightly packed or not: stride is the bytes from one row to the next.
Ocr_Image :: struct {
@ -40,6 +40,7 @@ Ocr_Engine :: struct {
env: *OrtEnv;
memory_info: *OrtMemoryInfo;
det, rec: Ocr_Session;
#if OS == .MACOS det_coreml, rec_coreml: *Coreml_Model; // the detector and recognizer, when Core ML runs them
orientation: Ocr_Session; // the text line orientation classifier; none when its session is null
orientation_width, orientation_height: s32; // its input size
dictionary: [] string; // class n is dictionary[n - 1]
@ -117,11 +118,24 @@ ocr_create :: (models_folder: string, model_name: string, prefer_gpu := true, op
cores := cast(s32) max(get_number_of_processors(.ALL_LOGICAL), 1); // efficiency cores included
use_gpu := false;
if prefer_gpu {
#if OS == .MACOS if prefer_gpu {
// Core ML on the GPU: the detector and recognizer, the orientation classifier on the CPU. On any
// failure, onnxruntime on the CPU below, with the reason.
coreml_ok, coreml_error := ocr_create_coreml(engine, models_folder, cores);
if coreml_ok {
engine.provider = .COREML;
engine.workers = max(options.gpu_workers, 1);
engine.rec_batch = max(engine.model.coreml_rec_batch, 1);
return engine, true, "";
}
ocr_release_coreml(engine);
engine.gpu_explanation = sprint("OCR runs on the CPU: %", coreml_error);
}
if prefer_gpu && OS != .MACOS {
available, explanation := ort_cuda_check("OCR"); // modules/onnxruntime/cuda.jai
use_gpu = available;
if !available engine.gpu_explanation = copy_string(explanation);
} else {
} else if !prefer_gpu {
engine.gpu_explanation = copy_string("OCR runs on the CPU, as it was asked to.");
}
@ -160,7 +174,16 @@ ocr_provider :: (engine: *Ocr_Engine) -> Ocr_Provider {
// Whether the GPU is in use, and when not, why, and what to install so it can be.
ocr_gpu_status :: (engine: *Ocr_Engine) -> in_use: bool, explanation: string {
return engine.provider == .CUDA, engine.gpu_explanation;
return engine.provider == .CUDA || engine.provider == .COREML, engine.gpu_explanation;
}
// What runs the detector and recognizer, for people and the files it reads: "CUDA", "Core ML" or "CPU".
ocr_provider_name :: (engine: *Ocr_Engine) -> string {
if engine.provider == {
case .CUDA; return "CUDA";
case .COREML; return "Core ML";
}
return "CPU";
}
ocr_destroy :: (engine: *Ocr_Engine) {
@ -169,6 +192,7 @@ ocr_destroy :: (engine: *Ocr_Engine) {
if engine.provider == .CUDA ort_gpu_lock();
ocr_release_sessions(engine);
if engine.provider == .CUDA ort_gpu_unlock();
ocr_release_coreml(engine);
if engine.memory_info ort.api.ReleaseMemoryInfo(engine.memory_info);
if engine.env ort.api.ReleaseEnv(engine.env);
free(engine.gpu_explanation,, context.default_allocator);
@ -458,6 +482,7 @@ run_on_workers :: (engine: *Ocr_Engine, count: s64, proc: Work_Proc, data: *void
#import "Atomics";
#import "onnxruntime";
Default_Allocator :: #import "Default_Allocator"; // rpmalloc (modules/Default_Allocator), for thread_idle where the workers wait
#if OS == .MACOS #import "Core_ML";
#if OS == .WINDOWS {
#import "Windows";
#import "Windows_Utf8";

39
modules/OCR/recognize.jai
View File

@ -29,21 +29,44 @@ ocr_recognize_batch :: (engine: *Ocr_Engine, jobs: [] *Line_Job) -> ok: bool, er
step := max(engine.options.rec_width_step, 1);
width = (width + step - 1) / step * step;
}
rows := cast(s64) jobs.count;
#if OS == .MACOS if engine.rec_coreml {
// The sizes the Core ML package takes: coreml_rec_batch lines (the rest padding), at its width step.
width = align_up(width, engine.model.coreml_rec_width_step);
rows = max(rows, cast(s64) engine.model.coreml_rec_batch);
}
plane := cast(s64) height * width;
input := NewArray(plane * 3 * jobs.count, float, initialized = true,, context.default_allocator); // 0 is the padding
input := NewArray(plane * 3 * rows, float, initialized = true,, context.default_allocator); // 0 is the padding
defer array_free(input,, context.default_allocator);
for job, n: jobs {
ocr_sample_crop(job.image, job.corners, job.width, height, input.data + plane * 3 * n, width, plane, REC_MEAN, REC_STD);
}
shape := s64.[jobs.count, 3, height, width];
out, ok, error := ocr_run(*engine.rec, engine.memory_info, input, shape);
if !ok return false, error;
shape := s64.[rows, 3, height, width];
scores: *float;
dims: [] s64;
out: Ocr_Output;
defer ocr_release_output(*out);
if out.dim_count != 3 || out.dims[0] != jobs.count return false, tprint("The recognizer gave an output of shape %.", array_view(out.dims, 0, out.dim_count));
steps := out.dims[1];
classes := out.dims[2];
#if OS == .MACOS {
coreml_out: Coreml_Output;
defer coreml_release_output(*coreml_out);
}
#if OS == .MACOS if engine.rec_coreml {
ok: bool; error: string;
coreml_out, ok, error = coreml_predict(engine.rec_coreml, input, shape);
if !ok return false, error;
scores, dims = coreml_out.data, array_view(coreml_out.dims, 0, coreml_out.dim_count);
}
if !scores {
ok: bool; error: string;
out, ok, error = ocr_run(*engine.rec, engine.memory_info, input, shape);
if !ok return false, error;
scores, dims = out.data, array_view(out.dims, 0, out.dim_count);
}
if dims.count != 3 || dims[0] != rows return false, tprint("The recognizer gave an output of shape %.", dims);
steps := dims[1];
classes := dims[2];
if classes != engine.dictionary.count + 2 {
return false, tprint("The recognizer has % classes, but its dictionary % characters (plus blank and space, % expected).", classes, engine.dictionary.count, engine.dictionary.count + 2);
}
@ -55,7 +78,7 @@ ocr_recognize_batch :: (engine: *Ocr_Engine, jobs: [] *Line_Job) -> ok: bool, er
probability_sum: float64;
characters := 0;
for t: 0..steps - 1 {
row := out.data + (n * steps + t) * classes;
row := scores + (n * steps + t) * classes;
best := 0;
best_p := row[0];
for c: 1..classes - 1 if row[c] > best_p { best = c; best_p = row[c]; }

77
modules/Screen_Record/audio_mix.jai Normal file
View File

@ -0,0 +1,77 @@
// The sound of a recording, mixed: each source's packets put on the recording's timeline and added
// together, and an AAC frame encoded once every source has covered it. Shared by the Windows and
// macOS recorders (windows.jai, macos.jai), whose Recorder each has the fields used here: moved here
// from module.jai unchanged.
#scope_module
// The sound so far, from its base: one AAC frame's worth is encoded at a time.
Audio_Mix :: struct {
samples: [..] float; // interleaved stereo, from base on
base: s64; // the frame (in samples, on the timeline) samples[0] is
next: [2] s64; // each source's next frame, so packets that follow on are not moved by jitter
started: [2] bool;
frame_size: s32;
}
AUDIO_SLACK :: AUDIO_RATE / 5; // 200 ms: how late a source's packets may come and still be in the mix
AUDIO_FOLLOW_ON :: AUDIO_RATE / 33; // 30 ms: a packet that close to where its source left off follows on
// A packet into the mix at its place on the timeline, added to what the other source put there.
audio_mix_add :: (mix: *Audio_Mix, r: *Recorder, source: s64, packet: Audio_Packet) {
at := record_timeline(r, packet.ticks) * AUDIO_RATE / r.frequency;
if mix.started[source] && abs(at - mix.next[source]) <= AUDIO_FOLLOW_ON at = mix.next[source];
mix.started[source] = true;
mix.next[source] = at + packet.frames;
end := (at + packet.frames - mix.base) * AUDIO_CHANNELS;
if end > mix.samples.count {
old := mix.samples.count;
array_resize(*mix.samples, end, initialize = false);
for old..end - 1 mix.samples[it] = 0;
}
if packet.silent || !packet.samples.count return;
for frame: max(at, mix.base)..at + packet.frames - 1 {
into := (frame - mix.base) * AUDIO_CHANNELS;
from := (frame - at) * AUDIO_CHANNELS;
mix.samples[into] += packet.samples[from];
mix.samples[into + 1] += packet.samples[from + 1];
}
}
// Encodes the frames every open source has covered: up to where each has sent, or, for one that has
// sent nothing lately (silence), to AUDIO_SLACK ago. final: everything sent.
audio_mix_encode :: (mix: *Audio_Mix, r: *Recorder, final: bool) -> ok: bool, error: string {
// Paused, the timeline stands still where the pause began.
now := ifx atomic_read(*r.paused) then atomic_read(*r.pause_began) else ticks_now();
now_frame := record_timeline(r, now) * AUDIO_RATE / r.frequency;
ready := now_frame - AUDIO_SLACK;
for source, index: r.audio_sources {
if !source.open continue;
covered := max(mix.next[index], now_frame - AUDIO_SLACK);
ready = min(ready, covered);
}
if final { ready = mix.base; for mix.next ready = max(ready, it); }
size := mix.frame_size;
frame := NewArray(size * AUDIO_CHANNELS, float, initialized = false,, temp);
done: s64;
while mix.base + size <= ready {
for i: 0..frame.count - 1 {
at := done * AUDIO_CHANNELS + i;
value := ifx at < mix.samples.count then mix.samples[at] else 0.0;
frame[i] = clamp(value, -1.0, 1.0);
}
ok, error := encoder_write_audio(*r.encoder, frame.data, mix.base);
if !ok return false, error;
mix.base += size;
done += size;
lock(*r.mutex); r.stats.audio_frames += 1; unlock(*r.mutex);
}
// What was encoded leaves the front.
gone := min(done * AUDIO_CHANNELS, mix.samples.count);
if gone {
memcpy(mix.samples.data, mix.samples.data + gone, (mix.samples.count - gone) * size_of(float));
mix.samples.count -= gone;
}
return true, "";
}

17
modules/Screen_Record/encoder.jai
View File

@ -1,6 +1,7 @@
// The encoder thread's half: frames of BGRA pixels to H.264 in a fragmented MP4, with FFmpeg
// (modules/ffmpeg). The first encoder in RECORD_ENCODERS that opens at the size wanted is used:
// NVENC, AMD's AMF, Intel's QSV, Windows' Media Foundation, then libx264 on the CPU. NVENC takes the
// NVENC, AMD's AMF, Intel's QSV, Windows' Media Foundation, then libx264 on the CPU; on macOS,
// VideoToolbox (Apple silicon's media engine), at a constant quality. NVENC takes the
// BGRA as it is and converts it on the GPU; the others are given NV12 or YUV 4:2:0, converted with
// swscale (BT.709, limited range, as tagged).
//
@ -12,7 +13,8 @@
// span any time: a keyframe is made every RECORD_KEYFRAME_SECONDS of time instead, which bounds what a
// crash loses. remux.jai makes the file a regular MP4 once it is done.
RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"];
#if OS == .MACOS RECORD_ENCODERS :: string.["h264_videotoolbox"]; // Apple's, in every Mac; the macOS FFmpeg has no libx264
else RECORD_ENCODERS :: string.["h264_nvenc", "h264_amf", "h264_qsv", "h264_mf", "libx264"];
RECORD_KEYFRAME_SECONDS :: 2;
RECORD_AUDIO_BIT_RATE :: 160_000;
@ -170,6 +172,7 @@ encoder_close :: (e: *Encoder) -> ok: bool, error: string {
#scope_file
RECORD_TIME_BASE :: 90000; // pts ticks a second: MPEG's
VIDEOTOOLBOX_QUALITY :: 65; // 0-100: sharp text at a few MB a minute for a screen
encoder_try :: (e: *Encoder, name: string, width: s32, height: s32, fps: s32) -> ok: bool, error: string {
codec := avcodec_find_encoder_by_name(temp_c_string(name));
@ -203,6 +206,13 @@ encoder_try :: (e: *Encoder, name: string, width: s32, height: s32, fps: s32) ->
case "h264_qsv"; set("global_quality", "23"); set("preset", "medium");
case "h264_mf"; set("rate_control", "quality"); set("quality", "70"); set("scenario", "display_remoting");
case "libx264"; set("preset", "veryfast"); set("crf", "23"); set("forced-idr", "1");
case "h264_videotoolbox";
// A constant quality (VideoToolbox's 0-1, as 0-100), which Apple silicon takes: FFmpeg's default
// bit rate, 200 kb/s, would blur a screen. Real time, as frames come as they are recorded.
set("realtime", "1");
set("allow_sw", "0");
c.flags |= AV_CODEC_FLAG_QSCALE;
c.global_quality = FF_QP2LAMBDA * VIDEOTOOLBOX_QUALITY;
}
result := avcodec_open2(c, codec, *options);
av_dict_free(*options);
@ -231,7 +241,8 @@ encoder_input_format :: (codec: *AVCodec) -> AVPixelFormat {
// The packets an encoder (the video's, unless another is given) has ready, into the file.
encoder_drain :: (e: *Encoder, codec: *AVCodecContext = null, stream: *AVStream = null, packet: *AVPacket = null) -> ok: bool, error: string {
AVERROR_EAGAIN :: -11;
#if OS == .MACOS AVERROR_EAGAIN :: -35; // AVERROR(EAGAIN): -EAGAIN, whose errno differs
else AVERROR_EAGAIN :: -11;
if !codec codec, stream, packet = e.codec, e.stream, e.packet;
while true {
result := avcodec_receive_packet(codec, packet);

526
modules/Screen_Record/macos.jai Normal file
View File

@ -0,0 +1,526 @@
// The macOS recorder (module.jai has what is shared; windows.jai is the Windows one, which this follows).
// A ScreenCaptureKit stream (modules/Screen_Record/macos/capture.m) of one display's region gives the
// frames, already scaled to the region's pixels and with the cursor drawn in when asked, and the sound:
// what the computer plays, and on macOS 15 and later the microphone, both from the same stream, so
// there is no device to open here. mcap's own sound is left out.
//
// Three threads, as on Windows. The capture thread takes the newest frame, at most fps a second and
// one a second while nothing changes, so players can seek; each carries the host time it was shown,
// less the time paused. The encoder thread hands them to FFmpeg (encoder.jai, with VideoToolbox), the
// oldest dropped when it falls RECORD_QUEUE behind. The audio thread puts each source's packets on the
// same timeline into the mix (audio_mix.jai), and turns the sources on and off in the stream's
// configuration as record_set_audio asks.
//
// Times are the host clock's (mach_absolute_time, CLOCK_UPTIME_RAW) in nanoseconds, which
// ScreenCaptureKit's sample times are on too.
AUDIO_RATE :: 48000;
AUDIO_CHANNELS :: 2;
Audio_Source_Kind :: enum u8 { SYSTEM; MICROPHONE; }
// A source of sound in the stream: open while the stream is set to give it.
Audio_Source :: struct {
kind: Audio_Source_Kind;
name: string; // "System sound", "Microphone"
open: bool;
lost_at: s64; // the mix's; never lost here
}
Audio_Packet :: struct {
samples: [] float; // interleaved stereo, AUDIO_CHANNELS a frame
frames: s64;
ticks: s64; // the host time its first frame was heard, in nanoseconds
silent: bool;
}
Record_Options :: struct {
path: string; // the .mp4 written
display_id: u32; // the display (CGDirectDisplayID)
x, y, w, h: float64; // the region, in the display's points from its top left
width, height: s32; // and in its pixels: rounded down to even here
fps: s32 = 30;
cursor := true;
encoder: string; // "" tries RECORD_ENCODERS
system_audio: bool; // what the computer plays
microphone: bool; // the default input (macOS 15 and later); both at the start: record_set_audio turns them on and off
audio_track: bool; // a sound track whatever is on at the start, so a source can be turned on later
exclude_windows: [] u32; // window numbers kept out of the frames: the border and toolbar
}
Record_Stats :: struct {
source: string; // "ScreenCaptureKit, display 1"
fell_back_from: string; // never, here
encoder: string;
width, height: s32;
captured: s64; // new frames from the stream
pointer_moves: s64;
emitted: s64; // frames sent to be encoded
repeats: s64; // of them, the once-a-second repeat of an unchanged screen
written: s64; // encoded
dropped: s64; // the encoder was too far behind
lost: s64;
paused_seconds: float64; // left out of the video
audio: string; // the sound recorded: "System sound; Microphone", or ""
audio_frames: s64; // AAC frames encoded
audio_errors: string; // sources that could not be turned on
bytes: s64; // encoded
seconds: float64; // from start to stop
}
Recorder :: struct {
options: Record_Options;
stream: *Mcap_Sck_Stream;
encoder: Encoder;
width, height: s32;
frequency: s64 = 1_000_000_000; // ticks a second: nanoseconds
start_ticks: s64;
capture_thread, encoder_thread, audio_thread: Thread;
capture_started, encoder_started, audio_started: bool;
audio_sources: [2] Audio_Source;
audio_open: bool; // the MP4 has sound
audio_on: [2] bool; // each source is wanted (by Audio_Source_Kind): record_set_audio
audio_failed: [2] bool; // it could not be turned on: record_audio_failed says once
zero_ticks: s64; // the recording's time 0
paused_ticks: s64; // time paused so far; set by record_pause
pause_began: s64;
running: bool;
stopping: bool;
paused: bool;
capture_ended: bool;
mutex: Mutex;
ready: Semaphore; // signalled for each frame queued, and once when capture ends
latest: [] u8; // the stream's newest frame, BGRA: the capture thread's
frames: [RECORD_BUFFERS] Record_Frame;
free: [RECORD_BUFFERS] s32;
free_count: s32;
queue: [RECORD_BUFFERS] s32; // oldest first
queue_count: s32;
last_frame: [] u8; // the last frame queued, for record_last_frame
stats: Record_Stats;
error: string; // why it stopped by itself; heap
}
Record_Frame :: struct {
pixels: [] u8; // BGRA, rows packed
ticks: s64;
}
// Starts the stream, the encoder and the file, and starts recording. fall_back is never set: there is
// one way of taking the screen here.
record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool {
r.* = .{};
r.options = options;
r.options.path = copy_string(options.path);
r.options.encoder = copy_string(options.encoder);
r.options.exclude_windows = array_copy(options.exclude_windows);
if r.options.fps <= 0 r.options.fps = 30;
if !record_size_allowed(options.width, options.height)
return false, tprint("% x % is smaller than % x %", options.width, options.height, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false;
// Even both ways: H.264's 4:2:0 colour needs it. The region in points shrinks to match.
r.width = options.width & cast(s32) ~1;
r.height = options.height & cast(s32) ~1;
scale := options.w / options.width;
w := r.width * scale;
h := r.height * scale;
r.stats.width, r.stats.height = r.width, r.height;
init(*r.mutex, "record");
init(*r.ready);
for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false);
for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it;
r.free_count = RECORD_BUFFERS;
r.latest = NewArray(cast(s64) r.width * r.height * 4, u8);
r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8);
// The microphone only where the stream can give it.
microphone := options.microphone && mcap_sck_microphone_supported() != 0;
if options.microphone && !microphone {
r.stats.audio_errors = copy_string("microphone: it needs macOS 15 or later");
r.audio_failed[Audio_Source_Kind.MICROPHONE] = true;
}
error_text: [512] u8;
excluded := r.options.exclude_windows;
r.stream = mcap_sck_stream_start(options.display_id, options.x, options.y, w, h, r.width, r.height, r.options.fps,
xx ifx options.cursor then 1 else 0, xx ifx options.system_audio then 1 else 0, xx ifx microphone then 1 else 0,
excluded.data, xx excluded.count, error_text.data, error_text.count);
if !r.stream {
error := tprint("ScreenCaptureKit could not record: %", to_string(error_text.data));
record_free(r);
return false, error, false;
}
r.stats.source = sprint("ScreenCaptureKit, display %", options.display_id);
r.start_ticks = ticks_now();
r.zero_ticks = r.start_ticks;
r.audio_sources[0] = .{kind = .SYSTEM, name = "System sound", open = options.system_audio};
r.audio_sources[1] = .{kind = .MICROPHONE, name = "Microphone", open = microphone};
r.audio_on = .[options.system_audio, microphone];
r.audio_open = options.system_audio || microphone || options.audio_track;
record_note_audio(r);
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder, audio = r.audio_open);
if tried log("[record] Encoders that did not open: %", tried);
if !ok {
error = copy_string(error,, temp);
record_free(r);
return false, error, false;
}
r.stats.encoder = copy_string(r.encoder.name);
r.running = true;
thread_init(*r.capture_thread, record_capture_proc, name = "record capture");
r.capture_started = true;
thread_start(*r.capture_thread, r);
thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder");
r.encoder_started = true;
thread_start(*r.encoder_thread, r);
if r.audio_open {
thread_init(*r.audio_thread, record_audio_proc, name = "record audio");
r.audio_started = true;
thread_start(*r.audio_thread, r);
}
return true, "", false;
}
// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an
// error, or the file could not be finished; what was recorded up to then is in the file all the same.
record_stop :: (r: *Recorder) -> ok: bool, error: string {
if !r.running return false, "not recording";
atomic_swap(*r.stopping, true);
thread_is_done(*r.capture_thread, -1);
thread_is_done(*r.encoder_thread, -1);
if r.audio_started thread_is_done(*r.audio_thread, -1);
r.running = false;
closed, close_error := encoder_close(*r.encoder);
r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency;
r.stats.bytes = r.encoder.bytes;
error := ifx r.error then copy_string(r.error,, temp) else "";
record_free(r);
if error return false, error;
if !closed return false, close_error;
return true, "";
}
// Pauses or resumes; the time paused is left out of the video, sound and all.
record_pause :: (r: *Recorder, pause: bool) {
if !r.running || atomic_read(*r.paused) == pause return;
now := ticks_now();
if pause {
atomic_write(*r.pause_began, now);
atomic_swap(*r.paused, true);
} else {
atomic_add(*r.paused_ticks, now - atomic_read(*r.pause_began));
atomic_swap(*r.paused, false);
}
}
// Turns a source of sound on or off while recording; the stream is told on the audio thread. False
// when the recording has no sound track to put it in.
record_set_audio :: (r: *Recorder, kind: Audio_Source_Kind, on: bool) -> bool {
if !r.running || !r.audio_open return false;
atomic_swap(*r.audio_on[kind], on);
return true;
}
record_audio_on :: (r: *Recorder, kind: Audio_Source_Kind) -> bool {
return r.running && atomic_read(*r.audio_on[kind]);
}
// Whether a source could not be turned on (it is off again then), and why; once.
record_audio_failed :: (r: *Recorder, kind: Audio_Source_Kind) -> failed: bool, error: string {
if !r.running || !atomic_read(*r.audio_failed[kind]) return false, "";
atomic_swap(*r.audio_failed[kind], false);
lock(*r.mutex);
defer unlock(*r.mutex);
return true, copy_string(r.stats.audio_errors,, temp);
}
// Ticks on the recording's timeline: time 0 at its start, less the time paused.
record_timeline :: (r: *Recorder, ticks: s64) -> s64 {
return ticks - r.zero_ticks - atomic_read(*r.paused_ticks);
}
record_paused :: (r: *Recorder) -> bool {
return r.running && atomic_read(*r.paused);
}
record_stats :: (r: *Recorder) -> Record_Stats {
if !r.running return r.stats;
lock(*r.mutex);
defer unlock(*r.mutex);
return r.stats;
}
// The last frame queued, BGRA (temp), cursor and all.
record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 {
if r.running lock(*r.mutex);
pixels := array_copy(r.last_frame,, temp);
if r.running unlock(*r.mutex);
return pixels, r.width, r.height;
}
record_failed :: (r: *Recorder) -> bool, string {
lock(*r.mutex);
defer unlock(*r.mutex);
return r.error.count > 0, r.error;
}
// The host clock, in nanoseconds: ScreenCaptureKit's sample times are on it.
ticks_now :: () -> s64 {
CLOCK_UPTIME_RAW :: 8;
return cast(s64) clock_gettime_nsec_np(CLOCK_UPTIME_RAW);
}
#scope_file
// The sources heard, for the stats: "System sound; Microphone".
record_note_audio :: (r: *Recorder) {
names: [..] string;
names.allocator = temp;
for r.audio_sources if it.open array_add(*names, it.name);
lock(*r.mutex);
free(r.stats.audio);
r.stats.audio = join(..names, "; ");
unlock(*r.mutex);
}
record_free :: (r: *Recorder) {
atomic_swap(*r.stopping, true);
if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; }
if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; }
if r.audio_started { thread_is_done(*r.audio_thread, -1); thread_deinit(*r.audio_thread); r.audio_started = false; }
if r.stream { mcap_sck_stream_stop(r.stream); r.stream = null; }
if r.encoder.format encoder_close(*r.encoder);
for * r.frames array_free(it.pixels);
array_free(r.latest);
array_free(r.last_frame);
r.latest, r.last_frame = .[], .[];
array_free(r.options.exclude_windows);
r.options.exclude_windows = .[];
destroy(*r.ready);
destroy(*r.mutex);
free(r.error);
r.error = "";
}
// Takes the stream's frames as they come, and sends one when its turn is due: at most fps a second
// (a tenth of a frame early counts as on time), and one a second while nothing changes.
record_capture_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
period := r.frequency / r.options.fps;
early := period / 10;
turn := ticks_now();
last_emit: s64 = -r.frequency;
have_image, changed := false, false;
frame_time: s64;
was_paused := false;
error_text: [512] u8;
while !atomic_read(*r.stopping) {
now := ticks_now();
due := ifx changed then turn - early else last_emit + r.frequency;
wait := cast(s32) clamp((due - now) * 1000 / r.frequency, 1, 100);
shown: s64;
got := mcap_sck_next_frame(r.stream, r.latest.data, *shown, wait, error_text.data, error_text.count);
if got < 0 {
record_fail(r, tprint("ScreenCaptureKit stopped: %", to_string(error_text.data)));
break;
}
if got > 0 {
have_image, changed = true, true;
frame_time = shown;
lock(*r.mutex); r.stats.captured += 1; unlock(*r.mutex);
}
now = ticks_now();
if atomic_read(*r.paused) {
was_paused = true;
lock(*r.mutex); r.stats.paused_seconds = cast(float64) (atomic_read(*r.paused_ticks) + now - atomic_read(*r.pause_began)) / r.frequency; unlock(*r.mutex);
continue;
}
if was_paused {
was_paused = false;
changed = true;
turn = now;
frame_time = now; // the screen as it is, at the moment of resuming
lock(*r.mutex); r.stats.paused_seconds = cast(float64) atomic_read(*r.paused_ticks) / r.frequency; unlock(*r.mutex);
}
if !have_image continue;
repeat := !changed && now - last_emit >= r.frequency;
if !(changed && now >= turn - early) && !repeat continue;
turn = max(turn + period, now - period);
record_queue_frame(r, record_timeline(r, ifx repeat then now else frame_time));
last_emit = now;
changed = false;
lock(*r.mutex);
r.stats.emitted += 1;
if repeat r.stats.repeats += 1;
unlock(*r.mutex);
}
atomic_swap(*r.capture_ended, true);
signal(*r.ready);
return 0;
}
// The newest frame into a free buffer, queued for the encoder; the oldest waiting dropped when every
// buffer is taken.
record_queue_frame :: (r: *Recorder, ticks: s64) {
lock(*r.mutex);
if !r.free_count {
r.free[r.free_count] = r.queue[0];
r.free_count += 1;
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
r.stats.dropped += 1;
}
r.free_count -= 1;
index := r.free[r.free_count];
unlock(*r.mutex);
frame := *r.frames[index];
memcpy(frame.pixels.data, r.latest.data, frame.pixels.count);
frame.ticks = ticks;
lock(*r.mutex);
r.queue[r.queue_count] = index;
r.queue_count += 1;
memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count);
unlock(*r.mutex);
signal(*r.ready);
}
record_encoder_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
while true {
wait_for(*r.ready, 100);
lock(*r.mutex);
index: s32 = -1;
if r.queue_count {
index = r.queue[0];
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
}
unlock(*r.mutex);
if index < 0 {
if atomic_read(*r.capture_ended) break;
continue;
}
frame := *r.frames[index];
pts := max(frame.ticks, 0) * 90000 / r.frequency; // on the timeline already
ok, error := encoder_write(*r.encoder, frame.pixels.data, pts);
lock(*r.mutex);
r.free[r.free_count] = index;
r.free_count += 1;
if ok r.stats.written += 1;
r.stats.bytes = r.encoder.bytes;
unlock(*r.mutex);
if !ok {
record_fail(r, error);
while !atomic_read(*r.capture_ended) {
wait_for(*r.ready, 100);
lock(*r.mutex);
for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; }
r.queue_count = 0;
unlock(*r.mutex);
}
break;
}
}
return 0;
}
// Every 10 ms: the sources turned on or off as asked, their packets into the mix, and what every
// source has covered encoded.
record_audio_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
if !r.encoder.audio return 0;
mix: Audio_Mix;
mix.frame_size = r.encoder.audio_frame_size;
defer array_free(mix.samples);
MAX_FRAMES :: 4096;
samples := NewArray(MAX_FRAMES * AUDIO_CHANNELS, float);
defer array_free(samples);
error_text: [512] u8;
while true {
stopping := atomic_read(*r.stopping);
// The stream's sources as the toolbar wants them.
want := r.audio_on;
for * want it.* = atomic_read(it);
if !stopping && (want[0] != r.audio_sources[0].open || want[1] != r.audio_sources[1].open) {
if want[1] && !r.audio_sources[1].open && !mcap_sck_microphone_supported() {
atomic_swap(*r.audio_on[Audio_Source_Kind.MICROPHONE], false);
lock(*r.mutex); free(r.stats.audio_errors); r.stats.audio_errors = copy_string("microphone: it needs macOS 15 or later"); unlock(*r.mutex);
atomic_swap(*r.audio_failed[Audio_Source_Kind.MICROPHONE], true);
want[1] = false;
}
if mcap_sck_stream_set_audio(r.stream, xx ifx want[0] then 1 else 0, xx ifx want[1] then 1 else 0, error_text.data, error_text.count) {
for 0..1 {
if want[it] && !r.audio_sources[it].open mix.started[it] = false;
r.audio_sources[it].open = want[it];
}
record_note_audio(r);
log("[record] Sound now: %.", ifx r.stats.audio then r.stats.audio else "none");
} else {
why := to_string(error_text.data);
log_error("[record] The sound could not be changed: %", why);
for 0..1 if want[it] && !r.audio_sources[it].open {
atomic_swap(*r.audio_on[it], false);
lock(*r.mutex); free(r.stats.audio_errors); r.stats.audio_errors = sprint("% could not be turned on: %", r.audio_sources[it].name, why); unlock(*r.mutex);
atomic_swap(*r.audio_failed[it], true);
}
}
}
for * source, index: r.audio_sources {
if !source.open continue;
while true {
shown: s64;
frames := mcap_sck_next_audio(r.stream, xx index, samples.data, MAX_FRAMES, *shown);
if frames <= 0 break;
if atomic_read(*r.paused) continue;
packet := Audio_Packet.{samples = array_view(samples, 0, frames * AUDIO_CHANNELS), frames = frames, ticks = shown};
audio_mix_add(*mix, r, index, packet);
}
}
ok, error := audio_mix_encode(*mix, r, final = stopping);
if !ok { record_fail(r, error); break; }
if stopping break;
reset_temporary_storage();
sleep_milliseconds(10);
}
return 0;
}
record_fail :: (r: *Recorder, error: string) {
log_error("[record] %", error);
lock(*r.mutex);
if !r.error r.error = copy_string(error);
unlock(*r.mutex);
atomic_swap(*r.stopping, true);
}
#scope_module
// The shim (macos/capture.h).
Mcap_Sck_Stream :: struct { _: u8; }
mcap_capture :: #library,no_dll "macos/capture";
mcap_sck_stream_start :: (display_id: u32, x: float64, y: float64, w: float64, h: float64, width: s32, height: s32, fps: s32, show_cursor: s32,
system_audio: s32, microphone: s32, excluded: *u32, excluded_count: s32, error: *u8, error_size: s32) -> *Mcap_Sck_Stream #foreign mcap_capture;
mcap_sck_next_frame :: (s: *Mcap_Sck_Stream, pixels: *u8, host_time_ns: *s64, timeout_ms: s32, error: *u8, error_size: s32) -> s32 #foreign mcap_capture;
mcap_sck_next_audio :: (s: *Mcap_Sck_Stream, source: s32, samples: *float, max_frames: s32, host_time_ns: *s64) -> s32 #foreign mcap_capture;
mcap_sck_stream_set_audio :: (s: *Mcap_Sck_Stream, system_audio: s32, microphone: s32, error: *u8, error_size: s32) -> s32 #foreign mcap_capture;
mcap_sck_microphone_supported :: () -> s32 #foreign mcap_capture;
mcap_sck_stream_stop :: (s: *Mcap_Sck_Stream) #foreign mcap_capture;
ScreenCaptureKit_Framework :: #library,system,link_always "ScreenCaptureKit";
CoreMedia_Framework :: #library,system,link_always "CoreMedia";
CoreVideo_Framework :: #library,system,link_always "CoreVideo";
clock_gettime_nsec_np :: (clock: s32) -> u64 #foreign libc_for_clock;
libc_for_clock :: #library,system "libc";

53
modules/Screen_Record/macos/capture.h Normal file
View File

@ -0,0 +1,53 @@
// mcap's ScreenCaptureKit shim (capture.m): plain C, for Jai (src/app/screen_capture_macos.jai).
#pragma once
#include <stdint.h>
// One screen to copy. display_id, width and height (its size in pixels) are given; ScreenCaptureKit's
// own width and height and the pixels (packed RGBA, opaque, sRGB; free with mcap_sck_free) come back.
typedef struct {
uint32_t display_id;
int32_t width, height;
uint8_t *pixels;
} Mcap_Sck_Shot;
// Copies each screen, all at once, and waits for them. 1 when every one was copied; else 0, with why
// in error (pixels already made are left set, to be freed).
int mcap_sck_screenshots(Mcap_Sck_Shot *shots, int count, char *error, int error_size);
void mcap_sck_free(void *pointer);
// ---------------------------------------------------------------- recording (macos-plan, phase 7)
// A stream of one display's region, with the sound wanted. Frames are kept one at a time, the
// newest; sound is queued, per source, until taken.
typedef struct Mcap_Sck_Stream Mcap_Sck_Stream;
enum { MCAP_SCK_SYSTEM = 0, MCAP_SCK_MICROPHONE = 1 };
// The region (x, y, w, h) in the display's points from its top left; width and height the frames'
// size in pixels. excluded: window numbers kept out of the frames (mcap's border and toolbar).
// The microphone comes from the stream itself, which macOS 15 has; on 14 it cannot be turned on.
// NULL, with why in error, when it could not start.
Mcap_Sck_Stream *mcap_sck_stream_start(uint32_t display_id, double x, double y, double w, double h,
int32_t width, int32_t height, int32_t fps, int show_cursor,
int system_audio, int microphone,
const uint32_t *excluded, int excluded_count,
char *error, int error_size);
// The newest frame since the last taken, as packed BGRA into pixels (width * height * 4), with the
// host time it was shown at (mach_absolute_time, in nanoseconds). 1 for a frame; 0 when none came in
// timeout_ms; -1 when the stream stopped by itself (error says why).
int mcap_sck_next_frame(Mcap_Sck_Stream *s, uint8_t *pixels, int64_t *host_time_ns, int timeout_ms, char *error, int error_size);
// The oldest sound packet of a source not yet taken: up to max_frames frames of 48 kHz stereo float,
// interleaved, into samples, and the host time of its first frame. Its frame count; 0 for none.
int mcap_sck_next_audio(Mcap_Sck_Stream *s, int source, float *samples, int max_frames, int64_t *host_time_ns);
// Turns the sources on and off while it runs (the stream's configuration changed). 1 when done.
int mcap_sck_stream_set_audio(Mcap_Sck_Stream *s, int system_audio, int microphone, char *error, int error_size);
// Whether this macOS can take the microphone from the stream.
int mcap_sck_microphone_supported(void);
// Stops it, waits for it, and frees it.
void mcap_sck_stream_stop(Mcap_Sck_Stream *s);

392
modules/Screen_Record/macos/capture.m Normal file
View File

@ -0,0 +1,392 @@
// ScreenCaptureKit for mcap on macOS, behind a plain C interface (capture.h): ScreenCaptureKit is
// asynchronous, with completion blocks and its own queues, which is simpler to wait on here than from
// Jai. first.jai compiles this into capture.a beside it. Stills for the screen capture overlay, and
// a stream for recording (modules/Screen_Record/macos.jai): its frames and sound, on host time.
#import "capture.h"
#import <ScreenCaptureKit/ScreenCaptureKit.h>
#import <CoreGraphics/CoreGraphics.h>
#import <CoreMedia/CoreMedia.h>
#import <CoreVideo/CoreVideo.h>
#include <stdlib.h>
#include <string.h>
static void set_error(char *error, int error_size, NSString *text) {
if (!error || error_size <= 0) return;
const char *utf8 = text.UTF8String ? text.UTF8String : "unknown error";
strncpy(error, utf8, (size_t) error_size - 1);
error[error_size - 1] = 0;
}
// The image as packed RGBA, opaque, in sRGB (which the capture is already in): malloc'd.
static uint8_t *rgba_from_image(CGImageRef image, int32_t width, int32_t height) {
size_t row = (size_t) width * 4;
uint8_t *pixels = malloc(row * (size_t) height);
if (!pixels) return NULL;
CGColorSpaceRef srgb = CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
CGContextRef bitmap = CGBitmapContextCreate(pixels, (size_t) width, (size_t) height, 8, row, srgb,
kCGBitmapByteOrder32Big | kCGImageAlphaNoneSkipLast);
CGColorSpaceRelease(srgb);
if (!bitmap) { free(pixels); return NULL; }
CGContextSetBlendMode(bitmap, kCGBlendModeCopy);
CGContextDrawImage(bitmap, CGRectMake(0, 0, width, height), image);
CGContextRelease(bitmap);
for (size_t i = 0; i < (size_t) width * (size_t) height; i++) pixels[i * 4 + 3] = 255;
return pixels;
}
int mcap_sck_screenshots(Mcap_Sck_Shot *shots, int count, char *error, int error_size) {
@autoreleasepool {
for (int i = 0; i < count; i++) shots[i].pixels = NULL;
__block SCShareableContent *content = nil;
__block NSError *content_error = nil;
dispatch_semaphore_t got = dispatch_semaphore_create(0);
[SCShareableContent getShareableContentExcludingDesktopWindows:NO onScreenWindowsOnly:YES
completionHandler:^(SCShareableContent *c, NSError *e) {
content = c;
content_error = e;
dispatch_semaphore_signal(got);
}];
if (dispatch_semaphore_wait(got, dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC)) != 0) {
set_error(error, error_size, @"ScreenCaptureKit did not list the screens in time");
return 0;
}
if (!content) {
set_error(error, error_size, content_error ? content_error.localizedDescription : @"ScreenCaptureKit listed no screens");
return 0;
}
// Every screen at once; each completion fills its own entry.
dispatch_group_t group = dispatch_group_create();
__block NSString *failure = nil;
NSLock *lock = [NSLock new];
for (int i = 0; i < count; i++) {
SCDisplay *display = nil;
for (SCDisplay *d in content.displays) if (d.displayID == shots[i].display_id) display = d;
if (!display) {
failure = [NSString stringWithFormat:@"ScreenCaptureKit has no display %u", shots[i].display_id];
continue;
}
SCContentFilter *filter = [[SCContentFilter alloc] initWithDisplay:display excludingWindows:@[]];
SCStreamConfiguration *configuration = [SCStreamConfiguration new];
configuration.width = (size_t) shots[i].width;
configuration.height = (size_t) shots[i].height;
configuration.showsCursor = NO;
configuration.colorSpaceName = kCGColorSpaceSRGB;
configuration.captureResolution = SCCaptureResolutionBest;
Mcap_Sck_Shot *shot = &shots[i];
dispatch_group_enter(group);
[SCScreenshotManager captureImageWithFilter:filter configuration:configuration
completionHandler:^(CGImageRef image, NSError *e) {
if (image) {
int32_t w = (int32_t) CGImageGetWidth(image), h = (int32_t) CGImageGetHeight(image);
shot->width = w;
shot->height = h;
shot->pixels = rgba_from_image(image, w, h);
}
if (!shot->pixels) {
[lock lock];
failure = e ? e.localizedDescription : @"ScreenCaptureKit gave no image";
[lock unlock];
}
dispatch_group_leave(group);
}];
}
if (dispatch_group_wait(group, dispatch_time(DISPATCH_TIME_NOW, 10 * NSEC_PER_SEC)) != 0) {
failure = @"ScreenCaptureKit did not copy the screens in time";
}
if (failure) {
// Entries still being filled after a timeout are left to leak rather than raced.
set_error(error, error_size, failure);
return 0;
}
return 1;
}
}
void mcap_sck_free(void *pointer) {
free(pointer);
}
// ---------------------------------------------------------------- recording
#define AUDIO_RATE 48000
// Whether this is macOS 15 or later, for the microphone in the stream. Asked of NSProcessInfo rather
// than with @available, whose check comes from clang's runtime library, which Jai's link leaves out;
// the warnings for the calls it guards are off below.
static BOOL macos_15(void) {
static int known = -1;
if (known < 0) known = [[NSProcessInfo processInfo] isOperatingSystemAtLeastVersion:(NSOperatingSystemVersion) {15, 0, 0}] ? 1 : 0;
return known == 1;
}
#pragma clang diagnostic ignored "-Wunguarded-availability-new"
// A source's packets, waiting to be taken: interleaved stereo float at AUDIO_RATE.
@interface McapAudioPacket : NSObject
@property (nonatomic) NSData *samples;
@property (nonatomic) int64_t hostTime;
@end
@implementation McapAudioPacket
@end
@interface McapStream : NSObject <SCStreamOutput, SCStreamDelegate>
@property (nonatomic) SCStream *stream;
@property (nonatomic) SCStreamConfiguration *configuration;
@property (nonatomic) dispatch_queue_t queue;
@property (nonatomic) NSCondition *condition; // over everything below
@property (nonatomic) NSMutableData *frame; // the newest frame, packed BGRA
@property (nonatomic) int64_t frameTime;
@property (nonatomic) BOOL fresh; // a frame not yet taken
@property (nonatomic) int32_t width, height;
@property (nonatomic) NSString *failure; // the stream stopped by itself
@property (nonatomic) NSMutableArray<McapAudioPacket *> *system, *microphone;
@end
static int64_t host_nanoseconds(CMTime time) {
return (int64_t) (CMTimeGetSeconds(time) * 1e9);
}
@implementation McapStream
- (void)stream:(SCStream *)stream didOutputSampleBuffer:(CMSampleBufferRef)buffer ofType:(SCStreamOutputType)type {
if (!CMSampleBufferIsValid(buffer)) return;
if (type == SCStreamOutputTypeScreen) [self takeFrame:buffer];
else if (type == SCStreamOutputTypeAudio) [self takeAudio:buffer into:self.system];
else if (macos_15()) { if (type == SCStreamOutputTypeMicrophone) [self takeAudio:buffer into:self.microphone]; }
}
- (void)stream:(SCStream *)stream didStopWithError:(NSError *)error {
[self.condition lock];
self.failure = error.localizedDescription ?: @"the screen stopped being recorded";
[self.condition broadcast];
[self.condition unlock];
}
// A complete frame (not an idle one, which repeats the last), packed into self.frame.
- (void)takeFrame:(CMSampleBufferRef)buffer {
CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(buffer, false);
if (attachments && CFArrayGetCount(attachments) > 0) {
NSDictionary *info = (__bridge NSDictionary *) CFArrayGetValueAtIndex(attachments, 0);
NSNumber *status = info[SCStreamFrameInfoStatus];
if (status && status.integerValue != SCFrameStatusComplete) return;
}
CVPixelBufferRef image = CMSampleBufferGetImageBuffer(buffer);
if (!image) return;
CVPixelBufferLockBaseAddress(image, kCVPixelBufferLock_ReadOnly);
const uint8_t *base = CVPixelBufferGetBaseAddress(image);
size_t stride = CVPixelBufferGetBytesPerRow(image);
size_t w = MIN((size_t) self.width, CVPixelBufferGetWidth(image));
size_t h = MIN((size_t) self.height, CVPixelBufferGetHeight(image));
[self.condition lock];
uint8_t *out = self.frame.mutableBytes;
size_t row = (size_t) self.width * 4;
if (base) for (size_t y = 0; y < h; y++) memcpy(out + y * row, base + y * stride, w * 4);
self.frameTime = host_nanoseconds(CMSampleBufferGetPresentationTimeStamp(buffer));
self.fresh = YES;
[self.condition signal];
[self.condition unlock];
CVPixelBufferUnlockBaseAddress(image, kCVPixelBufferLock_ReadOnly);
}
// A sound buffer as interleaved stereo float at AUDIO_RATE, onto the source's queue.
- (void)takeAudio:(CMSampleBufferRef)buffer into:(NSMutableArray<McapAudioPacket *> *)queue {
CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(buffer);
const AudioStreamBasicDescription *asbd = format ? CMAudioFormatDescriptionGetStreamBasicDescription(format) : NULL;
if (!asbd || !(asbd->mFormatFlags & kAudioFormatFlagIsFloat) || asbd->mBitsPerChannel != 32) return;
CMItemCount frames = CMSampleBufferGetNumSamples(buffer);
if (frames <= 0) return;
size_t needed = 0;
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(buffer, &needed, NULL, 0, NULL, NULL, 0, NULL);
AudioBufferList *list = malloc(needed);
CMBlockBufferRef block = NULL;
OSStatus status = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(buffer, NULL, list, needed, NULL, NULL,
kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment, &block);
if (status != noErr) { free(list); return; }
int channels = (int) asbd->mChannelsPerFrame;
BOOL planar = (asbd->mFormatFlags & kAudioFormatFlagIsNonInterleaved) != 0;
// Each frame's left and right: a mono source on both.
float (^sample)(CMItemCount, int) = ^float(CMItemCount i, int c) {
int channel = channels >= 2 ? c : 0;
if (planar) return ((float *) list->mBuffers[channel < (int) list->mNumberBuffers ? channel : 0].mData)[i];
return ((float *) list->mBuffers[0].mData)[i * channels + channel];
};
double ratio = asbd->mSampleRate > 0 ? AUDIO_RATE / asbd->mSampleRate : 1;
CMItemCount out_frames = (CMItemCount) (frames * ratio + 0.5);
NSMutableData *samples = [NSMutableData dataWithLength:(size_t) out_frames * 2 * sizeof(float)];
float *out = samples.mutableBytes;
for (CMItemCount i = 0; i < out_frames; i++) {
// Linearly between the source's frames, when its rate is not AUDIO_RATE.
double at = ratio == 1 ? (double) i : i / ratio;
CMItemCount a = (CMItemCount) at;
CMItemCount b = MIN(a + 1, frames - 1);
float t = (float) (at - (double) a);
for (int c = 0; c < 2; c++) out[i * 2 + c] = sample(MIN(a, frames - 1), c) * (1 - t) + sample(b, c) * t;
}
CFRelease(block);
free(list);
McapAudioPacket *packet = [McapAudioPacket new];
packet.samples = samples;
packet.hostTime = host_nanoseconds(CMSampleBufferGetPresentationTimeStamp(buffer));
[self.condition lock];
[queue addObject:packet];
[self.condition unlock];
}
@end
static void configure_audio(SCStreamConfiguration *configuration, int system_audio, int microphone) {
configuration.capturesAudio = system_audio ? YES : NO;
configuration.sampleRate = AUDIO_RATE;
configuration.channelCount = 2;
configuration.excludesCurrentProcessAudio = YES;
if (macos_15()) configuration.captureMicrophone = microphone ? YES : NO;
}
int mcap_sck_microphone_supported(void) {
if (macos_15()) return 1;
return 0;
}
Mcap_Sck_Stream *mcap_sck_stream_start(uint32_t display_id, double x, double y, double w, double h,
int32_t width, int32_t height, int32_t fps, int show_cursor,
int system_audio, int microphone,
const uint32_t *excluded, int excluded_count,
char *error, int error_size) {
@autoreleasepool {
__block SCShareableContent *content = nil;
__block NSError *content_error = nil;
dispatch_semaphore_t got = dispatch_semaphore_create(0);
[SCShareableContent getShareableContentExcludingDesktopWindows:NO onScreenWindowsOnly:NO
completionHandler:^(SCShareableContent *c, NSError *e) {
content = c;
content_error = e;
dispatch_semaphore_signal(got);
}];
if (dispatch_semaphore_wait(got, dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC)) != 0 || !content) {
set_error(error, error_size, content_error ? content_error.localizedDescription : @"ScreenCaptureKit did not list the screens");
return NULL;
}
SCDisplay *display = nil;
for (SCDisplay *d in content.displays) if (d.displayID == display_id) display = d;
if (!display) { set_error(error, error_size, [NSString stringWithFormat:@"ScreenCaptureKit has no display %u", display_id]); return NULL; }
NSMutableArray<SCWindow *> *leave_out = [NSMutableArray array];
for (SCWindow *window in content.windows)
for (int i = 0; i < excluded_count; i++) if (window.windowID == excluded[i]) [leave_out addObject:window];
SCContentFilter *filter = [[SCContentFilter alloc] initWithDisplay:display excludingWindows:leave_out];
SCStreamConfiguration *configuration = [SCStreamConfiguration new];
configuration.sourceRect = CGRectMake(x, y, w, h);
configuration.width = (size_t) width;
configuration.height = (size_t) height;
configuration.minimumFrameInterval = CMTimeMake(1, fps > 0 ? fps : 30);
configuration.pixelFormat = kCVPixelFormatType_32BGRA;
configuration.colorSpaceName = kCGColorSpaceSRGB;
configuration.showsCursor = show_cursor ? YES : NO;
configuration.queueDepth = 6;
configure_audio(configuration, system_audio, microphone);
McapStream *holder = [McapStream new];
holder.configuration = configuration;
holder.condition = [NSCondition new];
holder.width = width;
holder.height = height;
holder.frame = [NSMutableData dataWithLength:(size_t) width * (size_t) height * 4];
holder.system = [NSMutableArray array];
holder.microphone = [NSMutableArray array];
holder.queue = dispatch_queue_create("mcap.record", DISPATCH_QUEUE_SERIAL);
holder.stream = [[SCStream alloc] initWithFilter:filter configuration:configuration delegate:holder];
NSError *add_error = nil;
BOOL added = [holder.stream addStreamOutput:holder type:SCStreamOutputTypeScreen sampleHandlerQueue:holder.queue error:&add_error];
if (added) added = [holder.stream addStreamOutput:holder type:SCStreamOutputTypeAudio sampleHandlerQueue:holder.queue error:&add_error];
if (added) if (macos_15()) added = [holder.stream addStreamOutput:holder type:SCStreamOutputTypeMicrophone sampleHandlerQueue:holder.queue error:&add_error];
if (!added) { set_error(error, error_size, add_error.localizedDescription ?: @"ScreenCaptureKit would not take the stream's outputs"); return NULL; }
__block NSError *start_error = nil;
dispatch_semaphore_t started = dispatch_semaphore_create(0);
[holder.stream startCaptureWithCompletionHandler:^(NSError *e) { start_error = e; dispatch_semaphore_signal(started); }];
if (dispatch_semaphore_wait(started, dispatch_time(DISPATCH_TIME_NOW, 10 * NSEC_PER_SEC)) != 0) {
set_error(error, error_size, @"ScreenCaptureKit did not start recording in time");
return NULL;
}
if (start_error) { set_error(error, error_size, start_error.localizedDescription); return NULL; }
return (__bridge_retained Mcap_Sck_Stream *) holder;
}
}
int mcap_sck_next_frame(Mcap_Sck_Stream *s, uint8_t *pixels, int64_t *host_time_ns, int timeout_ms, char *error, int error_size) {
McapStream *holder = (__bridge McapStream *) s;
[holder.condition lock];
NSDate *until = [NSDate dateWithTimeIntervalSinceNow:timeout_ms / 1000.0];
while (!holder.fresh && !holder.failure) if (![holder.condition waitUntilDate:until]) break;
int result = 0;
if (holder.failure) {
set_error(error, error_size, holder.failure);
result = -1;
} else if (holder.fresh) {
memcpy(pixels, holder.frame.bytes, holder.frame.length);
*host_time_ns = holder.frameTime;
holder.fresh = NO;
result = 1;
}
[holder.condition unlock];
return result;
}
int mcap_sck_next_audio(Mcap_Sck_Stream *s, int source, float *samples, int max_frames, int64_t *host_time_ns) {
McapStream *holder = (__bridge McapStream *) s;
NSMutableArray<McapAudioPacket *> *queue = source == MCAP_SCK_MICROPHONE ? holder.microphone : holder.system;
[holder.condition lock];
int frames = 0;
if (queue.count) {
McapAudioPacket *packet = queue[0];
int available = (int) (packet.samples.length / (2 * sizeof(float)));
frames = MIN(available, max_frames);
memcpy(samples, packet.samples.bytes, (size_t) frames * 2 * sizeof(float));
*host_time_ns = packet.hostTime;
if (frames == available) [queue removeObjectAtIndex:0];
else { // the rest stays, its time moved on
packet.samples = [packet.samples subdataWithRange:NSMakeRange((size_t) frames * 2 * sizeof(float), packet.samples.length - (size_t) frames * 2 * sizeof(float))];
packet.hostTime += (int64_t) frames * 1000000000LL / AUDIO_RATE;
}
}
[holder.condition unlock];
return frames;
}
int mcap_sck_stream_set_audio(Mcap_Sck_Stream *s, int system_audio, int microphone, char *error, int error_size) {
@autoreleasepool {
McapStream *holder = (__bridge McapStream *) s;
if (microphone && !mcap_sck_microphone_supported()) { set_error(error, error_size, @"the microphone needs macOS 15 or later"); return 0; }
configure_audio(holder.configuration, system_audio, microphone);
__block NSError *update_error = nil;
dispatch_semaphore_t done = dispatch_semaphore_create(0);
[holder.stream updateConfiguration:holder.configuration completionHandler:^(NSError *e) { update_error = e; dispatch_semaphore_signal(done); }];
if (dispatch_semaphore_wait(done, dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC)) != 0) {
set_error(error, error_size, @"ScreenCaptureKit did not change the sound in time");
return 0;
}
if (update_error) { set_error(error, error_size, update_error.localizedDescription); return 0; }
// A source turned off keeps nothing queued.
[holder.condition lock];
if (!system_audio) [holder.system removeAllObjects];
if (!microphone) [holder.microphone removeAllObjects];
[holder.condition unlock];
return 1;
}
}
void mcap_sck_stream_stop(Mcap_Sck_Stream *s) {
if (!s) return;
McapStream *holder = (__bridge_transfer McapStream *) s;
@autoreleasepool {
dispatch_semaphore_t stopped = dispatch_semaphore_create(0);
[holder.stream stopCaptureWithCompletionHandler:^(NSError *e) { dispatch_semaphore_signal(stopped); }];
dispatch_semaphore_wait(stopped, dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC));
[holder.stream removeStreamOutput:holder type:SCStreamOutputTypeScreen error:nil];
[holder.stream removeStreamOutput:holder type:SCStreamOutputTypeAudio error:nil];
if (macos_15()) [holder.stream removeStreamOutput:holder type:SCStreamOutputTypeMicrophone error:nil];
dispatch_sync(holder.queue, ^{}); // the last sample handler is done
}
}

772
modules/Screen_Record/module.jai
View File

@ -1,33 +1,14 @@
// Screen_Record: a region of one screen recorded to H.264 in an MP4, for the mcap app's screen
// recording and tools/record_probe.jai (video-capture-plan.html). Windows only.
//
// Two threads. The capture thread takes the screen with DXGI Desktop Duplication (dxgi_source.jai),
// or with Windows.Graphics.Capture (wgc_source.jai) where duplication cannot run, each on a D3D11
// device of its own; copies the region on the GPU; reads it back through a ring of staging textures,
// so Map never waits on the GPU (capture_gpu.jai); and, for duplication, draws the cursor in
// (cursor.jai). The encoder thread hands the frames to FFmpeg (encoder.jai). Between them is a pool of
// RECORD_BUFFERS frames; when the encoder falls RECORD_QUEUE frames behind, the oldest waiting is
// dropped and counted.
//
// A frame is taken at most fps times a second, and only when the screen or the cursor changed, with
// one a second while nothing does, so players can seek. Each carries the time it was taken, less the
// time spent paused (record_pause), so a pause leaves no gap in the video.
//
// Sound (system_audio, microphone) has a thread of its own (audio_source.jai's WASAPI): each packet is
// put on the same timeline as the frames, by when it was heard, less the same pauses, into one mix,
// and an AAC frame of it is encoded once every source has had its say up to there. A source that sends
// nothing (the computer playing nothing) is silence. Either source can be turned on and off while
// recording (record_set_audio); with audio_track, the MP4 has a sound track even when no source is on
// at the start, so one can be.
// recording and tools/record_probe.jai (video-capture-plan.html, macos-plan phase 7). On Windows with
// Desktop Duplication or Windows.Graphics.Capture and WASAPI (windows.jai); on macOS with
// ScreenCaptureKit, its frames and sound alike, through modules/Screen_Record/macos/capture.m
// (macos.jai). Either way to FFmpeg (encoder.jai), the sound mixed onto the video's timeline
// (audio_mix.jai), and made a regular MP4 at the end (remux.jai).
//
// recorder: Recorder;
// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0});
// ok, error := record_start(*recorder, .{path = "out.mp4", ...});
// ...
// ok, error = record_stop(*recorder);
//
// With capture .AUTO (the default), Desktop Duplication is tried first and Windows.Graphics.Capture
// takes over when it cannot run: on a laptop with two GPUs, on a rotated screen, or with too many
// duplications of the screen running. fall_back is only returned for .DXGI, which does not fall back.
RECORD_MIN_WIDTH :: 240; // a smaller region is refused
RECORD_MIN_HEIGHT :: 180;
@ -40,750 +21,15 @@ record_size_allowed :: (width: s64, height: s64) -> bool {
return width >= RECORD_MIN_WIDTH && height >= RECORD_MIN_HEIGHT;
}
Record_Screen :: struct {
adapter_index, output_index: u32;
adapter: string; // "NVIDIA GeForce RTX 5080 Laptop GPU"
name: string; // "\\.\DISPLAY1"
desktop: RECT; // on the virtual desktop
rotation: DXGI_MODE_ROTATION;
monitor: HMONITOR;
}
Record_Capture :: enum u8 {
AUTO; // Desktop Duplication, else Windows.Graphics.Capture
DXGI; // Desktop Duplication only
WGC; // Windows.Graphics.Capture only
}
// For record-probe: paths a machine may not take by itself.
Record_Testing :: enum_flags u8 {
DXGI_UNSUPPORTED; // Desktop Duplication fails as on a laptop with two GPUs (DXGI_ERROR_UNSUPPORTED)
WGC_POLL; // Windows.Graphics.Capture polled, as when it will not take the FrameArrived handler
KEEP_BORDER; // Windows.Graphics.Capture's yellow border left on
LOSE_DUPLICATION; // Desktop Duplication lost every 3 s, as a mode change or the secure desktop loses it
}
Record_Options :: struct {
path: string; // the .mp4 written
screen: s32 = -1; // into record_list_screens(); -1: the one holding region
region: RECT; // on the virtual desktop; empty: the whole screen
fps: s32 = 30;
cursor := true;
encoder: string; // "" tries RECORD_ENCODERS in turn
capture: Record_Capture;
legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe
testing: Record_Testing;
system_audio: bool; // what the computer plays (the default output device)
microphone: bool; // the default input device; both at the start: record_set_audio turns them on and off
audio_track: bool; // a sound track whatever is on at the start, so a source can be turned on later
}
Record_Stats :: struct {
source: string; // "DXGI DuplicateOutput1 on <adapter>", "Windows.Graphics.Capture on <adapter>, border off"
fell_back_from: string; // why Desktop Duplication could not run, when it fell back
encoder: string;
width, height: s32;
captured: s64; // new images of the screen
pointer_moves: s64; // cursor changes with no new image (Desktop Duplication)
emitted: s64; // frames sent to be encoded
repeats: s64; // of them, the once-a-second repeat of an unchanged screen
written: s64; // encoded
dropped: s64; // the encoder was too far behind
lost: s64; // times the duplication was lost and made again
paused_seconds: float64; // left out of the video
audio: string; // the sound recorded: "Speakers (Realtek(R) Audio); Microphone (USB Audio)", or ""
audio_frames: s64; // AAC frames encoded
audio_errors: string; // sources that could not be opened, or were lost
copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks
bytes: s64; // encoded
seconds: float64; // from start to stop
}
Recorder :: struct {
options: Record_Options;
screen: Record_Screen;
local: RECT; // the region, in the screen's pixels
dxgi: Dxgi_Source; // the capture thread's: one of these is open
wgc: Wgc_Source;
using_wgc: bool;
encoder: Encoder;
width, height: s32;
frequency: s64; // QueryPerformanceCounter's
start_ticks: s64;
capture_thread, encoder_thread: Thread;
capture_started, encoder_started, audio_started: bool;
audio_thread: Thread;
audio_opened: Semaphore; // the audio thread has opened its sources
audio_go: Semaphore; // the encoder is open: the audio thread may write
audio_sources: [2] Audio_Source;
audio_open: bool; // any source opened, or audio_track: the MP4 has sound
audio_on: [2] bool; // each source is wanted (by Audio_Source_Kind): record_set_audio; cleared when it cannot be opened
audio_failed: [2] bool; // it could not be opened after being turned on: record_audio_failed says once
zero_ticks: s64; // the recording's time 0, for frames and sound alike
paused_ticks: s64; // time paused so far, for the capture and audio threads; set by record_pause
pause_began: s64;
running: bool;
stopping: bool; // set by record_stop; the capture thread ends, then the encoder
paused: bool; // set by record_pause; the capture thread takes nothing while it is
capture_ended: bool;
opened: Semaphore; // the capture thread has opened its source, or failed to
open_ok, open_fall_back: bool;
open_error: string;
mutex: Mutex;
ready: Semaphore; // signalled for each frame queued, and once when capture ends
frames: [RECORD_BUFFERS] Record_Frame;
free: [RECORD_BUFFERS] s32;
free_count: s32;
queue: [RECORD_BUFFERS] s32; // oldest first
queue_count: s32;
last_frame: [] u8; // the last frame queued, BGRA, for record_last_frame
stats: Record_Stats;
error: string; // why it stopped by itself; heap
}
Record_Frame :: struct {
pixels: [] u8; // BGRA, rows packed
ticks: s64;
}
// Opens the screen, the encoder and the file, and starts recording.
record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool {
r.* = .{};
r.options = options;
r.options.path = copy_string(options.path);
r.options.encoder = copy_string(options.encoder);
if r.options.fps <= 0 r.options.fps = 30;
screens := record_list_screens();
if !screens.count return false, "no screen to record", false;
screen_index := options.screen;
region := options.region;
has_region := region.right > region.left && region.bottom > region.top;
if screen_index < 0 {
if !has_region screen_index = 0;
else for screens if rect_inside(region, it.desktop) { screen_index = xx it_index; break; }
if screen_index < 0 return false, "the region is not on one screen", false;
}
if screen_index >= screens.count return false, tprint("there is no screen %", screen_index), false;
r.screen = screens[screen_index];
r.screen.adapter = copy_string(r.screen.adapter);
r.screen.name = copy_string(r.screen.name);
if !has_region region = r.screen.desktop;
if !rect_inside(region, r.screen.desktop) return false, "the region is not on that screen", false;
// The minimum first, then even both ways, in the screen's pixels: H.264's 4:2:0 colour needs it.
// RECORD_MIN_WIDTH and _HEIGHT are even, so what passes stays at least that size.
#assert RECORD_MIN_WIDTH % 2 == 0 && RECORD_MIN_HEIGHT % 2 == 0;
if !record_size_allowed(region.right - region.left, region.bottom - region.top)
return false, tprint("% x % is smaller than % x %", region.right - region.left, region.bottom - region.top, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false;
d := r.screen.desktop;
r.local = .{region.left - d.left, region.top - d.top, region.right - d.left, region.bottom - d.top};
r.local.right = r.local.left + ((r.local.right - r.local.left) & cast(s32) ~1);
r.local.bottom = r.local.top + ((r.local.bottom - r.local.top) & cast(s32) ~1);
r.width = r.local.right - r.local.left;
r.height = r.local.bottom - r.local.top;
r.stats.width, r.stats.height = r.width, r.height;
init(*r.mutex, "record");
init(*r.ready);
init(*r.opened);
for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false);
for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it;
r.free_count = RECORD_BUFFERS;
r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8);
frequency: LARGE_INTEGER;
QueryPerformanceFrequency(*frequency);
r.frequency = frequency.QuadPart;
r.start_ticks = ticks_now();
r.zero_ticks = r.start_ticks;
// The source is opened on the capture thread, which uses it: WinRT's objects belong to the
// apartment they are made in.
thread_init(*r.capture_thread, record_capture_proc, name = "record capture");
r.capture_started = true;
thread_start(*r.capture_thread, r);
wait_for(*r.opened);
if !r.open_ok {
error, fall_back := copy_string(r.open_error,, temp), r.open_fall_back;
record_free(r);
return false, error, fall_back;
}
// The sound's sources are opened on their own thread (COM, for WASAPI), before the encoder, which
// has a sound stream only when one opened.
r.audio_on = .[r.options.system_audio, r.options.microphone];
if r.options.system_audio || r.options.microphone || r.options.audio_track {
init(*r.audio_opened);
init(*r.audio_go);
thread_init(*r.audio_thread, record_audio_proc, name = "record audio");
r.audio_started = true;
thread_start(*r.audio_thread, r);
wait_for(*r.audio_opened);
}
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder, audio = r.audio_open);
if tried log("[record] Encoders that did not open: %", tried);
if !ok {
error = copy_string(error,, temp);
record_free(r);
return false, error, false;
}
r.stats.encoder = copy_string(r.encoder.name);
if r.audio_started signal(*r.audio_go);
thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder");
r.encoder_started = true;
r.running = true;
thread_start(*r.encoder_thread, r);
return true, "", false;
}
// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an
// error, or the file could not be finished; what was recorded up to then is in the file all the same.
record_stop :: (r: *Recorder) -> ok: bool, error: string {
if !r.running return false, "not recording";
atomic_swap(*r.stopping, true);
thread_is_done(*r.capture_thread, -1);
thread_is_done(*r.encoder_thread, -1);
if r.audio_started thread_is_done(*r.audio_thread, -1);
r.running = false;
closed, close_error := encoder_close(*r.encoder);
r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency;
r.stats.bytes = r.encoder.bytes;
error := ifx r.error then copy_string(r.error,, temp) else "";
record_free(r);
if error return false, error;
if !closed return false, close_error;
return true, "";
}
// Pauses or resumes. While paused nothing is taken, and the time is left out of the video, sound and
// all: the time paused is kept here, once, for the capture and audio threads both.
record_pause :: (r: *Recorder, pause: bool) {
if !r.running || atomic_read(*r.paused) == pause return;
now := ticks_now();
if pause {
atomic_write(*r.pause_began, now);
atomic_swap(*r.paused, true);
} else {
atomic_add(*r.paused_ticks, now - atomic_read(*r.pause_began)); // before paused clears, so no thread sees it unpaused with the old total
atomic_swap(*r.paused, false);
}
}
// Turns a source of sound on or off while recording. It is opened only while on, so Windows shows the
// microphone in use only then. False when the recording has no sound track to put it in.
record_set_audio :: (r: *Recorder, kind: Audio_Source_Kind, on: bool) -> bool {
if !r.running || !r.audio_open return false;
atomic_swap(*r.audio_on[kind], on);
return true;
}
record_audio_on :: (r: *Recorder, kind: Audio_Source_Kind) -> bool {
return r.running && atomic_read(*r.audio_on[kind]);
}
// Whether a source could not be opened when turned on (it is off again then), and why; once.
record_audio_failed :: (r: *Recorder, kind: Audio_Source_Kind) -> failed: bool, error: string {
if !r.running || !atomic_read(*r.audio_failed[kind]) return false, "";
atomic_swap(*r.audio_failed[kind], false);
lock(*r.mutex);
defer unlock(*r.mutex);
return true, copy_string(r.stats.audio_errors,, temp);
}
// Ticks on the recording's timeline: time 0 at its start, less the time paused.
record_timeline :: (r: *Recorder, ticks: s64) -> s64 {
return ticks - r.zero_ticks - atomic_read(*r.paused_ticks);
}
record_paused :: (r: *Recorder) -> bool {
return r.running && atomic_read(*r.paused);
}
// As they are now; strings are the recorder's.
record_stats :: (r: *Recorder) -> Record_Stats {
if !r.running return r.stats;
lock(*r.mutex);
defer unlock(*r.mutex);
return r.stats;
}
// The last frame queued, BGRA (temp), cursor and all.
record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 {
if r.running lock(*r.mutex);
pixels := array_copy(r.last_frame,, temp);
if r.running unlock(*r.mutex);
return pixels, r.width, r.height;
}
// Whether recording stopped by itself (the screen went away, the encoder failed); why, then.
record_failed :: (r: *Recorder) -> bool, string {
lock(*r.mutex);
defer unlock(*r.mutex);
return r.error.count > 0, r.error;
}
#scope_file
Staged :: struct { pending: bool; ticks: s64; pointer: POINT; pointer_visible: bool; }
// Stops the capture thread if it runs, and frees what record_start made. The stats' strings are kept.
record_free :: (r: *Recorder) {
atomic_swap(*r.stopping, true);
if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; }
if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; }
if r.audio_started {
signal(*r.audio_go); // in case it waits for an encoder that did not open
thread_is_done(*r.audio_thread, -1);
thread_deinit(*r.audio_thread);
destroy(*r.audio_opened);
destroy(*r.audio_go);
r.audio_started = false;
}
if r.encoder.format encoder_close(*r.encoder);
for * r.frames array_free(it.pixels);
array_free(r.last_frame);
r.last_frame = .[];
destroy(*r.opened);
destroy(*r.ready);
destroy(*r.mutex);
free(r.open_error);
free(r.error);
r.open_error, r.error = "", "";
}
// Desktop Duplication, unless asked for Windows.Graphics.Capture, which takes over when duplication
// cannot run here and the options allow.
record_open_source :: (r: *Recorder) -> bool {
if r.options.capture != .WGC {
ok, error, fall_back := dxgi_open(*r.dxgi, r.screen, r.local, r.options.legacy_duplicate);
if ok && r.options.testing & .DXGI_UNSUPPORTED {
ok, error, fall_back = false, "Desktop Duplication (testing): 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", true;
}
if ok {
r.stats.source = sprint("DXGI % on %", r.dxgi.method, r.screen.adapter);
return true;
}
dxgi_close(*r.dxgi);
if r.options.capture == .DXGI || !fall_back {
r.open_error, r.open_fall_back = copy_string(error), fall_back;
return false;
}
log("[record] Desktop Duplication cannot run here (%); Windows.Graphics.Capture instead.", error);
r.stats.fell_back_from = copy_string(error);
}
ok, error := wgc_open(*r.wgc, r.screen, r.local, r.options.cursor, r.options.testing);
if !ok {
wgc_close(*r.wgc);
r.open_error = copy_string(error);
return false;
}
r.using_wgc = true;
r.stats.source = sprint("Windows.Graphics.Capture on %, %%", r.screen.adapter, r.wgc.border, ifx r.wgc.polling then ", polled" else "");
return true;
}
record_capture_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
// Aware of each monitor's DPI, whatever the process is: Desktop Duplication refuses a thread that is
// not (DXGI_ERROR_UNSUPPORTED), and the mcap app's ImGui makes only its main thread aware.
gpu_thread_dpi_aware();
wgc_thread_start();
defer wgc_thread_end();
r.open_ok = record_open_source(r);
signal(*r.opened);
if !r.open_ok return 0;
gpu: *Capture_Gpu = ifx r.using_wgc then *r.wgc.gpu else *r.dxgi.gpu;
defer if r.using_wgc wgc_close(*r.wgc); else dxgi_close(*r.dxgi);
// A change is sent as soon as it comes once its turn is due: turns are a frame's time apart on a
// schedule that moves on a frame at a time, so the rate never passes fps, and a tenth of a frame
// early counts as on time, so a 60 Hz screen's frames, which jitter, all make a 60 fps cap.
period := r.frequency / r.options.fps;
early := period / 10;
turn := ticks_now();
last_emit: s64 = -r.frequency;
have_image, changed := false, false;
staged: [RECORD_STAGING] Staged;
emits: s64;
was_paused := false;
last_lost := ticks_now(); // for Record_Testing.LOSE_DUPLICATION
while !atomic_read(*r.stopping) {
now := ticks_now();
due := ifx changed then turn - early else last_emit + r.frequency; // the change, or the repeat
wait := cast(u32) clamp((due - now) * 1000 / r.frequency, 0, 100);
next: Capture_Next;
error: string;
if r.using_wgc next, error = wgc_next(*r.wgc, wait);
else next, error = dxgi_next(*r.dxgi, wait);
if r.options.testing & .LOSE_DUPLICATION && !r.using_wgc && ticks_now() - last_lost >= 3 * r.frequency {
next, last_lost = Capture_Next.LOST, ticks_now();
}
if next == {
case .FAILED;
record_fail(r, error);
break;
case .LOST;
// A mode change, the secure desktop, a game going fullscreen: duplicated again, and on.
lock(*r.mutex); r.stats.lost += 1; unlock(*r.mutex);
while !atomic_read(*r.stopping) {
hr := dxgi_duplicate(*r.dxgi);
if SUCCEEDED(hr) break;
if !hresult_is(hr, DXGI_ERROR_ACCESS_LOST) && !hresult_is(hr, 0x80070005) && !hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE) {
record_fail(r, hresult_text("the screen could not be duplicated again", hr));
break;
}
sleep_milliseconds(250);
}
if atomic_read(*r.stopping) || r.error break;
continue;
case .IMAGE; #through;
case .IMAGE_AND_POINTER;
have_image, changed = true, true;
lock(*r.mutex);
r.stats.captured += 1;
r.stats.copy_ticks = gpu.copy_ticks;
unlock(*r.mutex);
case .POINTER;
if r.options.cursor && have_image {
changed = true;
lock(*r.mutex); r.stats.pointer_moves += 1; unlock(*r.mutex);
}
}
// Paused: the source is still drained, so it never stalls, but nothing is sent. On resuming, the
// screen as it is now is sent at once.
now = ticks_now();
if atomic_read(*r.paused) {
was_paused = true;
lock(*r.mutex); r.stats.paused_seconds = cast(float64) (atomic_read(*r.paused_ticks) + now - atomic_read(*r.pause_began)) / r.frequency; unlock(*r.mutex);
continue;
}
if was_paused {
was_paused = false;
changed = true;
turn = now;
lock(*r.mutex); r.stats.paused_seconds = cast(float64) atomic_read(*r.paused_ticks) / r.frequency; unlock(*r.mutex);
}
if !have_image continue;
repeat := !changed && now - last_emit >= r.frequency;
if !(changed && now >= turn - early) && !repeat continue;
turn = max(turn + period, now - period); // after a quiet spell, no burst to catch up
// Copied to staging now; read back two frames on, when the GPU is done with it.
slot := emits % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
ID3D11DeviceContext_CopyResource(gpu.device_context, gpu.staging[slot], gpu.latest);
staged[slot] = .{pending = true, ticks = record_timeline(r, now)};
if !r.using_wgc staged[slot].pointer, staged[slot].pointer_visible = r.dxgi.pointer_position, r.dxgi.pointer_visible;
emits += 1;
last_emit = now;
changed = false;
lock(*r.mutex);
r.stats.emitted += 1;
if repeat r.stats.repeats += 1;
unlock(*r.mutex);
}
// What is still staged, oldest first.
for 0..RECORD_STAGING - 1 {
slot := (emits + it) % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
}
atomic_swap(*r.capture_ended, true);
signal(*r.ready);
return 0;
}
// Copies a staging texture into a free frame, draws the cursor (Desktop Duplication leaves it out)
// and queues it.
record_read_back :: (r: *Recorder, gpu: *Capture_Gpu, slot: s64, staged: Staged) {
t := ticks_now();
lock(*r.mutex);
if !r.free_count { // every frame is waiting: drop the oldest
r.free[r.free_count] = r.queue[0];
r.free_count += 1;
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
r.stats.dropped += 1;
}
r.free_count -= 1;
index := r.free[r.free_count];
unlock(*r.mutex);
frame := *r.frames[index];
mapped: D3D11_MAPPED_SUBRESOURCE;
hr := ID3D11DeviceContext_Map(gpu.device_context, gpu.staging[slot], 0, .READ, 0, *mapped);
if FAILED(hr) {
lock(*r.mutex); r.free[r.free_count] = index; r.free_count += 1; unlock(*r.mutex);
record_fail(r, hresult_text("reading the frame back", hr));
return;
}
row := cast(s64) r.width * 4;
for y: 0..r.height - 1 memcpy(frame.pixels.data + y * row, cast(*u8) mapped.pData + cast(s64) y * mapped.RowPitch, row);
ID3D11DeviceContext_Unmap(gpu.device_context, gpu.staging[slot], 0);
if r.options.cursor && staged.pointer_visible {
cursor_draw(frame.pixels.data, r.width, r.height, *r.dxgi.shape,
staged.pointer.x - cast(s32) gpu.box.left, staged.pointer.y - cast(s32) gpu.box.top);
}
frame.ticks = staged.ticks;
lock(*r.mutex);
r.queue[r.queue_count] = index;
r.queue_count += 1;
memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count);
r.stats.readback_ticks += ticks_now() - t;
unlock(*r.mutex);
signal(*r.ready);
}
record_encoder_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
while true {
wait_for(*r.ready, 100);
lock(*r.mutex);
index: s32 = -1;
if r.queue_count {
index = r.queue[0];
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
}
unlock(*r.mutex);
if index < 0 {
if atomic_read(*r.capture_ended) break;
continue;
}
frame := *r.frames[index];
pts := max(frame.ticks, 0) * 90000 / r.frequency; // frame.ticks is on the timeline already
t := ticks_now();
ok, error := encoder_write(*r.encoder, frame.pixels.data, pts);
lock(*r.mutex);
r.free[r.free_count] = index;
r.free_count += 1;
r.stats.encode_ticks += ticks_now() - t;
if ok r.stats.written += 1;
r.stats.bytes = r.encoder.bytes;
unlock(*r.mutex);
if !ok {
record_fail(r, error);
// Frames still come until the capture thread sees it; they are let go.
while !atomic_read(*r.capture_ended) {
wait_for(*r.ready, 100);
lock(*r.mutex);
for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; }
r.queue_count = 0;
unlock(*r.mutex);
}
break;
}
}
return 0;
}
// The audio thread: opens the sources asked for, waits for the encoder, then every 10 ms takes their
// packets into the mix and encodes what every source has covered.
record_audio_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
audio_thread_start();
defer audio_thread_end();
kinds: [2] Audio_Source_Kind = .[.SYSTEM, .MICROPHONE];
wanted: [2] bool = .[r.options.system_audio, r.options.microphone];
names, errors: [..] string; // every source heard, and every failure, for the stats
names.allocator, errors.allocator = context.default_allocator, context.default_allocator;
defer { for names free(it); array_free(names); for errors free(it); array_free(errors); }
note :: (list: *[..] string, text: string, field: *string) #expand {
for list.* if it == text return;
array_add(list, copy_string(text));
lock(*`r.mutex);
free(field.*);
field.* = join(..list.*, "; ");
unlock(*`r.mutex);
}
for 0..1 {
if !wanted[it] continue;
s := *r.audio_sources[it];
ok, error := audio_open(s, kinds[it]);
what := ifx kinds[it] == .SYSTEM then "system sound" else "microphone";
if ok { r.audio_open = true; note(*names, s.name, *r.stats.audio); }
else { audio_close(s); note(*errors, tprint("%: %", what, error), *r.stats.audio_errors); log_error("[record] No %: %", what, error); }
}
for r.audio_sources if !it.open atomic_swap(*r.audio_on[it_index], false);
if r.options.audio_track r.audio_open = true;
signal(*r.audio_opened);
wait_for(*r.audio_go);
if !r.audio_open || !r.encoder.audio || atomic_read(*r.stopping) {
for * r.audio_sources audio_free(it);
return 0;
}
mix: Audio_Mix;
mix.frame_size = r.encoder.audio_frame_size;
defer array_free(mix.samples);
while true {
stopping := atomic_read(*r.stopping);
// Each source, turned on or off from the toolbar: opened only while on.
for * source, index: r.audio_sources {
on := atomic_read(*r.audio_on[index]) && !stopping;
what := ifx kinds[index] == .SYSTEM then "System sound" else "Microphone";
if on && !source.open && !wanted[index] {
ok, error := audio_open(source, kinds[index]);
if ok {
log("[record] % on: %.", what, source.name);
note(*names, source.name, *r.stats.audio);
mix.started[index] = false;
} else {
audio_close(source);
log_error("[record] % could not be opened: %", what, error);
note(*errors, tprint("%: %", to_lower_copy(what,, temp), error), *r.stats.audio_errors);
atomic_swap(*r.audio_on[index], false);
atomic_swap(*r.audio_failed[index], true);
}
} else if !on && source.open {
log("[record] % off.", what);
audio_close(source);
source.lost_at = 0;
}
wanted[index] = source.open || (wanted[index] && on);
}
for * source, index: r.audio_sources {
if !wanted[index] continue;
if !source.open { // lost: opened again a second on, from the default device then
if source.lost_at && ticks_now() - source.lost_at >= r.frequency && !stopping {
if audio_open(source, kinds[index]) log("[record] % is back: %.", ifx kinds[index] == .SYSTEM then "System sound" else "The microphone", source.name);
else { audio_close(source); source.lost_at = ticks_now(); }
}
continue;
}
packets, lost := audio_take(source, r.frequency);
for packets if !atomic_read(*r.paused) audio_mix_add(*mix, r, index, it);
if lost {
log_error("[record] % was lost (unplugged, or the default device changed); trying again.", source.name);
audio_close(source);
source.lost_at = ticks_now();
}
}
ok, error := audio_mix_encode(*mix, r, final = stopping);
if !ok { record_fail(r, error); break; }
if stopping break;
reset_temporary_storage();
sleep_milliseconds(10);
}
for * r.audio_sources audio_free(it);
return 0;
}
// The sound so far, from its base: one AAC frame's worth is encoded at a time.
Audio_Mix :: struct {
samples: [..] float; // interleaved stereo, from base on
base: s64; // the frame (in samples, on the timeline) samples[0] is
next: [2] s64; // each source's next frame, so packets that follow on are not moved by jitter
started: [2] bool;
frame_size: s32;
}
AUDIO_SLACK :: AUDIO_RATE / 5; // 200 ms: how late a source's packets may come and still be in the mix
AUDIO_FOLLOW_ON :: AUDIO_RATE / 33; // 30 ms: a packet that close to where its source left off follows on
// A packet into the mix at its place on the timeline, added to what the other source put there.
audio_mix_add :: (mix: *Audio_Mix, r: *Recorder, source: s64, packet: Audio_Packet) {
at := record_timeline(r, packet.ticks) * AUDIO_RATE / r.frequency;
if mix.started[source] && abs(at - mix.next[source]) <= AUDIO_FOLLOW_ON at = mix.next[source];
mix.started[source] = true;
mix.next[source] = at + packet.frames;
end := (at + packet.frames - mix.base) * AUDIO_CHANNELS;
if end > mix.samples.count {
old := mix.samples.count;
array_resize(*mix.samples, end, initialize = false);
for old..end - 1 mix.samples[it] = 0;
}
if packet.silent || !packet.samples.count return;
for frame: max(at, mix.base)..at + packet.frames - 1 {
into := (frame - mix.base) * AUDIO_CHANNELS;
from := (frame - at) * AUDIO_CHANNELS;
mix.samples[into] += packet.samples[from];
mix.samples[into + 1] += packet.samples[from + 1];
}
}
// Encodes the frames every open source has covered: up to where each has sent, or, for one that has
// sent nothing lately (silence), to AUDIO_SLACK ago. final: everything sent.
audio_mix_encode :: (mix: *Audio_Mix, r: *Recorder, final: bool) -> ok: bool, error: string {
// Paused, the timeline stands still where the pause began.
now := ifx atomic_read(*r.paused) then atomic_read(*r.pause_began) else ticks_now();
now_frame := record_timeline(r, now) * AUDIO_RATE / r.frequency;
ready := now_frame - AUDIO_SLACK;
for source, index: r.audio_sources {
if !source.open continue;
covered := max(mix.next[index], now_frame - AUDIO_SLACK);
ready = min(ready, covered);
}
if final { ready = mix.base; for mix.next ready = max(ready, it); }
size := mix.frame_size;
frame := NewArray(size * AUDIO_CHANNELS, float, initialized = false,, temp);
done: s64;
while mix.base + size <= ready {
for i: 0..frame.count - 1 {
at := done * AUDIO_CHANNELS + i;
value := ifx at < mix.samples.count then mix.samples[at] else 0.0;
frame[i] = clamp(value, -1.0, 1.0);
}
ok, error := encoder_write_audio(*r.encoder, frame.data, mix.base);
if !ok return false, error;
mix.base += size;
done += size;
lock(*r.mutex); r.stats.audio_frames += 1; unlock(*r.mutex);
}
// What was encoded leaves the front.
gone := min(done * AUDIO_CHANNELS, mix.samples.count);
if gone {
memcpy(mix.samples.data, mix.samples.data + gone, (mix.samples.count - gone) * size_of(float));
mix.samples.count -= gone;
}
return true, "";
}
// Ends the recording from one of its threads; the first reason is kept.
record_fail :: (r: *Recorder, error: string) {
log_error("[record] %", error);
lock(*r.mutex);
if !r.error r.error = copy_string(error);
unlock(*r.mutex);
atomic_swap(*r.stopping, true);
}
rect_inside :: (inner: RECT, outer: RECT) -> bool {
return inner.left >= outer.left && inner.top >= outer.top && inner.right <= outer.right && inner.bottom <= outer.bottom;
}
#scope_export
#load "capture_gpu.jai";
#load "dxgi_source.jai";
#load "wgc_source.jai";
#if OS == .WINDOWS #load "windows.jai";
#if OS == .MACOS #load "macos.jai";
#load "audio_mix.jai";
#load "encoder.jai";
#load "remux.jai";
#load "audio_source.jai";
#load "cursor.jai";
#import "Basic";
#import "String";
#import "Atomics";
#import "Math";
#import "Thread";
#import "Windows";
#import "Windows_Utf8";
#import "dxgi";
#import "d3d11";
#import "ffmpeg";

701
modules/Screen_Record/windows.jai Normal file
View File

@ -0,0 +1,701 @@
// The Windows recorder (module.jai has what is shared): moved here from module.jai unchanged.
//
// A region of one screen recorded to H.264 in an MP4, for the mcap app's screen
// recording and tools/record_probe.jai (video-capture-plan.html). Windows only.
//
// Two threads. The capture thread takes the screen with DXGI Desktop Duplication (dxgi_source.jai),
// or with Windows.Graphics.Capture (wgc_source.jai) where duplication cannot run, each on a D3D11
// device of its own; copies the region on the GPU; reads it back through a ring of staging textures,
// so Map never waits on the GPU (capture_gpu.jai); and, for duplication, draws the cursor in
// (cursor.jai). The encoder thread hands the frames to FFmpeg (encoder.jai). Between them is a pool of
// RECORD_BUFFERS frames; when the encoder falls RECORD_QUEUE frames behind, the oldest waiting is
// dropped and counted.
//
// A frame is taken at most fps times a second, and only when the screen or the cursor changed, with
// one a second while nothing does, so players can seek. Each carries the time it was taken, less the
// time spent paused (record_pause), so a pause leaves no gap in the video.
//
// Sound (system_audio, microphone) has a thread of its own (audio_source.jai's WASAPI): each packet is
// put on the same timeline as the frames, by when it was heard, less the same pauses, into one mix,
// and an AAC frame of it is encoded once every source has had its say up to there. A source that sends
// nothing (the computer playing nothing) is silence. Either source can be turned on and off while
// recording (record_set_audio); with audio_track, the MP4 has a sound track even when no source is on
// at the start, so one can be.
//
// recorder: Recorder;
// ok, error, fall_back := record_start(*recorder, .{path = "out.mp4", screen = 0});
// ...
// ok, error = record_stop(*recorder);
//
// With capture .AUTO (the default), Desktop Duplication is tried first and Windows.Graphics.Capture
// takes over when it cannot run: on a laptop with two GPUs, on a rotated screen, or with too many
// duplications of the screen running. fall_back is only returned for .DXGI, which does not fall back.
Record_Screen :: struct {
adapter_index, output_index: u32;
adapter: string; // "NVIDIA GeForce RTX 5080 Laptop GPU"
name: string; // "\\.\DISPLAY1"
desktop: RECT; // on the virtual desktop
rotation: DXGI_MODE_ROTATION;
monitor: HMONITOR;
}
Record_Capture :: enum u8 {
AUTO; // Desktop Duplication, else Windows.Graphics.Capture
DXGI; // Desktop Duplication only
WGC; // Windows.Graphics.Capture only
}
// For record-probe: paths a machine may not take by itself.
Record_Testing :: enum_flags u8 {
DXGI_UNSUPPORTED; // Desktop Duplication fails as on a laptop with two GPUs (DXGI_ERROR_UNSUPPORTED)
WGC_POLL; // Windows.Graphics.Capture polled, as when it will not take the FrameArrived handler
KEEP_BORDER; // Windows.Graphics.Capture's yellow border left on
LOSE_DUPLICATION; // Desktop Duplication lost every 3 s, as a mode change or the secure desktop loses it
}
Record_Options :: struct {
path: string; // the .mp4 written
screen: s32 = -1; // into record_list_screens(); -1: the one holding region
region: RECT; // on the virtual desktop; empty: the whole screen
fps: s32 = 30;
cursor := true;
encoder: string; // "" tries RECORD_ENCODERS in turn
capture: Record_Capture;
legacy_duplicate: bool; // the plain DuplicateOutput, not DuplicateOutput1: for record-probe
testing: Record_Testing;
system_audio: bool; // what the computer plays (the default output device)
microphone: bool; // the default input device; both at the start: record_set_audio turns them on and off
audio_track: bool; // a sound track whatever is on at the start, so a source can be turned on later
}
Record_Stats :: struct {
source: string; // "DXGI DuplicateOutput1 on <adapter>", "Windows.Graphics.Capture on <adapter>, border off"
fell_back_from: string; // why Desktop Duplication could not run, when it fell back
encoder: string;
width, height: s32;
captured: s64; // new images of the screen
pointer_moves: s64; // cursor changes with no new image (Desktop Duplication)
emitted: s64; // frames sent to be encoded
repeats: s64; // of them, the once-a-second repeat of an unchanged screen
written: s64; // encoded
dropped: s64; // the encoder was too far behind
lost: s64; // times the duplication was lost and made again
paused_seconds: float64; // left out of the video
audio: string; // the sound recorded: "Speakers (Realtek(R) Audio); Microphone (USB Audio)", or ""
audio_frames: s64; // AAC frames encoded
audio_errors: string; // sources that could not be opened, or were lost
copy_ticks, readback_ticks, encode_ticks: s64; // totals, in QueryPerformanceCounter ticks
bytes: s64; // encoded
seconds: float64; // from start to stop
}
Recorder :: struct {
options: Record_Options;
screen: Record_Screen;
local: RECT; // the region, in the screen's pixels
dxgi: Dxgi_Source; // the capture thread's: one of these is open
wgc: Wgc_Source;
using_wgc: bool;
encoder: Encoder;
width, height: s32;
frequency: s64; // QueryPerformanceCounter's
start_ticks: s64;
capture_thread, encoder_thread: Thread;
capture_started, encoder_started, audio_started: bool;
audio_thread: Thread;
audio_opened: Semaphore; // the audio thread has opened its sources
audio_go: Semaphore; // the encoder is open: the audio thread may write
audio_sources: [2] Audio_Source;
audio_open: bool; // any source opened, or audio_track: the MP4 has sound
audio_on: [2] bool; // each source is wanted (by Audio_Source_Kind): record_set_audio; cleared when it cannot be opened
audio_failed: [2] bool; // it could not be opened after being turned on: record_audio_failed says once
zero_ticks: s64; // the recording's time 0, for frames and sound alike
paused_ticks: s64; // time paused so far, for the capture and audio threads; set by record_pause
pause_began: s64;
running: bool;
stopping: bool; // set by record_stop; the capture thread ends, then the encoder
paused: bool; // set by record_pause; the capture thread takes nothing while it is
capture_ended: bool;
opened: Semaphore; // the capture thread has opened its source, or failed to
open_ok, open_fall_back: bool;
open_error: string;
mutex: Mutex;
ready: Semaphore; // signalled for each frame queued, and once when capture ends
frames: [RECORD_BUFFERS] Record_Frame;
free: [RECORD_BUFFERS] s32;
free_count: s32;
queue: [RECORD_BUFFERS] s32; // oldest first
queue_count: s32;
last_frame: [] u8; // the last frame queued, BGRA, for record_last_frame
stats: Record_Stats;
error: string; // why it stopped by itself; heap
}
Record_Frame :: struct {
pixels: [] u8; // BGRA, rows packed
ticks: s64;
}
// Opens the screen, the encoder and the file, and starts recording.
record_start :: (r: *Recorder, options: Record_Options) -> ok: bool, error: string, fall_back: bool {
r.* = .{};
r.options = options;
r.options.path = copy_string(options.path);
r.options.encoder = copy_string(options.encoder);
if r.options.fps <= 0 r.options.fps = 30;
screens := record_list_screens();
if !screens.count return false, "no screen to record", false;
screen_index := options.screen;
region := options.region;
has_region := region.right > region.left && region.bottom > region.top;
if screen_index < 0 {
if !has_region screen_index = 0;
else for screens if rect_inside(region, it.desktop) { screen_index = xx it_index; break; }
if screen_index < 0 return false, "the region is not on one screen", false;
}
if screen_index >= screens.count return false, tprint("there is no screen %", screen_index), false;
r.screen = screens[screen_index];
r.screen.adapter = copy_string(r.screen.adapter);
r.screen.name = copy_string(r.screen.name);
if !has_region region = r.screen.desktop;
if !rect_inside(region, r.screen.desktop) return false, "the region is not on that screen", false;
// The minimum first, then even both ways, in the screen's pixels: H.264's 4:2:0 colour needs it.
// RECORD_MIN_WIDTH and _HEIGHT are even, so what passes stays at least that size.
#assert RECORD_MIN_WIDTH % 2 == 0 && RECORD_MIN_HEIGHT % 2 == 0;
if !record_size_allowed(region.right - region.left, region.bottom - region.top)
return false, tprint("% x % is smaller than % x %", region.right - region.left, region.bottom - region.top, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT), false;
d := r.screen.desktop;
r.local = .{region.left - d.left, region.top - d.top, region.right - d.left, region.bottom - d.top};
r.local.right = r.local.left + ((r.local.right - r.local.left) & cast(s32) ~1);
r.local.bottom = r.local.top + ((r.local.bottom - r.local.top) & cast(s32) ~1);
r.width = r.local.right - r.local.left;
r.height = r.local.bottom - r.local.top;
r.stats.width, r.stats.height = r.width, r.height;
init(*r.mutex, "record");
init(*r.ready);
init(*r.opened);
for * r.frames it.pixels = NewArray(cast(s64) r.width * r.height * 4, u8, initialized = false);
for 0..RECORD_BUFFERS - 1 r.free[it] = cast(s32) it;
r.free_count = RECORD_BUFFERS;
r.last_frame = NewArray(cast(s64) r.width * r.height * 4, u8);
frequency: LARGE_INTEGER;
QueryPerformanceFrequency(*frequency);
r.frequency = frequency.QuadPart;
r.start_ticks = ticks_now();
r.zero_ticks = r.start_ticks;
// The source is opened on the capture thread, which uses it: WinRT's objects belong to the
// apartment they are made in.
thread_init(*r.capture_thread, record_capture_proc, name = "record capture");
r.capture_started = true;
thread_start(*r.capture_thread, r);
wait_for(*r.opened);
if !r.open_ok {
error, fall_back := copy_string(r.open_error,, temp), r.open_fall_back;
record_free(r);
return false, error, fall_back;
}
// The sound's sources are opened on their own thread (COM, for WASAPI), before the encoder, which
// has a sound stream only when one opened.
r.audio_on = .[r.options.system_audio, r.options.microphone];
if r.options.system_audio || r.options.microphone || r.options.audio_track {
init(*r.audio_opened);
init(*r.audio_go);
thread_init(*r.audio_thread, record_audio_proc, name = "record audio");
r.audio_started = true;
thread_start(*r.audio_thread, r);
wait_for(*r.audio_opened);
}
ok, error, tried := encoder_open(*r.encoder, r.options.path, r.width, r.height, r.options.fps, r.options.encoder, audio = r.audio_open);
if tried log("[record] Encoders that did not open: %", tried);
if !ok {
error = copy_string(error,, temp);
record_free(r);
return false, error, false;
}
r.stats.encoder = copy_string(r.encoder.name);
if r.audio_started signal(*r.audio_go);
thread_init(*r.encoder_thread, record_encoder_proc, name = "record encoder");
r.encoder_started = true;
r.running = true;
thread_start(*r.encoder_thread, r);
return true, "", false;
}
// Stops, finishes the file and frees everything. ok is false when recording stopped by itself on an
// error, or the file could not be finished; what was recorded up to then is in the file all the same.
record_stop :: (r: *Recorder) -> ok: bool, error: string {
if !r.running return false, "not recording";
atomic_swap(*r.stopping, true);
thread_is_done(*r.capture_thread, -1);
thread_is_done(*r.encoder_thread, -1);
if r.audio_started thread_is_done(*r.audio_thread, -1);
r.running = false;
closed, close_error := encoder_close(*r.encoder);
r.stats.seconds = cast(float64) (ticks_now() - r.start_ticks) / r.frequency;
r.stats.bytes = r.encoder.bytes;
error := ifx r.error then copy_string(r.error,, temp) else "";
record_free(r);
if error return false, error;
if !closed return false, close_error;
return true, "";
}
// Pauses or resumes. While paused nothing is taken, and the time is left out of the video, sound and
// all: the time paused is kept here, once, for the capture and audio threads both.
record_pause :: (r: *Recorder, pause: bool) {
if !r.running || atomic_read(*r.paused) == pause return;
now := ticks_now();
if pause {
atomic_write(*r.pause_began, now);
atomic_swap(*r.paused, true);
} else {
atomic_add(*r.paused_ticks, now - atomic_read(*r.pause_began)); // before paused clears, so no thread sees it unpaused with the old total
atomic_swap(*r.paused, false);
}
}
// Turns a source of sound on or off while recording. It is opened only while on, so Windows shows the
// microphone in use only then. False when the recording has no sound track to put it in.
record_set_audio :: (r: *Recorder, kind: Audio_Source_Kind, on: bool) -> bool {
if !r.running || !r.audio_open return false;
atomic_swap(*r.audio_on[kind], on);
return true;
}
record_audio_on :: (r: *Recorder, kind: Audio_Source_Kind) -> bool {
return r.running && atomic_read(*r.audio_on[kind]);
}
// Whether a source could not be opened when turned on (it is off again then), and why; once.
record_audio_failed :: (r: *Recorder, kind: Audio_Source_Kind) -> failed: bool, error: string {
if !r.running || !atomic_read(*r.audio_failed[kind]) return false, "";
atomic_swap(*r.audio_failed[kind], false);
lock(*r.mutex);
defer unlock(*r.mutex);
return true, copy_string(r.stats.audio_errors,, temp);
}
// Ticks on the recording's timeline: time 0 at its start, less the time paused.
record_timeline :: (r: *Recorder, ticks: s64) -> s64 {
return ticks - r.zero_ticks - atomic_read(*r.paused_ticks);
}
record_paused :: (r: *Recorder) -> bool {
return r.running && atomic_read(*r.paused);
}
// As they are now; strings are the recorder's.
record_stats :: (r: *Recorder) -> Record_Stats {
if !r.running return r.stats;
lock(*r.mutex);
defer unlock(*r.mutex);
return r.stats;
}
// The last frame queued, BGRA (temp), cursor and all.
record_last_frame :: (r: *Recorder) -> pixels: [] u8, width: s32, height: s32 {
if r.running lock(*r.mutex);
pixels := array_copy(r.last_frame,, temp);
if r.running unlock(*r.mutex);
return pixels, r.width, r.height;
}
// Whether recording stopped by itself (the screen went away, the encoder failed); why, then.
record_failed :: (r: *Recorder) -> bool, string {
lock(*r.mutex);
defer unlock(*r.mutex);
return r.error.count > 0, r.error;
}
#scope_file
Staged :: struct { pending: bool; ticks: s64; pointer: POINT; pointer_visible: bool; }
// Stops the capture thread if it runs, and frees what record_start made. The stats' strings are kept.
record_free :: (r: *Recorder) {
atomic_swap(*r.stopping, true);
if r.capture_started { thread_is_done(*r.capture_thread, -1); thread_deinit(*r.capture_thread); r.capture_started = false; }
if r.encoder_started { thread_is_done(*r.encoder_thread, -1); thread_deinit(*r.encoder_thread); r.encoder_started = false; }
if r.audio_started {
signal(*r.audio_go); // in case it waits for an encoder that did not open
thread_is_done(*r.audio_thread, -1);
thread_deinit(*r.audio_thread);
destroy(*r.audio_opened);
destroy(*r.audio_go);
r.audio_started = false;
}
if r.encoder.format encoder_close(*r.encoder);
for * r.frames array_free(it.pixels);
array_free(r.last_frame);
r.last_frame = .[];
destroy(*r.opened);
destroy(*r.ready);
destroy(*r.mutex);
free(r.open_error);
free(r.error);
r.open_error, r.error = "", "";
}
// Desktop Duplication, unless asked for Windows.Graphics.Capture, which takes over when duplication
// cannot run here and the options allow.
record_open_source :: (r: *Recorder) -> bool {
if r.options.capture != .WGC {
ok, error, fall_back := dxgi_open(*r.dxgi, r.screen, r.local, r.options.legacy_duplicate);
if ok && r.options.testing & .DXGI_UNSUPPORTED {
ok, error, fall_back = false, "Desktop Duplication (testing): 0x887A0004 (DXGI_ERROR_UNSUPPORTED)", true;
}
if ok {
r.stats.source = sprint("DXGI % on %", r.dxgi.method, r.screen.adapter);
return true;
}
dxgi_close(*r.dxgi);
if r.options.capture == .DXGI || !fall_back {
r.open_error, r.open_fall_back = copy_string(error), fall_back;
return false;
}
log("[record] Desktop Duplication cannot run here (%); Windows.Graphics.Capture instead.", error);
r.stats.fell_back_from = copy_string(error);
}
ok, error := wgc_open(*r.wgc, r.screen, r.local, r.options.cursor, r.options.testing);
if !ok {
wgc_close(*r.wgc);
r.open_error = copy_string(error);
return false;
}
r.using_wgc = true;
r.stats.source = sprint("Windows.Graphics.Capture on %, %%", r.screen.adapter, r.wgc.border, ifx r.wgc.polling then ", polled" else "");
return true;
}
record_capture_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
// Aware of each monitor's DPI, whatever the process is: Desktop Duplication refuses a thread that is
// not (DXGI_ERROR_UNSUPPORTED), and the mcap app's ImGui makes only its main thread aware.
gpu_thread_dpi_aware();
wgc_thread_start();
defer wgc_thread_end();
r.open_ok = record_open_source(r);
signal(*r.opened);
if !r.open_ok return 0;
gpu: *Capture_Gpu = ifx r.using_wgc then *r.wgc.gpu else *r.dxgi.gpu;
defer if r.using_wgc wgc_close(*r.wgc); else dxgi_close(*r.dxgi);
// A change is sent as soon as it comes once its turn is due: turns are a frame's time apart on a
// schedule that moves on a frame at a time, so the rate never passes fps, and a tenth of a frame
// early counts as on time, so a 60 Hz screen's frames, which jitter, all make a 60 fps cap.
period := r.frequency / r.options.fps;
early := period / 10;
turn := ticks_now();
last_emit: s64 = -r.frequency;
have_image, changed := false, false;
staged: [RECORD_STAGING] Staged;
emits: s64;
was_paused := false;
last_lost := ticks_now(); // for Record_Testing.LOSE_DUPLICATION
while !atomic_read(*r.stopping) {
now := ticks_now();
due := ifx changed then turn - early else last_emit + r.frequency; // the change, or the repeat
wait := cast(u32) clamp((due - now) * 1000 / r.frequency, 0, 100);
next: Capture_Next;
error: string;
if r.using_wgc next, error = wgc_next(*r.wgc, wait);
else next, error = dxgi_next(*r.dxgi, wait);
if r.options.testing & .LOSE_DUPLICATION && !r.using_wgc && ticks_now() - last_lost >= 3 * r.frequency {
next, last_lost = Capture_Next.LOST, ticks_now();
}
if next == {
case .FAILED;
record_fail(r, error);
break;
case .LOST;
// A mode change, the secure desktop, a game going fullscreen: duplicated again, and on.
lock(*r.mutex); r.stats.lost += 1; unlock(*r.mutex);
while !atomic_read(*r.stopping) {
hr := dxgi_duplicate(*r.dxgi);
if SUCCEEDED(hr) break;
if !hresult_is(hr, DXGI_ERROR_ACCESS_LOST) && !hresult_is(hr, 0x80070005) && !hresult_is(hr, DXGI_ERROR_NOT_CURRENTLY_AVAILABLE) {
record_fail(r, hresult_text("the screen could not be duplicated again", hr));
break;
}
sleep_milliseconds(250);
}
if atomic_read(*r.stopping) || r.error break;
continue;
case .IMAGE; #through;
case .IMAGE_AND_POINTER;
have_image, changed = true, true;
lock(*r.mutex);
r.stats.captured += 1;
r.stats.copy_ticks = gpu.copy_ticks;
unlock(*r.mutex);
case .POINTER;
if r.options.cursor && have_image {
changed = true;
lock(*r.mutex); r.stats.pointer_moves += 1; unlock(*r.mutex);
}
}
// Paused: the source is still drained, so it never stalls, but nothing is sent. On resuming, the
// screen as it is now is sent at once.
now = ticks_now();
if atomic_read(*r.paused) {
was_paused = true;
lock(*r.mutex); r.stats.paused_seconds = cast(float64) (atomic_read(*r.paused_ticks) + now - atomic_read(*r.pause_began)) / r.frequency; unlock(*r.mutex);
continue;
}
if was_paused {
was_paused = false;
changed = true;
turn = now;
lock(*r.mutex); r.stats.paused_seconds = cast(float64) atomic_read(*r.paused_ticks) / r.frequency; unlock(*r.mutex);
}
if !have_image continue;
repeat := !changed && now - last_emit >= r.frequency;
if !(changed && now >= turn - early) && !repeat continue;
turn = max(turn + period, now - period); // after a quiet spell, no burst to catch up
// Copied to staging now; read back two frames on, when the GPU is done with it.
slot := emits % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
ID3D11DeviceContext_CopyResource(gpu.device_context, gpu.staging[slot], gpu.latest);
staged[slot] = .{pending = true, ticks = record_timeline(r, now)};
if !r.using_wgc staged[slot].pointer, staged[slot].pointer_visible = r.dxgi.pointer_position, r.dxgi.pointer_visible;
emits += 1;
last_emit = now;
changed = false;
lock(*r.mutex);
r.stats.emitted += 1;
if repeat r.stats.repeats += 1;
unlock(*r.mutex);
}
// What is still staged, oldest first.
for 0..RECORD_STAGING - 1 {
slot := (emits + it) % RECORD_STAGING;
if staged[slot].pending record_read_back(r, gpu, slot, staged[slot]);
}
atomic_swap(*r.capture_ended, true);
signal(*r.ready);
return 0;
}
// Copies a staging texture into a free frame, draws the cursor (Desktop Duplication leaves it out)
// and queues it.
record_read_back :: (r: *Recorder, gpu: *Capture_Gpu, slot: s64, staged: Staged) {
t := ticks_now();
lock(*r.mutex);
if !r.free_count { // every frame is waiting: drop the oldest
r.free[r.free_count] = r.queue[0];
r.free_count += 1;
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
r.stats.dropped += 1;
}
r.free_count -= 1;
index := r.free[r.free_count];
unlock(*r.mutex);
frame := *r.frames[index];
mapped: D3D11_MAPPED_SUBRESOURCE;
hr := ID3D11DeviceContext_Map(gpu.device_context, gpu.staging[slot], 0, .READ, 0, *mapped);
if FAILED(hr) {
lock(*r.mutex); r.free[r.free_count] = index; r.free_count += 1; unlock(*r.mutex);
record_fail(r, hresult_text("reading the frame back", hr));
return;
}
row := cast(s64) r.width * 4;
for y: 0..r.height - 1 memcpy(frame.pixels.data + y * row, cast(*u8) mapped.pData + cast(s64) y * mapped.RowPitch, row);
ID3D11DeviceContext_Unmap(gpu.device_context, gpu.staging[slot], 0);
if r.options.cursor && staged.pointer_visible {
cursor_draw(frame.pixels.data, r.width, r.height, *r.dxgi.shape,
staged.pointer.x - cast(s32) gpu.box.left, staged.pointer.y - cast(s32) gpu.box.top);
}
frame.ticks = staged.ticks;
lock(*r.mutex);
r.queue[r.queue_count] = index;
r.queue_count += 1;
memcpy(r.last_frame.data, frame.pixels.data, frame.pixels.count);
r.stats.readback_ticks += ticks_now() - t;
unlock(*r.mutex);
signal(*r.ready);
}
record_encoder_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
while true {
wait_for(*r.ready, 100);
lock(*r.mutex);
index: s32 = -1;
if r.queue_count {
index = r.queue[0];
for 1..r.queue_count - 1 r.queue[it - 1] = r.queue[it];
r.queue_count -= 1;
}
unlock(*r.mutex);
if index < 0 {
if atomic_read(*r.capture_ended) break;
continue;
}
frame := *r.frames[index];
pts := max(frame.ticks, 0) * 90000 / r.frequency; // frame.ticks is on the timeline already
t := ticks_now();
ok, error := encoder_write(*r.encoder, frame.pixels.data, pts);
lock(*r.mutex);
r.free[r.free_count] = index;
r.free_count += 1;
r.stats.encode_ticks += ticks_now() - t;
if ok r.stats.written += 1;
r.stats.bytes = r.encoder.bytes;
unlock(*r.mutex);
if !ok {
record_fail(r, error);
// Frames still come until the capture thread sees it; they are let go.
while !atomic_read(*r.capture_ended) {
wait_for(*r.ready, 100);
lock(*r.mutex);
for 0..r.queue_count - 1 { r.free[r.free_count] = r.queue[it]; r.free_count += 1; }
r.queue_count = 0;
unlock(*r.mutex);
}
break;
}
}
return 0;
}
// The audio thread: opens the sources asked for, waits for the encoder, then every 10 ms takes their
// packets into the mix and encodes what every source has covered.
record_audio_proc :: (thread: *Thread) -> s64 {
r := cast(*Recorder) thread.data;
audio_thread_start();
defer audio_thread_end();
kinds: [2] Audio_Source_Kind = .[.SYSTEM, .MICROPHONE];
wanted: [2] bool = .[r.options.system_audio, r.options.microphone];
names, errors: [..] string; // every source heard, and every failure, for the stats
names.allocator, errors.allocator = context.default_allocator, context.default_allocator;
defer { for names free(it); array_free(names); for errors free(it); array_free(errors); }
note :: (list: *[..] string, text: string, field: *string) #expand {
for list.* if it == text return;
array_add(list, copy_string(text));
lock(*`r.mutex);
free(field.*);
field.* = join(..list.*, "; ");
unlock(*`r.mutex);
}
for 0..1 {
if !wanted[it] continue;
s := *r.audio_sources[it];
ok, error := audio_open(s, kinds[it]);
what := ifx kinds[it] == .SYSTEM then "system sound" else "microphone";
if ok { r.audio_open = true; note(*names, s.name, *r.stats.audio); }
else { audio_close(s); note(*errors, tprint("%: %", what, error), *r.stats.audio_errors); log_error("[record] No %: %", what, error); }
}
for r.audio_sources if !it.open atomic_swap(*r.audio_on[it_index], false);
if r.options.audio_track r.audio_open = true;
signal(*r.audio_opened);
wait_for(*r.audio_go);
if !r.audio_open || !r.encoder.audio || atomic_read(*r.stopping) {
for * r.audio_sources audio_free(it);
return 0;
}
mix: Audio_Mix;
mix.frame_size = r.encoder.audio_frame_size;
defer array_free(mix.samples);
while true {
stopping := atomic_read(*r.stopping);
// Each source, turned on or off from the toolbar: opened only while on.
for * source, index: r.audio_sources {
on := atomic_read(*r.audio_on[index]) && !stopping;
what := ifx kinds[index] == .SYSTEM then "System sound" else "Microphone";
if on && !source.open && !wanted[index] {
ok, error := audio_open(source, kinds[index]);
if ok {
log("[record] % on: %.", what, source.name);
note(*names, source.name, *r.stats.audio);
mix.started[index] = false;
} else {
audio_close(source);
log_error("[record] % could not be opened: %", what, error);
note(*errors, tprint("%: %", to_lower_copy(what,, temp), error), *r.stats.audio_errors);
atomic_swap(*r.audio_on[index], false);
atomic_swap(*r.audio_failed[index], true);
}
} else if !on && source.open {
log("[record] % off.", what);
audio_close(source);
source.lost_at = 0;
}
wanted[index] = source.open || (wanted[index] && on);
}
for * source, index: r.audio_sources {
if !wanted[index] continue;
if !source.open { // lost: opened again a second on, from the default device then
if source.lost_at && ticks_now() - source.lost_at >= r.frequency && !stopping {
if audio_open(source, kinds[index]) log("[record] % is back: %.", ifx kinds[index] == .SYSTEM then "System sound" else "The microphone", source.name);
else { audio_close(source); source.lost_at = ticks_now(); }
}
continue;
}
packets, lost := audio_take(source, r.frequency);
for packets if !atomic_read(*r.paused) audio_mix_add(*mix, r, index, it);
if lost {
log_error("[record] % was lost (unplugged, or the default device changed); trying again.", source.name);
audio_close(source);
source.lost_at = ticks_now();
}
}
ok, error := audio_mix_encode(*mix, r, final = stopping);
if !ok { record_fail(r, error); break; }
if stopping break;
reset_temporary_storage();
sleep_milliseconds(10);
}
for * r.audio_sources audio_free(it);
return 0;
}
// Ends the recording from one of its threads; the first reason is kept.
record_fail :: (r: *Recorder, error: string) {
log_error("[record] %", error);
lock(*r.mutex);
if !r.error r.error = copy_string(error);
unlock(*r.mutex);
atomic_swap(*r.stopping, true);
}
rect_inside :: (inner: RECT, outer: RECT) -> bool {
return inner.left >= outer.left && inner.top >= outer.top && inner.right <= outer.right && inner.bottom <= outer.bottom;
}
#scope_export
#load "capture_gpu.jai";
#load "dxgi_source.jai";
#load "wgc_source.jai";
#load "audio_source.jai";
#load "cursor.jai";
#import "Windows";
#import "Windows_Utf8";
#import "dxgi";
#import "d3d11";

5
modules/Segment/README.md
View File

@ -18,7 +18,10 @@ conda run -n mcap-seg python tools/export_mask2former.py --check FOLDER_OF_IMAGE
```
This writes `models/mask2former-swin-base-coco-panoptic.onnx` (462 MB) and its class list, then checks
the .onnx against PyTorch on the images. Building the app copies `models.config` and the models into
the .onnx against PyTorch on the images. On macOS, `tools/export_mask2former_coreml.py --check
FOLDER_OF_IMAGES` then makes `models/mask2former-swin-base-coco-panoptic.mlpackage`, which runs on the
GPU through Core ML (`modules/Core_ML`), one image a run, about four times as fast as the CPU with the
same segments. Building the app copies `models.config` and the models into
`bin/segment-models`. The models folder is not in git.
## Using it

1
modules/Segment/models.config
View File

@ -6,6 +6,7 @@
[model mask2former-swin-base-coco-panoptic]
title: "Mask2Former Swin-B, COCO panoptic (133 classes)"
model: "mask2former-swin-base-coco-panoptic.onnx"
model_coreml: "mask2former-swin-base-coco-panoptic.mlpackage" # macOS: on the GPU (tools/export_mask2former_coreml.py makes it)
classes: "mask2former-swin-base-coco-panoptic.classes.txt"
image_size: 384 # squashed to 384 x 384, not keeping its shape, as the checkpoint's processor does
mean: 0.485 0.456 0.406

99
modules/Segment/module.jai
View File

@ -11,7 +11,7 @@
// transformers' post_process_panoptic_segmentation does (tools/export_mask2former.py checks that
// they match).
Segment_Provider :: enum { CPU; CUDA; }
Segment_Provider :: enum { CPU; CUDA; COREML; }
// Tightly packed or not: stride is the bytes from one row to the next.
Segment_Image :: struct {
@ -23,6 +23,7 @@ Segment_Model_Info :: struct {
name: string;
title: string;
model: string; // the .onnx, in the models folder
model_coreml: string; // macOS: the model as a Core ML package (tools/export_mask2former_coreml.py), run on the GPU; "" for none
classes: string; // the class list, in the models folder
image_size: s32 = 384; // the input's width and height: images are resized to it, not keeping their shape
mean := float.[0.485, 0.456, 0.406]; // ImageNet's, per channel, after scaling to 0..1
@ -50,6 +51,7 @@ Segment_Engine :: struct {
env: *OrtEnv;
memory_info: *OrtMemoryInfo;
session: Ort_Session;
#if OS == .MACOS coreml: *Coreml_Model; // the model, when Core ML runs it (one image a run)
cancel: *bool; // when set, and true, segment_images stops between batches: the rest fail as "cancelled"
stats: Segment_Stats; // since it was made
}
@ -114,6 +116,7 @@ segment_list_models :: (models_folder: string) -> [] Segment_Model_Info, error:
case "title"; model.title = copy_string(value);
case "model"; model.model = copy_string(value);
case "classes"; model.classes = copy_string(value);
case "model_coreml"; model.model_coreml = copy_string(value);
case "image_size"; if number_ok model.image_size = cast(s32) number;
case "mean"; segment_parse_three(value, *model.mean, path, line_number);
case "std"; segment_parse_three(value, *model.std, path, line_number);
@ -155,11 +158,29 @@ segment_create :: (models_folder: string, model_name: string, prefer_gpu := true
HINT :: "make it with tools/export_mask2former.py (see modules/Segment/README.md) and rebuild";
path := tprint("%/%", models_folder, model.model);
use_gpu := false;
if prefer_gpu {
#if OS == .MACOS if prefer_gpu {
// Core ML on the GPU; on any failure, onnxruntime on the CPU below, with the reason.
coreml_path := tprint("%/%", models_folder, model.model_coreml);
if !model.model_coreml || !file_exists(coreml_path) {
engine.gpu_explanation = sprint("Segmentation runs on the CPU: % is not there; tools/export_mask2former_coreml.py makes it.", ifx model.model_coreml then model.model_coreml else "its Core ML model");
} else {
engine.coreml, ok, error = coreml_load(coreml_path, "pixel_values", "class_queries_logits");
if ok {
engine.provider = .COREML;
ok, error = segment_warm_up(engine);
if ok return engine, true, "";
coreml_free(engine.coreml);
engine.coreml = null;
engine.provider = .CPU;
}
engine.gpu_explanation = sprint("Segmentation runs on the CPU: Core ML could not run % on the GPU:\n%", model.model_coreml, error);
}
}
if prefer_gpu && OS != .MACOS {
available, explanation := ort_cuda_check("Segmentation");
use_gpu = available;
if !available engine.gpu_explanation = copy_string(explanation);
} else {
} else if !prefer_gpu {
engine.gpu_explanation = copy_string("Segmentation runs on the CPU, as it was asked to.");
}
if use_gpu {
@ -190,6 +211,7 @@ segment_destroy :: (engine: *Segment_Engine) {
if engine.provider == .CUDA ort_gpu_lock();
ort_release_session(*engine.session);
if engine.provider == .CUDA ort_gpu_unlock();
#if OS == .MACOS coreml_free(engine.coreml);
if engine.memory_info ort.api.ReleaseMemoryInfo(engine.memory_info);
if engine.env ort.api.ReleaseEnv(engine.env);
for engine.classes { free(it.name); free(it.model_name); }
@ -198,6 +220,24 @@ segment_destroy :: (engine: *Segment_Engine) {
free(engine);
}
// Images a run: Core ML's model takes one.
segment_batch :: (engine: *Segment_Engine) -> s32 {
if engine.provider == {
case .CUDA; return engine.model.batch_gpu;
case .COREML; return 1;
}
return engine.model.batch_cpu;
}
// What runs the model, for people: "CUDA", "Core ML" or "CPU".
segment_provider_name :: (engine: *Segment_Engine) -> string {
if engine.provider == {
case .CUDA; return "CUDA";
case .COREML; return "Core ML";
}
return "CPU";
}
// The map size for an image of width x height: its shape, the longest side at most max_side.
segment_map_size :: (width: s32, height: s32, max_side: s32) -> s32, s32 {
scale := min(cast(float64) 1, cast(float64) max_side / max(width, height));
@ -210,7 +250,7 @@ segment_images :: (engine: *Segment_Engine, images: [] Segment_Image, allocator
results := NewArray(images.count, Segment_Result,, allocator);
size := engine.model.image_size;
plane := cast(s64) size * size;
batch := cast(s64) max(ifx engine.provider == .CUDA then engine.model.batch_gpu else engine.model.batch_cpu, 1);
batch := cast(s64) max(segment_batch(engine), 1);
input := NewArray(plane * 3 * batch, float,, context.default_allocator);
defer array_free(input,, context.default_allocator);
@ -224,10 +264,10 @@ segment_images :: (engine: *Segment_Engine, images: [] Segment_Image, allocator
start := current_time_monotonic();
for 0..count - 1 segment_prepare_image(engine.model, images[first + it], input.data + plane * 3 * it);
shape := s64.[count, 3, size, size];
outs: [2] Ort_Output;
outs: [2] Segment_Output;
ran := current_time_monotonic();
engine.stats.prepare_us += to_microseconds(ran - start);
ok, error := ort_run_float_outputs(*engine.session, engine.memory_info, array_view(input, 0, plane * 3 * count), shape, .["class_queries_logits", "masks_queries_logits"], outs);
ok, error := segment_run(engine, array_view(input, 0, plane * 3 * count), shape, outs);
posted := current_time_monotonic();
engine.stats.model_us += to_microseconds(posted - ran);
engine.stats.images += count;
@ -249,8 +289,7 @@ segment_images :: (engine: *Segment_Engine, images: [] Segment_Image, allocator
cast(s32) queries, cast(s32) labels, cast(s32) mask_w, cast(s32) mask_h, image.width, image.height, allocator);
}
}
ort_release_output(*outs[0]);
ort_release_output(*outs[1]);
segment_release_outputs(engine, outs);
}
first += count;
}
@ -524,14 +563,48 @@ segment_warm_up :: (engine: *Segment_Engine) -> ok: bool, error: string {
size := engine.model.image_size;
input := NewArray(3 * size * size, float, initialized = true,, context.default_allocator);
defer array_free(input,, context.default_allocator);
outs: [2] Ort_Output;
ok, error := ort_run_float_outputs(*engine.session, engine.memory_info, input, s64.[1, 3, size, size], .["class_queries_logits", "masks_queries_logits"], outs);
outs: [2] Segment_Output;
ok, error := segment_run(engine, input, s64.[1, 3, size, size], outs);
if !ok return false, tprint("The first run on the GPU failed: %", error);
ort_release_output(*outs[0]);
ort_release_output(*outs[1]);
segment_release_outputs(engine, outs);
return true, "";
}
// The two outputs, class and mask logits, as onnxruntime or Core ML gives them.
Segment_Output :: struct {
data: *float;
dims: [4] s64;
dim_count: s64;
ort: Ort_Output;
#if OS == .MACOS coreml: Coreml_Output;
}
segment_run :: (engine: *Segment_Engine, input: [] float, shape: [] s64, outs: [] Segment_Output) -> ok: bool, error: string {
NAMES :: string.["class_queries_logits", "masks_queries_logits"];
#if OS == .MACOS if engine.coreml {
coreml_outs: [2] Coreml_Output;
ok, error := coreml_predict_outputs(engine.coreml, input, shape, NAMES, coreml_outs);
if !ok return false, error;
for * outs {
it.* = .{data = coreml_outs[it_index].data, dim_count = min(coreml_outs[it_index].dim_count, 4), coreml = coreml_outs[it_index]};
for d: 0..it.dim_count - 1 it.dims[d] = coreml_outs[it_index].dims[d];
}
return true, "";
}
ort_outs: [2] Ort_Output;
ok, error := ort_run_float_outputs(*engine.session, engine.memory_info, input, shape, NAMES, ort_outs);
if !ok return false, error;
for * outs it.* = .{data = ort_outs[it_index].data, dims = ort_outs[it_index].dims, dim_count = ort_outs[it_index].dim_count, ort = ort_outs[it_index]};
return true, "";
}
segment_release_outputs :: (engine: *Segment_Engine, outs: [] Segment_Output) {
for * outs {
ort_release_output(*it.ort);
#if OS == .MACOS coreml_release_output(*it.coreml);
}
}
// The class list: `class: INDEX thing|stuff "model's name" "name"`, a line each, in index order.
segment_load_classes :: (engine: *Segment_Engine, path: string) -> ok: bool, error: string {
text, ok := read_entire_file(path, log_errors = false,, temp);
@ -597,9 +670,11 @@ segment_strip_comment :: (line: string) -> string {
#import "Basic";
#import "String";
#import "File";
#import "File_Utilities";
#import "Math";
#import "System";
#import "Hash";
#import "Sort";
#import "Capture_File"; // segment_mcap_chunks
#import "onnxruntime";
#if OS == .MACOS #import "Core_ML";

13
modules/ffmpeg/README.md
View File

@ -2,6 +2,19 @@
FFmpeg bindings for jai and some examples decoding and play videos.
## In mcap
FFmpeg 6.0 on both systems, so the two sets of bindings match:
- **Windows:** `windows/` holds the DLLs and import libraries of a GPL "full" build from gyan.dev
(`windows/LICENSE-BUILD.txt`), and `windows.jai` the bindings made from its headers (Setup below).
- **macOS:** `macos/` holds `libavcodec`, `libavformat`, `libavutil`, `libswresample` and `libswscale`,
an LGPL build for Apple silicon made from the release's source by `jai build_macos.jai` (about a
minute; `macos/LICENSE-BUILD.txt` has its configure options). It also installs the headers in
`macos-build/` (not in git), from which `jai generate.jai` on a Mac makes `macos.jai`. The libraries
are linked with `no_dll`, so the compiler does not load them, and copied beside the executable,
where its `@loader_path` rpath finds them.
## Setup
The FFmpeg libaries and header files are not included in this repo as I didn't want to have to deal with license

117
modules/ffmpeg/build_macos.jai Normal file
View File

@ -0,0 +1,117 @@
// Builds the FFmpeg 6.0 libraries mcap uses on macOS (avformat, avcodec, avutil, swscale and
// swresample) from the release's source, its SHA-256 checked, and puts them in macos/ with the
// headers the bindings are made from (generate.jai) in macos-build/install/include. The same version
// as the Windows DLLs (windows/), so the two sets of bindings match.
//
// cd modules/ffmpeg && jai build_macos.jai
//
// An LGPL build: no GPL parts (the Windows DLLs are a GPL build with libx264), nothing from outside
// macOS (--disable-autodetect, pkg-config pointed nowhere, so nothing from Homebrew is linked), and
// VideoToolbox and AudioToolbox for the hardware codecs. Needs Apple's command line tools: curl,
// tar, make and clang. A few minutes on Apple silicon.
FFMPEG_VERSION :: "6.0";
FFMPEG_ARCHIVE :: "ffmpeg-6.0.tar.xz";
FFMPEG_SHA256 :: "57be87c22d9b49c112b6d24bc67d42508660e6b718b3db89c44e47e289137082";
BUILD_FOLDER :: "macos-build"; // the source, the build and the installed headers; not in git
OUTPUT_FOLDER :: "macos"; // the libraries, in git as windows/ is
// Each library as its major version names it, which is how they find each other (@rpath/...).
LIBRARIES :: string.["libavcodec.60.dylib", "libavformat.60.dylib", "libavutil.58.dylib", "libswresample.4.dylib", "libswscale.7.dylib"];
CONFIGURE_OPTIONS :: string.[
"--install-name-dir=@rpath",
"--enable-shared", "--disable-static",
"--disable-programs", "--disable-doc",
"--disable-avdevice", "--disable-avfilter", "--disable-postproc", "--disable-network",
"--disable-autodetect",
"--enable-videotoolbox", "--enable-audiotoolbox", "--enable-zlib", "--enable-bzlib",
"--extra-cflags=-mmacosx-version-min=14.0", "--extra-ldflags=-mmacosx-version-min=14.0",
];
#run,stallable {
set_build_options_dc(.{do_output = false});
#if OS != .MACOS {
log_error("build_macos.jai builds on a Mac only.");
return;
}
if !build() compiler_set_workspace_status(.FAILED);
}
build :: () -> bool {
here := get_working_directory();
build_folder := tprint("%/%", here, BUILD_FOLDER);
make_directory_if_it_does_not_exist(build_folder);
archive := tprint("%/%", build_folder, FFMPEG_ARCHIVE);
if !file_exists(archive) || sha256_of(archive) != FFMPEG_SHA256 {
url := tprint("https://ffmpeg.org/releases/%", FFMPEG_ARCHIVE);
log("Downloading %...", url);
result := run_command("curl", "-L", "--fail", "--progress-bar", "-o", archive, url);
if result.exit_code != 0 { log_error("curl failed (exit code %).", result.exit_code); return false; }
hash := sha256_of(archive);
if hash != FFMPEG_SHA256 {
log_error("% has SHA-256 %, expected %; it was not used.", archive, hash, FFMPEG_SHA256);
return false;
}
}
source := tprint("%/ffmpeg-%", build_folder, FFMPEG_VERSION);
install := tprint("%/install", build_folder);
run_command("rm", "-rf", source, install);
result := run_command("tar", "xf", archive, "-C", build_folder);
if result.exit_code != 0 { log_error("tar could not unpack %.", archive); return false; }
log("Configuring...");
arguments: [..] string;
array_add(*arguments, "env", "PKG_CONFIG_LIBDIR=", "PKG_CONFIG_PATH=", "./configure", tprint("--prefix=%", install));
array_add(*arguments, ..CONFIGURE_OPTIONS);
result = run_command(..arguments, working_directory = source);
if result.exit_code != 0 { log_error("configure failed; see %/ffbuild/config.log.", source); return false; }
log("Building...");
cpus := get_number_of_processors();
result = run_command("make", tprint("-j%", cpus), working_directory = source);
if result.exit_code != 0 { log_error("make failed."); return false; }
result = run_command("make", "install", working_directory = source);
if result.exit_code != 0 { log_error("make install failed."); return false; }
output := tprint("%/%", here, OUTPUT_FOLDER);
run_command("rm", "-rf", output);
make_directory_if_it_does_not_exist(output);
for LIBRARIES {
// The major-version name is a link to the full one; the file itself is copied under that name.
if !copy_file(tprint("%/lib/%", install, it), tprint("%/%", output, it)) { log_error("Could not copy %.", it); return false; }
run_command("strip", "-x", tprint("%/%", output, it));
}
write_entire_file(tprint("%/LICENSE-BUILD.txt", output), tprint(LICENSE_TEXT, FFMPEG_VERSION, FFMPEG_ARCHIVE, FFMPEG_SHA256, join(..CONFIGURE_OPTIONS, "\n ")));
log("Put % libraries in % and the headers in %/include (FFmpeg %, LGPL).", LIBRARIES.count, output, install, FFMPEG_VERSION);
return true;
}
LICENSE_TEXT :: #string TEXT
FFmpeg % shared libraries for macOS on Apple silicon, built by modules/ffmpeg/build_macos.jai.
Source: https://ffmpeg.org/releases/%
SHA-256: %
License: LGPL version 2.1 or later (no GPL or non-free parts). The source is at the address above;
the libraries are built from it unchanged, with these configure options:
%
TEXT
// Lowercase hex, or "" when it could not be worked out.
sha256_of :: (path: string) -> string {
result, output := run_command("shasum", "-a", "256", path, capture_and_return_output = true);
if result.exit_code != 0 return "";
return to_lower_copy(trim(split(output, " ")[0]));
}
#import "Basic";
#import "Compiler";
#import "File";
#import "File_Utilities";
#import "Process";
#import "String";
#import "System";

59
modules/ffmpeg/generate.jai
View File

@ -46,10 +46,13 @@ FOOTER :: #string DONE
DONE
INCLUDE_PATH :: "src";
// Windows: the headers of the downloaded build, copied to src. macOS: those build_macos.jai installs.
#if OS == .MACOS INCLUDE_PATH :: "macos-build/install/include";
else INCLUDE_PATH :: "src";
generate_bindings :: () -> bool {
output_filename := "windows.jai";
#if OS == .MACOS output_filename := "macos.jai";
else output_filename := "windows.jai";
opts: Generate_Bindings_Options;
{
@ -67,19 +70,31 @@ generate_bindings :: () -> bool {
// doesn't work, just get a load of __UnknownLib dont you :roll:.
// strip_flags &= ~.SYMBOLS_WITH_UNKNOWN_FOREIGN_LIBS;
array_add(*libpaths, "windows");
libraries := string.[
"avcodec-60",
"avdevice-60",
"avfilter-9",
"avformat-60",
"avutil-58",
"postproc-57",
"swresample-4",
"swscale-7",
];
for libraries array_add(*libnames, it);
#if OS == .MACOS {
// The five libraries build_macos.jai makes, as their major versions name them; with the
// extension, since the generator takes a name with a dot in it for a whole file name.
array_add(*libpaths, "macos");
library_names :: string.[
"libavcodec.60.dylib",
"libavformat.60.dylib",
"libavutil.58.dylib",
"libswresample.4.dylib",
"libswscale.7.dylib",
];
} else {
array_add(*libpaths, "windows");
library_names :: string.[
"avcodec-60",
"avdevice-60",
"avfilter-9",
"avformat-60",
"avutil-58",
"postproc-57",
"swresample-4",
"swscale-7",
];
}
for library_names array_add(*libnames, it);
array_add(*include_paths, INCLUDE_PATH);
@ -89,7 +104,7 @@ generate_bindings :: () -> bool {
// pick the headers I want!
#if false {
dirs_to_visit: [..] string;
for libraries array_add(*dirs_to_visit, tprint("%/lib%", INCLUDE_PATH, it));
for library_names array_add(*dirs_to_visit, tprint("%/lib%", INCLUDE_PATH, it));
file_visitor :: (info: *File_Visit_Info, source_files : *[..] string) {
assert(info != null, "info (%) != null", info);
@ -121,7 +136,7 @@ generate_bindings :: () -> bool {
array_add(*source_files, tprint("%/libavformat/avformat.h", INCLUDE_PATH));
array_add(*source_files, tprint("%/libswscale/swscale.h", INCLUDE_PATH));
array_add(*system_include_paths, GENERATOR_DEFAULT_SYSTEM_INCLUDE_PATH);
// system_include_paths left empty: the generator fills in this system's own.
builder: String_Builder;
append(*builder, HEADER);
@ -134,6 +149,14 @@ generate_bindings :: () -> bool {
print("Generating ffmpeg bindings...\n");
result := generate_bindings(opts, output_filename);
// The compiler loads each #library to run code at compile time, but there the dylibs cannot find
// each other through @rpath; nothing of FFmpeg's is run then, so none is loaded (no_dll).
#if OS == .MACOS if result {
text, ok := read_entire_file(output_filename);
if !ok return false;
result = write_entire_file(output_filename, replace(text, " :: #library \"macos/", " :: #library,no_dll \"macos/"));
}
return result;
}
@ -243,7 +266,7 @@ generate_error_codes :: (b: *String_Builder) {
for info_struct.members longest_name_length = max(longest_name_length, it.name.count);
for info_struct.members {
slot := cast (*u8) *dummy + it.offset_in_bytes;
value := << cast(*s32) slot;
value := (cast(*s32) slot).*;
append(b, it.name);
num_spaces := longest_name_length - it.name.count;
for 1..num_spaces append(b, " ");

12947
modules/ffmpeg/macos.jai Normal file
View File

File diff suppressed because it is too large Load Diff

23
modules/ffmpeg/macos/LICENSE-BUILD.txt Normal file
View File

@ -0,0 +1,23 @@
FFmpeg 6.0 shared libraries for macOS on Apple silicon, built by modules/ffmpeg/build_macos.jai.
Source: https://ffmpeg.org/releases/ffmpeg-6.0.tar.xz
SHA-256: 57be87c22d9b49c112b6d24bc67d42508660e6b718b3db89c44e47e289137082
License: LGPL version 2.1 or later (no GPL or non-free parts). The source is at the address above;
the libraries are built from it unchanged, with these configure options:
--install-name-dir=@rpath
--enable-shared
--disable-static
--disable-programs
--disable-doc
--disable-avdevice
--disable-avfilter
--disable-postproc
--disable-network
--disable-autodetect
--enable-videotoolbox
--enable-audiotoolbox
--enable-zlib
--enable-bzlib
--extra-cflags=-mmacosx-version-min=14.0
--extra-ldflags=-mmacosx-version-min=14.0

BIN
modules/ffmpeg/macos/libavcodec.60.dylib Normal file
View File

Binary file not shown.

BIN
modules/ffmpeg/macos/libavformat.60.dylib Normal file
View File

Binary file not shown.

BIN
modules/ffmpeg/macos/libavutil.58.dylib Normal file
View File

Binary file not shown.

BIN
modules/ffmpeg/macos/libswresample.4.dylib Normal file
View File

Binary file not shown.

BIN
modules/ffmpeg/macos/libswscale.7.dylib Normal file
View File

Binary file not shown.

4
modules/ffmpeg/module.jai
View File

@ -7,7 +7,9 @@ size_t :: u64;
#if OS == .WINDOWS {
#load "windows.jai";
} else #if OS == .MACOS {
#load "macos.jai"; // generated from build_macos.jai's headers; its libraries are in macos/
} else {
#assert false, "Only windows has bindings atm, sorry bros!";
#assert false, "FFmpeg has bindings for Windows and macOS only.";
}

8
modules/onnxruntime/cuda.jai
View File

@ -49,8 +49,14 @@ Cuda_Part :: enum { DRIVER; TOOLKIT; CUDNN; }
// Whether the CUDA provider should be tried, and when not, an explanation for people: what is
// missing, and the steps that fix it, starting "<what> runs on the CPU". Temp.
ort_cuda_check :: (what: string) -> available: bool, explanation: string {
// CoreML, onnxruntime's provider for Apple's GPU and Neural Engine, was measured on an M5 Pro
// (onnxruntime 1.30.0) and is slower than the CPU for these models: OCR's take a new image size
// nearly every run, which CoreML compiles for each time, and SigLIP 2 converts only with every
// dimension fixed (its ML Program conversion fails on the patch embedding), then runs at about
// half the CPU's speed. So onnxruntime runs them on the CPU on macOS; OCR and labels use native
// Core ML instead when their converted models are there (modules/Core_ML).
#if OS == .MACOS {
return false, tprint("% runs on the CPU: onnxruntime has no CUDA on macOS.", what);
return false, tprint("% runs on the CPU: onnxruntime has no CUDA on macOS, and its CoreML provider was measured slower than the CPU on Apple silicon.", what);
}
provider_path := tprint("%/%", path_strip_filename(ort.library_path), CUDA_PROVIDER_LIBRARY);

3
modules/onnxruntime/fetch.jai
View File

@ -113,7 +113,8 @@ sha256_of :: (path: string) -> string {
if lines.count < 2 return "";
return to_lower_copy(replace(trim(lines[1]), " ", "")); // line 2 is the hash
} else {
command := ifx OS == .MACOS then string.["shasum", "-a", "256", path] else string.["sha256sum", path];
#if OS == .MACOS command := string.["shasum", "-a", "256", path];
else command := string.["sha256sum", path];
result, output := run_command(..command, capture_and_return_output = true);
if result.exit_code != 0 return "";
return to_lower_copy(trim(split(output, " ")[0]));

90
modules/onnxruntime/include/coreml_provider_factory.h Normal file
View File

@ -0,0 +1,90 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
#pragma once
#include "onnxruntime_c_api.h"
// COREMLFlags are bool options we want to set for CoreML EP
// This enum is defined as bit flags, and cannot have negative value
// To generate an uint32_t coreml_flags for using with OrtSessionOptionsAppendExecutionProvider_CoreML below,
// uint32_t coreml_flags = 0;
// coreml_flags |= COREML_FLAG_USE_CPU_ONLY;
enum COREMLFlags {
COREML_FLAG_USE_NONE = 0x000,
// Using CPU only in CoreML EP, this may decrease the perf but will provide
// reference output value without precision loss, which is useful for validation
COREML_FLAG_USE_CPU_ONLY = 0x001,
// Enable CoreML EP on subgraph
COREML_FLAG_ENABLE_ON_SUBGRAPH = 0x002,
// By default CoreML Execution provider will be enabled for all compatible Apple devices
// Enable this option will only enable CoreML EP for Apple devices with ANE (Apple Neural Engine)
// Please note, enable this option does not guarantee the entire model to be executed using ANE only
COREML_FLAG_ONLY_ENABLE_DEVICE_WITH_ANE = 0x004,
// Only allow CoreML EP to take nodes with inputs with static shapes. By default it will also allow inputs with
// dynamic shapes. However, the performance may be negatively impacted if inputs have dynamic shapes.
COREML_FLAG_ONLY_ALLOW_STATIC_INPUT_SHAPES = 0x008,
// Create an MLProgram. By default it will create a NeuralNetwork model. Requires Core ML 5 or later.
COREML_FLAG_CREATE_MLPROGRAM = 0x010,
// https://developer.apple.com/documentation/coreml/mlcomputeunits?language=objc
// there are four compute units:
// MLComputeUnitsCPUAndNeuralEngine|MLComputeUnitsCPUAndGPU|MLComputeUnitsCPUOnly|MLComputeUnitsAll
// different CU will have different performance and power consumption
COREML_FLAG_USE_CPU_AND_GPU = 0x020,
// Keep COREML_FLAG_LAST at the end of the enum definition
// And assign the last COREMLFlag to it
COREML_FLAG_LAST = COREML_FLAG_USE_CPU_AND_GPU,
};
// MLComputeUnits can be one of the following values:
// 'MLComputeUnitsCPUAndNeuralEngine|MLComputeUnitsCPUAndGPU|MLComputeUnitsCPUOnly|MLComputeUnitsAll'
// these values are intended to be used with Ort::SessionOptions::AppendExecutionProvider (C++ API)
// and SessionOptionsAppendExecutionProvider (C API). For the old API, use COREMLFlags instead.
static const char* const kCoremlProviderOption_MLComputeUnits = "MLComputeUnits";
static const char* const kCoremlProviderOption_ModelFormat = "ModelFormat";
// same as COREML_FLAG_ONLY_ALLOW_STATIC_INPUT_SHAPES
static const char* const kCoremlProviderOption_RequireStaticInputShapes = "RequireStaticInputShapes";
static const char* const kCoremlProviderOption_EnableOnSubgraphs = "EnableOnSubgraphs";
// provided by https://developer.apple.com/documentation/coreml/mloptimizationhints-swift.struct/specializationstrategy-swift.property
// Core ML segments the model’s compute graph and specializes each segment for the target compute device.
// This process can affect the model loading time and the prediction latency.
// Use this option to tailor the specialization strategy for your model.
static const char* const kCoremlProviderOption_SpecializationStrategy = "SpecializationStrategy";
// Profile the Core ML MLComputePlan.
// This logs the hardware each operator is dispatched to and the estimated execution time.
// Intended for developer usage but provide useful diagnostic information if performance is not as expected.
static const char* const kCoremlProviderOption_ProfileComputePlan = "ProfileComputePlan";
// please refer to https://developer.apple.com/documentation/coreml/mlmodelconfiguration/allowlowprecisionaccumulationongpu
static const char* const kCoremlProviderOption_AllowLowPrecisionAccumulationOnGPU = "AllowLowPrecisionAccumulationOnGPU";
// Specify the directory to cache any CoreML models created from the ONNX model in.
// CoreML EP will convert onnx subgraph to CoreML model and save to disk.
// If this path is not specified, the model will be saved to a temp directory and deleted after the session is closed.
// otherwise, the model will be saved to the specified path and User should manage to delete the model.
// we do NOT detect if the onnx model has changed and no longer matches the cached model.
// the user should carefully manage the cache if modifying/replacing a model.
// The cache key is generated by
// 1. User provided key in metadata_props if found (preferred)
// 2. Hash of the model url the inference session was created with
// 3. Hash of the input/output names of the model
// Please find out how to set metadata_props in the onnxruntime API documentation. https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html#configuration-options
static const char* const kCoremlProviderOption_ModelCacheDirectory = "ModelCacheDirectory";
// User provided cache-key in metadata_props.
static const char* const kCOREML_CACHE_KEY = "COREML_CACHE_KEY";
#ifdef __cplusplus
extern "C" {
#endif
ORT_EXPORT ORT_API_STATUS(OrtSessionOptionsAppendExecutionProvider_CoreML,
_In_ OrtSessionOptions* options, uint32_t coreml_flags);
#ifdef __cplusplus
}
#endif

14
modules/onnxruntime/include/webgpu_provider_factory.h Normal file
View File

@ -0,0 +1,14 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
// Dummy file to provide a signal in the ONNX Runtime C cocoapod as to whether the WebGPU EP was included in the build.
// If it was, this file will be included in the cocoapod, and a test like this can be used:
//
// #if __has_include(<onnxruntime/webgpu_provider_factory.h>)
// #define WEBGPU_EP_AVAILABLE 1
// #else
// #define WEBGPU_EP_AVAILABLE 0
// #endif
// The WebGPU EP can be enabled via the generic SessionOptionsAppendExecutionProvider method, so no direct usage of
// the provider factory is required.

10
modules/onnxruntime/module.jai
View File

@ -14,15 +14,17 @@ ort: struct {
log_level := OrtLoggingLevel.ORT_LOGGING_LEVEL_WARNING;
}
// Loads onnxruntime from the first folder that has it: beside the running executable, then each
// of folders. A full path is always used: Windows keeps an older onnxruntime.dll in System32 for
// its own use, which a bare name could find first.
// Loads onnxruntime from the first folder that has it: beside the running executable (or in a macOS
// bundle's Contents/Frameworks), then each of folders. A full path is always used: Windows keeps an
// older onnxruntime.dll in System32 for its own use, which a bare name could find first.
ort_load :: (folders: [] string = .[]) -> ok: bool, error: string {
if ort.api return true, "";
candidates: [..] string;
candidates.allocator = temp;
array_add(*candidates, path_strip_filename(get_path_of_running_executable()));
executable_folder := path_strip_filename(get_path_of_running_executable());
array_add(*candidates, executable_folder);
#if OS == .MACOS if ends_with(executable_folder, "/Contents/MacOS/") array_add(*candidates, tprint("%../Frameworks", executable_folder));
for folders array_add(*candidates, it);
tried: [..] string;

14
src/App.jai
View File

@ -31,7 +31,9 @@ USING_WINDOWS :: OS == .WINDOWS;
#load "app/screen_capture.jai";
#load "app/screen_record.jai";
#if OS == .WINDOWS #load "app/screen_capture_win32.jai";
#if OS == .MACOS #load "app/screen_capture_macos.jai";
#if OS == .WINDOWS #load "app/screen_record_win32.jai";
#if OS == .MACOS #load "app/screen_record_macos.jai";
#load "base/circle_buffer.jai";
#load "base/debug.jai";
@ -78,7 +80,7 @@ Shared_Channel :: #import "Shared_Memory_Channel";
#import "Capture_Text";
#import "Base64";
#import "onnxruntime";
#if OS == .WINDOWS #import "Screen_Record"; // screen recording: Desktop Duplication or Windows.Graphics.Capture, to NVENC
#if OS == .WINDOWS || OS == .MACOS #import "Screen_Record"; // screen recording: Desktop Duplication or Windows.Graphics.Capture, to NVENC
// Basic's arenas import Secure_Memory at its defaults; this import adds its test suite.
using Secure_Memory :: #import "Secure_Memory"()(LOAD_TESTS=COMPILE_TESTS);
@ -199,7 +201,7 @@ App :: struct {
run_tests: bool;
headless: bool; // -headless (or a no_crash_dialog build): no message boxes on failures, and run_tests exits when done
exit_code: s32; // the process's exit code when the main loop ends
library_folder: string; // -library FOLDER: the .mcap folder the viewer shows, in place of VIEWER_FOLDER
library_folder: string; // the .mcap folder the viewer shows: -library FOLDER, else [library] folder, else LIBRARY_FOLDER_DEFAULT
scan_and_exit: bool; // -scan: scan the library with the chosen OCR model once it has loaded, then exit (0, or 1 if any capture failed)
stats_and_exit: bool; // -stats: log the Statistics window's numbers once the catalog has loaded, then exit
catalog_threads: s64; // -catalog-threads N: workers loading the catalog at once, in place of all of the background group's (for measuring)
@ -309,6 +311,14 @@ app_running_from_app_bundle :: () -> bool {
return app_from_app_bundle;
}
// The folder the models' folders (ocr-models, label-models, segment-models) are in: the bundle's
// Contents/Resources on macOS, else beside the executable. Ends in a separator.
app_resource_directory :: () -> string {
directory := path_strip_filename(get_path_of_running_executable());
#if OS == .MACOS if app_running_from_app_bundle() return tprint("%../Resources/", directory);
return directory;
}
#scope_file
app_install_directory_path: string;
app_from_app_bundle: bool;

6
src/app/import.jai
View File

@ -376,7 +376,7 @@ import_file :: (path: string) -> Import_Outcome, text: string, new_path: string
// A video: hashed and copied into its .mcap a few MB at a time, so it can be larger than memory.
import_video_file :: (path: string, name: string, head: [] u8, size: s64, modified: Apollo_Time) -> Import_Outcome, text: string, new_path: string {
if OS != .WINDOWS return .SKIPPED, "video (Windows only)", "";
if OS == .LINUX return .SKIPPED, "video (Windows and macOS only)", "";
hash, hashed := import_hash_file(.{path = path, offset = 0, size = size});
if !hashed return .FAILED, "cannot be read", "";
@ -1038,9 +1038,9 @@ import_window :: () { using ImGui;
import_source_changed();
}
SameLine();
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
if Button("Browse...") {
chosen := windows_choose_folder("Choose a folder to import");
chosen := platform_choose_folder("Choose a folder to import");
if chosen {
count := min(chosen.count, importer.source_buffer.count - 1);
memcpy(importer.source_buffer.data, chosen.data, count);

6
src/app/labels_ui.jai
View File

@ -53,7 +53,7 @@ labels_ui: struct {
};
labels_models_folder :: () -> string {
return tprint("%label-models", path_strip_filename(get_path_of_running_executable()));
return tprint("%label-models", app_resource_directory());
}
labels_list_path :: () -> string {
@ -244,7 +244,7 @@ labels_ui_task :: (task: Ocr_Task) {
engine, ok, error := labels_create(labels_models_folder(), LABELS_MODEL);
free(labels_ui.engine_error);
labels_ui.engine_error = ifx ok then "" else copy_string(error);
if ok log("[Labels] % on %.%", LABELS_MODEL, engine.provider, ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
if ok log("[Labels] % on %.%", LABELS_MODEL, labels_provider_name(engine), ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
else log_error("[Labels] Could not start %: %", LABELS_MODEL, error);
labels_ui.engine = engine;
labels_ui.engine_tried = true;
@ -273,7 +273,7 @@ labels_window_contents :: () { using ImGui;
TextWrapped("Run `jai fetch_models.jai` in modules/Labels, then rebuild.");
} else if labels_ui.engine {
in_use, explanation := labels_gpu_status(labels_ui.engine);
Text("%: images on %, text on the CPU", labels_ui.engine.model.title, ifx in_use then "the GPU (CUDA)" else "the CPU");
Text("%: images on %, text on the CPU", labels_ui.engine.model.title, ifx in_use then tprint("the GPU (%)", labels_provider_name(labels_ui.engine)) else "the CPU");
if explanation text_with_links(explanation);
} else if !labels_ui.engine_tried {
TextColored(app_theme.palette.dim, "Not loaded: it loads when a capture is read or labelled, or a like: search runs, and unloads after % minutes unused.", MODELS_IDLE_UNLOAD_SECONDS / 60);

9
src/app/main.jai
View File

@ -18,12 +18,7 @@ main :: () {
app.headless = array_find(app.args, "-headless") || NO_CRASH_DIALOG;
runtime_support_headless = app.headless;
app.allow_multiple_instances = array_find(app.args, "-allow-multiple-instances");
for app.args if it == "-library" && it_index + 1 < app.args.count {
// With / between its folders and none at the end, as the paths found in it are made.
app.library_folder = copy_string(app.args[it_index + 1]);
path_overwrite_separators(app.library_folder, #char "/");
while app.library_folder.count > 3 && app.library_folder[app.library_folder.count - 1] == #char "/" app.library_folder.count -= 1;
}
for app.args if it == "-library" && it_index + 1 < app.args.count app.library_folder = library_folder_normalize(app.args[it_index + 1]);
app.scan_and_exit = array_find(app.args, "-scan");
app.segment_and_exit = array_find(app.args, "-segment");
for app.args if it == "-record" && it_index + 1 < app.args.count app.record_and_exit = max(string_to_float64(app.args[it_index + 1]), 1.0);
@ -193,6 +188,8 @@ main :: () {
settings_write_back := app_load_user_settings_file() && !app.view_mode; // the running instance owns the file
logger_set_event_log_capacity(app_user_settings.event_log_capacity);
library_folder_choose();
log("Library: %", app.library_folder);
// #Startup:CreateWindow
#if USING_SDL {

12
src/app/ocr_ui.jai
View File

@ -106,7 +106,7 @@ Scan_Item :: struct {
}
ocr_models_folder :: () -> string {
return tprint("%ocr-models", path_strip_filename(get_path_of_running_executable()));
return tprint("%ocr-models", app_resource_directory());
}
// ---------------------------------------------------------------- startup and shutdown
@ -283,7 +283,7 @@ ocr_ui_thread_proc :: (thread: *Thread) -> s64 {
ocr_ui.engine = engine;
free(ocr_ui.engine_error);
ocr_ui.engine_error = ifx ok then "" else copy_string(error);
if ok log("[OCR] % on %.%", task.model, engine.provider, ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
if ok log("[OCR] % on %.%", task.model, ocr_provider_name(engine), ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
else log_error("[OCR] Could not start %: %", task.model, error);
ocr_ui.engine_tried = true;
@ -446,7 +446,7 @@ ocr_read_and_store :: (items: [] Scan_Item, set: *Label_Set) -> [] string {
vectors, label_error := labels_embed_images(labels, label_images,, temp);
if label_error for label_of if !errors[it] errors[it] = sprint("labelling failed: %", label_error);
dims := labels.dimensions;
label_engine := tprint("onnxruntime %, %", ort.version, ifx labels.provider == .CUDA then "CUDA" else "CPU");
label_engine := ifx labels.provider == .COREML then "Core ML, GPU" else tprint("onnxruntime %, %", ort.version, labels_provider_name(labels));
for n: 0..items.count - 1 {
if errors[n] continue;
vector: [] float;
@ -463,7 +463,7 @@ ocr_read_and_store :: (items: [] Scan_Item, set: *Label_Set) -> [] string {
}
ocr_results := ocr_process(ocr_ui.engine, ocr_images,, temp);
engine_text := tprint("onnxruntime %, %", ort.version, ifx ocr_ui.engine.provider == .CUDA then "CUDA" else "CPU");
engine_text := ifx ocr_ui.engine.provider == .COREML then "Core ML, GPU" else tprint("onnxruntime %, %", ort.version, ocr_provider_name(ocr_ui.engine));
for n: 0..items.count - 1 {
if errors[n] continue;
section: Mcap_Ocr_Section;
@ -715,7 +715,7 @@ ocr_scan_progress_ui :: () { using ImGui;
ProgressBar(cast(float) done / max(total, 1), .{-FLOAT32_MIN, 0}, temp_c_string(ocr_scan_progress_text(done, total)));
if scan.erase_ocr || scan.erase_labels || scan.erase_segments Text("% elapsed, % left, % captures/s, % failed", ocr_duration_text(elapsed), ocr_duration_text(left), FF(rate, 1), failed);
else if scan.segmenting Text("% elapsed, % left, % images/s, % failed", ocr_duration_text(elapsed), ocr_duration_text(left), FF(rate, 1), failed); // the engine is the OCR thread's while it runs
else Text("% elapsed, % left, % captures/s, % failed, on %", ocr_duration_text(elapsed), ocr_duration_text(left), FF(rate, 1), failed, ifx ocr_ui.engine && ocr_ui.engine.provider == .CUDA then "the GPU" else "the CPU");
else Text("% elapsed, % left, % captures/s, % failed, on %", ocr_duration_text(elapsed), ocr_duration_text(left), FF(rate, 1), failed, ifx ocr_ui.engine && ocr_gpu_status(ocr_ui.engine) then "the GPU" else "the CPU");
if scan.paused {
if Button("Resume") { scan.paused_seconds += to_float64_seconds(current_time_monotonic() - scan.pause_start); atomic_swap(*scan.paused, false); }
} else {
@ -811,7 +811,7 @@ ocr_ui_frame :: () { using ImGui;
ocr_scan_ui();
} else {
in_use, explanation := ocr_gpu_status(ocr_ui.engine);
Text("Running on: %", ifx in_use then "the GPU (CUDA)" else "the CPU");
Text("Running on: %", ifx in_use then tprint("the GPU (%)", ocr_provider_name(ocr_ui.engine)) else "the CPU");
SameLine();
TextColored(app_theme.palette.dim, "onnxruntime %, % worker threads", ort.version, ocr_ui.engine.workers);
Separator();

10
src/app/open_file.jai
View File

@ -183,7 +183,7 @@ open_file_list_folders :: () {
array_reset_keeping_memory(*open_file.folders);
visit_files(viewer_folder(), false, *open_file.folders, (info: *File_Visit_Info, folders: *[..] string) {
if !info.is_directory || !import_name_is_valid(info.short_name) return; // not hidden ones (.thumbnails)
#if OS != .WINDOWS if info.short_name == "deleted" return; // where deleted captures go there
#if OS == .LINUX if info.short_name == "deleted" return; // where deleted captures go there
array_add(folders, copy_string(info.short_name,, context.default_allocator));
}, visit_files = false, visit_directories = true);
quick_sort(open_file.folders, compare);
@ -254,7 +254,7 @@ open_file_proc :: (thread: *Thread) -> s64 {
}
case .VIDEO;
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
image, video, ok, poster_error := video_poster(path);
if !ok { error = poster_error; }
else {
@ -265,7 +265,7 @@ open_file_proc :: (thread: *Thread) -> s64 {
}
to_hash = .{path = path, offset = 0, size = open_file.size};
} else {
error = "videos are read on Windows only";
error = "videos are read on Windows and macOS only";
}
case;
@ -359,7 +359,7 @@ open_file_copy_path :: () {
}
open_file_copy_image :: () {
#if OS != .WINDOWS {
#if OS == .LINUX {
toast_show(.Error, "Copying images is not available on this system yet");
return;
}
@ -492,7 +492,7 @@ open_file_library_ui :: (stage: Open_File_Stage) { using ImGui;
return;
}
if open_file.kind == .UNKNOWN { TextColored(app_theme.palette.warning, "mcap does not import this kind of file."); return; }
if OS != .WINDOWS && open_file.kind == .VIDEO { TextColored(app_theme.palette.warning, "Videos are imported on Windows only."); return; }
if OS == .LINUX && open_file.kind == .VIDEO { TextColored(app_theme.palette.warning, "Videos are imported on Windows and macOS only."); return; }
if open_file.importing {
if importer.phase == .INDEXING Text("Indexing the library (once; this finds duplicates): %", ocr_scan_progress_text(importer.index_done, importer.index_total));

8
src/app/screen_capture.jai
View File

@ -1,6 +1,6 @@
// Screen capture, as Snagit does it: PrintScreen (the capture_region global hotkey, from any app,
// even with mcap hidden in the tray) freezes every screen, and a rectangle dragged over them is taken
// (app/screen_capture_win32.jai; Windows only so far). What is taken goes onto the clipboard at once,
// Screen capture, as Snagit does it: PrintScreen on Windows, Cmd+Shift+2 on macOS (the capture_region
// global hotkey, from any app, even with mcap hidden in the tray) freezes every screen, and a rectangle
// dragged over them is taken (app/screen_capture_win32.jai, app/screen_capture_macos.jai). What is taken goes onto the clipboard at once,
// and into the library as an .mcap in the folder SCREEN_CAPTURE_FOLDER: the pixels as a PNG, with
// the window it was taken from in metadata.json, its title as WindowName and its program as AppName,
// as Snagit records them, so searches find it the same way. mcap then comes to the front showing
@ -219,7 +219,7 @@ screen_capture_out_path :: (name: string) -> path: string, error: string {
return path, "";
}
#if OS != .WINDOWS {
#if OS == .LINUX {
Screen_Shot :: struct {
pixels: [] u8;
width, height: s32;

827
src/app/screen_capture_macos.jai Normal file
View File

@ -0,0 +1,827 @@
// The screen capture overlay on macOS (app/screen_capture.jai is the rest; the Windows one is
// screen_capture_win32.jai, which this follows). The moment the hotkey fires, every screen is copied
// with ScreenCaptureKit (modules/Screen_Record/macos/capture.m), and each is shown frozen and darkened
// in a window of its own: a window cannot span screens on macOS, as the one Windows overlay does. A
// rectangle dragged shows at full brightness, and letting go takes it; a click takes the whole screen
// the cursor is on. A magnifier beside the cursor shows the pixels under it, with the position or the
// selection's size, so a region can be chosen to the pixel; the arrow keys move the cursor a pixel at
// a time (ten with Shift). Escape or a right click cancels; Return takes what is dragged.
//
// For a recording (video), the outline and magnifier are red, and a region smaller than
// RECORD_MIN_WIDTH x RECORD_MIN_HEIGHT is outlined dashed, with the minimum in the label, and letting
// go of it takes nothing: the overlay stays to drag again. No pixels are copied out for a recording,
// which app/screen_record.jai has ScreenCaptureKit stream from the display and region the shot names.
//
// A drag stays on the screen it began on: its pixels are that screen's, two to a point on a Retina
// screen. Everything is worked out in each screen's own pixels, which the cursor's position in points
// is turned into.
//
// The overlay windows are non-activating panels at the overlay level, on every Space and over
// full-screen apps, as Ex4's overlays are (Ex4-Jai/src/base/platform/macos_overlay.jai): the app you
// were in stays the active one. They are drawn with Core Graphics, only where something changed, and
// run in an event loop of their own: platform_screen_capture_select returns once a region is taken or
// the capture is cancelled.
//
// Needs the Screen Recording permission: without it, macOS gives only the desktop, so mcap asks for
// it and opens its pane in System Settings instead. The window it came from is found in the window
// server's list, whose titles need that permission too.
// The region taken, and the window it was taken from.
Screen_Shot :: struct {
pixels: [] u8; // RGBA, rows packed; heap
width, height: s32; // in pixels
left, top: s32; // in points, from the main screen's top left
whole_screen: bool; // a click took the screen under the cursor
window_title: string; // the window under the region's middle; heap, as are the two below
app_name: string; // its app ("Safari")
app_path: string; // its app's bundle, or executable
// The display it is on, and the region in that display's points from its top left: what a
// recording streams (modules/Screen_Record).
display_id: u32;
scale: float64; // its pixels a point
local_x, local_y, local_width, local_height: float64;
screen_left, screen_top, screen_width, screen_height: float64; // the display, in points from the main screen's top left
}
screen_shot_free :: (shot: *Screen_Shot) {
push_allocator(context.default_allocator);
array_free(shot.pixels);
free(shot.window_title);
free(shot.app_name);
free(shot.app_path);
shot.* = .{};
}
// Freezes the screens and lets a region be chosen: for a still, or with video for a recording. With
// immediate, no overlay: the whole screen under the cursor is taken at once. ok is false when it was
// cancelled (error "") or could not start (error says why).
platform_screen_capture_select :: (video := false, immediate := false) -> Screen_Shot, ok: bool, error: string {
shot: Screen_Shot;
init_objective_c();
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
if !CGPreflightScreenCaptureAccess() {
// Asked for once (macOS shows its prompt the first time only), and its pane opened, as Ex4 does.
CGRequestScreenCaptureAccess();
run_command("open", SCREEN_RECORDING_PANE);
return shot, false, "mcap needs the Screen Recording permission: turn it on for mcap in System Settings > Privacy & Security > Screen Recording, then capture again";
}
// The app in front, before the overlay is up: what the capture is of when no window is found.
front_name, front_path := frontmost_app();
ok, error := capture_freeze();
defer capture_release();
if !ok return shot, false, error;
capture.video = video;
if immediate {
x, y := capture_cursor_points();
capture.result_screen = capture_screen_at(x, y);
capture.result = .{0, 0, capture.screens[capture.result_screen].pixel_width, capture.screens[capture.result_screen].pixel_height};
capture.whole_screen = true;
} else {
if !capture_open() return shot, false, "the capture windows could not be opened";
capture_choose();
capture_close();
if capture.cancelled return shot, false, "";
}
screen := *capture.screens[capture.result_screen];
r := capture.result;
shot.width = r.right - r.left;
shot.height = r.bottom - r.top;
if shot.width <= 0 || shot.height <= 0 return shot, false, "";
shot.left = cast(s32) (screen.left + r.left / screen.scale);
shot.top = cast(s32) (screen.top + r.top / screen.scale);
shot.whole_screen = capture.whole_screen;
shot.display_id = screen.display_id;
shot.scale = screen.scale;
shot.screen_left, shot.screen_top, shot.screen_width, shot.screen_height = screen.left, screen.top, screen.width, screen.height;
shot.local_x, shot.local_y = r.left / screen.scale, r.top / screen.scale;
shot.local_width, shot.local_height = (r.right - r.left) / screen.scale, (r.bottom - r.top) / screen.scale;
if !video {
shot.pixels = NewArray(cast(s64) shot.width * shot.height * 4, u8, initialized = false,, context.default_allocator);
row_bytes := cast(s64) shot.width * 4;
for y: 0..shot.height - 1 {
from := screen.pixels + (cast(s64) (r.top + y) * screen.pixel_width + r.left) * 4;
memcpy(shot.pixels.data + y * row_bytes, from, row_bytes);
}
}
// The window under the region's middle, now that the overlay is gone; else the app that was in front.
center_x := screen.left + (r.left + r.right) / 2.0 / screen.scale;
center_y := screen.top + (r.top + r.bottom) / 2.0 / screen.scale;
window_info(*shot, center_x, center_y, front_name, front_path);
return shot, true, "";
}
SCREEN_RECORDING_PANE :: "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture";
#scope_file
Capture_Screen :: struct {
left, top, width, height: float64; // in points, from the main screen's top left
scale: float64; // pixels a point
display_id: u32;
pixel_width, pixel_height: s32;
pixels: *u8; // RGBA, from the shim (mcap_sck_free)
image: CGImageRef; // over pixels
window: *NSWindow;
view: *NSView;
dirty: [..] CGRect; // what is drawn now that is not the plain dimmed screen, in its points
}
Pixel :: struct { x, y: s32; }
Pixel_Rect :: struct { left, top, right, bottom: s32; }
capture: struct {
screens: [..] Capture_Screen;
video: bool; // choosing a region to record
cursor_screen: s64; // the screen the cursor is on, and its pixel there
cursor: Pixel;
dragging: bool;
anchor_screen: s64; // where the drag began
anchor: Pixel;
done, cancelled: bool;
result_screen: s64;
result: Pixel_Rect; // in that screen's pixels
whole_screen: bool;
}
MAGNIFIER_PIXELS :: 21; // the pixels it shows across: odd, so one is in the middle
MAGNIFIER_ZOOM :: 6.0; // points each is shown as
MAGNIFIER_OFFSET :: 24.0; // points from the cursor
BORDER :: 2.0; // the selection's outline, in points
CLICK_SLOP :: 3; // a drag smaller than this either way, in pixels, is a click
DIM :: 0.5; // how dark the frozen screens are
ACCENT :: Vector4.{1, 0.647, 0, 1}; // orange, as on Windows
RECORD :: Vector4.{0.91, 0.157, 0.157, 1}; // red: choosing a region to record
DASH : float64 : 6; // a too-small region's dashes and gaps, in points
GAP : float64 : 4;
LABEL :: Vector4.{0.125, 0.125, 0.125, 0.92};
TEXT :: Vector4.{1, 1, 1, 1};
// Every screen: its place and size, then its pixels, all copied at once.
capture_freeze :: () -> ok: bool, error: string {
screens := objc_msgSend_typed(objc_getClass("NSScreen"), selector("screens"));
count := ifx screens then cast(s64) objc_msgSend_typed(screens, selector("count")) else 0;
if !count return false, "there are no screens";
main_height: float64;
capture.screens.allocator = context.default_allocator;
for 0..count - 1 {
ns := msg_object_at(screens, it);
frame := msg_rect(ns, "frame");
if it == 0 main_height = frame.size.height;
s := array_add(*capture.screens);
s.left = frame.origin.x;
s.top = main_height - frame.origin.y - frame.size.height;
s.width = frame.size.width;
s.height = frame.size.height;
s.scale = msg_float(ns, "backingScaleFactor");
s.pixel_width = cast(s32) (s.width * s.scale + 0.5);
s.pixel_height = cast(s32) (s.height * s.scale + 0.5);
description := objc_msgSend_typed(ns, selector("deviceDescription"));
number := msg_object_for_key(description, "NSScreenNumber");
s.display_id = cast(u32) msg_integer(number, "unsignedIntValue");
}
shots := NewArray(capture.screens.count, Mcap_Sck_Shot,, temp);
for capture.screens shots[it_index] = .{display_id = it.display_id, width = it.pixel_width, height = it.pixel_height};
error_text: [512] u8;
copied := mcap_sck_screenshots(shots.data, xx shots.count, error_text.data, error_text.count);
for * capture.screens {
it.pixels = shots[it_index].pixels;
if it.pixels { it.pixel_width, it.pixel_height = shots[it_index].width, shots[it_index].height; }
}
if !copied return false, tprint("the screens could not be copied: %", to_string(error_text.data));
srgb := CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
defer CGColorSpaceRelease(srgb);
for * capture.screens {
provider := CGDataProviderCreateWithData(null, it.pixels, cast(u64) it.pixel_width * cast(u64) it.pixel_height * 4, null);
it.image = CGImageCreate(xx it.pixel_width, xx it.pixel_height, 8, 32, cast(u64) it.pixel_width * 4, srgb, kCGImageAlphaNoneSkipLast | kCGBitmapByteOrder32Big, provider, null, false, kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
if !it.image return false, "the copied screens could not be shown";
}
return true, "";
}
capture_release :: () {
for * capture.screens {
if it.image CGImageRelease(it.image);
if it.pixels mcap_sck_free(it.pixels);
array_free(it.dirty);
}
array_free(capture.screens);
capture = .{};
}
// A window over each screen, the one under the cursor taking the keys.
capture_open :: () -> bool {
view_class, panel_class := capture_classes();
if !view_class || !panel_class return false;
init_core_graphics_runtime_constants();
capture.cursor_screen = -1;
capture_follow_cursor();
for * screen, index: capture.screens {
main_height := capture.screens[0].height;
frame := NSMakeRect(screen.left, main_height - screen.top - screen.height, screen.width, screen.height);
NSWindowStyleMaskNonactivatingPanel :: 1 << 7;
window := NSWindow.initWithContentRect(cast(*NSWindow) objc_alloc(panel_class), frame, NSWindowStyleMaskBorderless | NSWindowStyleMaskNonactivatingPanel, NSBackingStoreBuffered, NO);
if !window return false;
NSWindow.setReleasedWhenClosed(window, NO);
msg_bool(window, "setHasShadow:", false);
msg_bool(window, "setOpaque:", true);
NSWindow.setLevel(window, kCGOverlayWindowLevel);
NSWindow.setCollectionBehavior(window, NSWindowCollectionBehaviorCanJoinAllSpaces | NSWindowCollectionBehaviorStationary |
NSWindowCollectionBehaviorFullScreenAuxiliary | NSWindowCollectionBehaviorIgnoresCycle);
NSWindowAnimationBehaviorNone :: 2;
msg_integer_set(window, "setAnimationBehavior:", NSWindowAnimationBehaviorNone);
view := cast(*NSView) objc_init_with_frame(objc_alloc(view_class), NSMakeRect(0, 0, screen.width, screen.height));
(cast(*s64) object_getIndexedIvars(view)).* = index;
NSWindow.setContentView(window, view);
release(view);
screen.window, screen.view = window, view;
NSWindow.display(window); // drawn before it shows, so its first frame is not an empty one
objc_msgSend_typed(window, selector("orderFrontRegardless"));
}
objc_msgSend_typed(capture.screens[max(capture.cursor_screen, 0)].window, selector("makeKeyWindow"));
capture_update();
return true;
}
capture_close :: () {
for * capture.screens if it.window {
NSWindow.orderOut(it.window, null);
NSWindow.close(it.window);
release(it.window);
it.window, it.view = null;
}
objc_msgSend_typed(objc_msgSend_typed(objc_getClass("NSCursor"), selector("arrowCursor")), selector("set"));
}
// The overlay's own event loop: the cursor followed, clicks and keys taken, until it is done.
capture_choose :: () {
application := NSApplication.sharedApplication();
crosshair := objc_msgSend_typed(objc_getClass("NSCursor"), selector("crosshairCursor"));
while !capture.done {
objc_msgSend_typed(crosshair, selector("set")); // another app's cursor rects may have set theirs
func_date :: #type (*void, Selector, float64) -> *NSObject #c_call;
until := (cast(func_date) objc_msgSend)(objc_getClass("NSDate"), selector("dateWithTimeIntervalSinceNow:"), 1.0 / 120);
event := NSApplication.nextEventMatchingMask(application, NSEventMaskAny, xx until, NSDefaultRunLoopMode, YES);
if event {
type := NSEvent.type(event);
if type == {
case NSEventTypeLeftMouseDown;
capture_follow_cursor();
capture.dragging = true;
capture.anchor_screen = capture.cursor_screen;
capture.anchor = capture.cursor;
capture_update();
case NSEventTypeLeftMouseUp;
if capture.dragging { capture_follow_cursor(); capture_take(); }
case NSEventTypeRightMouseDown;
if capture.dragging { capture.dragging = false; capture_update(); } // let go, and start again
else capture_cancel();
case NSEventTypeKeyDown;
capture_key(NSEvent.keyCode(event), (NSEvent.modifierFlags(event) & NSEventModifierFlagShift) != 0);
case NSEventTypeLeftMouseDragged; #through;
case NSEventTypeMouseMoved; #through;
case NSEventTypeKeyUp; #through;
case NSEventTypeFlagsChanged; #through;
case NSEventTypeRightMouseUp;
// Taken here, not handed on: the windows have nothing to do with them.
case;
NSApplication.sendEvent(application, event);
}
}
if capture_follow_cursor() capture_update();
}
}
capture_cancel :: () {
capture.cancelled = true;
capture.done = true;
}
// Escape cancels, Return takes the drag (or the screen), the arrows move the cursor a pixel.
capture_key :: (key_code: u16, shift: bool) {
KEY_RETURN :: 36; KEY_ENTER :: 76; KEY_ESCAPE :: 53;
KEY_LEFT :: 123; KEY_RIGHT :: 124; KEY_DOWN :: 125; KEY_UP :: 126;
step: s32 = 1;
if shift step = 10;
dx, dy: s32;
if key_code == {
case KEY_ESCAPE; capture_cancel(); return;
case KEY_RETURN; #through;
case KEY_ENTER; capture_take(); return;
case KEY_LEFT; dx = -step;
case KEY_RIGHT; dx = step;
case KEY_UP; dy = -step;
case KEY_DOWN; dy = step;
case; return;
}
// The cursor moved to the middle of the pixel; following it does the rest.
screen := *capture.screens[capture.cursor_screen];
x := clamp(capture.cursor.x + dx, 0, screen.pixel_width - 1);
y := clamp(capture.cursor.y + dy, 0, screen.pixel_height - 1);
CGWarpMouseCursorPosition(.{screen.left + (x + 0.5) / screen.scale, screen.top + (y + 0.5) / screen.scale});
CGAssociateMouseAndMouseCursorPosition(1);
}
// Where the cursor is now, in points from the main screen's top left.
capture_cursor_points :: () -> float64, float64 {
location := NSEvent.mouseLocation();
return location.x, capture.screens[0].height - location.y;
}
capture_screen_at :: (x: float64, y: float64) -> s64 {
for capture.screens if x >= it.left && x < it.left + it.width && y >= it.top && y < it.top + it.height return it_index;
return 0;
}
// The cursor's screen and pixel, from where it is; true when either changed.
capture_follow_cursor :: () -> bool {
x, y := capture_cursor_points();
index := capture_screen_at(x, y);
screen := *capture.screens[index];
pixel := Pixel.{clamp(cast(s32) floor((x - screen.left) * screen.scale), 0, screen.pixel_width - 1),
clamp(cast(s32) floor((y - screen.top) * screen.scale), 0, screen.pixel_height - 1)};
changed := index != capture.cursor_screen || pixel.x != capture.cursor.x || pixel.y != capture.cursor.y;
capture.cursor_screen, capture.cursor = index, pixel;
return changed;
}
// From the anchor to the cursor, both pixels in it, on the anchor's screen.
capture_selection :: () -> Pixel_Rect {
screen := *capture.screens[capture.anchor_screen];
c := capture.cursor;
if capture.cursor_screen != capture.anchor_screen {
// The cursor on another screen: where it would be on this one, kept to its edges.
other := *capture.screens[capture.cursor_screen];
gx := other.left + capture.cursor.x / other.scale;
gy := other.top + capture.cursor.y / other.scale;
c = .{cast(s32) floor((gx - screen.left) * screen.scale), cast(s32) floor((gy - screen.top) * screen.scale)};
}
c.x = clamp(c.x, 0, screen.pixel_width - 1);
c.y = clamp(c.y, 0, screen.pixel_height - 1);
a := capture.anchor;
return .{min(a.x, c.x), min(a.y, c.y), max(a.x, c.x) + 1, max(a.y, c.y) + 1};
}
// The selection, or on a click (or Return without a drag) the screen under the cursor.
capture_take :: () {
if capture.dragging {
selection := capture_selection();
if selection.right - selection.left >= CLICK_SLOP || selection.bottom - selection.top >= CLICK_SLOP {
if capture.video && !record_size_allowed(selection.right - selection.left, selection.bottom - selection.top) {
capture.dragging = false; // too small to record: nothing taken, the overlay stays
capture_update();
return;
}
capture.result_screen = capture.anchor_screen;
capture.result = selection;
capture.dragging = false;
capture.done = true;
return;
}
}
capture.dragging = false;
capture.result_screen = capture.cursor_screen;
screen := capture.screens[capture.cursor_screen];
capture.result = .{0, 0, screen.pixel_width, screen.pixel_height};
capture.whole_screen = true;
capture.done = true;
}
// What each screen draws now beyond the dimmed copy, and which parts of it to draw again: where they
// were and where they are.
capture_update :: () {
for * screen, index: capture.screens {
if !screen.view continue;
now: [..] CGRect;
now.allocator = context.default_allocator;
for capture_parts(index) array_add(*now, it);
for screen.dirty NSView_setNeedsDisplayInRect(screen.view, it);
for now NSView_setNeedsDisplayInRect(screen.view, it);
array_free(screen.dirty);
screen.dirty = now;
}
}
// The parts of screen index that differ from the dimmed copy, in its points: the selection and its
// outline, or the crosshair; and the magnifier and its label.
capture_parts :: (index: s64) -> [] CGRect {
parts: [..] CGRect;
parts.allocator = temp;
screen := *capture.screens[index];
if capture.dragging && index == capture.anchor_screen {
s := points_of(screen, capture_selection());
array_add(*parts, .{.{s.origin.x - BORDER, s.origin.y - BORDER}, .{s.size.width + 2 * BORDER, s.size.height + 2 * BORDER}});
}
if index == capture.cursor_screen {
if !capture.dragging {
x, y := capture.cursor.x / screen.scale, capture.cursor.y / screen.scale;
array_add(*parts, .{.{0, y}, .{screen.width, 1 / screen.scale}});
array_add(*parts, .{.{x, 0}, .{1 / screen.scale, screen.height}});
}
magnifier, label := magnifier_layout(screen);
array_add(*parts, .{.{magnifier.origin.x - 1, magnifier.origin.y - 1}, .{magnifier.size.width + 2, magnifier.size.height + 2}});
array_add(*parts, label);
}
return parts;
}
points_of :: (screen: *Capture_Screen, r: Pixel_Rect) -> CGRect {
return .{.{r.left / screen.scale, r.top / screen.scale}, .{(r.right - r.left) / screen.scale, (r.bottom - r.top) / screen.scale}};
}
// The magnifier below and right of the cursor, or on the other side where the screen ends, and the
// label under it.
magnifier_layout :: (screen: *Capture_Screen) -> magnifier: CGRect, label: CGRect {
side: float64 = MAGNIFIER_PIXELS * MAGNIFIER_ZOOM + 2;
label_height: float64 = 22;
label_width := max(side, text_size(label_text(screen)).width + 12);
x := capture.cursor.x / screen.scale + MAGNIFIER_OFFSET;
y := capture.cursor.y / screen.scale + MAGNIFIER_OFFSET;
if x + max(side, label_width) > screen.width x = capture.cursor.x / screen.scale - MAGNIFIER_OFFSET - max(side, label_width);
if y + side + label_height > screen.height y = capture.cursor.y / screen.scale - MAGNIFIER_OFFSET - side - label_height;
return .{.{x, y}, .{side, side}}, .{.{x, y + side}, .{label_width, label_height}};
}
// The position (in the screen's pixels), or the selection's size.
label_text :: (screen: *Capture_Screen) -> string {
if capture.dragging {
s := capture_selection();
text := tprint("% × %", s.right - s.left, s.bottom - s.top);
if capture_too_small(s) text = tprint("% · at least % × %", text, RECORD_MIN_WIDTH, RECORD_MIN_HEIGHT);
return text;
}
return tprint("%, %", capture.cursor.x, capture.cursor.y);
}
// To record, smaller than the minimum (and not a click, which takes the whole screen).
capture_too_small :: (s: Pixel_Rect) -> bool {
w, h := s.right - s.left, s.bottom - s.top;
if !capture.video || (w < CLICK_SLOP && h < CLICK_SLOP) return false;
return !record_size_allowed(w, h);
}
// drawRect: for screen index's view, in its points (flipped: y goes down).
capture_draw :: (index: s64, cg: CGContextRef) {
screen := *capture.screens[index];
accent := ifx capture.video then RECORD else ACCENT;
whole := CGRect.{.{0, 0}, .{screen.width, screen.height}};
draw_image(cg, screen.image, whole);
CGContextSetRGBFillColor(cg, 0, 0, 0, DIM);
CGContextFillRect(cg, whole);
fill :: (cg: CGContextRef, r: CGRect, color: Vector4) {
CGContextSetRGBFillColor(cg, color.x, color.y, color.z, color.w);
CGContextFillRect(cg, r);
}
if capture.dragging && index == capture.anchor_screen {
s := points_of(screen, capture_selection());
CGContextSaveGState(cg);
CGContextClipToRect(cg, s);
draw_image(cg, screen.image, whole);
CGContextRestoreGState(cg);
// The outline just outside it, so every pixel taken shows as it is; dashed when too small to record.
dashed := capture_too_small(capture_selection());
outline := CGRect.[
.{.{s.origin.x - BORDER, s.origin.y - BORDER}, .{s.size.width + 2 * BORDER, BORDER}},
.{.{s.origin.x - BORDER, s.origin.y + s.size.height}, .{s.size.width + 2 * BORDER, BORDER}},
.{.{s.origin.x - BORDER, s.origin.y}, .{BORDER, s.size.height}},
.{.{s.origin.x + s.size.width, s.origin.y}, .{BORDER, s.size.height}},
];
for outline {
if !dashed { fill(cg, it, accent); continue; }
across := it.size.width > it.size.height;
start := ifx across then it.origin.x else it.origin.y;
end := start + (ifx across then it.size.width else it.size.height);
position := start;
while position < end {
piece := it;
if across { piece.origin.x = position; piece.size.width = min(DASH, end - position); }
else { piece.origin.y = position; piece.size.height = min(DASH, end - position); }
fill(cg, piece, accent);
position += DASH + GAP;
}
}
}
if index != capture.cursor_screen return;
pixel := 1 / screen.scale;
cx, cy := capture.cursor.x * pixel, capture.cursor.y * pixel;
if !capture.dragging {
fill(cg, .{.{0, cy}, .{screen.width, pixel}}, accent);
fill(cg, .{.{cx, 0}, .{pixel, screen.height}}, accent);
}
// The magnifier: the pixels around the cursor, as copied, each MAGNIFIER_ZOOM points wide; past
// the screen's edges it is black.
magnifier, label := magnifier_layout(screen);
fill(cg, magnifier, .{1, 1, 1, 1});
inner := CGRect.{.{magnifier.origin.x + 1, magnifier.origin.y + 1}, .{magnifier.size.width - 2, magnifier.size.height - 2}};
fill(cg, inner, .{0, 0, 0, 1});
half :: MAGNIFIER_PIXELS / 2;
left := max(capture.cursor.x - half, 0);
top := max(capture.cursor.y - half, 0);
right := min(capture.cursor.x + half + 1, screen.pixel_width);
bottom := min(capture.cursor.y + half + 1, screen.pixel_height);
if right > left && bottom > top {
part := CGImageCreateWithImageInRect(screen.image, .{.{xx left, xx top}, .{xx (right - left), xx (bottom - top)}});
if part {
CGContextSetInterpolationQuality(cg, kCGInterpolationNone);
draw_image(cg, part, .{.{inner.origin.x + (left - (capture.cursor.x - half)) * MAGNIFIER_ZOOM, inner.origin.y + (top - (capture.cursor.y - half)) * MAGNIFIER_ZOOM},
.{(right - left) * MAGNIFIER_ZOOM, (bottom - top) * MAGNIFIER_ZOOM}});
CGImageRelease(part);
}
}
// The pixel under the cursor, outlined.
c := CGRect.{.{inner.origin.x + half * MAGNIFIER_ZOOM, inner.origin.y + half * MAGNIFIER_ZOOM}, .{MAGNIFIER_ZOOM, MAGNIFIER_ZOOM}};
fill(cg, .{.{c.origin.x - 1, c.origin.y - 1}, .{c.size.width + 2, 1}}, accent);
fill(cg, .{.{c.origin.x - 1, c.origin.y + c.size.height}, .{c.size.width + 2, 1}}, accent);
fill(cg, .{.{c.origin.x - 1, c.origin.y}, .{1, c.size.height}}, accent);
fill(cg, .{.{c.origin.x + c.size.width, c.origin.y}, .{1, c.size.height}}, accent);
// Under it, the position, or the selection's size.
fill(cg, label, LABEL);
draw_text(label_text(screen), label.origin.x + 6, label.origin.y + 3);
}
// image upright in rect, in a flipped context.
draw_image :: (cg: CGContextRef, image: CGImageRef, rect: CGRect) {
CGContextSaveGState(cg);
CGContextTranslateCTM(cg, rect.origin.x, rect.origin.y + rect.size.height);
CGContextScaleCTM(cg, 1, -1);
CGContextDrawImage(cg, .{.{0, 0}, rect.size}, image);
CGContextRestoreGState(cg);
}
// The label's font: the system's, semibold, 13 points, white.
text_attributes :: () -> *NSObject {
func_font :: #type (*void, Selector, CGFloat, CGFloat) -> *NSObject #c_call;
font := (cast(func_font) objc_msgSend)(objc_getClass("NSFont"), selector("monospacedDigitSystemFontOfSize:weight:"), 13, 0.3);
func_color :: #type (*void, Selector, CGFloat, CGFloat, CGFloat, CGFloat) -> *NSObject #c_call;
color := (cast(func_color) objc_msgSend)(objc_getClass("NSColor"), selector("colorWithSRGBRed:green:blue:alpha:"), TEXT.x, TEXT.y, TEXT.z, TEXT.w);
keys := *void.[NSFontAttributeName, NSForegroundColorAttributeName];
values := *void.[font, color];
func_dictionary :: #type (*void, Selector, **void, **void, NSUInteger) -> *NSObject #c_call;
return (cast(func_dictionary) objc_msgSend)(objc_getClass("NSDictionary"), selector("dictionaryWithObjects:forKeys:count:"), values.data, keys.data, 2);
}
text_size :: (text: string) -> NSSize {
func :: #type (*void, Selector, *NSObject) -> NSSize #c_call;
return (cast(func) objc_msgSend)(NSString.getTempString(text), selector("sizeWithAttributes:"), text_attributes());
}
draw_text :: (text: string, x: float64, y: float64) {
func :: #type (*void, Selector, NSPoint, *NSObject) -> void #c_call;
(cast(func) objc_msgSend)(NSString.getTempString(text), selector("drawAtPoint:withAttributes:"), NSMakePoint(x, y), text_attributes());
}
// ---------------------------------------------------------------- the window it came from
// The front-most window, other than mcap's, whose bounds hold the point; its title and app. Else the
// app that was in front.
window_info :: (shot: *Screen_Shot, x: float64, y: float64, front_name: string, front_path: string) {
push_allocator(context.default_allocator);
windows := CGWindowListCopyWindowInfo(kCGWindowListOptionOnScreenOnly | kCGWindowListExcludeDesktopElements, 0);
own := getpid();
found := false;
if windows {
defer CFRelease(windows);
for 0..CFArrayGetCount(windows) - 1 {
info := CFArrayGetValueAtIndex(windows, it);
layer := cf_integer(CFDictionaryGetValue(info, kCGWindowLayer));
pid := cf_integer(CFDictionaryGetValue(info, kCGWindowOwnerPID));
if layer != 0 || pid == own continue; // the menu bar, the Dock, overlays; and mcap's own
bounds: CGRect;
if !CGRectMakeWithDictionaryRepresentation(CFDictionaryGetValue(info, kCGWindowBounds), *bounds) continue;
if x < bounds.origin.x || y < bounds.origin.y || x >= bounds.origin.x + bounds.size.width || y >= bounds.origin.y + bounds.size.height continue;
shot.window_title = copy_string(cf_string(CFDictionaryGetValue(info, kCGWindowName)));
name, path := app_of(cast(s32) pid);
if !name name = cf_string(CFDictionaryGetValue(info, kCGWindowOwnerName));
shot.app_name = copy_string(name);
shot.app_path = copy_string(path);
found = true;
break;
}
}
if !found {
shot.app_name = copy_string(front_name);
shot.app_path = copy_string(front_path);
}
}
// The app in front: its name and bundle. Temp.
frontmost_app :: () -> name: string, path: string {
workspace := objc_msgSend_typed(objc_getClass("NSWorkspace"), selector("sharedWorkspace"));
app := objc_msgSend_typed(workspace, selector("frontmostApplication"));
name, path := app_name_and_path(app);
return name, path;
}
app_of :: (pid: s32) -> name: string, path: string {
func :: #type (*void, Selector, s32) -> *NSObject #c_call;
app := (cast(func) objc_msgSend)(objc_getClass("NSRunningApplication"), selector("runningApplicationWithProcessIdentifier:"), pid);
name, path := app_name_and_path(app);
return name, path;
}
app_name_and_path :: (app: *void) -> name: string, path: string {
if !app return "", "";
name := ns_string(objc_msgSend_typed(app, selector("localizedName")));
url := objc_msgSend_typed(app, selector("bundleURL"));
if !url url = objc_msgSend_typed(app, selector("executableURL"));
path := ifx url then ns_string(objc_msgSend_typed(url, selector("path"))) else "";
return name, path;
}
ns_string :: (s: *void) -> string {
if !s return "";
return copy_string(to_string(cast(*NSString) s),, temp);
}
cf_integer :: (number: *void) -> s64 {
value: s64;
if number CFNumberGetValue(number, kCFNumberSInt64Type, *value);
return value;
}
cf_string :: (s: *void) -> string {
if !s return "";
buffer: [1024] u8;
if !CFStringGetCString(s, buffer.data, buffer.count, kCFStringEncodingUTF8) return "";
return copy_string(to_string(buffer.data),, temp);
}
// ---------------------------------------------------------------- the classes, and messages
// McapCaptureView, an NSView whose drawRect: draws its screen and whose y goes down, its extra bytes
// its screen's index; and McapCapturePanel, an NSPanel that can be the key window with no title bar.
capture_classes :: () -> view: Class, panel: Class {
if view_class && panel_class return view_class, panel_class;
view_class = objc_getClass("McapCaptureView");
if !view_class {
view_class = objc_allocateClassPair(objc_getClass("NSView"), "McapCaptureView", size_of(s64));
if !view_class return null, null;
draw_rect :: (self: id, _sel: Selector, dirty: NSRect) #c_call {
push_context capture_context {
index := (cast(*s64) object_getIndexedIvars(self)).*;
if index < 0 || index >= capture.screens.count return;
context_object := objc_msgSend_typed(objc_getClass("NSGraphicsContext"), selector("currentContext"));
cg := ifx context_object then cast(CGContextRef) objc_msgSend_typed(context_object, selector("CGContext")) else null;
if cg capture_draw(index, cg);
}
}
yes :: (self: id, _sel: Selector) -> BOOL #c_call { return YES; }
yes_for_event :: (self: id, _sel: Selector, event: *NSEvent) -> BOOL #c_call { return YES; }
class_addMethod(view_class, selector("drawRect:"), xx draw_rect, "v@:{CGRect={CGPoint=dd}{CGSize=dd}}");
class_addMethod(view_class, selector("isFlipped"), xx yes, "c@:");
class_addMethod(view_class, selector("isOpaque"), xx yes, "c@:");
class_addMethod(view_class, selector("acceptsFirstMouse:"), xx yes_for_event, "c@:@");
objc_registerClassPair(view_class);
}
panel_class = objc_getClass("McapCapturePanel");
if !panel_class {
panel_class = objc_allocateClassPair(objc_getClass("NSPanel"), "McapCapturePanel", 0);
if !panel_class return null, null;
yes :: (self: id, _sel: Selector) -> BOOL #c_call { return YES; }
no :: (self: id, _sel: Selector) -> BOOL #c_call { return NO; }
class_addMethod(panel_class, selector("canBecomeKeyWindow"), xx yes, "c@:");
class_addMethod(panel_class, selector("canBecomeMainWindow"), xx no, "c@:");
objc_registerClassPair(panel_class);
}
capture_context = context;
return view_class, panel_class;
}
view_class, panel_class: Class;
capture_context: #Context; // drawRect: runs on the main thread, in the capture's own event loop
msg_object_at :: (array: *void, index: s64) -> *void {
func :: #type (*void, Selector, u64) -> *void #c_call;
return (cast(func) objc_msgSend)(array, selector("objectAtIndex:"), xx index);
}
msg_object_for_key :: (dictionary: *void, key: string) -> *void {
func :: #type (*void, Selector, *NSString) -> *void #c_call;
return (cast(func) objc_msgSend)(dictionary, selector("objectForKey:"), NSString.getTempString(key));
}
msg_rect :: (object: *void, name: string) -> NSRect {
func :: #type (*void, Selector) -> NSRect #c_call;
return (cast(func) objc_msgSend)(object, selector(name));
}
msg_float :: (object: *void, name: string) -> float64 {
func :: #type (*void, Selector) -> float64 #c_call;
return (cast(func) objc_msgSend)(object, selector(name));
}
msg_integer :: (object: *void, name: string) -> s64 {
func :: #type (*void, Selector) -> s64 #c_call;
return (cast(func) objc_msgSend)(object, selector(name));
}
msg_integer_set :: (object: *void, name: string, value: s64) {
func :: #type (*void, Selector, s64) -> void #c_call;
(cast(func) objc_msgSend)(object, selector(name), value);
}
msg_bool :: (object: *void, name: string, value: bool) {
func :: #type (*void, Selector, BOOL) -> void #c_call;
(cast(func) objc_msgSend)(object, selector(name), ifx value then YES else NO);
}
objc_init_with_frame :: (object: *void, frame: NSRect) -> *void {
func :: #type (*void, Selector, NSRect) -> *void #c_call;
return (cast(func) objc_msgSend)(object, selector("initWithFrame:"), frame);
}
NSView_setNeedsDisplayInRect :: (view: *NSView, rect: CGRect) {
func :: #type (*void, Selector, NSRect) -> void #c_call;
(cast(func) objc_msgSend)(view, selector("setNeedsDisplayInRect:"), NSMakeRect(rect.origin.x, rect.origin.y, rect.size.width, rect.size.height));
}
// ---------------------------------------------------------------- foreign
Mcap_Sck_Shot :: struct {
display_id: u32;
width, height: s32;
pixels: *u8;
}
mcap_capture :: #library,no_dll "../../modules/Screen_Record/macos/capture";
mcap_sck_screenshots :: (shots: *Mcap_Sck_Shot, count: s32, error: *u8, error_size: s32) -> s32 #foreign mcap_capture;
mcap_sck_free :: (pointer: *void) #foreign mcap_capture;
ScreenCaptureKit_Framework :: #library,system,link_always "ScreenCaptureKit";
CGContextRef :: *void;
CGImageRef :: *void;
CGPoint :: struct { x, y: float64; }
CGSize :: struct { width, height: float64; }
CGRect :: struct { origin: CGPoint; size: CGSize; }
kCGImageAlphaNoneSkipLast :: 5;
kCGBitmapByteOrder32Big :: 4 << 12;
kCGRenderingIntentDefault :: 0;
kCGInterpolationNone :: 1;
kCGWindowListOptionOnScreenOnly :: 1 << 0;
kCGWindowListExcludeDesktopElements :: 1 << 4;
kCFNumberSInt64Type :: 4;
kCFStringEncodingUTF8 :: 0x0800_0100;
CoreGraphics_Framework :: #library,system "CoreGraphics";
CoreFoundation_Framework :: #library,system "CoreFoundation";
kCGColorSpaceSRGB: *void #elsewhere CoreGraphics_Framework;
kCGWindowLayer: *void #elsewhere CoreGraphics_Framework;
kCGWindowBounds: *void #elsewhere CoreGraphics_Framework;
kCGWindowName: *void #elsewhere CoreGraphics_Framework;
kCGWindowOwnerName: *void #elsewhere CoreGraphics_Framework;
kCGWindowOwnerPID: *void #elsewhere CoreGraphics_Framework;
AppKit_Framework :: #library,system "AppKit"; // the Objective_C module keeps its own to itself
NSFontAttributeName: *NSString #elsewhere AppKit_Framework;
NSForegroundColorAttributeName: *NSString #elsewhere AppKit_Framework;
CGPreflightScreenCaptureAccess :: () -> bool #foreign CoreGraphics_Framework;
CGRequestScreenCaptureAccess :: () -> bool #foreign CoreGraphics_Framework;
CGWarpMouseCursorPosition :: (point: CGPoint) -> s32 #foreign CoreGraphics_Framework;
CGAssociateMouseAndMouseCursorPosition :: (connected: s32) -> s32 #foreign CoreGraphics_Framework;
CGColorSpaceCreateWithName :: (name: *void) -> *void #foreign CoreGraphics_Framework;
CGColorSpaceRelease :: (space: *void) #foreign CoreGraphics_Framework;
CGDataProviderCreateWithData :: (info: *void, data: *void, size: u64, release: *void) -> *void #foreign CoreGraphics_Framework;
CGDataProviderRelease :: (provider: *void) #foreign CoreGraphics_Framework;
CGImageCreate :: (width: u64, height: u64, bits_per_component: u64, bits_per_pixel: u64, bytes_per_row: u64, space: *void, bitmap_info: u32, provider: *void, decode: *float64, interpolate: bool, intent: s32) -> CGImageRef #foreign CoreGraphics_Framework;
CGImageCreateWithImageInRect :: (image: CGImageRef, rect: CGRect) -> CGImageRef #foreign CoreGraphics_Framework;
CGImageRelease :: (image: CGImageRef) #foreign CoreGraphics_Framework;
CGContextDrawImage :: (c: CGContextRef, rect: CGRect, image: CGImageRef) #foreign CoreGraphics_Framework;
CGContextSaveGState :: (c: CGContextRef) #foreign CoreGraphics_Framework;
CGContextRestoreGState :: (c: CGContextRef) #foreign CoreGraphics_Framework;
CGContextTranslateCTM :: (c: CGContextRef, x: float64, y: float64) #foreign CoreGraphics_Framework;
CGContextScaleCTM :: (c: CGContextRef, x: float64, y: float64) #foreign CoreGraphics_Framework;
CGContextClipToRect :: (c: CGContextRef, rect: CGRect) #foreign CoreGraphics_Framework;
CGContextSetRGBFillColor :: (c: CGContextRef, r: float64, g: float64, b: float64, a: float64) #foreign CoreGraphics_Framework;
CGContextFillRect :: (c: CGContextRef, rect: CGRect) #foreign CoreGraphics_Framework;
CGContextSetInterpolationQuality :: (c: CGContextRef, quality: s32) #foreign CoreGraphics_Framework;
CGWindowListCopyWindowInfo :: (option: u32, relative_to: u32) -> *void #foreign CoreGraphics_Framework;
CGRectMakeWithDictionaryRepresentation :: (dictionary: *void, rect: *CGRect) -> bool #foreign CoreGraphics_Framework;
CFArrayGetCount :: (array: *void) -> s64 #foreign CoreFoundation_Framework;
CFArrayGetValueAtIndex :: (array: *void, index: s64) -> *void #foreign CoreFoundation_Framework;
CFDictionaryGetValue :: (dictionary: *void, key: *void) -> *void #foreign CoreFoundation_Framework;
CFNumberGetValue :: (number: *void, type: s64, value: *void) -> bool #foreign CoreFoundation_Framework;
CFStringGetCString :: (s: *void, buffer: *u8, size: s64, encoding: u32) -> bool #foreign CoreFoundation_Framework;
CFRelease :: (object: *void) #foreign CoreFoundation_Framework;
getpid :: () -> s32 #foreign libc;
libc :: #library,system "libc";
#import "Objective_C";
#import "Objective_C/AppKit";
#import "Process";

58
src/app/screen_record.jai
View File

@ -1,6 +1,7 @@
// Screen recording (video-capture-plan.html): Ctrl+PrintScreen (the record_region global hotkey, from
// any app, even with mcap hidden in the tray) freezes the screens as PrintScreen does, in the
// overlay's video mode (app/screen_capture_win32.jai): a region of one screen, at least 240 x 180, or
// Screen recording (video-capture-plan.html, macos-plan phase 7): Ctrl+PrintScreen on Windows,
// Opt+Cmd+Shift+2 on macOS (the record_region global hotkey, from any app, even with mcap hidden in the
// tray) freezes the screens as a still's key does, in the overlay's video mode
// (app/screen_capture_win32.jai, app/screen_capture_macos.jai): a region of one screen, at least 240 x 180, or
// a click for the whole screen. It is recorded with modules/Screen_Record to an MP4 in the temporary
// folder until Ctrl+PrintScreen is pressed again. The MP4 then goes into the library as a video .mcap
// in SCREEN_CAPTURE_FOLDER, beside the stills, through app/screen_capture.jai's jobs: the MP4 as its
@ -8,7 +9,8 @@
// recorded from in metadata.json, as for a still, and how it was recorded in capture.json.
//
// While it records, a red border is drawn just outside the region and a toolbar beside it gives the
// time, Pause, Stop and Discard (app/screen_record_win32.jai); both are kept out of the recording.
// time, Pause, Stop and Discard (app/screen_record_win32.jai, app/screen_record_macos.jai); both are
// kept out of the recording.
// Time paused is left out of the video. With sound: what the computer plays and the microphone, as
// [video] in the settings file says, as are the frame rate, the cursor and how the screen is taken.
// The toolbar's System sound and Mic buttons turn each on and off while recording, and are kept in
@ -19,7 +21,7 @@
// start (screen_record_leftovers_popup). PrintScreen while recording takes the screen under the cursor
// at once, without the overlay, whose frozen screen would be in the recording.
//
// Windows only, as modules/Screen_Record is.
// Windows and macOS, as modules/Screen_Record is.
SCREEN_RECORD_PREFIX :: "recording-"; // the MP4s in the temporary folder: recording-2026-10-10_07-12-43.mp4
@ -32,7 +34,7 @@ screen_record: struct {
shot: Screen_Shot; // the region and the window it is of; no pixels
date: string; // when it started, as CaptureDate; heap
path: string; // the MP4 being written; heap
#if OS == .WINDOWS recorder: Recorder;
#if OS == .WINDOWS || OS == .MACOS recorder: Recorder;
}
// From app_frame when the hotkey fired, outside the ImGui frame: starts recording, or stops.
@ -44,8 +46,9 @@ screen_record_toggle :: () {
// Every frame (from screen_capture_frame): a recording that stopped by itself is finished and kept.
screen_record_frame :: () {
if app.record_and_exit > 0 screen_record_flag_frame();
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
if !screen_record.recording return;
#if OS == .MACOS record_ui_tick();
failed, why := record_failed(*screen_record.recorder);
if failed {
toast_show(.Error, "The recording stopped: %", why);
@ -122,7 +125,7 @@ screen_record_shutdown :: () {
// At startup: recordings a crash left in the temporary folder, to offer (screen_record_leftovers_popup).
screen_record_startup :: () {
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
screen_record.leftovers.allocator = context.default_allocator;
for file_list(screen_record_folder(),, temp) {
name := path_filename(it);
@ -190,21 +193,22 @@ screen_record_leftover_date :: (path: string) -> string {
}
screen_record_folder :: () -> string {
#if OS == .WINDOWS return tprint("%/%", windows_expand_environment("%TEMP%"), APP_FOLDER_NAME);
else return tprint("/tmp/%", APP_FOLDER_NAME);
return tprint("%/%", platform_temporary_folder(), APP_FOLDER_NAME);
}
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
// immediate: the screen under the cursor, without the overlay (-record).
screen_record_start :: (immediate := false) {
if app_exit_shutdown_started() return;
date := import_calendar_text(to_calendar(current_time_consensus(), .LOCAL));
shot, ok, error := platform_screen_capture_select(video = true, immediate = immediate);
r := app.record_region;
if ok && immediate && r.right > r.left { // -record-region
shot.left, shot.top, shot.width, shot.height = r.left, r.top, r.right - r.left, r.bottom - r.top;
shot.whole_screen = false;
#if OS == .WINDOWS {
r := app.record_region;
if ok && immediate && r.right > r.left { // -record-region
shot.left, shot.top, shot.width, shot.height = r.left, r.top, r.right - r.left, r.bottom - r.top;
shot.whole_screen = false;
}
}
if !ok {
if error { log_error("[record] %.", error); toast_show(.Error, "Screen recording: %", error); }
@ -221,18 +225,28 @@ screen_record_start :: (immediate := false) {
options: Record_Options;
options.path = path;
options.region = .{shot.left, shot.top, shot.left + shot.width, shot.top + shot.height};
options.fps = cast(s32) app_user_settings.video_fps;
options.cursor = app_user_settings.video_cursor;
options.system_audio = app_user_settings.video_system_audio;
options.microphone = app_user_settings.video_microphone;
options.audio_track = true; // so the toolbar can turn a source on mid-recording
if app_user_settings.video_capture == {
case "dxgi"; options.capture = .DXGI;
case "wgc"; options.capture = .WGC;
#if OS == .MACOS {
// The display's region, and the border and toolbar first: ScreenCaptureKit leaves them out of the
// recording by their window numbers.
options.display_id = shot.display_id;
options.x, options.y, options.w, options.h = shot.local_x, shot.local_y, shot.local_width, shot.local_height;
options.width, options.height = shot.width, shot.height;
options.exclude_windows = record_ui_open(shot, shot.width & cast(s32) ~1, shot.height & cast(s32) ~1);
} else {
options.region = .{shot.left, shot.top, shot.left + shot.width, shot.top + shot.height};
if app_user_settings.video_capture == {
case "dxgi"; options.capture = .DXGI;
case "wgc"; options.capture = .WGC;
}
}
started, start_error := record_start(*screen_record.recorder, options);
if !started {
#if OS == .MACOS record_ui_close();
log_error("[record] Could not record % x % at %, %: %", shot.width, shot.height, shot.left, shot.top, start_error);
toast_show(.Error, "Could not record: %", start_error);
screen_shot_free(*shot);
@ -254,7 +268,7 @@ screen_record_start :: (immediate := false) {
} else {
toast_show(.Success, "Recording % x %: % stops", stats.width, stats.height, screen_record_hotkey_text());
}
record_ui_open(options.region, shot.whole_screen, stats.width, stats.height);
#if OS == .WINDOWS record_ui_open(options.region, shot.whole_screen, stats.width, stats.height);
}
// A sound button's state, into [video] for the next recording.
@ -412,13 +426,13 @@ screen_record_hotkey_text :: () -> string {
screen_record_start :: (immediate := false) {
toast_show(.Error, "Screen recording is not available on this system yet");
if app.record_and_exit > 0 screen_record_saved("", "screen recording is Windows only");
if app.record_and_exit > 0 screen_record_saved("", "screen recording is on Windows and macOS only");
}
screen_record_stop :: (shutdown := false) {}
screen_record_write :: (job: *Screen_Capture_Job) -> path: string, error: string {
return "", "screen recording is Windows only";
return "", "screen recording is on Windows and macOS only";
}
}

365
src/app/screen_record_macos.jai Normal file
View File

@ -0,0 +1,365 @@
// While recording (app/screen_record.jai), on macOS: a thin red border just outside the region, and a
// toolbar beside it with the time, the size, frames dropped, and System sound, Mic, Pause, Stop and
// Discard, as app/screen_record_win32.jai has on Windows. Both are non-activating panels above other
// windows, on every Space and over full-screen apps, so they work while mcap is hidden in the menu
// bar and a click on the toolbar leaves the window being recorded in front. Both are kept out of the
// recording: their window numbers go to ScreenCaptureKit's filter (record_ui_open returns them), so
// they are made before the recording starts. The border is outside the region anyway.
//
// The border lets clicks through. A toolbar button only asks (record_ui.asked); screen_record_frame
// does it. Discard asks first, in the toolbar itself. Hover and the time are kept up from
// screen_record_frame (record_ui_tick), as the panels get no events while the mouse only passes.
Record_UI_Ask :: enum u8 { NONE; SYSTEM_SOUND; MICROPHONE; PAUSE; STOP; DISCARD; }
record_ui: struct {
border, toolbar: *NSWindow;
toolbar_view: *NSView;
toolbar_frame: CGRect; // in points from the main screen's top left, for the hover
asked: Record_UI_Ask; // a button pressed, for screen_record_frame
confirming: bool; // Discard pressed once: "Discard it? Yes No"
hovered: s32 = -1; // the button under the mouse
buttons: [5] Record_UI_Button;
button_count: s32;
started: Apollo_Time;
drawn: Apollo_Time; // the toolbar last drawn
size_text: string; // "1920 × 1080"; heap
}
Record_UI_Button :: struct { rect: CGRect; label: string; ask: Record_UI_Ask; confirm: bool; }
// Shows the border around the shot's region and the toolbar beside it; their window numbers (temp),
// to keep them out of the recording. width and height: what is recorded, in pixels.
record_ui_open :: (shot: Screen_Shot, width: s32, height: s32) -> [] u32 {
record_ui_close();
init_objective_c();
init_core_graphics_runtime_constants();
record_ui.started = current_time_monotonic();
record_ui.drawn = record_ui.started;
record_ui.asked = .NONE;
record_ui.confirming = false;
record_ui.hovered = -1;
free(record_ui.size_text,, context.default_allocator);
record_ui.size_text = sprint("% × %", width, height,, context.default_allocator);
region := CGRect.{.{shot.screen_left + shot.local_x, shot.screen_top + shot.local_y}, .{shot.local_width, shot.local_height}};
screen := CGRect.{.{shot.screen_left, shot.screen_top}, .{shot.screen_width, shot.screen_height}};
// Below the region, at its right; else above; else inside its top right (kept out of the recording).
toolbar_width, toolbar_height := record_toolbar_size();
x := region.origin.x + region.size.width - toolbar_width;
y := region.origin.y + region.size.height + BORDER + GAP;
if y + toolbar_height > screen.origin.y + screen.size.height y = region.origin.y - BORDER - GAP - toolbar_height;
if y < screen.origin.y || shot.whole_screen {
y = region.origin.y + GAP;
x = region.origin.x + region.size.width - toolbar_width - GAP;
}
x = clamp(x, screen.origin.x, screen.origin.x + screen.size.width - toolbar_width);
record_ui.toolbar_frame = .{.{x, y}, .{toolbar_width, toolbar_height}};
record_ui.toolbar, record_ui.toolbar_view = record_ui_panel(record_ui.toolbar_frame, TOOLBAR, clicks = true);
// The border: BORDER points wide just outside the region. Not for a whole screen, where it would be
// on the screen beside it.
if !shot.whole_screen {
outer := CGRect.{.{region.origin.x - BORDER, region.origin.y - BORDER}, .{region.size.width + 2 * BORDER, region.size.height + 2 * BORDER}};
record_ui.border = record_ui_panel(outer, BORDER_VIEW, clicks = false);
}
numbers: [..] u32;
numbers.allocator = temp;
if array_find(app.args, "-record-ui-visible") return numbers; // to look at them in a recording
for window: *NSWindow.[record_ui.toolbar, record_ui.border] if window {
func :: #type (*void, Selector) -> s64 #c_call;
array_add(*numbers, cast(u32) (cast(func) objc_msgSend)(window, selector("windowNumber")));
}
return numbers;
}
record_ui_close :: () {
for window: *NSWindow.[record_ui.toolbar, record_ui.border] if window {
NSWindow.orderOut(window, null);
NSWindow.close(window);
release(window);
}
record_ui.toolbar, record_ui.border, record_ui.toolbar_view = null, null, null;
}
// From another thread (the shutdown's): hidden on the main thread, the only one AppKit takes.
record_ui_hide :: () {
hide :: (data: *void) #c_call {
new_context: #Context;
push_context new_context {
for window: *NSWindow.[record_ui.toolbar, record_ui.border] if window NSWindow.orderOut(window, null);
}
}
dispatch_async_f(*_dispatch_main_q, null, hide);
}
// The button pressed since the last call, once.
record_ui_take_ask :: () -> Record_UI_Ask {
ask := record_ui.asked;
record_ui.asked = .NONE;
return ask;
}
record_ui_refresh :: () {
set_needs_display :: #type (*void, Selector, BOOL) -> void #c_call;
send := cast(set_needs_display) objc_msgSend;
if record_ui.toolbar_view send(record_ui.toolbar_view, selector("setNeedsDisplay:"), YES);
}
// Each frame while recording: the button under the mouse lit, and the time moved on four times a second.
record_ui_tick :: () {
if !record_ui.toolbar_view return;
location := NSEvent.mouseLocation();
main_height := main_screen_height();
f := record_ui.toolbar_frame;
point := Vector2.{xx (location.x - f.origin.x), xx (main_height - location.y - f.origin.y)};
hovered := record_toolbar_hit(point);
now := current_time_monotonic();
if hovered != record_ui.hovered || to_float64_seconds(now - record_ui.drawn) >= 0.25 {
record_ui.hovered = hovered;
record_ui.drawn = now;
record_ui_refresh();
}
}
#scope_file
BORDER : float64 : 3.0; // points
GAP : float64 : 6.0;
TOOLBAR :: 1; // the view's kind, in its extra bytes
BORDER_VIEW :: 2;
RED :: Vector4.{0.91, 0.157, 0.157, 1}; // as the overlay's video mode
DIM_RED:: Vector4.{0.38, 0.125, 0.125, 1};
GREY :: Vector4.{0.565, 0.565, 0.565, 1};
AMBER :: Vector4.{1, 0.706, 0, 1};
BACK :: Vector4.{0.125, 0.125, 0.125, 0.96};
BUTTON :: Vector4.{0.22, 0.22, 0.22, 1};
HOVER :: Vector4.{0.314, 0.314, 0.314, 1};
TEXT :: Vector4.{0.94, 0.94, 0.94, 1};
PADDING : float64 : 10.0;
SPACE : float64 : 8.0;
HEIGHT : float64 : 34.0;
DOT : float64 : 12.0;
main_screen_height :: () -> float64 {
screens := objc_msgSend_typed(objc_getClass("NSScreen"), selector("screens"));
func_at :: #type (*void, Selector, u64) -> *void #c_call;
first := (cast(func_at) objc_msgSend)(screens, selector("objectAtIndex:"), 0);
func_frame :: #type (*void, Selector) -> NSRect #c_call;
return (cast(func_frame) objc_msgSend)(first, selector("frame")).size.height;
}
// A borderless, non-activating panel at frame (points from the main screen's top left) above other
// windows, its content a McapRecordView of kind; letting clicks through unless clicks.
record_ui_panel :: (frame: CGRect, kind: s64, clicks: bool) -> *NSWindow, *NSView {
view_class := record_view_class();
if !view_class return null, null;
main_height := main_screen_height();
NSWindowStyleMaskNonactivatingPanel :: 1 << 7;
window := NSWindow.initWithContentRect(cast(*NSWindow) objc_alloc(objc_getClass("NSPanel")),
NSMakeRect(frame.origin.x, main_height - frame.origin.y - frame.size.height, frame.size.width, frame.size.height),
NSWindowStyleMaskBorderless | NSWindowStyleMaskNonactivatingPanel, NSBackingStoreBuffered, NO);
if !window return null, null;
set_bool :: (object: *void, name: string, value: bool) {
func :: #type (*void, Selector, BOOL) -> void #c_call;
(cast(func) objc_msgSend)(object, selector(name), ifx value then YES else NO);
}
NSWindow.setReleasedWhenClosed(window, NO);
set_bool(window, "setOpaque:", false);
set_bool(window, "setHasShadow:", kind == TOOLBAR);
set_bool(window, "setHidesOnDeactivate:", false);
set_bool(window, "setIgnoresMouseEvents:", !clicks);
set_bool(window, "setBecomesKeyOnlyIfNeeded:", true);
NSWindow.setBackgroundColor(window, objc_msgSend_typed(objc_getClass("NSColor"), selector("clearColor")));
NSStatusWindowLevel :: 25;
NSWindow.setLevel(window, NSStatusWindowLevel);
NSWindow.setCollectionBehavior(window, NSWindowCollectionBehaviorCanJoinAllSpaces | NSWindowCollectionBehaviorStationary |
NSWindowCollectionBehaviorFullScreenAuxiliary | NSWindowCollectionBehaviorIgnoresCycle);
func_init :: #type (*void, Selector, NSRect) -> *NSView #c_call;
view := (cast(func_init) objc_msgSend)(objc_alloc(view_class), selector("initWithFrame:"), NSMakeRect(0, 0, frame.size.width, frame.size.height));
(cast(*s64) object_getIndexedIvars(view)).* = kind;
NSWindow.setContentView(window, view);
release(view);
objc_msgSend_typed(window, selector("orderFrontRegardless"));
return window, view;
}
// McapRecordView: draws the border or the toolbar (its kind in its extra bytes), y going down, and
// takes the toolbar's clicks without making mcap active.
record_view_class :: () -> Class {
if view_class return view_class;
view_class = objc_getClass("McapRecordView");
if view_class return view_class;
view_class = objc_allocateClassPair(objc_getClass("NSView"), "McapRecordView", size_of(s64));
if !view_class return null;
draw_rect :: (self: id, _sel: Selector, dirty: NSRect) #c_call {
push_context record_context {
kind := (cast(*s64) object_getIndexedIvars(self)).*;
context_object := objc_msgSend_typed(objc_getClass("NSGraphicsContext"), selector("currentContext"));
cg := ifx context_object then cast(*void) objc_msgSend_typed(context_object, selector("CGContext")) else null;
if !cg return;
func_bounds :: #type (*void, Selector) -> NSRect #c_call;
bounds := (cast(func_bounds) objc_msgSend)(self, selector("bounds"));
if kind == BORDER_VIEW draw_border(cg, bounds.size.width, bounds.size.height);
else draw_toolbar(cg, bounds.size.width, bounds.size.height);
}
}
mouse_down :: (self: id, _sel: Selector, event: *NSEvent) #c_call {
push_context record_context {
func_location :: #type (*void, Selector) -> NSPoint #c_call;
location := (cast(func_location) objc_msgSend)(event, selector("locationInWindow"));
height := record_ui.toolbar_frame.size.height;
hit := record_toolbar_hit(.{xx location.x, xx (height - location.y)});
if hit >= 0 {
button := record_ui.buttons[hit];
if button.confirm record_ui.confirming = !record_ui.confirming;
else if button.ask != .NONE { record_ui.asked = button.ask; record_ui.confirming = false; }
record_ui_refresh();
}
}
}
yes :: (self: id, _sel: Selector) -> BOOL #c_call { return YES; }
yes_for_event :: (self: id, _sel: Selector, event: *NSEvent) -> BOOL #c_call { return YES; }
class_addMethod(view_class, selector("drawRect:"), xx draw_rect, "v@:{CGRect={CGPoint=dd}{CGSize=dd}}");
class_addMethod(view_class, selector("mouseDown:"), xx mouse_down, "v@:@");
class_addMethod(view_class, selector("isFlipped"), xx yes, "c@:");
class_addMethod(view_class, selector("acceptsFirstMouse:"), xx yes_for_event, "c@:@");
objc_registerClassPair(view_class);
record_context = context;
return view_class;
}
view_class: Class;
record_context: #Context; // AppKit calls the view on the main thread, during SDL's event pump
fill :: (cg: *void, r: CGRect, color: Vector4) {
CGContextSetRGBFillColor(cg, color.x, color.y, color.z, color.w);
CGContextFillRect(cg, r);
}
draw_border :: (cg: *void, width: float64, height: float64) {
fill(cg, .{.{0, 0}, .{width, BORDER}}, RED);
fill(cg, .{.{0, height - BORDER}, .{width, BORDER}}, RED);
fill(cg, .{.{0, BORDER}, .{BORDER, height - 2 * BORDER}}, RED);
fill(cg, .{.{width - BORDER, BORDER}, .{BORDER, height - 2 * BORDER}}, RED);
}
draw_toolbar :: (cg: *void, width: float64, height: float64) {
fill(cg, .{.{0, 0}, .{width, height}}, BACK);
paused := record_paused(*screen_record.recorder);
// The dot: red while recording, blinking each half second; grey while paused.
blink := (cast(s64) (to_float64_seconds(current_time_monotonic() - record_ui.started) * 2)) % 2 == 0;
dot := ifx paused then GREY else ifx blink then RED else DIM_RED;
CGContextSetRGBFillColor(cg, dot.x, dot.y, dot.z, dot.w);
CGContextFillEllipseInRect(cg, .{.{PADDING, (height - DOT) / 2}, .{DOT, DOT}});
status, dropped := record_toolbar_texts();
x := PADDING + DOT + SPACE;
draw_text(status, x, height, TEXT);
x += text_size(status).width + SPACE;
if dropped draw_text(dropped, x, height, AMBER);
// The buttons, from the right.
buttons := record_toolbar_buttons();
button_height := height - 10;
right := width - PADDING;
for < buttons {
label_width := text_size(it.label).width;
r := CGRect.{.{right - label_width - 2 * PADDING, (height - button_height) / 2}, .{label_width + 2 * PADDING, button_height}};
record_ui.buttons[it_index].rect = r;
lit := (it.ask == .SYSTEM_SOUND && record_audio_on(*screen_record.recorder, .SYSTEM)) || (it.ask == .MICROPHONE && record_audio_on(*screen_record.recorder, .MICROPHONE));
fill(cg, r, ifx lit then RED else ifx it_index == record_ui.hovered then HOVER else BUTTON);
draw_text(it.label, r.origin.x + PADDING, height, ifx it.ask == .DISCARD then AMBER else TEXT);
right = r.origin.x - SPACE;
}
if record_ui.confirming draw_text("Discard it?", right - text_size("Discard it?").width, height, AMBER);
}
// What the toolbar shows now, left to right: the status, then what went wrong.
record_toolbar_texts :: () -> status: string, dropped: string {
stats := record_stats(*screen_record.recorder);
paused := record_paused(*screen_record.recorder);
seconds := max(to_float64_seconds(current_time_monotonic() - record_ui.started) - stats.paused_seconds, 0.0);
whole := cast(s64) seconds;
two :: (n: s64) -> string { return tprint("%", formatInt(n, minimum_digits = 2)); }
time := ifx whole >= 3600 then tprint("%:%:%", whole / 3600, two(whole / 60 % 60), two(whole % 60)) else tprint("%:%", two(whole / 60), two(whole % 60));
status := tprint("% % %", ifx paused then "PAUSED" else "REC", time, record_ui.size_text);
dropped := ifx stats.dropped then tprint("% dropped", stats.dropped) else "";
return status, dropped;
}
record_toolbar_buttons :: () -> [] Record_UI_Button {
if record_ui.confirming {
record_ui.buttons[0] = .{label = "Yes, discard", ask = .DISCARD};
record_ui.buttons[1] = .{label = "No", confirm = true};
record_ui.button_count = 2;
} else {
recorder := *screen_record.recorder;
record_ui.buttons[0] = .{label = ifx record_audio_on(recorder, .SYSTEM) then "System sound: on" else "System sound: off", ask = .SYSTEM_SOUND};
record_ui.buttons[1] = .{label = ifx record_audio_on(recorder, .MICROPHONE) then "Mic: on" else "Mic: off", ask = .MICROPHONE};
record_ui.buttons[2] = .{label = ifx record_paused(recorder) then "Resume" else "Pause", ask = .PAUSE};
record_ui.buttons[3] = .{label = "Stop", ask = .STOP};
record_ui.buttons[4] = .{label = "Discard", confirm = true};
record_ui.button_count = 5;
}
return array_view(record_ui.buttons, 0, record_ui.button_count);
}
// The toolbar's size, for the longest it is likely to be.
record_toolbar_size :: () -> width: float64, height: float64 {
status_width := text_size(tprint("PAUSED 00:00:00 %", record_ui.size_text)).width;
dropped_width := text_size("999 dropped").width;
buttons_width: float64 = 4 * SPACE + 5 * 2 * PADDING;
for string.["System sound: off", "Mic: off", "Resume", "Stop", "Yes, discard"] buttons_width += text_size(it).width;
return PADDING + DOT + SPACE + status_width + SPACE + dropped_width + SPACE + buttons_width + PADDING, HEIGHT;
}
record_toolbar_hit :: (p: Vector2) -> s32 {
for array_view(record_ui.buttons, 0, record_ui.button_count) {
r := it.rect;
if p.x >= r.origin.x && p.x < r.origin.x + r.size.width && p.y >= r.origin.y && p.y < r.origin.y + r.size.height return cast(s32) it_index;
}
return -1;
}
text_attributes :: (color: Vector4) -> *NSObject {
func_font :: #type (*void, Selector, CGFloat, CGFloat) -> *NSObject #c_call;
font := (cast(func_font) objc_msgSend)(objc_getClass("NSFont"), selector("monospacedDigitSystemFontOfSize:weight:"), 13, 0.3);
func_color :: #type (*void, Selector, CGFloat, CGFloat, CGFloat, CGFloat) -> *NSObject #c_call;
ns_color := (cast(func_color) objc_msgSend)(objc_getClass("NSColor"), selector("colorWithSRGBRed:green:blue:alpha:"), color.x, color.y, color.z, color.w);
keys := *void.[NSFontAttributeName, NSForegroundColorAttributeName];
values := *void.[font, ns_color];
func_dictionary :: #type (*void, Selector, **void, **void, NSUInteger) -> *NSObject #c_call;
return (cast(func_dictionary) objc_msgSend)(objc_getClass("NSDictionary"), selector("dictionaryWithObjects:forKeys:count:"), values.data, keys.data, 2);
}
text_size :: (text: string) -> NSSize {
func :: #type (*void, Selector, *NSObject) -> NSSize #c_call;
return (cast(func) objc_msgSend)(NSString.getTempString(text), selector("sizeWithAttributes:"), text_attributes(TEXT));
}
// Text at x, centred in height.
draw_text :: (text: string, x: float64, height: float64, color: Vector4) {
size := text_size(text);
func :: #type (*void, Selector, NSPoint, *NSObject) -> void #c_call;
(cast(func) objc_msgSend)(NSString.getTempString(text), selector("drawAtPoint:withAttributes:"), NSMakePoint(x, (height - size.height) / 2), text_attributes(color));
}
CGRect :: struct { origin: struct { x, y: float64; }; size: struct { width, height: float64; }; }
CoreGraphics_Library :: #library,system "CoreGraphics";
CGContextSetRGBFillColor :: (c: *void, r: float64, g: float64, b: float64, a: float64) #foreign CoreGraphics_Library;
CGContextFillRect :: (c: *void, rect: CGRect) #foreign CoreGraphics_Library;
CGContextFillEllipseInRect :: (c: *void, rect: CGRect) #foreign CoreGraphics_Library;
AppKit_Library :: #library,system "AppKit";
NSFontAttributeName: *NSString #elsewhere AppKit_Library;
NSForegroundColorAttributeName: *NSString #elsewhere AppKit_Library;
libSystem_Library :: #library,system "libSystem";
_dispatch_main_q: u8 #elsewhere libSystem_Library;
dispatch_async_f :: (queue: *void, data: *void, work: (data: *void) #c_call) #foreign libSystem_Library;
#import "Objective_C";
#import "Objective_C/AppKit";

6
src/app/segment_ui.jai
View File

@ -114,7 +114,7 @@ segment_colour_set :: (class: string, rgb: [3] u8, default := false, save := tru
}
segment_models_folder :: () -> string {
return tprint("%segment-models", path_strip_filename(get_path_of_running_executable()));
return tprint("%segment-models", app_resource_directory());
}
// ---------------------------------------------------------------- startup and shutdown
@ -198,7 +198,7 @@ segment_ui_scan_task :: () {
return;
}
segment_ui.engine = engine;
log("[Segment] % on % in % ms.%", SEGMENT_MODEL, engine.provider, to_milliseconds(current_time_monotonic() - start), ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
log("[Segment] % on % in % ms.%", SEGMENT_MODEL, segment_provider_name(engine), to_milliseconds(current_time_monotonic() - start), ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else "");
}
engine := segment_ui.engine;
// Cancelling (or closing the app) stops it between batches of the engine's, rather than waiting the
@ -335,7 +335,7 @@ segment_window_contents :: () { using ImGui;
// The model.
model_file := tprint("%/%.onnx", segment_models_folder(), SEGMENT_MODEL);
if !busy && segment_ui.engine {
Text("% on %", SEGMENT_MODEL, ifx segment_ui.engine.provider == .CUDA then "the GPU" else "the CPU");
Text("% on %", SEGMENT_MODEL, ifx segment_ui.engine.provider != .CPU then tprint("the GPU (%)", segment_provider_name(segment_ui.engine)) else "the CPU");
if segment_ui.engine.gpu_explanation TextColored(app_theme.palette.dim, "%", segment_ui.engine.gpu_explanation);
} else if !busy && segment_ui.engine_error {
TextColored(app_theme.palette.warning, "% could not start: %", SEGMENT_MODEL, segment_ui.engine_error);

67
src/app/viewer.jai
View File

@ -16,10 +16,43 @@
// appends the new ones. The library is watched (viewer_watch_frame), so files added or deleted
// outside the app come and go on their own; Refresh is still there for a watcher that missed some.
VIEWER_FOLDER :: "D:/McapIndex/mcap"; // hardcoded for now; -library FOLDER shows another
// The library when [library] folder in settings.config names none; -library FOLDER overrides both.
#if OS == .WINDOWS LIBRARY_FOLDER_DEFAULT :: "D:/McapIndex/mcap";
else LIBRARY_FOLDER_DEFAULT :: "~/Pictures/mcap";
// Set once at startup (library_folder_choose), so the library never changes under a running app.
viewer_folder :: () -> string {
return ifx app.library_folder then app.library_folder else VIEWER_FOLDER;
return app.library_folder;
}
// -library, else the setting, else the default. Called once settings.config is read.
library_folder_choose :: () {
if app.library_folder return;
folder := ifx app_user_settings.library_folder.count then app_user_settings.library_folder else LIBRARY_FOLDER_DEFAULT;
app.library_folder = library_folder_normalize(folder);
// The default is made on first use, so the catalog cache and imports have somewhere to go. Any
// other folder is left to the user: a mistyped one is not made.
default := library_folder_normalize(LIBRARY_FOLDER_DEFAULT);
defer free(default,, context.default_allocator);
if app.library_folder == default && !file_exists(app.library_folder) {
if make_directory_if_it_does_not_exist(app.library_folder, recursive = true) log("[library] Made the library folder %.", app.library_folder);
else log_error("[library] Could not make the library folder %.", app.library_folder);
}
}
// With / between its folders and none at the end, as the paths found in it are made, and a leading
// ~/ for the home folder. Heap-allocated.
library_folder_normalize :: (folder: string) -> string {
expanded := folder;
#if OS != .WINDOWS if begins_with(folder, "~/") || folder == "~" {
home := to_string(POSIX.getenv("HOME"));
if home.count expanded = tprint("%1%2", trim_right(home, "/"), advance(folder, 1));
}
result := copy_string(expanded,, context.default_allocator);
path_overwrite_separators(result, #char "/");
while result.count > 3 && result[result.count - 1] == #char "/" result.count -= 1;
return result;
}
VIEWER_WORKER_COUNT :: 4;
VIEWER_MAX_TEXTURES :: 1200; // previews on the GPU at once, about 200 KB each
@ -620,13 +653,13 @@ viewer_list_files :: () -> [..] Listed_File {
}
// Whether path, a file or folder, is where the library keeps captures: under it, and not hidden or
// in the deleted folder viewer_delete_capture moves files to off Windows.
// in the deleted folder viewer_delete_capture moves files to on Linux.
viewer_is_library_path :: (path: string) -> bool {
folder := viewer_folder();
if path.count <= folder.count + 1 || !begins_with(path, folder) || path[folder.count] != #char "/" return false;
relative := advance(path, folder.count + 1);
if begins_with(relative, ".") || contains(relative, "/.") return false;
#if OS != .WINDOWS if relative == "deleted" || begins_with(relative, "deleted/") return false;
#if OS == .LINUX if relative == "deleted" || begins_with(relative, "deleted/") return false;
return true;
}
@ -804,13 +837,13 @@ viewer_mark_removed :: (index: s32) {
}
Viewer_Delete_Mode :: enum u8 {
RECYCLE; // to the Recycle Bin, or the deleted folder elsewhere: can be undone
RECYCLE; // to the Recycle Bin, the Trash, or the deleted folder on Linux: can be undone
OVERWRITE; // overwritten with random bytes, renamed, then deleted for good
}
// Deletes the capture's file, then takes it out of the grid. RECYCLE: to the Recycle Bin on Windows
// (Windows asks first when the drive has none and it would be deleted for good), elsewhere into a
// deleted folder beside it. OVERWRITE: viewer_overwrite_and_delete. False, with why, when the file
// (Windows asks first when the drive has none and it would be deleted for good), the Trash on macOS,
// and on Linux into a deleted folder in the library. OVERWRITE: viewer_overwrite_and_delete. False, with why, when the file
// is still there.
viewer_delete_capture :: (index: s32, mode: Viewer_Delete_Mode) -> ok: bool, error: string {
capture := *viewer.captures[index];
@ -827,6 +860,9 @@ viewer_delete_capture :: (index: s32, mode: Viewer_Delete_Mode) -> ok: bool, err
result := SHFileOperationW(*operation);
if operation.fAnyOperationsAborted return false, "cancelled";
if result != 0 return false, tprint("the shell could not delete it (error 0x%)", formatInt(result, base = 16));
} else #if OS == .MACOS {
ok, error := macos_move_to_trash(capture.path);
if !ok return false, tprint("it could not be moved to the Trash: %", error);
} else {
folder := tprint("%/deleted", viewer_folder());
make_directory_if_it_does_not_exist(folder);
@ -887,8 +923,9 @@ viewer_delete_now :: (index: s32) {
else toast_show(.Success, "Overwrote and deleted %", name);
}
#if OS == .WINDOWS VIEWER_TRASH_NAME :: "Recycle Bin";
else VIEWER_TRASH_NAME :: "deleted folder";
#if OS == .WINDOWS VIEWER_TRASH_NAME :: "Recycle Bin";
else #if OS == .MACOS VIEWER_TRASH_NAME :: "Trash";
else VIEWER_TRASH_NAME :: "deleted folder";
// The confirmation for viewer.delete_index, as a modal popup.
viewer_delete_popup :: () { using ImGui;
@ -1126,7 +1163,7 @@ viewer_ui :: () { using ImGui;
if IsItemHovered() {
engine: *Ocr_Engine;
if !ocr_ui.thread_busy engine = ocr_ui.engine; // the background thread owns it while busy
if engine SetTooltip("OCR with %, on %", engine.model.name, ifx engine.provider == .CUDA then "the GPU" else "the CPU");
if engine SetTooltip("OCR with %, on %", engine.model.name, ifx ocr_gpu_status(engine) then tprint("the GPU (%)", ocr_provider_name(engine)) else "the CPU");
else SetTooltip("OCR is not loaded, is loading, or could not start; the OCR window says which");
}
@ -1241,9 +1278,7 @@ viewer_open_proc :: (thread: *Thread) -> s64 {
exporting := viewer.open_folder.count > 0;
folder := viewer.open_folder;
if !exporting {
#if OS == .WINDOWS base := windows_expand_environment("%TEMP%");
else base := "/tmp";
folder = tprint("%/%", base, APP_FOLDER_NAME);
folder = tprint("%/%", platform_temporary_folder(), APP_FOLDER_NAME);
}
make_directory_if_it_does_not_exist(folder, recursive = true);
if !is_directory(folder) { viewer.open_error = copy_string(tprint("% is not a folder that can be written to", folder)); return 0; }
@ -1366,10 +1401,10 @@ viewer_export_popup :: () { using ImGui;
if custom {
SetNextItemWidth(GetFontSize() * 30);
InputTextWithHint("##folder", "The folder to export to", viewer.export_buffer.data, viewer.export_buffer.count);
#if OS == .WINDOWS {
#if OS == .WINDOWS || OS == .MACOS {
SameLine();
if Button("Browse...") {
chosen := windows_choose_folder("Choose a folder to export to");
chosen := platform_choose_folder("Choose a folder to export to");
if chosen viewer_export_set_buffer(chosen);
}
}
@ -1431,7 +1466,7 @@ viewer_copy_image :: () {
index := viewer_selected();
if index < 0 { toast_show(.Error, "Select a capture to copy its image"); return; }
capture := *viewer.captures[index];
#if OS != .WINDOWS {
#if OS == .LINUX {
toast_show(.Error, "Copying images is not available on this system yet");
return;
}

7
src/base/platform/platform_linux.jai
View File

@ -9,6 +9,13 @@ platform_get_path_config_data :: inline () -> string {
return XDG.get_path(.CONFIG_HOME);
}
// $TMPDIR, else /tmp, without a separator at the end. Temp.
platform_temporary_folder :: () -> string {
folder := posix_getenv("TMPDIR");
if !folder return "/tmp";
return trim_right(copy_string(folder,, temp), "/");
}
// A tag such as ja-JP from the locale variables, or "". Temp.
platform_preferred_language :: () -> string {
for string.["LC_ALL", "LC_MESSAGES", "LANG"] {

87
src/base/platform/platform_macos.jai
View File

@ -70,6 +70,93 @@ platform_cjk_font_candidates :: () -> [] Cjk_Font {
return list;
}
// ---------------------------------------------------------------- folders, the Trash and the clipboard
// The user's own temporary folder, NSTemporaryDirectory() (under /var/folders, cleared by macOS),
// not /tmp, which every user shares. No separator at the end. Temp.
platform_temporary_folder :: () -> string {
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
folder := copy_string(to_string(NSTemporaryDirectory()),, temp); // the NSString goes with the pool
return trim_right(folder, "/");
}
// Asks for a folder with the system's open panel, over the app's window: its path (temp), or ""
// when none was chosen. Main thread; it runs modally until the panel is closed.
platform_choose_folder :: (title: string) -> string {
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
panel := NSOpenPanel.openPanel();
if !panel return "";
func_bool :: #type (*void, Selector, BOOL) -> void #c_call;
func_string :: #type (*void, Selector, *NSString) -> void #c_call;
(cast(func_bool) objc_msgSend)(panel, selector("setCanChooseDirectories:"), YES);
(cast(func_bool) objc_msgSend)(panel, selector("setCanChooseFiles:"), NO);
(cast(func_bool) objc_msgSend)(panel, selector("setAllowsMultipleSelection:"), NO);
(cast(func_bool) objc_msgSend)(panel, selector("setCanCreateDirectories:"), YES);
(cast(func_string) objc_msgSend)(panel, selector("setMessage:"), NSString.getTempString(title));
(cast(func_string) objc_msgSend)(panel, selector("setPrompt:"), NSString.getTempString("Choose"));
save_panel := cast(*NSSavePanel) panel;
if NSSavePanel.runModal(save_panel) != NSModalResponseOK return "";
url := NSSavePanel.URL(save_panel);
if !url return "";
return copy_string(to_string(NSURL.path(url)),, temp);
}
// Moves a file to the Trash, where Finder's Put Back returns it to where it was. Error is "" or why not.
macos_move_to_trash :: (path: string) -> ok: bool, error: string {
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
url := NSURL.fileURLWithPath(NSString.getTempString(path));
error: *NSError;
func :: #type (*void, Selector, *NSURL, **NSURL, **NSError) -> BOOL #c_call;
if (cast(func) objc_msgSend)(NSFileManager.defaultManager(), selector("trashItemAtURL:resultingItemURL:error:"), url, null, *error) == YES return true, "";
why := ifx error then copy_string(to_string(NSError.localizedDescription(error)),, temp) else "no reason given";
return false, why;
}
// Puts RGBA pixels, top row first, on the clipboard as PNG and TIFF, in sRGB and opaque, as the
// Windows version makes them. Every app that pastes images takes one of the two.
platform_copy_image_to_clipboard :: (pixels: [] u8, width: s32, height: s32) -> bool {
if width <= 0 || height <= 0 || pixels.count < cast(s64) width * height * 4 return false;
pool := objc_new(NSAutoreleasePool);
defer NSAutoreleasePool.drain(pool);
// AppKit allocates the pixels (planes null) and frees them with the rep.
rep := NSBitmapImageRep.initWithBitmapDataPlanes(objc_alloc(NSBitmapImageRep), null, width, height, 8, 4, YES, NO, NSDeviceRGBColorSpace, cast(s64) width * 4, 32);
if !rep return false;
autorelease(rep);
out := NSBitmapImageRep.bitmapData(rep);
if !out return false;
memcpy(out, pixels.data, cast(s64) width * height * 4);
for 0..cast(s64) width * height - 1 out[it * 4 + 3] = 255;
// Tagged sRGB, as the pixels are, so a paste doesn't shift their colors on a wide-gamut screen.
func_object :: #type (*void, Selector) -> *NSObject #c_call;
func_retag :: #type (*void, Selector, *NSObject) -> *NSBitmapImageRep #c_call;
srgb := (cast(func_object) objc_msgSend)(objc_getClass("NSColorSpace"), selector("sRGBColorSpace"));
tagged := (cast(func_retag) objc_msgSend)(rep, selector("bitmapImageRepByRetaggingWithColorSpace:"), srgb);
if tagged rep = tagged;
NSBitmapImageFileTypePNG :: 4;
func_png :: #type (*void, Selector, NSUInteger, *NSObject) -> *NSData #c_call;
properties := (cast(func_object) objc_msgSend)(objc_getClass("NSDictionary"), selector("dictionary"));
png := (cast(func_png) objc_msgSend)(rep, selector("representationUsingType:properties:"), NSBitmapImageFileTypePNG, properties);
tiff := cast(*NSData) (cast(func_object) objc_msgSend)(rep, selector("TIFFRepresentation"));
if !png && !tiff return false;
pasteboard := NSPasteboard.generalPasteboard();
NSPasteboard.clearContents(pasteboard);
func_set :: #type (*void, Selector, *NSData, NSPasteboardType) -> BOOL #c_call;
set := false;
if png set = (cast(func_set) objc_msgSend)(pasteboard, selector("setData:forType:"), png, NSPasteboardTypePNG) == YES || set;
if tiff set = (cast(func_set) objc_msgSend)(pasteboard, selector("setData:forType:"), tiff, NSPasteboardTypeTIFF) == YES || set;
return set;
}
NSTemporaryDirectory :: () -> *NSString #foreign macos_foundation;
macos_foundation :: #library,system "Foundation"; // the Objective_C module keeps its own to itself
Mach :: #import "macOS"; // task_info
#import "Objective_C";

7
src/base/platform/platform_windows.jai
View File

@ -846,6 +846,13 @@ os_log_error :: () {
error_code := GetLastError();
log_error(" > GetLastError code: %, %", error_code, get_error_string(error_code));
}
// %TEMP%, without a separator at the end. Temp.
platform_temporary_folder :: () -> string {
return trim_right(windows_expand_environment("%TEMP%"), "/\\");
}
platform_choose_folder :: (title: string) -> string { return windows_choose_folder(title); }
// Asks for a folder with Explorer's own picker, over the app's window: its path (temp), or "" when
// none was chosen.
windows_choose_folder :: (title: string) -> string {

34
src/test/import_tests.jai
View File

@ -152,6 +152,39 @@ exif_tests :: () {
expect(exif_read(cast([] u8) "\xFF\xD8\xFF\xDA").orientation == 1, "a JPEG with none is upright");
}
// heif_info: a HEIF's primary item's size, rotation and mirroring, from a meta box made here.
heif_tests :: () {
group("heif");
box :: (type: string, contents: [] u8) -> [] u8 {
out: [..] u8;
out.allocator = temp;
size := contents.count + 8;
array_add(*out, cast(u8) (size >> 24), cast(u8) (size >> 16), cast(u8) (size >> 8), cast(u8) size);
array_add(*out, ..cast([] u8) type);
array_add(*out, ..contents);
return out;
}
join :: (parts: ..[] u8) -> [] u8 {
out: [..] u8;
out.allocator = temp;
for parts array_add(*out, ..it);
return out;
}
for turns: 0..3 {
ispe := box("ispe", u8.[0,0,0,0, 0,0,0,64, 0,0,0,48]);
irot := box("irot", u8.[cast(u8) turns]);
imir := box("imir", u8.[1]);
ipco := box("ipco", join(ispe, irot, imir));
ipma := box("ipma", u8.[0,0,0,0, 0,0,0,1, 0,1, 3, 0x81, 0x82, 0x03]); // item 1: properties 1-3
pitm := box("pitm", u8.[0,0,0,0, 0,1]);
meta := box("meta", join(u8.[0,0,0,0], pitm, box("iprp", join(ipco, ipma))));
heif := heif_info(join(box("ftyp", cast([] u8) "heic\0\0\0\0mif1heic"), meta));
expect(heif.width == 64 && heif.height == 48, "irot %: the size, % x %", turns, heif.width, heif.height);
expect(heif.rotation == turns * 90, "irot %: % degrees (expected %)", turns, heif.rotation, turns * 90);
expect(heif.mirror == 1, "irot %: imir read", turns);
}
}
orientation_tests :: () {
group("orientation");
// 3 x 2, pixels numbered by where they are: 0 1 2 / 3 4 5. Each orientation's result, row by row.
@ -861,6 +894,7 @@ run_import_tests :: (builder: *String_Builder = null) -> bool {
push_allocator(context.default_allocator);
kind_tests();
exif_tests();
heif_tests();
orientation_tests();
date_tests();
folder_tests();

29
src/test/labels_tests.jai
View File

@ -83,7 +83,34 @@ labels_tests :: () {
expect(created, "the label engine starts: %", error);
if !created return;
defer labels_destroy(engine);
emit(" images on %; % dimensions\n", ifx engine.provider == .CUDA then "CUDA" else "the CPU", engine.dimensions);
emit(" images on %; % dimensions\n", labels_provider_name(engine), engine.dimensions);
#if OS == .MACOS if engine.provider == .COREML {
// Core ML's vectors against onnxruntime's on the CPU, for an image of bands and a gradient: the
// float16 model on the GPU is within cosine 0.99999 of the CPU's on real images.
group("core ml");
cpu, cpu_created, cpu_error := labels_create(folder, LABELS_MODEL, prefer_gpu = false);
expect(cpu_created && cpu.provider == .CPU, "a CPU engine to compare with starts: %", cpu_error);
if cpu_created {
defer labels_destroy(cpu);
W :: 320; H :: 200;
pixels := NewArray(W * H * 4, u8,, temp);
for y: 0..H - 1 for x: 0..W - 1 {
p := pixels.data + (y * W + x) * 4;
p[0] = cast,trunc(u8) (x * 255 / W);
p[1] = cast(u8) (ifx (y / 25) % 2 then 220 else 40);
p[2] = cast,trunc(u8) (y * 255 / H);
p[3] = 255;
}
image := Labels_Image.{pixels.data, W, H, W * 4};
on_gpu, gpu_error := labels_embed_images(engine, .[image],, temp);
on_cpu, cpu_run_error := labels_embed_images(cpu, .[image],, temp);
expect(!gpu_error && !cpu_run_error, "both make the image's vector (% / %)", gpu_error, cpu_run_error);
cosine: float;
for 0..engine.dimensions - 1 cosine += on_gpu[it] * on_cpu[it];
expect(cosine > 0.999, "Core ML's vector is the CPU's: cosine %", cosine);
}
}
group("tokenizer");
for KNOWN {

2
src/test/ocr_tests.jai
View File

@ -169,7 +169,7 @@ turn_tests :: () {
if !created return;
defer ocr_destroy(engine);
expect(engine.orientation.session != null, "% has an orientation classifier", OCR_DEFAULT_MODEL);
emit(" running on %; classifier input % x %\n", ifx engine.provider == .CUDA then "CUDA" else "the CPU", engine.orientation_width, engine.orientation_height);
emit(" running on %; classifier input % x %\n", ocr_provider_name(engine), engine.orientation_width, engine.orientation_height);
level, drawn := drawn_lines();
expect(drawn, "the test text is drawn");

4
src/ui/imgui_keyboard.jai
View File

@ -349,8 +349,8 @@ GLOBAL_HOTKEY_ACTION_COUNT :: 3; // sizes the platforms' registration a
global_hotkey_actions_build :: (out: *[..] Hotkey_Action) {
array_add(out, .{"bring_to_foreground", *bring_to_foreground_hotkey, false});
array_add(out, .{"capture_region", *capture_region_hotkey, true}); // app/screen_capture.jai
array_add(out, .{"record_region", *record_region_hotkey, true}); // app/screen_record.jai
array_add(out, .{"capture_region", *capture_region_hotkey, OS == .LINUX}); // app/screen_capture.jai: Windows and macOS
array_add(out, .{"record_region", *record_region_hotkey, OS == .LINUX}); // app/screen_record.jai: Windows and macOS
}
global_hotkey_actions: [..] Hotkey_Action;

31
src/ui/imgui_theme_registry.jai
View File

@ -296,6 +296,9 @@ App_User_Settings :: struct {
// [export] folder, one line each: the folders Export offers for a capture's original, the first
// the default. None means the Downloads folder. Heap-allocated.
export_folders: [..] string;
// [library] folder: the library, set by hand and read at startup; "" means LIBRARY_FOLDER_DEFAULT.
// Heap-allocated.
library_folder: string;
// Of the monitor the window first opens on.
window_width_fraction: float = 0.80;
@ -328,6 +331,7 @@ settings_free :: (s: *App_User_Settings) {
free(s.ocr_model,, context.default_allocator);
for s.export_folders free(it,, context.default_allocator);
array_free(s.export_folders);
free(s.library_folder,, context.default_allocator);
free(s.video_capture,, context.default_allocator);
array_free(s.hotkey_bindings);
array_free(s.global_hotkey_bindings);
@ -387,11 +391,11 @@ app_load_user_settings_file :: () -> write_back: bool {
if hotkeys_missing.count || global_missing.count return true;
#if OS == .LINUX seen.global_hotkeys = true; // not written there: Linux has no global hotkeys yet
return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.export_folder && seen.video_fps && seen.video_cursor && seen.video_capture && seen.video_system_audio && seen.video_microphone);
return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model && seen.export_folder && seen.library_folder && seen.video_fps && seen.video_cursor && seen.video_capture && seen.video_system_audio && seen.video_microphone);
}
Settings_Seen :: struct {
theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys, ocr_model, export_folder: bool;
theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys, ocr_model, export_folder, library_folder: bool;
video_fps, video_cursor, video_capture, video_system_audio, video_microphone: bool;
}
@ -522,6 +526,10 @@ settings_parse :: (path: string, file_data: string, out: *App_User_Settings) ->
folder := settings_unquote(value); // an empty line says the list is empty
if folder.count array_add(*out.export_folders, copy_string(folder,, context.default_allocator));
seen.export_folder = true;
} else if section == "[library]" && key == "folder" {
free(out.library_folder,, context.default_allocator);
out.library_folder = copy_string(settings_unquote(value),, context.default_allocator);
seen.library_folder = true;
} else if section == "[ui]" && key == "theme" {
free(out.theme,, context.default_allocator);
out.theme = copy_string(settings_unquote(value),, context.default_allocator);
@ -686,6 +694,10 @@ app_save_user_settings_file :: () -> bool {
if seen.width app_user_settings.window_width_fraction = on_disk.window_width_fraction;
if seen.height app_user_settings.window_height_fraction = on_disk.window_height_fraction;
if seen.event_log app_user_settings.event_log_capacity = on_disk.event_log_capacity;
if seen.library_folder {
free(app_user_settings.library_folder,, context.default_allocator);
app_user_settings.library_folder = copy_string(on_disk.library_folder,, context.default_allocator);
}
if seen.video_fps app_user_settings.video_fps = on_disk.video_fps;
if seen.video_cursor app_user_settings.video_cursor = on_disk.video_cursor;
// video_system_audio and video_microphone are the app's: the recording toolbar's buttons set them.
@ -719,9 +731,13 @@ app_save_user_settings_file :: () -> bool {
for export_folders print_to_builder(*builder, "folder: %\n", settings_quote_if_needed(it));
if !export_folders append(*builder, "folder:\n");
append(*builder, "\n");
append(*builder, "[library]\n");
append(*builder, "# The folder of .mcap files mcap shows and imports into; ~/ is the home folder. Read when mcap\n");
append(*builder, "# starts; -library FOLDER on the command line overrides it.\n");
print_to_builder(*builder, "folder: %\n\n", settings_quote_if_needed(ifx app_user_settings.library_folder.count then app_user_settings.library_folder else LIBRARY_FOLDER_DEFAULT));
append(*builder, "[video]\n");
append(*builder, "# Screen recording (Ctrl+PrintScreen; Windows). fps: frames a second at most, written only when the\n");
append(*builder, "# screen changes. cursor: true or false. capture: auto (Desktop Duplication, else\n");
append(*builder, "# Screen recording (Ctrl+PrintScreen; Opt+Cmd+Shift+2 on macOS). fps: frames a second at most, written only when the\n");
append(*builder, "# screen changes. cursor: true or false. capture (Windows): auto (Desktop Duplication, else\n");
append(*builder, "# Windows.Graphics.Capture), dxgi or wgc. system_audio: what the computer plays; microphone: the\n");
append(*builder, "# default input device. Each true or false.\n");
print_to_builder(*builder, "fps: %\n", app_user_settings.video_fps);
@ -853,6 +869,10 @@ app_reload_user_settings_if_changed :: () {
if fresh.event_log_capacity != old.event_log_capacity logger_set_event_log_capacity(fresh.event_log_capacity);
if fresh.ocr_model != old.ocr_model ocr_ui_settings_changed(); // app/ocr_ui.jai
if fresh.library_folder != old.library_folder {
log("[settings] The library folder is now %; mcap shows it from its next start.", ifx fresh.library_folder.count then fresh.library_folder else LIBRARY_FOLDER_DEFAULT);
toast_show(.Success, "The new library folder is shown when mcap starts again");
}
bindings: [] Hotkey_Binding = hotkey_defaults();
if seen.hotkeys bindings = app_user_settings.hotkey_bindings;
@ -881,7 +901,8 @@ settings_watch: struct {
app_settings_watch_start :: () {
dir := app_program_data_dir();
if !Watcher.init(*settings_watch.watcher, settings_watch_callback, watch_recursively = false) {
// macOS's FSEvents always watches recursively, and the module logs an error when asked not to.
if !Watcher.init(*settings_watch.watcher, settings_watch_callback, watch_recursively = OS == .MACOS) {
log_error("[settings] Could not watch %. Changes to % apply at the next start.", dir, APP_USER_SETTINGS_FILE_NAME);
return;
}

173
tools/export_mask2former_coreml.py Normal file
View File

@ -0,0 +1,173 @@
# Converts Mask2Former (facebook/mask2former-swin-base-coco-panoptic) to a Core ML model for
# modules/Segment on macOS, then checks it against the ONNX model on a folder of images, segment by
# segment. Run once, on a Mac, after tools/export_mask2former.py; mcap only uses the .mlpackage it writes.
#
# python3 -m venv ~/mcap-coreml
# ~/mcap-coreml/bin/pip install torch==2.7.0 transformers coremltools==9.0 onnxruntime pillow scipy
# ~/mcap-coreml/bin/pip install --no-deps timm
# ~/mcap-coreml/bin/python tools/export_mask2former_coreml.py --check FOLDER_OF_IMAGES
#
# The model as tools/export_mask2former.py exports it (pixel_values [1, 3, 384, 384] to
# class_queries_logits [1, 100, 134] and masks_queries_logits [1, 100, 96, 96]), as an ML Program in
# float32, one image a run (a batch of 4 did not convert). Three things coremltools 9.0 cannot take
# are worked around for the conversion only, none changing what the model computes:
# - it casts a one-element tensor with int(array), which NumPy 2 refuses;
# - its meshgrid refuses the inputs Mask2Former's traced linspaces come out as;
# - Core ML takes tensors of at most 5 dimensions, and the multi-scale deformable attention makes 6
# (batch, queries, heads, levels, points, xy): it is run a level at a time instead, which gives
# the same output in PyTorch (checked below) with at most 5.
# Measured on an M5 Pro: 42 ms an image on the GPU against 166 ms for onnxruntime on the CPU, with
# the same segments (IoU 1.0) on every test image.
import argparse, glob, os, sys, time
import numpy as np
import torch
from torch import nn
from PIL import Image
import coremltools as ct
from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil.frontend.torch import ops as ct_ops
from coremltools.converters.mil.frontend.torch.torch_op_registry import register_torch_op
from transformers import AutoImageProcessor, Mask2FormerForUniversalSegmentation
from transformers.utils import logging
import transformers.models.mask2former.modeling_mask2former as modeling
logging.set_verbosity_error()
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from export_mask2former import Logits
CHECKPOINT = "facebook/mask2former-swin-base-coco-panoptic"
MODELS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "modules", "Segment", "models")
ONNX = os.path.join(MODELS, "mask2former-swin-base-coco-panoptic.onnx")
OUTPUT = os.path.join(MODELS, "mask2former-swin-base-coco-panoptic.mlpackage")
SIZE = 384
_original_cast = ct_ops._cast
def _cast_one_element(context, node, dtype, dtype_name):
x = ct_ops._get_inputs(context, node, expected=1)[0]
if x.can_be_folded_to_const() and np.ndim(x.val) > 0 and np.size(x.val) == 1:
context.add(ct_ops.mb.const(val=dtype(np.asarray(x.val).item()), name=node.name), node.name)
return
_original_cast(context, node, dtype, dtype_name)
ct_ops._cast = _cast_one_element
@register_torch_op(override=True)
def meshgrid(context, node):
"""torch.meshgrid, from its inputs flattened to 1-d first."""
inputs = ct_ops._get_inputs(context, node, expected=[1, 2])
indexing = inputs[1].val if len(inputs) > 1 else "ij"
indexing = ct_ops._get_kwinputs(context, node, "indexing", default=[indexing])[0]
if hasattr(indexing, "val"):
indexing = indexing.val
tensors = [mb.reshape(x=t, shape=[-1], name=f"{node.name}_flat_{i}") for i, t in enumerate(inputs[0])]
sizes = [t.shape[0] for t in tensors]
grids = []
for i, t in enumerate(tensors):
view = [1] * len(tensors); view[i] = -1
reps = list(sizes); reps[i] = 1
grid = mb.tile(x=mb.reshape(x=t, shape=view, name=f"{node.name}_view_{i}"), reps=reps, name=f"{node.name}_grid_{i}")
if indexing == "xy":
grid = mb.transpose(x=grid, perm=[1, 0] + list(range(2, len(tensors))), name=f"{node.name}_xy_{i}")
grids.append(grid)
context.add(tuple(grids), node.name)
def deformable_attention_rank5(self, hidden_states, attention_mask=None, encoder_hidden_states=None, encoder_attention_mask=None,
position_embeddings=None, reference_points=None, spatial_shapes_list=None, level_start_index=None,
output_attentions=False):
"""Mask2FormerPixelDecoderEncoderMultiscaleDeformableAttention.forward, a level at a time."""
if position_embeddings is not None:
hidden_states = self.with_pos_embed(hidden_states, position_embeddings)
if reference_points.shape[-1] != 2:
raise ValueError("only 2-d reference points are rewritten")
B, Q, _ = hidden_states.shape
_, S, _ = encoder_hidden_states.shape
H, L, P, D = self.n_heads, self.n_levels, self.n_points, self.d_model // self.n_heads
value = self.value_proj(encoder_hidden_states)
if attention_mask is not None:
value = value.masked_fill(attention_mask[..., None], float(0))
value = value.view(B, S, H, D)
offsets = self.sampling_offsets(hidden_states).view(B, Q, H, L, P * 2)
weights = nn.functional.softmax(self.attention_weights(hidden_states).view(B, Q, H, L * P), -1)
weights = weights.transpose(1, 2).reshape(B * H, 1, Q, L * P)
values = value.split([h * w for h, w in spatial_shapes_list], dim=1)
sampled = []
for level, (h, w) in enumerate(spatial_shapes_list):
value_l = values[level].flatten(2).transpose(1, 2).reshape(B * H, D, h, w)
offset_l = offsets[:, :, :, level].reshape(B, Q, H, P, 2)
location = reference_points[:, :, None, level, None, :] + offset_l / torch.tensor([w, h], dtype=offset_l.dtype)
grid = (2 * location - 1).transpose(1, 2).flatten(0, 1)
sampled.append(nn.functional.grid_sample(value_l, grid, mode="bilinear", padding_mode="zeros", align_corners=False))
output = (torch.stack(sampled, dim=-2).flatten(-2) * weights).sum(-1).view(B, H * D, Q).transpose(1, 2).contiguous()
return self.output_proj(output), None
def segments_of(processor, class_logits, mask_logits):
holder = type("Outputs", (), {"class_queries_logits": torch.from_numpy(class_logits), "masks_queries_logits": torch.from_numpy(mask_logits)})()
return processor.post_process_panoptic_segmentation(holder, target_sizes=[(SIZE, SIZE)])[0]
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--check", metavar="FOLDER", help="images to check the Core ML model against the ONNX one on")
args = parser.parse_args()
processor = AutoImageProcessor.from_pretrained(CHECKPOINT)
model = Mask2FormerForUniversalSegmentation.from_pretrained(CHECKPOINT).eval()
example = torch.zeros(1, 3, SIZE, SIZE)
with torch.no_grad():
original = Logits(model)(torch.randn(1, 3, SIZE, SIZE, generator=torch.Generator().manual_seed(0)))
modeling.Mask2FormerPixelDecoderEncoderMultiscaleDeformableAttention.forward = deformable_attention_rank5
rewritten = Logits(model)(torch.randn(1, 3, SIZE, SIZE, generator=torch.Generator().manual_seed(0)))
difference = max((a - b).abs().max().item() for a, b in zip(original, rewritten))
print(f"The attention a level at a time, against the original in PyTorch: max difference {difference:.2e}")
if difference > 1e-4:
sys.exit("The rewritten attention does not give the same output.")
traced = torch.jit.trace(Logits(model).eval(), example)
start = time.perf_counter()
ml = ct.convert(traced, inputs=[ct.TensorType(name="pixel_values", shape=(1, 3, SIZE, SIZE))],
outputs=[ct.TensorType(name="class_queries_logits"), ct.TensorType(name="masks_queries_logits")],
convert_to="mlprogram", compute_precision=ct.precision.FLOAT32, minimum_deployment_target=ct.target.macOS14)
ml.short_description = f"{CHECKPOINT}, class and mask logits, float32, for mcap"
ml.save(OUTPUT)
print(f"Wrote {OUTPUT} in {time.perf_counter() - start:.0f} s.")
if not args.check:
return
import onnxruntime as ort
paths = sorted(p for p in glob.glob(os.path.join(args.check, "**", "*"), recursive=True)
if p.lower().endswith((".png", ".jpg", ".jpeg", ".webp")))
if not paths:
sys.exit(f"No images in {args.check}.")
session = ort.InferenceSession(ONNX, providers=["CPUExecutionProvider"])
coreml = ct.models.MLModel(OUTPUT, compute_units=ct.ComputeUnit.CPU_AND_GPU)
total, worst_iou, unmatched, times = 0, 1.0, 0, []
for path in paths:
pixels = processor(images=Image.open(path).convert("RGB"), return_tensors="np")["pixel_values"].astype(np.float32)
ref = session.run(["class_queries_logits", "masks_queries_logits"], {"pixel_values": pixels})
start = time.perf_counter()
got = coreml.predict({"pixel_values": pixels})
times.append(time.perf_counter() - start)
a = segments_of(processor, ref[0], ref[1])
b = segments_of(processor, got["class_queries_logits"].astype(np.float32), got["masks_queries_logits"].astype(np.float32))
for sa in a["segments_info"]:
total += 1
best = 0.0
for sb in b["segments_info"]:
if sb["label_id"] == sa["label_id"]:
ma, mbk = (a["segmentation"] == sa["id"]).numpy(), (b["segmentation"] == sb["id"]).numpy()
best = max(best, (ma & mbk).sum() / max((ma | mbk).sum(), 1))
if best == 0:
unmatched += 1
else:
worst_iou = min(worst_iou, best)
print(f"{len(paths)} images, {total} segments: worst IoU against the ONNX model {worst_iou:.4f}, unmatched {unmatched}; "
f"{np.median(times[1:] or times) * 1000:.0f} ms an image on the GPU")
if unmatched or worst_iou < 0.98:
sys.exit("The Core ML model segments differently from the ONNX one.")
if __name__ == "__main__":
main()

207
tools/export_ppocr_coreml.py Normal file
View File

@ -0,0 +1,207 @@
# Converts a PP-OCRv5 detector and recognizer to Core ML models for modules/OCR on macOS, then checks
# each against the ONNX model on the CPU. Run once, on a Mac, after modules/OCR/fetch_models.jai;
# mcap only uses the .mlpackage files it writes (into modules/OCR/models, beside the ONNX ones).
#
# python3 -m venv ~/mcap-coreml
# ~/mcap-coreml/bin/pip install torch==2.7.0 coremltools==9.0 onnx onnxruntime pillow
# ~/mcap-coreml/bin/pip install --no-deps onnx2torch torchvision==0.22.0
# ~/mcap-coreml/bin/python tools/export_ppocr_coreml.py --model mobile
# ~/mcap-coreml/bin/python tools/export_ppocr_coreml.py --model server
#
# There is no PyTorch source for PP-OCR (it is PaddlePaddle's), and coremltools no longer reads ONNX, so
# each model goes ONNX -> PyTorch (onnx2torch) -> Core ML, as an ML Program in float32: float16 moves
# the detector's text pixels and changes recognized text, and the Neural Engine gets it wrong.
#
# The shapes. Core ML on the GPU cannot take a flexible height and width for the detector (its
# depthwise convolutions fail to compile: "invalid axis" in mps.expand_dims), so both models take a
# list of sizes (at most 128 a model), which mcap pads its inputs up to; models.config's coreml_ keys
# say the same steps, and must be kept in step with these:
# detector: 1 x 3 x H x W, H a multiple of 128 up to 1536 (det_tile_height), W of 256 up to 2560 (det_max_side)
# recognizer: 16 x 3 x 48 x W, W a multiple of 64 up to 4096 (REC_MAX_WIDTH); fewer lines are padded to 16
# The server models pad some 2 x 2 convolutions and max pools with auto_pad SAME_UPPER, which onnx2torch
# does not take; at stride 1 it is a fixed pixel after, written out as pads first.
# Reshapes that compute their target shape from tensor sizes (the recognizers', the server detector's)
# are frozen by tracing at the traced size; they are rewritten first as constant shapes of 0 (copy the
# dimension) and -1 (infer it), worked out from the shapes onnxruntime gives at several sizes, and
# checked to give the same output.
#
# Measured on an M5 Pro against onnxruntime on the CPU, each at the same (padded) size: the detector
# about 5 times as fast, the recognizer about 4 times for 16 lines, with the same text pixels and text.
# Whole images through mcap (ocr-bench, 8 text pages): mobile 323 ms an image against 536, server 940
# against 2030.
import argparse, itertools, os, sys, time
import numpy as np
import onnx, onnxruntime as ort
from onnx import helper, numpy_helper
import torch, onnx2torch
import coremltools as ct
HERE = os.path.dirname(os.path.abspath(__file__))
MODELS = os.path.join(HERE, "..", "modules", "OCR", "models")
DET_HEIGHTS = list(range(128, 1536 + 1, 128))
DET_WIDTHS = list(range(256, 2560 + 1, 256))
REC_BATCH = 16
REC_HEIGHT = 48
REC_WIDTHS = list(range(64, 4096 + 1, 64))
# coremltools 9.0 casts a folded one-element tensor with int(array), which NumPy 2 refuses ("only
# 0-dimensional arrays can be converted"); the same cast, with the element taken out first.
from coremltools.converters.mil.frontend.torch import ops as ct_ops
_original_cast = ct_ops._cast
def _cast_one_element(context, node, dtype, dtype_name):
x = ct_ops._get_inputs(context, node, expected=1)[0]
if x.can_be_folded_to_const() and np.ndim(x.val) > 0 and np.size(x.val) == 1:
context.add(ct_ops.mb.const(val=dtype(np.asarray(x.val).item()), name=node.name), node.name)
return
_original_cast(context, node, dtype, dtype_name)
ct_ops._cast = _cast_one_element
def constant_reshapes(model, input_shapes):
"""The model with each computed Reshape target made a constant of 0s and -1s."""
graph = model.graph
initializers = {i.name for i in graph.initializer}
producers = {o: n for n in graph.node for o in n.output}
computed = [n for n in graph.node if n.op_type == "Reshape" and n.input[1] not in initializers
and not (n.input[1] in producers and producers[n.input[1]].op_type == "Constant")]
if not computed:
return model
probe = onnx.ModelProto(); probe.CopyFrom(model)
names = []
for n in computed:
for t in (n.input[0], n.output[0]):
if t not in names:
names.append(t)
probe.graph.output.append(helper.make_tensor_value_info(t, onnx.TensorProto.FLOAT, None))
session = ort.InferenceSession(probe.SerializeToString(), providers=["CPUExecutionProvider"])
name = session.get_inputs()[0].name
shapes = []
for shape in input_shapes:
outs = session.run(names, {name: np.zeros(shape, np.float32)})
shapes.append({t: o.shape for t, o in zip(names, outs)})
for n in computed:
ins = [s[n.input[0]] for s in shapes]
outs = [s[n.output[0]] for s in shapes]
target = []
for d in range(len(outs[0])):
sizes = {o[d] for o in outs}
if len(sizes) == 1:
target.append(sizes.pop())
elif all(len(i) > d and i[d] == o[d] for i, o in zip(ins, outs)):
target.append(0)
else:
target.append(-1)
if target.count(-1) > 1:
sys.exit(f"{n.name}: more than one dimension to infer ({target})")
constant = n.name + "_constant_shape"
graph.initializer.append(numpy_helper.from_array(np.array(target, dtype=np.int64), constant))
n.input[1] = constant
print(f" {n.name}: target shape {target}")
return model
def explicit_padding(model):
"""The model with auto_pad SAME_UPPER and SAME_LOWER, which onnx2torch does not take, written out as
pads. Only stride 1 is rewritten: there the padding does not depend on the input's size (the server
models' 2 x 2 convolutions and max pools: one pixel after, none before)."""
for n in model.graph.node:
attrs = {a.name: a for a in n.attribute}
mode = attrs["auto_pad"].s.decode() if "auto_pad" in attrs else "NOTSET"
if mode not in ("SAME_UPPER", "SAME_LOWER"):
continue
kernel = list(attrs["kernel_shape"].ints) if "kernel_shape" in attrs else None
if kernel is None:
sys.exit(f"{n.name}: auto_pad {mode} with no kernel_shape")
strides = list(attrs["strides"].ints) if "strides" in attrs else [1] * len(kernel)
dilations = list(attrs["dilations"].ints) if "dilations" in attrs else [1] * len(kernel)
if any(s != 1 for s in strides):
sys.exit(f"{n.name}: auto_pad {mode} with strides {strides}; only stride 1 is rewritten")
totals = [(k - 1) * d for k, d in zip(kernel, dilations)]
small = [t // 2 for t in totals]
begins = small if mode == "SAME_UPPER" else [t - b for t, b in zip(totals, small)]
ends = [t - b for t, b in zip(totals, begins)]
n.attribute.remove(attrs["auto_pad"])
if "pads" in attrs:
n.attribute.remove(attrs["pads"])
n.attribute.append(helper.make_attribute("pads", begins + ends))
print(f" {n.name} ({n.op_type}): auto_pad {mode} -> pads {begins + ends}")
return model
def convert(onnx_model, example, shape, path):
module = onnx2torch.convert(onnx_model).eval()
with torch.no_grad():
traced = torch.jit.trace(module, torch.from_numpy(example))
ml = ct.convert(traced, inputs=[ct.TensorType(name="x", shape=shape)], outputs=[ct.TensorType(name="y")],
convert_to="mlprogram", compute_precision=ct.precision.FLOAT32,
minimum_deployment_target=ct.target.macOS14)
ml.save(path)
return ct.models.MLModel(path, compute_units=ct.ComputeUnit.CPU_AND_GPU)
def ctc(logits):
texts = []
for row in logits.argmax(-1):
seq, last = [], -1
for i in row:
if i != last and i != 0:
seq.append(int(i))
last = i
texts.append(seq)
return texts
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--model", choices=["mobile", "server"], default="mobile")
args = parser.parse_args()
rng = np.random.default_rng(0)
# Detector.
stem = f"ch_PP-OCRv5_det_{args.model}"
onnx_path = os.path.join(MODELS, stem + ".onnx")
out_path = os.path.join(MODELS, stem + ".mlpackage")
shapes = [(1, 3, h, w) for h in DET_HEIGHTS for w in DET_WIDTHS]
start = time.perf_counter()
model = constant_reshapes(explicit_padding(onnx.load(onnx_path)), [(1, 3, h, w) for h in (256, 512) for w in (512, 768)])
det = convert(model, np.zeros((1, 3, 768, 1280), np.float32),
ct.EnumeratedShapes(shapes=shapes, default=(1, 3, 768, 1280)), out_path)
print(f"Wrote {out_path} ({len(shapes)} sizes) in {time.perf_counter() - start:.1f} s.")
reference = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
worst = 0.0
for (h, w) in [(768, 1280), (1536, 2560), (128, 256)]:
x = rng.standard_normal((1, 3, h, w)).astype(np.float32)
a = reference.run(None, {"x": x})[0]
b = det.predict({"x": x})["y"]
flips = float(np.mean((a > 0.3) != (b > 0.3)))
worst = max(worst, flips)
print(f" {h} x {w}: max difference {np.abs(a - b).max():.2e}, pixels across 0.3: {flips * 100:.4f}%")
if worst > 0.0001:
sys.exit("The Core ML detector is too far from the ONNX one.")
# Recognizer.
stem = f"ch_PP-OCRv5_rec_{args.model}"
onnx_path = os.path.join(MODELS, stem + ".onnx")
out_path = os.path.join(MODELS, stem + ".mlpackage")
print("Recognizer: making its reshapes constant")
model = constant_reshapes(explicit_padding(onnx.load(onnx_path)), [(n, 3, REC_HEIGHT, w) for n in (1, 2) for w in (320, 640)])
shapes = [(REC_BATCH, 3, REC_HEIGHT, w) for w in REC_WIDTHS]
start = time.perf_counter()
rec = convert(model, np.zeros((REC_BATCH, 3, REC_HEIGHT, 640), np.float32),
ct.EnumeratedShapes(shapes=shapes, default=(REC_BATCH, 3, REC_HEIGHT, 640)), out_path)
print(f"Wrote {out_path} ({len(shapes)} widths) in {time.perf_counter() - start:.1f} s.")
reference = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
for w in (64, 640, 4096):
x = rng.standard_normal((REC_BATCH, 3, REC_HEIGHT, w)).astype(np.float32) * 0.5
a = reference.run(None, {"x": x})[0]
b = rec.predict({"x": x})["y"]
same = np.mean([p == q for p, q in zip(ctc(a), ctc(b))])
print(f" width {w}: max difference {np.abs(a - b).max():.2e}, decoded the same {same * 100:.0f}%")
if same < 1:
sys.exit("The Core ML recognizer reads differently from the ONNX one.")
if __name__ == "__main__":
main()

115
tools/export_siglip_coreml.py Normal file
View File

@ -0,0 +1,115 @@
# Converts SigLIP 2's vision encoder to a Core ML model for modules/Labels on macOS, then checks it
# against the ONNX model mcap runs elsewhere, on a folder of images. Run once, on a Mac; mcap only uses
# the .mlpackage it writes (into modules/Labels/models, where the build copies it from).
#
# python3 -m venv ~/mcap-coreml
# ~/mcap-coreml/bin/pip install torch==2.7.0 transformers coremltools==9.0 onnxruntime pillow sentencepiece protobuf
# ~/mcap-coreml/bin/python tools/export_siglip_coreml.py --check FOLDER_OF_IMAGES
#
# torch 2.7.0 is the newest coremltools 9.0 was tested with. The model is the vision tower of
# google/siglip2-base-patch16-384, from pixel_values [1, 3, 384, 384] (RGB resized to 384 x 384,
# scaled to -1..1) to pooler_output [1, 768], as the ONNX one; an ML Program in float16, for the GPU.
# Measured on an M5 Pro: 6.7 ms an image on the GPU, against 67 ms for onnxruntime on the CPU, with
# vectors within cosine 0.99994 of the CPU's. The Neural Engine gets float16 badly wrong for this
# model (cosine 0.72), so mcap runs it on the CPU and GPU only.
import argparse, glob, os, sys, time
import numpy as np
import torch
from PIL import Image
import coremltools as ct
from transformers import AutoModel
from transformers.utils import logging
logging.set_verbosity_error()
CHECKPOINT = "google/siglip2-base-patch16-384"
SIZE = 384
HERE = os.path.dirname(os.path.abspath(__file__))
MODELS = os.path.join(HERE, "..", "modules", "Labels", "models")
OUTPUT = os.path.join(MODELS, "siglip2-base-patch16-384-vision.mlpackage")
ONNX = os.path.join(MODELS, "siglip2-base-patch16-384-vision.onnx")
# coremltools 9.0 casts a folded one-element tensor with int(array), which NumPy 2 refuses ("only
# 0-dimensional arrays can be converted"); the same cast, with the element taken out first.
from coremltools.converters.mil.frontend.torch import ops as ct_ops
_original_cast = ct_ops._cast
def _cast_one_element(context, node, dtype, dtype_name):
x = ct_ops._get_inputs(context, node, expected=1)[0]
if x.can_be_folded_to_const() and np.ndim(x.val) > 0 and np.size(x.val) == 1:
context.add(ct_ops.mb.const(val=dtype(np.asarray(x.val).item()), name=node.name), node.name)
return
_original_cast(context, node, dtype, dtype_name)
ct_ops._cast = _cast_one_element
class Pooled(torch.nn.Module):
"""The vision tower, from pixel values to the pooled vector, as the ONNX graph has it."""
def __init__(self, model):
super().__init__()
self.model = model
def forward(self, pixel_values):
return self.model(pixel_values=pixel_values).pooler_output
def load_images(folder, limit):
paths = sorted(p for p in glob.glob(os.path.join(folder, "**", "*"), recursive=True)
if p.lower().endswith((".png", ".jpg", ".jpeg", ".webp")))[:limit]
images = []
for p in paths:
image = Image.open(p).convert("RGB").resize((SIZE, SIZE), Image.BICUBIC)
a = np.asarray(image, dtype=np.float32) / 255.0
images.append(((a - 0.5) / 0.5).transpose(2, 0, 1))
return np.stack(images).astype(np.float32)
def worst_cosine(a, b):
a = a / np.linalg.norm(a, axis=-1, keepdims=True)
b = b / np.linalg.norm(b, axis=-1, keepdims=True)
return float(np.min(np.sum(a * b, axis=-1)))
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--check", metavar="FOLDER", help="images to check the Core ML model against the ONNX one on")
parser.add_argument("--limit", type=int, default=32)
args = parser.parse_args()
model = Pooled(AutoModel.from_pretrained(CHECKPOINT, torch_dtype=torch.float32).vision_model.eval()).eval()
example = torch.zeros(1, 3, SIZE, SIZE)
with torch.no_grad():
traced = torch.jit.trace(model, example)
start = time.perf_counter()
ml = ct.convert(traced, inputs=[ct.TensorType(name="pixel_values", shape=(1, 3, SIZE, SIZE))],
outputs=[ct.TensorType(name="pooler_output")],
convert_to="mlprogram", compute_precision=ct.precision.FLOAT16,
minimum_deployment_target=ct.target.macOS14)
ml.short_description = f"{CHECKPOINT} vision encoder (pooler_output), float16, for mcap"
ml.save(OUTPUT)
print(f"Wrote {OUTPUT} in {time.perf_counter() - start:.1f} s.")
if not args.check:
return
import onnxruntime as ort
images = load_images(args.check, args.limit)
if not len(images):
sys.exit(f"No images in {args.check}.")
session = ort.InferenceSession(ONNX, providers=["CPUExecutionProvider"])
name = session.get_inputs()[0].name
reference = np.concatenate([session.run(["pooler_output"], {name: images[i:i + 1]})[0] for i in range(len(images))])
coreml = ct.models.MLModel(OUTPUT, compute_units=ct.ComputeUnit.CPU_AND_GPU)
coreml.predict({"pixel_values": images[:1]})
start = time.perf_counter()
got = np.concatenate([coreml.predict({"pixel_values": images[i:i + 1]})["pooler_output"] for i in range(len(images))])
per = (time.perf_counter() - start) / len(images)
cosine = worst_cosine(got.astype(np.float32), reference)
print(f"{len(images)} images: {per * 1000:.1f} ms an image on the GPU, worst cosine to the ONNX model on the CPU {cosine:.5f}.")
if cosine < 0.999:
sys.exit("The Core ML model is too far from the ONNX one.")
if __name__ == "__main__":
main()

2
tools/image_probe.jai
View File

@ -66,7 +66,7 @@ main :: () {
start := current_time_monotonic();
image, decoded, error := image_decode(bytes, kind);
if !decoded { print(" cannot decode: %\n", error); failures += 1; continue; }
print(" decoded: % x % in % ms (%)\n", image.width, image.height, FF(to_float64_seconds(current_time_monotonic() - start) * 1000, 1), ifx image.from_stb then "stb_image" else "WIC or turned");
print(" decoded: % x % in % ms (%)\n", image.width, image.height, FF(to_float64_seconds(current_time_monotonic() - start) * 1000, 1), ifx image.from_stb then "stb_image" else "WIC, ImageIO or turned");
if out_folder {
name := path_strip_extension(path_filename(path));
full := stbi_write_png(temp_c_string(tprint("%/%.png", out_folder, name)), image.width, image.height, 4, image.pixels, image.width * 4);

4
tools/segment_probe.jai
View File

@ -33,7 +33,7 @@ main :: () {
engine, ok, error := segment_create(models_folder, model_name, prefer_gpu);
if !ok { print("Cannot start %: %\n", model_name, error); exit(1); }
defer segment_destroy(engine);
print("% on %, % classes (made in % ms)%\n", model_name, engine.provider, engine.classes.count, ms(start),
print("% on %, % classes (made in % ms)%\n", model_name, segment_provider_name(engine), engine.classes.count, ms(start),
ifx engine.gpu_explanation then tprint(": %", engine.gpu_explanation) else "");
failures := 0;
@ -129,7 +129,7 @@ main :: () {
print("\n% images: decoding % ms each, segmenting % ms each (model and post-processing, batch %)\n", done,
formatFloat(decode_ms / max(done, 1), trailing_width = 0), formatFloat(model_ms / max(done, 1), trailing_width = 0),
ifx engine.provider == .CUDA then engine.model.batch_gpu else engine.model.batch_cpu);
segment_batch(engine));
per := max(engine.stats.images, 1) * 1000.0;
print(" of which: preparing % ms, the model % ms, making segments % ms, per image\n", formatFloat(engine.stats.prepare_us / per, trailing_width = 1),
formatFloat(engine.stats.model_us / per, trailing_width = 1), formatFloat(engine.stats.post_us / per, trailing_width = 1));