mcap on macOS: building again, then feature parity
+Planning document · 2026-10-10 · for mcap after 0.1.11 · not started
+
+ In short
+
+
+
+
+-
+
- Where it stands: mcap was built to compile on macOS (Apple silicon, SDL3 and Metal), but it hasn't been built on a Mac since at least 0.0.8. Two build breaks were found and fixed in 0.1.11 without a Mac. Everything added since the template is Windows first, and much of it Windows only. +
- Proposed Eight phases, smallest risk first:
+
-
+
- Build and run on a Mac. +
- Fill the small platform gaps: choosing a folder, the Trash, the clipboard, the temporary folder. +
- Images through ImageIO. +
- Video through FFmpeg for macOS. +
- The models on the GPU through CoreML. +
- Screen capture stills. +
- Screen recording. +
- A signed release. +
+ - Proposed One codebase, the same file formats. An
.mcapmade on either system opens on the other. Where Windows uses WIC, DXGI and WASAPI, macOS uses ImageIO and ScreenCaptureKit. FFmpeg, onnxruntime, the encoder, the remux and the mixer stay shared.
+ - Proposed ScreenCaptureKit through a small Objective-C shim. It is asynchronous: completion blocks, delegates, dispatch queues. A few hundred lines of Objective-C, compiled with clang into a static library with a plain C interface, is far safer than doing that from Jai. Open question 6. +
- The big unknowns need a Mac to settle. Which macOS to require (14 Sonoma is proposed). Signing: macOS remembers the Screen Recording permission per signature, and an ad hoc signature changes with every build. What the capture keys should be: Mac keyboards have no PrintScreen. See open questions. +
1. Where it stands
+From the code; none of this has been run on a Mac in this project's history.
+| Part | macOS today |
|---|---|
| Window, ImGui, themes, fonts | SDL3 with SDL_GPU on Metal (src/backends/sdl3_gpu3.jai). The menu-bar tray is SDL's. Fonts and CJK fallbacks have macOS paths (platform_macos.jai). |
| Build | jai first.jai makes bin/mcap-macos-debug, and with - release bin/mcap.app, signed ad hoc. Apple silicon only, with a native jai. The .app holds only the executable and libSDL3.0.dylib. onnxruntime and the models are copied beside it, not into it. |
| Type check from Windows | Not possible: jai check_targets.jai - macos runs only on a Mac, because the Objective-C module runs code at compile time. |
| Known breaks, fixed in 0.1.11 |
+
|
| Library location | Hard-coded as D:/McapIndex/mcap (VIEWER_FOLDER). On a Mac only -library works. |
| Global hotkeys | Carbon RegisterEventHotKey (macos_global_hotkeys.jai). capture_region and record_region default to none. |
| OCR, labels, segmentation | onnxruntime 1.30.0 for macOS arm64 is in fetch.jai. Sessions use CUDA or the CPU (session.jai), so on a Mac it's CPU only, several times slower than a GPU. |
2. What is Windows only
+| Feature | Windows | macOS today | macOS plan |
|---|---|---|---|
| Images other than PNG, JPEG, GIF, BMP, PSD, TGA | WIC (Image_Decode/wic.jai) | Refused | ImageIO, CGImageSource (phase 3) |
| Videos: import, frame for the thumbnail | FFmpeg (modules/ffmpeg, Windows bindings only) | Skipped | FFmpeg for macOS (phase 4) |
| Import: choosing a folder | Explorer's picker (windows_choose_folder) | Typed path only | NSOpenPanel (phase 2) |
| Delete to the bin | Recycle Bin (SHFileOperationW) | A "deleted" folder beside the file | NSFileManager trashItemAtURL (phase 2) |
| Temporary folder (open original, recordings) | %TEMP% | /tmp | NSTemporaryDirectory() (phase 2) |
| Image to the clipboard | CF_DIB | None | NSPasteboard, PNG (phase 2) |
| Models on the GPU | CUDA | CPU | CoreML execution provider (phase 5) |
| Screen capture stills | GDI overlay, BitBlt | None | Cocoa overlay, ScreenCaptureKit screenshots (phase 6) |
| Screen recording, border, toolbar | DXGI or Windows.Graphics.Capture, GDI windows | None | ScreenCaptureKit stream, Cocoa panels (phase 7) |
| Sound | WASAPI | None | ScreenCaptureKit audio, plus its microphone or AVAudioEngine (phase 7) |
| Release | release.jai: a zip with CUDA | None | A signed .app in a .dmg (phase 8) |
Already shared and fine as they are:
+-
+
- The
.mcapformat, catalog, search, Stats.
+ .snagximport, EXIF (exif.jaireads it itself).
+ - The settings and hotkey files. +
- The encoder, remux and mixer in
modules/Screen_Record. Their capture sources aren't shared.
+
3. Approach
+-
+
- Same interfaces, a second implementation. Each Windows-only piece already sits behind a function the app calls. The macOS version goes beside it, in its own file:
+
-
+
platform_screen_capture_select,platform_copy_image_to_clipboard.
+ image_decode's WIC branch.
+ video_poster.
+ Screen_Record's source: a third source besidedxgi_sourceandwgc_source.
+
screen_capture.jai,screen_record.jai,import.jai) shouldn't need to know which system they're on beyond what they do now.
+ - FFmpeg on macOS too, rather than AVFoundation. Video import, the thumbnail frame, the recording encoder and the remux are FFmpeg code already. On a Mac the encoder would be
h264_videotoolbox, Apple's hardware encoder, put first inRECORD_ENCODERS. The cost is shipping FFmpeg's dylibs in the.app, and bindings for their macOS names.
+ - The Objective-C shim (
modules/Screen_Record/macos/capture.m) holds ScreenCaptureKit and nothing else. It has a C interface: +-
+
mcap_sck_screenshot
+ mcap_sck_stream_start
+ mcap_sck_next_frame
+ mcap_sck_next_audio
+ mcap_sck_stream_stop
+
first.jaiwith clang when building on a Mac, and linked statically. AppKit work that is plain message sends (NSOpenPanel,NSPasteboard, the overlay and toolbar windows) stays in Jai through the Objective-C module the app already uses.
+ - The build is checked on every phase with
jai check_targets.jai - macosand the test suites on the Mac.jai check_targets.jai - linuxon Windows catches the shared files between Mac sessions.
+
4. Screen capture on macOS
+-
+
- Stills: the moment the key is pressed, each screen is copied:
+
-
+
- Copied with
SCScreenshotManager captureImageWithFilter, on macOS 14 and later.CGWindowListCreateImageis deprecated in macOS 15.
+ - Shown frozen in a borderless
NSWindowper screen, at the screen-saver level. A window can't span screens on macOS as the one Windows overlay does.
+ - The drag, magnifier and keys work as on Windows. The overlay's drawing (darkened copy, bright selection, outline, magnifier) becomes Core Graphics in each window's
drawRect, or anNSImagedrawn by the shim.
+
+ - Copied with
- Points and pixels: AppKit works in points. A Retina screen has 2 pixels a point, and screens can differ. The region is chosen in points and converted to each screen's pixels for the capture. The 240 × 180 minimum is in pixels, as on Windows. +
- Recording: an
SCStreamfor the region's display (sourceRect) delivers BGRACVPixelBuffers with timestamps on the host clock. +-
+
- The shim hands them over as the pipeline's frames, and the encoder and the rest are unchanged. +
- The cursor:
showsCursor.
+ - Keeping mcap's own border and toolbar out: an
SCContentFilterthat excludes the app's windows. That's cleaner than Windows' window flag, and doesn't depend on it.
+
+ - Sound:
+
-
+
- System sound comes from the same stream (
capturesAudio, macOS 13 on), as 48 kHz float.
+ - The microphone comes from the stream too on macOS 15 (
captureMicrophone), else fromAVAudioEngine's input.
+ - Both go into the mixer as packets with host-clock times, as WASAPI's do. +
+ - System sound comes from the same stream (
- The border and toolbar: non-activating
NSPanels at the floating level. The toolbar's buttons ask, and the app acts on its next frame, as on Windows.
+ - The window it was taken from: the frontmost app comes from
NSWorkspace, and window titles fromCGWindowListCopyWindowInfo. The titles need the Screen Recording permission, which capture needs anyway.
+ - Hotkeys: no PrintScreen on Mac keyboards. ⌘⇧3, 4 and 5 are the system's screenshot keys. See open question 3. +
5. Permissions and signing
+-
+
- Screen Recording (Privacy & Security) must be granted before any capture. The first capture asks; until it is granted, the overlay gets only the desktop, without other apps' windows. mcap will check (
CGPreflightScreenCaptureAccess) and say what to do instead of showing an empty capture.
+ - Microphone permission is asked the first time the Mic button is turned on.
Info.plistgainsNSMicrophoneUsageDescription.
+ - macOS remembers these per code signature. The release
.appis signed ad hoc today, and an ad hoc signature changes with every build, so each build would ask again, or silently stay refused. A stable signing identity solves it: an Apple Development certificate for builds, and a Developer ID for releases, notarized so Gatekeeper opens it. See open question 4.
+
6. Phases
+Each phase ends with something to run and check on the Mac, and is a commit. They are ordered so a usable mcap comes first: by the end of phase 4 the library browser, search, imports and OCR all work.
+ +
+
+
+Phase 1: Build and run on a Mac
+-
+
- The type check:
jai check_targets.jai - macos, fixing what it finds.
+ - The builds:
jai first.jaiand- release.
+ - A library folder for macOS: a default in place of
D:/McapIndex/mcap, open question 5.
+ - The
.appmade whole:libonnxruntime.dylibinContents/Frameworks, and the OCR, label and segmentation models inContents/Resources, found from there.
+ - Tests:
run_tests -headlesson the Mac.
+
Done when:
+-
+
- Both builds succeed and the test suites pass on the Mac. +
mcap.app, started from Finder, shows a library copied from Windows: thumbnails, the Capture window, search, Stats.
+ - An OCR scan reads captures on the CPU, and its time per capture is recorded. +
+
+
+Phase 2: The small platform gaps
+-
+
- Choosing a folder:
NSOpenPanelfor Import's Browse button.
+ - Delete: into the Trash with
NSFileManager trashItemAtURL, with "Move to Trash" in the delete dialog.
+ - Temporary files:
NSTemporaryDirectory()for opening an original and for recordings.
+ - The clipboard:
platform_copy_image_to_clipboardthroughNSPasteboard, as PNG and TIFF.
+
Done when: Browse picks a folder; a deleted capture is in the Trash and can be put back from there; Open plays a video's original; an image pasted from the clipboard (phase 6 is its first user, so a test fills it) pastes into Preview.
+
+
+
+Phase 3: Images through ImageIO
+Image_Decode/imageio.jai: CGImageSourceCreateWithData and CGImageSourceCreateImageAtIndex, drawn into an RGBA bitmap context. It takes the formats WIC takes on Windows:
-
+
- HEIC and AVIF (AVIF from macOS 13). +
- WebP (from macOS 11). +
- TIFF. +
- The camera RAW formats. +
EXIF orientation stays exif.jai's. ImageIO is asked for the image as stored, so it isn't turned twice.
Done when: image-probe on the Mac decodes the same test images as on Windows, the same way up and the same size, and importing an iPhone photo folder makes the same .mcap files, except for the import date.
+
+
+Phase 4: Video through FFmpeg for macOS
+-
+
- FFmpeg itself: a shared FFmpeg 6 build for macOS arm64, with VideoToolbox and the native AAC encoder. Its dylibs and their SHA-256 sums go in
modules/ffmpeg/fetch.jai, as onnxruntime's are.
+ - Bindings:
modules/ffmpeg/macos.jai, from the same headers with the dylib names, and#loaded on macOS.
+ - The video code:
video_ffmpeg.jaiand the import's video path, no longer Windows only.
+ - The
.app: the dylibs go in it.
+
Done when: importing a folder with MP4, MOV and HEVC iPhone videos gives the same .mcap files as on Windows, with thumbnails, and they play with Open.
+
+
+Phase 5: The models on the GPU, with CoreML
+-
+
- CoreML first:
session.jaiasks onnxruntime for the CoreML execution provider on macOS (SessionOptionsAppendExecutionProviderwith"CoreML", the ML Program format), and the CPU for what it can't take.
+ - Measured per model: PP-OCR detection and recognition, SigLIP 2, Mask2Former. Each is timed against the CPU and checked to give the same output as on Windows, within the tolerances the Windows probes already use. +
- The status line says CoreML, as it says CUDA on Windows. +
Done when: OCR, labels and segmentation give the same results as on Windows on the probes' test images, and each is faster than the CPU, or is left on the CPU with the reason recorded.
+
+
+
+Phase 6: Screen capture stills
+-
+
- The shim's screenshot (
mcap_sck_screenshot, per display).
+ - The overlay: a window per screen, with the drag, magnifier, keys, and a click for the whole screen. +
- The window it came from: its title and app. +
- The clipboard (phase 2). +
- The capture hotkey (open question 3). +
- Permission: the check, and what mcap says when Screen Recording isn't granted. +
Done when: on a Mac with a Retina screen and an external non-Retina one:
+-
+
- The hotkey freezes both screens. +
- A region dragged on either is taken at the screen's own pixels, and is on the clipboard and in Screen Captures with its window's title and app. +
- With the permission refused, mcap says how to grant it. +
+
+
+Phase 7: Screen recording
+-
+
- The source:
modules/Screen_Record/sck_source.jaiover the shim's stream, as the third source. The encoder triesh264_videotoolboxfirst.
+ - Sound: system sound and the microphone, through the existing mixer. +
- The overlay's video mode, with the 240 × 180 minimum. +
- The border and toolbar as panels, kept out of the stream by the content filter. Pause, Discard, the sound buttons, and leftovers after a crash. +
Done when the same checks pass as on Windows:
+-
+
record-proberuns on the Mac, with-sync-test,-pauseand-mic-toggle.
+ - Neither border nor toolbar is in a recording. +
- A 10 s pause leaves no gap. +
- Beeps stay within about 25 ms of their flashes with no drift over 3 minutes. +
- A killed recording is offered at the next start. +
+
+
+Phase 8: A signed release
+-
+
release.jaifor macOS: the.appwith everything in it (SDL3, onnxruntime, FFmpeg, the models), signed with a Developer ID with the hardened runtime and the entitlements it needs.
+ - Notarized and stapled. +
- Packaged in a
.dmg, with its SHA-256 printed, as the Windows zip's is.
+
Done when: the .dmg, downloaded on a second Mac, opens without a Gatekeeper warning, asks once for Screen Recording, and keeps that permission across an update to the next release.
7. Risks
+No Mac in this loop yet. Everything here can be written on Windows, but only built, run and checked on a Mac. Each phase needs a session there, yours or a remote one (open question 1). The Linux type check catches the shared files in between, but not the Objective-C or the macOS-only files.
+CoreML and the models. CoreML takes some operators and not others. Mask2Former's attention and the panoptic post-processing may fall back to the CPU in parts, which can be slower than the CPU alone, as the data moves back and forth. Phase 5 measures each model, and keeps the CPU where CoreML doesn't win.
+Permissions and signing. Without a stable signature, Screen Recording has to be granted again after every build, and a refused permission gives a capture of the empty desktop rather than an error. Development needs a certificate early, by phase 6.
+FFmpeg licensing. The Windows FFmpeg is a GPL build (with libx264). On macOS an LGPL build is enough: VideoToolbox does H.264, and FFmpeg's own AAC encoder does sound. That avoids GPL obligations for the Mac release; the Windows release should be checked against the same question.
+Multi-screen overlays. One window per screen, with Spaces, full-screen apps and screens at different scales, is where macOS differs most from Windows. The overlay must sit above full-screen apps (
+
+NSWindowCollectionBehaviorCanJoinAllSpaces and FullScreenAuxiliary).8. Open questions
+-
+
- The Mac: which Mac and macOS version will the work be checked on? Can I build and run there (SSH, or a remote session), or will you run the builds and send back the output? +
- The oldest macOS: 14 Sonoma, for ScreenCaptureKit's screenshots, is proposed. 13 would need the deprecated
CGWindowListCreateImagefor stills; 15 would give the microphone in the stream itself.
+ - The keys: Mac keyboards have no PrintScreen, and ⌘⇧3/4/5 are the system's. Proposed: ⌘⇧2 for a still and ⌃⌘⇧2 to record, or F13 (PrintScreen on Apple's full keyboards) and ⌃F13, or your own. +
- Signing: do you have, or want, an Apple Developer account (99 USD a year) for a stable development signature now and Developer ID notarization for releases? Without it, phases 6 and 7 work but ask for permission after every build. +
- The library on a Mac: where should it default to? Proposed:
~/Pictures/mcap. Should the library folder become a setting on both systems, in place of the hard-codedD:/McapIndex/mcap?
+ - The shim: is a small Objective-C file built with clang acceptable in the repo, or should ScreenCaptureKit be done from Jai alone, at more cost and risk? +