mcap on macOS: building again, then feature parity

Planning document · 2026-10-10 · for mcap after 0.1.11 · not started

In short

Where it stands: mcap was built to compile on macOS (Apple silicon, SDL3 and Metal), but it hasn't been built on a Mac since at least 0.0.8. Two build breaks were found and fixed in 0.1.11 without a Mac. Everything added since the template is Windows first, and much of it Windows only.
Proposed Eight phases, smallest risk first:
1. Build and run on a Mac.
2. Fill the small platform gaps: choosing a folder, the Trash, the clipboard, the temporary folder.
3. Images through ImageIO.
4. Video through FFmpeg for macOS.
5. The models on the GPU through CoreML.
6. Screen capture stills.
7. Screen recording.
8. A signed release.
Proposed One codebase, the same file formats. An .mcap made on either system opens on the other. Where Windows uses WIC, DXGI and WASAPI, macOS uses ImageIO and ScreenCaptureKit. FFmpeg, onnxruntime, the encoder, the remux and the mixer stay shared.
Proposed ScreenCaptureKit through a small Objective-C shim. It is asynchronous: completion blocks, delegates, dispatch queues. A few hundred lines of Objective-C, compiled with clang into a static library with a plain C interface, is far safer than doing that from Jai. Open question 6.
The big unknowns need a Mac to settle. Which macOS to require (14 Sonoma is proposed). Signing: macOS remembers the Screen Recording permission per signature, and an ad hoc signature changes with every build. What the capture keys should be: Mac keyboards have no PrintScreen. See open questions.

1. Where it stands

From the code; none of this has been run on a Mac in this project's history.

Part	macOS today
Window, ImGui, themes, fonts	SDL3 with SDL_GPU on Metal (`src/backends/sdl3_gpu3.jai`). The menu-bar tray is SDL's. Fonts and CJK fallbacks have macOS paths (`platform_macos.jai`).
Build	`jai first.jai` makes `bin/mcap-macos-debug`, and with `- release` `bin/mcap.app`, signed ad hoc. Apple silicon only, with a native `jai`. The `.app` holds only the executable and `libSDL3.0.dylib`. onnxruntime and the models are copied beside it, not into it.
Type check from Windows	Not possible: `jai check_targets.jai - macos` runs only on a Mac, because the Objective-C module runs code at compile time.
Known breaks, fixed in 0.1.11	`unix_sdl3_gpu3.jai` pulled all of ImGui into the app's scope, so ImGui's `Image` clashed with Image_Decode's (broken since 0.0.8). A Windows `RECT` was in the shared `App` struct (since 0.1.6). The Linux check passes after both, and macOS shares the same files.
Library location	Hard-coded as `D:/McapIndex/mcap` (`VIEWER_FOLDER`). On a Mac only `-library` works.
Global hotkeys	Carbon `RegisterEventHotKey` (`macos_global_hotkeys.jai`). `capture_region` and `record_region` default to `none`.
OCR, labels, segmentation	onnxruntime 1.30.0 for macOS arm64 is in `fetch.jai`. Sessions use CUDA or the CPU (`session.jai`), so on a Mac it's CPU only, several times slower than a GPU.

2. What is Windows only

Feature	Windows	macOS today	macOS plan
Images other than PNG, JPEG, GIF, BMP, PSD, TGA	WIC (`Image_Decode/wic.jai`)	Refused	ImageIO, `CGImageSource` (phase 3)
Videos: import, frame for the thumbnail	FFmpeg (`modules/ffmpeg`, Windows bindings only)	Skipped	FFmpeg for macOS (phase 4)
Import: choosing a folder	Explorer's picker (`windows_choose_folder`)	Typed path only	`NSOpenPanel` (phase 2)
Delete to the bin	Recycle Bin (`SHFileOperationW`)	A "deleted" folder beside the file	`NSFileManager trashItemAtURL` (phase 2)
Temporary folder (open original, recordings)	`%TEMP%`	`/tmp`	`NSTemporaryDirectory()` (phase 2)
Image to the clipboard	`CF_DIB`	None	`NSPasteboard`, PNG (phase 2)
Models on the GPU	CUDA	CPU	CoreML execution provider (phase 5)
Screen capture stills	GDI overlay, `BitBlt`	None	Cocoa overlay, ScreenCaptureKit screenshots (phase 6)
Screen recording, border, toolbar	DXGI or Windows.Graphics.Capture, GDI windows	None	ScreenCaptureKit stream, Cocoa panels (phase 7)
Sound	WASAPI	None	ScreenCaptureKit audio, plus its microphone or AVAudioEngine (phase 7)
Release	`release.jai`: a zip with CUDA	None	A signed `.app` in a `.dmg` (phase 8)

Already shared and fine as they are:

The .mcap format, catalog, search, Stats.
.snagx import, EXIF (exif.jai reads it itself).
The settings and hotkey files.
The encoder, remux and mixer in modules/Screen_Record. Their capture sources aren't shared.

3. Approach

Same interfaces, a second implementation. Each Windows-only piece already sits behind a function the app calls. The macOS version goes beside it, in its own file:
- platform_screen_capture_select, platform_copy_image_to_clipboard.
- image_decode's WIC branch.
- video_poster.
- Screen_Record's source: a third source beside dxgi_source and wgc_source.
The app's own files (screen_capture.jai, screen_record.jai, import.jai) shouldn't need to know which system they're on beyond what they do now.
FFmpeg on macOS too, rather than AVFoundation. Video import, the thumbnail frame, the recording encoder and the remux are FFmpeg code already. On a Mac the encoder would be h264_videotoolbox, Apple's hardware encoder, put first in RECORD_ENCODERS. The cost is shipping FFmpeg's dylibs in the .app, and bindings for their macOS names.
The Objective-C shim (modules/Screen_Record/macos/capture.m) holds ScreenCaptureKit and nothing else. It has a C interface:
- mcap_sck_screenshot
- mcap_sck_stream_start
- mcap_sck_next_frame
- mcap_sck_next_audio
- mcap_sck_stream_stop
It is compiled by first.jai with clang when building on a Mac, and linked statically. AppKit work that is plain message sends (NSOpenPanel, NSPasteboard, the overlay and toolbar windows) stays in Jai through the Objective-C module the app already uses.
The build is checked on every phase with jai check_targets.jai - macos and the test suites on the Mac. jai check_targets.jai - linux on Windows catches the shared files between Mac sessions.

4. Screen capture on macOS

Stills: the moment the key is pressed, each screen is copied:
- Copied with SCScreenshotManager captureImageWithFilter, on macOS 14 and later. CGWindowListCreateImage is deprecated in macOS 15.
- Shown frozen in a borderless NSWindow per screen, at the screen-saver level. A window can't span screens on macOS as the one Windows overlay does.
- The drag, magnifier and keys work as on Windows. The overlay's drawing (darkened copy, bright selection, outline, magnifier) becomes Core Graphics in each window's drawRect, or an NSImage drawn by the shim.
Points and pixels: AppKit works in points. A Retina screen has 2 pixels a point, and screens can differ. The region is chosen in points and converted to each screen's pixels for the capture. The 240 × 180 minimum is in pixels, as on Windows.
Recording: an SCStream for the region's display (sourceRect) delivers BGRA CVPixelBuffers with timestamps on the host clock.
- The shim hands them over as the pipeline's frames, and the encoder and the rest are unchanged.
- The cursor: showsCursor.
- Keeping mcap's own border and toolbar out: an SCContentFilter that excludes the app's windows. That's cleaner than Windows' window flag, and doesn't depend on it.
Sound:
- System sound comes from the same stream (capturesAudio, macOS 13 on), as 48 kHz float.
- The microphone comes from the stream too on macOS 15 (captureMicrophone), else from AVAudioEngine's input.
- Both go into the mixer as packets with host-clock times, as WASAPI's do.
The border and toolbar: non-activating NSPanels at the floating level. The toolbar's buttons ask, and the app acts on its next frame, as on Windows.
The window it was taken from: the frontmost app comes from NSWorkspace, and window titles from CGWindowListCopyWindowInfo. The titles need the Screen Recording permission, which capture needs anyway.
Hotkeys: no PrintScreen on Mac keyboards. ⌘⇧3, 4 and 5 are the system's screenshot keys. See open question 3.

5. Permissions and signing

Screen Recording (Privacy & Security) must be granted before any capture. The first capture asks; until it is granted, the overlay gets only the desktop, without other apps' windows. mcap will check (CGPreflightScreenCaptureAccess) and say what to do instead of showing an empty capture.
Microphone permission is asked the first time the Mic button is turned on. Info.plist gains NSMicrophoneUsageDescription.
macOS remembers these per code signature. The release .app is signed ad hoc today, and an ad hoc signature changes with every build, so each build would ask again, or silently stay refused. A stable signing identity solves it: an Apple Development certificate for builds, and a Developer ID for releases, notarized so Gatekeeper opens it. See open question 4.

6. Phases

Each phase ends with something to run and check on the Mac, and is a commit. They are ordered so a usable mcap comes first: by the end of phase 4 the library browser, search, imports and OCR all work.

Phase 1: Build and run on a Mac

The type check: jai check_targets.jai - macos, fixing what it finds.
The builds: jai first.jai and - release.
A library folder for macOS: a default in place of D:/McapIndex/mcap, open question 5.
The .app made whole: libonnxruntime.dylib in Contents/Frameworks, and the OCR, label and segmentation models in Contents/Resources, found from there.
Tests: run_tests -headless on the Mac.

Done when:

Both builds succeed and the test suites pass on the Mac.
mcap.app, started from Finder, shows a library copied from Windows: thumbnails, the Capture window, search, Stats.
An OCR scan reads captures on the CPU, and its time per capture is recorded.

Phase 2: The small platform gaps

Choosing a folder: NSOpenPanel for Import's Browse button.
Delete: into the Trash with NSFileManager trashItemAtURL, with "Move to Trash" in the delete dialog.
Temporary files: NSTemporaryDirectory() for opening an original and for recordings.
The clipboard: platform_copy_image_to_clipboard through NSPasteboard, as PNG and TIFF.

Done when: Browse picks a folder; a deleted capture is in the Trash and can be put back from there; Open plays a video's original; an image pasted from the clipboard (phase 6 is its first user, so a test fills it) pastes into Preview.

Phase 3: Images through ImageIO

Image_Decode/imageio.jai: CGImageSourceCreateWithData and CGImageSourceCreateImageAtIndex, drawn into an RGBA bitmap context. It takes the formats WIC takes on Windows:

HEIC and AVIF (AVIF from macOS 13).
WebP (from macOS 11).
TIFF.
The camera RAW formats.

EXIF orientation stays exif.jai's. ImageIO is asked for the image as stored, so it isn't turned twice.

Done when: image-probe on the Mac decodes the same test images as on Windows, the same way up and the same size, and importing an iPhone photo folder makes the same .mcap files, except for the import date.

Phase 4: Video through FFmpeg for macOS

FFmpeg itself: a shared FFmpeg 6 build for macOS arm64, with VideoToolbox and the native AAC encoder. Its dylibs and their SHA-256 sums go in modules/ffmpeg/fetch.jai, as onnxruntime's are.
Bindings: modules/ffmpeg/macos.jai, from the same headers with the dylib names, and #loaded on macOS.
The video code: video_ffmpeg.jai and the import's video path, no longer Windows only.
The .app: the dylibs go in it.

Done when: importing a folder with MP4, MOV and HEVC iPhone videos gives the same .mcap files as on Windows, with thumbnails, and they play with Open.

Phase 5: The models on the GPU, with CoreML

CoreML first: session.jai asks onnxruntime for the CoreML execution provider on macOS (SessionOptionsAppendExecutionProvider with "CoreML", the ML Program format), and the CPU for what it can't take.
Measured per model: PP-OCR detection and recognition, SigLIP 2, Mask2Former. Each is timed against the CPU and checked to give the same output as on Windows, within the tolerances the Windows probes already use.
The status line says CoreML, as it says CUDA on Windows.

Done when: OCR, labels and segmentation give the same results as on Windows on the probes' test images, and each is faster than the CPU, or is left on the CPU with the reason recorded.

Phase 6: Screen capture stills

The shim's screenshot (mcap_sck_screenshot, per display).
The overlay: a window per screen, with the drag, magnifier, keys, and a click for the whole screen.
The window it came from: its title and app.
The clipboard (phase 2).
The capture hotkey (open question 3).
Permission: the check, and what mcap says when Screen Recording isn't granted.

Done when: on a Mac with a Retina screen and an external non-Retina one:

The hotkey freezes both screens.
A region dragged on either is taken at the screen's own pixels, and is on the clipboard and in Screen Captures with its window's title and app.
With the permission refused, mcap says how to grant it.

Phase 7: Screen recording

The source: modules/Screen_Record/sck_source.jai over the shim's stream, as the third source. The encoder tries h264_videotoolbox first.
Sound: system sound and the microphone, through the existing mixer.
The overlay's video mode, with the 240 × 180 minimum.
The border and toolbar as panels, kept out of the stream by the content filter. Pause, Discard, the sound buttons, and leftovers after a crash.

Done when the same checks pass as on Windows:

record-probe runs on the Mac, with -sync-test, -pause and -mic-toggle.
Neither border nor toolbar is in a recording.
A 10 s pause leaves no gap.
Beeps stay within about 25 ms of their flashes with no drift over 3 minutes.
A killed recording is offered at the next start.

Phase 8: A signed release

release.jai for macOS: the .app with everything in it (SDL3, onnxruntime, FFmpeg, the models), signed with a Developer ID with the hardened runtime and the entitlements it needs.
Notarized and stapled.
Packaged in a .dmg, with its SHA-256 printed, as the Windows zip's is.

Done when: the .dmg, downloaded on a second Mac, opens without a Gatekeeper warning, asks once for Screen Recording, and keeps that permission across an update to the next release.

7. Risks

No Mac in this loop yet. Everything here can be written on Windows, but only built, run and checked on a Mac. Each phase needs a session there, yours or a remote one (open question 1). The Linux type check catches the shared files in between, but not the Objective-C or the macOS-only files.

CoreML and the models. CoreML takes some operators and not others. Mask2Former's attention and the panoptic post-processing may fall back to the CPU in parts, which can be slower than the CPU alone, as the data moves back and forth. Phase 5 measures each model, and keeps the CPU where CoreML doesn't win.

Permissions and signing. Without a stable signature, Screen Recording has to be granted again after every build, and a refused permission gives a capture of the empty desktop rather than an error. Development needs a certificate early, by phase 6.

FFmpeg licensing. The Windows FFmpeg is a GPL build (with libx264). On macOS an LGPL build is enough: VideoToolbox does H.264, and FFmpeg's own AAC encoder does sound. That avoids GPL obligations for the Mac release; the Windows release should be checked against the same question.

Multi-screen overlays. One window per screen, with Spaces, full-screen apps and screens at different scales, is where macOS differs most from Windows. The overlay must sit above full-screen apps (NSWindowCollectionBehaviorCanJoinAllSpaces and FullScreenAuxiliary).

8. Open questions

The Mac: which Mac and macOS version will the work be checked on? Can I build and run there (SSH, or a remote session), or will you run the builds and send back the output?
The oldest macOS: 14 Sonoma, for ScreenCaptureKit's screenshots, is proposed. 13 would need the deprecated CGWindowListCreateImage for stills; 15 would give the microphone in the stream itself.
The keys: Mac keyboards have no PrintScreen, and ⌘⇧3/4/5 are the system's. Proposed: ⌘⇧2 for a still and ⌃⌘⇧2 to record, or F13 (PrintScreen on Apple's full keyboards) and ⌃F13, or your own.
Signing: do you have, or want, an Apple Developer account (99 USD a year) for a stable development signature now and Developer ID notarization for releases? Without it, phases 6 and 7 work but ask for permission after every build.
The library on a Mac: where should it default to? Proposed: ~/Pictures/mcap. Should the library folder become a setting on both systems, in place of the hard-coded D:/McapIndex/mcap?
The shim: is a small Objective-C file built with clang acceptable in the repo, or should ScreenCaptureKit be done from Jai alone, at more cost and risk?