From fd21e46a14be871e3e1774c9609879736d31de69 Mon Sep 17 00:00:00 2001 From: Musa Mahmood Date: Sat, 10 Oct 2026 17:49:38 +0100 Subject: [PATCH] [0.1.11] macOS plan: building again on a Mac, then feature parity - macos-plan.html: where mcap stands on macOS (built to compile there, never yet run in this project; two build breaks fixed in 0.1.11), what is Windows only, and eight phases to parity: build and run on a Mac; the small platform gaps (folder picker, Trash, temporary folder, clipboard); images through ImageIO; video through FFmpeg for macOS; the models on the GPU with CoreML; screen capture stills and recording through ScreenCaptureKit, behind a small Objective-C shim; and a signed, notarized release. Permissions and signing, risks, and six open questions: which Mac, the oldest macOS, the capture keys, signing, the library's place, the shim. --- macos-plan.html | 335 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 335 insertions(+) create mode 100644 macos-plan.html diff --git a/macos-plan.html b/macos-plan.html new file mode 100644 index 0000000..447b949 --- /dev/null +++ b/macos-plan.html @@ -0,0 +1,335 @@ + + + + + +macOS Port Plan + + + +
+ +
+

mcap on macOS: building again, then feature parity

+

Planning document · 2026-10-10 · for mcap after 0.1.11 · not started

+
+ +
+ In short +
    +
  • Where it stands: mcap was built to compile on macOS (Apple silicon, SDL3 and Metal), but it hasn't been built on a Mac since at least 0.0.8. Two build breaks were found and fixed in 0.1.11 without a Mac. Everything added since the template is Windows first, and much of it Windows only.
    • +
  • Proposed Eight phases, smallest risk first: +
      +
    1. Build and run on a Mac.
      2. +
    2. Fill the small platform gaps: choosing a folder, the Trash, the clipboard, the temporary folder.
      3. +
    3. Images through ImageIO.
      4. +
    4. Video through FFmpeg for macOS.
      5. +
    5. The models on the GPU through CoreML.
      6. +
    6. Screen capture stills.
      7. +
    7. Screen recording.
      8. +
    8. A signed release.
      9. +
    • +
  • Proposed One codebase, the same file formats. An .mcap made on either system opens on the other. Where Windows uses WIC, DXGI and WASAPI, macOS uses ImageIO and ScreenCaptureKit. FFmpeg, onnxruntime, the encoder, the remux and the mixer stay shared.
    • +
  • Proposed ScreenCaptureKit through a small Objective-C shim. It is asynchronous: completion blocks, delegates, dispatch queues. A few hundred lines of Objective-C, compiled with clang into a static library with a plain C interface, is far safer than doing that from Jai. Open question 6.
    • +
  • The big unknowns need a Mac to settle. Which macOS to require (14 Sonoma is proposed). Signing: macOS remembers the Screen Recording permission per signature, and an ad hoc signature changes with every build. What the capture keys should be: Mac keyboards have no PrintScreen. See open questions.
    • +
+
+ + + +

1. Where it stands

+

From the code; none of this has been run on a Mac in this project's history.

+
+ + + + + + + + +
PartmacOS today
Window, ImGui, themes, fontsSDL3 with SDL_GPU on Metal (src/backends/sdl3_gpu3.jai). The menu-bar tray is SDL's. Fonts and CJK fallbacks have macOS paths (platform_macos.jai).
Buildjai first.jai makes bin/mcap-macos-debug, and with - release bin/mcap.app, signed ad hoc. Apple silicon only, with a native jai. The .app holds only the executable and libSDL3.0.dylib. onnxruntime and the models are copied beside it, not into it.
Type check from WindowsNot possible: jai check_targets.jai - macos runs only on a Mac, because the Objective-C module runs code at compile time.
Known breaks, fixed in 0.1.11 +
    +
  • unix_sdl3_gpu3.jai pulled all of ImGui into the app's scope, so ImGui's Image clashed with Image_Decode's (broken since 0.0.8).
    • +
  • A Windows RECT was in the shared App struct (since 0.1.6).
    • +
+ The Linux check passes after both, and macOS shares the same files.
Library locationHard-coded as D:/McapIndex/mcap (VIEWER_FOLDER). On a Mac only -library works.
Global hotkeysCarbon RegisterEventHotKey (macos_global_hotkeys.jai). capture_region and record_region default to none.
OCR, labels, segmentationonnxruntime 1.30.0 for macOS arm64 is in fetch.jai. Sessions use CUDA or the CPU (session.jai), so on a Mac it's CPU only, several times slower than a GPU.
+ +

2. What is Windows only

+
+ + + + + + + + + + + + +
FeatureWindowsmacOS todaymacOS plan
Images other than PNG, JPEG, GIF, BMP, PSD, TGAWIC (Image_Decode/wic.jai)RefusedImageIO, CGImageSource (phase 3)
Videos: import, frame for the thumbnailFFmpeg (modules/ffmpeg, Windows bindings only)SkippedFFmpeg for macOS (phase 4)
Import: choosing a folderExplorer's picker (windows_choose_folder)Typed path onlyNSOpenPanel (phase 2)
Delete to the binRecycle Bin (SHFileOperationW)A "deleted" folder beside the fileNSFileManager trashItemAtURL (phase 2)
Temporary folder (open original, recordings)%TEMP%/tmpNSTemporaryDirectory() (phase 2)
Image to the clipboardCF_DIBNoneNSPasteboard, PNG (phase 2)
Models on the GPUCUDACPUCoreML execution provider (phase 5)
Screen capture stillsGDI overlay, BitBltNoneCocoa overlay, ScreenCaptureKit screenshots (phase 6)
Screen recording, border, toolbarDXGI or Windows.Graphics.Capture, GDI windowsNoneScreenCaptureKit stream, Cocoa panels (phase 7)
SoundWASAPINoneScreenCaptureKit audio, plus its microphone or AVAudioEngine (phase 7)
Releaserelease.jai: a zip with CUDANoneA signed .app in a .dmg (phase 8)
+

Already shared and fine as they are:

+ + +

3. Approach

+ + +

4. Screen capture on macOS

+ + +

5. Permissions and signing

+ + +

6. Phases

+

Each phase ends with something to run and check on the Mac, and is a commit. They are ordered so a usable mcap comes first: by the end of phase 4 the library browser, search, imports and OCR all work.

+ +
+

Phase 1: Build and run on a Mac

+
    +
  • The type check: jai check_targets.jai - macos, fixing what it finds.
    • +
  • The builds: jai first.jai and - release.
    • +
  • A library folder for macOS: a default in place of D:/McapIndex/mcap, open question 5.
    • +
  • The .app made whole: libonnxruntime.dylib in Contents/Frameworks, and the OCR, label and segmentation models in Contents/Resources, found from there.
    • +
  • Tests: run_tests -headless on the Mac.
    • +
+

Done when:

+
    +
  • Both builds succeed and the test suites pass on the Mac.
    • +
  • mcap.app, started from Finder, shows a library copied from Windows: thumbnails, the Capture window, search, Stats.
    • +
  • An OCR scan reads captures on the CPU, and its time per capture is recorded.
    • +
+
+ +
+

Phase 2: The small platform gaps

+
    +
  • Choosing a folder: NSOpenPanel for Import's Browse button.
    • +
  • Delete: into the Trash with NSFileManager trashItemAtURL, with "Move to Trash" in the delete dialog.
    • +
  • Temporary files: NSTemporaryDirectory() for opening an original and for recordings.
    • +
  • The clipboard: platform_copy_image_to_clipboard through NSPasteboard, as PNG and TIFF.
    • +
+

Done when: Browse picks a folder; a deleted capture is in the Trash and can be put back from there; Open plays a video's original; an image pasted from the clipboard (phase 6 is its first user, so a test fills it) pastes into Preview.

+
+ +
+

Phase 3: Images through ImageIO

+

Image_Decode/imageio.jai: CGImageSourceCreateWithData and CGImageSourceCreateImageAtIndex, drawn into an RGBA bitmap context. It takes the formats WIC takes on Windows:

+
    +
  • HEIC and AVIF (AVIF from macOS 13).
    • +
  • WebP (from macOS 11).
    • +
  • TIFF.
    • +
  • The camera RAW formats.
    • +
+

EXIF orientation stays exif.jai's. ImageIO is asked for the image as stored, so it isn't turned twice.

+

Done when: image-probe on the Mac decodes the same test images as on Windows, the same way up and the same size, and importing an iPhone photo folder makes the same .mcap files, except for the import date.

+
+ +
+

Phase 4: Video through FFmpeg for macOS

+
    +
  • FFmpeg itself: a shared FFmpeg 6 build for macOS arm64, with VideoToolbox and the native AAC encoder. Its dylibs and their SHA-256 sums go in modules/ffmpeg/fetch.jai, as onnxruntime's are.
    • +
  • Bindings: modules/ffmpeg/macos.jai, from the same headers with the dylib names, and #loaded on macOS.
    • +
  • The video code: video_ffmpeg.jai and the import's video path, no longer Windows only.
    • +
  • The .app: the dylibs go in it.
    • +
+

Done when: importing a folder with MP4, MOV and HEVC iPhone videos gives the same .mcap files as on Windows, with thumbnails, and they play with Open.

+
+ +
+

Phase 5: The models on the GPU, with CoreML

+
    +
  • CoreML first: session.jai asks onnxruntime for the CoreML execution provider on macOS (SessionOptionsAppendExecutionProvider with "CoreML", the ML Program format), and the CPU for what it can't take.
    • +
  • Measured per model: PP-OCR detection and recognition, SigLIP 2, Mask2Former. Each is timed against the CPU and checked to give the same output as on Windows, within the tolerances the Windows probes already use.
    • +
  • The status line says CoreML, as it says CUDA on Windows.
    • +
+

Done when: OCR, labels and segmentation give the same results as on Windows on the probes' test images, and each is faster than the CPU, or is left on the CPU with the reason recorded.

+
+ +
+

Phase 6: Screen capture stills

+
    +
  • The shim's screenshot (mcap_sck_screenshot, per display).
    • +
  • The overlay: a window per screen, with the drag, magnifier, keys, and a click for the whole screen.
    • +
  • The window it came from: its title and app.
    • +
  • The clipboard (phase 2).
    • +
  • The capture hotkey (open question 3).
    • +
  • Permission: the check, and what mcap says when Screen Recording isn't granted.
    • +
+

Done when: on a Mac with a Retina screen and an external non-Retina one:

+
    +
  • The hotkey freezes both screens.
    • +
  • A region dragged on either is taken at the screen's own pixels, and is on the clipboard and in Screen Captures with its window's title and app.
    • +
  • With the permission refused, mcap says how to grant it.
    • +
+
+ +
+

Phase 7: Screen recording

+
    +
  • The source: modules/Screen_Record/sck_source.jai over the shim's stream, as the third source. The encoder tries h264_videotoolbox first.
    • +
  • Sound: system sound and the microphone, through the existing mixer.
    • +
  • The overlay's video mode, with the 240 × 180 minimum.
    • +
  • The border and toolbar as panels, kept out of the stream by the content filter. Pause, Discard, the sound buttons, and leftovers after a crash.
    • +
+

Done when the same checks pass as on Windows:

+
    +
  • record-probe runs on the Mac, with -sync-test, -pause and -mic-toggle.
    • +
  • Neither border nor toolbar is in a recording.
    • +
  • A 10 s pause leaves no gap.
    • +
  • Beeps stay within about 25 ms of their flashes with no drift over 3 minutes.
    • +
  • A killed recording is offered at the next start.
    • +
+
+ +
+

Phase 8: A signed release

+
    +
  • release.jai for macOS: the .app with everything in it (SDL3, onnxruntime, FFmpeg, the models), signed with a Developer ID with the hardened runtime and the entitlements it needs.
    • +
  • Notarized and stapled.
    • +
  • Packaged in a .dmg, with its SHA-256 printed, as the Windows zip's is.
    • +
+

Done when: the .dmg, downloaded on a second Mac, opens without a Gatekeeper warning, asks once for Screen Recording, and keeps that permission across an update to the next release.

+
+ +

7. Risks

+
No Mac in this loop yet. Everything here can be written on Windows, but only built, run and checked on a Mac. Each phase needs a session there, yours or a remote one (open question 1). The Linux type check catches the shared files in between, but not the Objective-C or the macOS-only files.
+
CoreML and the models. CoreML takes some operators and not others. Mask2Former's attention and the panoptic post-processing may fall back to the CPU in parts, which can be slower than the CPU alone, as the data moves back and forth. Phase 5 measures each model, and keeps the CPU where CoreML doesn't win.
+
Permissions and signing. Without a stable signature, Screen Recording has to be granted again after every build, and a refused permission gives a capture of the empty desktop rather than an error. Development needs a certificate early, by phase 6.
+
FFmpeg licensing. The Windows FFmpeg is a GPL build (with libx264). On macOS an LGPL build is enough: VideoToolbox does H.264, and FFmpeg's own AAC encoder does sound. That avoids GPL obligations for the Mac release; the Windows release should be checked against the same question.
+
Multi-screen overlays. One window per screen, with Spaces, full-screen apps and screens at different scales, is where macOS differs most from Windows. The overlay must sit above full-screen apps (NSWindowCollectionBehaviorCanJoinAllSpaces and FullScreenAuxiliary).
+ +

8. Open questions

+
    +
  1. The Mac: which Mac and macOS version will the work be checked on? Can I build and run there (SSH, or a remote session), or will you run the builds and send back the output?
    2. +
  2. The oldest macOS: 14 Sonoma, for ScreenCaptureKit's screenshots, is proposed. 13 would need the deprecated CGWindowListCreateImage for stills; 15 would give the microphone in the stream itself.
    3. +
  3. The keys: Mac keyboards have no PrintScreen, and ⌘⇧3/4/5 are the system's. Proposed: ⌘⇧2 for a still and ⌃⌘⇧2 to record, or F13 (PrintScreen on Apple's full keyboards) and ⌃F13, or your own.
    4. +
  4. Signing: do you have, or want, an Apple Developer account (99 USD a year) for a stable development signature now and Developer ID notarization for releases? Without it, phases 6 and 7 work but ask for permission after every build.
    5. +
  5. The library on a Mac: where should it default to? Proposed: ~/Pictures/mcap. Should the library folder become a setting on both systems, in place of the hard-coded D:/McapIndex/mcap?
    6. +
  6. The shim: is a small Objective-C file built with clang acceptable in the repo, or should ScreenCaptureKit be done from Jai alone, at more cost and risk?
    7. +
+ +
+ +