[0.1.11] macOS plan: building again on a Mac, then feature parity
- macos-plan.html: where mcap stands on macOS (built to compile there, never yet run in this project; two build breaks fixed in 0.1.11), what is Windows only, and eight phases to parity: build and run on a Mac; the small platform gaps (folder picker, Trash, temporary folder, clipboard); images through ImageIO; video through FFmpeg for macOS; the models on the GPU with CoreML; screen capture stills and recording through ScreenCaptureKit, behind a small Objective-C shim; and a signed, notarized release. Permissions and signing, risks, and six open questions: which Mac, the oldest macOS, the capture keys, signing, the library's place, the shim.
This commit is contained in:
parent
693974bedd
commit
fd21e46a14
335
macos-plan.html
Normal file
335
macos-plan.html
Normal file
@ -0,0 +1,335 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>macOS Port Plan</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--bg: #fbfaf8; --surface: #ffffff; --text: #1d1d1f; --dim: #5d5d66; --line: #e3e1dc;
|
||||||
|
--accent: #3a5bd9; --accent-soft: #e9edfb; --warn: #9a5b00; --warn-soft: #fbf1e1;
|
||||||
|
--code-bg: #f2f0ec; --ok: #2e7d4f; --ok-soft: #e5f3ea;
|
||||||
|
}
|
||||||
|
@media (prefers-color-scheme: dark) {
|
||||||
|
:root:not([data-theme="light"]) {
|
||||||
|
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
|
||||||
|
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
|
||||||
|
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
:root[data-theme="dark"] {
|
||||||
|
--bg: #16171a; --surface: #1d1f23; --text: #e7e7ea; --dim: #a2a3ab; --line: #30333a;
|
||||||
|
--accent: #8aa2ff; --accent-soft: #232a45; --warn: #f0b45a; --warn-soft: #33281a;
|
||||||
|
--code-bg: #25272c; --ok: #6fcf97; --ok-soft: #1c2e24;
|
||||||
|
}
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
body {
|
||||||
|
margin: 0; background: var(--bg); color: var(--text);
|
||||||
|
font: 16px/1.6 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif;
|
||||||
|
}
|
||||||
|
main { max-width: 860px; margin: 0 auto; padding: 40px 16px 80px; }
|
||||||
|
header p { color: var(--dim); margin-top: 4px; }
|
||||||
|
h1 { font-size: 2rem; line-height: 1.2; margin: 0; }
|
||||||
|
h2 { font-size: 1.35rem; margin: 48px 0 8px; padding-top: 16px; border-top: 1px solid var(--line); }
|
||||||
|
h3 { font-size: 1.05rem; margin: 28px 0 6px; }
|
||||||
|
p, li { max-width: 72ch; }
|
||||||
|
a { color: var(--accent); }
|
||||||
|
code, pre { font-family: ui-monospace, "Cascadia Code", Consolas, monospace; font-size: 0.88em; }
|
||||||
|
code { background: var(--code-bg); padding: 1px 5px; border-radius: 4px; }
|
||||||
|
pre { background: var(--code-bg); padding: 12px 14px; border-radius: 8px; overflow-x: auto; line-height: 1.45; }
|
||||||
|
pre code { background: none; padding: 0; }
|
||||||
|
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 0.94rem; }
|
||||||
|
th, td { text-align: left; padding: 7px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||||
|
th { color: var(--dim); font-weight: 600; }
|
||||||
|
.table-wrap { overflow-x: auto; }
|
||||||
|
.summary { background: var(--surface); border: 1px solid var(--line); border-radius: 10px; padding: 16px 20px; margin-top: 24px; }
|
||||||
|
.summary ul { margin: 6px 0 0; padding-left: 20px; }
|
||||||
|
.note { background: var(--accent-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
|
||||||
|
.risk { background: var(--warn-soft); border-radius: 8px; padding: 10px 14px; margin: 14px 0; }
|
||||||
|
.risk strong { color: var(--warn); }
|
||||||
|
.tag { display: inline-block; font-size: 0.78rem; font-weight: 600; padding: 1px 8px; border-radius: 99px; background: var(--accent-soft); color: var(--accent); }
|
||||||
|
.tag.later { background: var(--code-bg); color: var(--dim); }
|
||||||
|
.tag.decided { background: var(--ok-soft); color: var(--ok); }
|
||||||
|
nav ol { columns: 2; column-gap: 32px; padding-left: 20px; margin: 8px 0 0; }
|
||||||
|
@media (max-width: 600px) { nav ol { columns: 1; } h1 { font-size: 1.6rem; } }
|
||||||
|
.phase { border-left: 3px solid var(--accent); padding: 2px 0 2px 14px; margin: 16px 0; }
|
||||||
|
.phase h3 { margin-top: 0; }
|
||||||
|
figure { margin: 18px 0; }
|
||||||
|
figcaption { color: var(--dim); font-size: 0.88rem; margin-top: 6px; }
|
||||||
|
svg text { fill: var(--text); font: 13px system-ui, sans-serif; }
|
||||||
|
svg .dim { fill: var(--dim); }
|
||||||
|
svg .box { fill: var(--surface); stroke: var(--line); }
|
||||||
|
svg .hl { fill: var(--accent-soft); stroke: var(--accent); }
|
||||||
|
svg .arrow { stroke: var(--dim); fill: none; marker-end: url(#a); }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<main>
|
||||||
|
|
||||||
|
<header>
|
||||||
|
<h1>mcap on macOS: building again, then feature parity</h1>
|
||||||
|
<p>Planning document · 2026-10-10 · for mcap after 0.1.11 · <strong>not started</strong></p>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<div class="summary">
|
||||||
|
<strong>In short</strong>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Where it stands:</strong> mcap was built to compile on macOS (Apple silicon, SDL3 and Metal), but it hasn't been built on a Mac since at least 0.0.8. Two build breaks were found and fixed in 0.1.11 without a Mac. Everything added since the template is Windows first, and much of it Windows only.</li>
|
||||||
|
<li><span class="tag decided">Proposed</span> <strong>Eight phases</strong>, smallest risk first:
|
||||||
|
<ol>
|
||||||
|
<li>Build and run on a Mac.</li>
|
||||||
|
<li>Fill the small platform gaps: choosing a folder, the Trash, the clipboard, the temporary folder.</li>
|
||||||
|
<li>Images through ImageIO.</li>
|
||||||
|
<li>Video through FFmpeg for macOS.</li>
|
||||||
|
<li>The models on the GPU through CoreML.</li>
|
||||||
|
<li>Screen capture stills.</li>
|
||||||
|
<li>Screen recording.</li>
|
||||||
|
<li>A signed release.</li>
|
||||||
|
</ol></li>
|
||||||
|
<li><span class="tag decided">Proposed</span> <strong>One codebase, the same file formats.</strong> An <code>.mcap</code> made on either system opens on the other. Where Windows uses WIC, DXGI and WASAPI, macOS uses ImageIO and ScreenCaptureKit. FFmpeg, onnxruntime, the encoder, the remux and the mixer stay shared.</li>
|
||||||
|
<li><span class="tag decided">Proposed</span> <strong>ScreenCaptureKit through a small Objective-C shim.</strong> It is asynchronous: completion blocks, delegates, dispatch queues. A few hundred lines of Objective-C, compiled with clang into a static library with a plain C interface, is far safer than doing that from Jai. Open question 6.</li>
|
||||||
|
<li><strong>The big unknowns</strong> need a Mac to settle. Which macOS to require (14 Sonoma is proposed). Signing: macOS remembers the Screen Recording permission per signature, and an ad hoc signature changes with every build. What the capture keys should be: Mac keyboards have no PrintScreen. See <a href="#questions">open questions</a>.</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<nav>
|
||||||
|
<ol>
|
||||||
|
<li><a href="#today">Where it stands</a></li>
|
||||||
|
<li><a href="#gaps">What is Windows only</a></li>
|
||||||
|
<li><a href="#approach">Approach</a></li>
|
||||||
|
<li><a href="#capture">Screen capture on macOS</a></li>
|
||||||
|
<li><a href="#permissions">Permissions and signing</a></li>
|
||||||
|
<li><a href="#phases">Phases</a></li>
|
||||||
|
<li><a href="#risks">Risks</a></li>
|
||||||
|
<li><a href="#questions">Open questions</a></li>
|
||||||
|
</ol>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
<h2 id="today">1. Where it stands</h2>
|
||||||
|
<p>From the code; none of this has been run on a Mac in this project's history.</p>
|
||||||
|
<div class="table-wrap"><table>
|
||||||
|
<tr><th>Part</th><th>macOS today</th></tr>
|
||||||
|
<tr><td>Window, ImGui, themes, fonts</td><td>SDL3 with SDL_GPU on Metal (<code>src/backends/sdl3_gpu3.jai</code>). The menu-bar tray is SDL's. Fonts and CJK fallbacks have macOS paths (<code>platform_macos.jai</code>).</td></tr>
|
||||||
|
<tr><td>Build</td><td><code>jai first.jai</code> makes <code>bin/mcap-macos-debug</code>, and with <code>- release</code> <code>bin/mcap.app</code>, signed ad hoc. Apple silicon only, with a native <code>jai</code>. The <code>.app</code> holds only the executable and <code>libSDL3.0.dylib</code>. onnxruntime and the models are copied beside it, not into it.</td></tr>
|
||||||
|
<tr><td>Type check from Windows</td><td>Not possible: <code>jai check_targets.jai - macos</code> runs only on a Mac, because the Objective-C module runs code at compile time.</td></tr>
|
||||||
|
<tr><td>Known breaks, fixed in 0.1.11</td><td>
|
||||||
|
<ul>
|
||||||
|
<li><code>unix_sdl3_gpu3.jai</code> pulled all of ImGui into the app's scope, so ImGui's <code>Image</code> clashed with Image_Decode's (broken since 0.0.8).</li>
|
||||||
|
<li>A Windows <code>RECT</code> was in the shared <code>App</code> struct (since 0.1.6).</li>
|
||||||
|
</ul>
|
||||||
|
The Linux check passes after both, and macOS shares the same files.</td></tr>
|
||||||
|
<tr><td>Library location</td><td>Hard-coded as <code>D:/McapIndex/mcap</code> (<code>VIEWER_FOLDER</code>). On a Mac only <code>-library</code> works.</td></tr>
|
||||||
|
<tr><td>Global hotkeys</td><td>Carbon <code>RegisterEventHotKey</code> (<code>macos_global_hotkeys.jai</code>). <code>capture_region</code> and <code>record_region</code> default to <code>none</code>.</td></tr>
|
||||||
|
<tr><td>OCR, labels, segmentation</td><td>onnxruntime 1.30.0 for macOS arm64 is in <code>fetch.jai</code>. Sessions use CUDA or the CPU (<code>session.jai</code>), so on a Mac it's CPU only, several times slower than a GPU.</td></tr>
|
||||||
|
</table></div>
|
||||||
|
|
||||||
|
<h2 id="gaps">2. What is Windows only</h2>
|
||||||
|
<div class="table-wrap"><table>
|
||||||
|
<tr><th>Feature</th><th>Windows</th><th>macOS today</th><th>macOS plan</th></tr>
|
||||||
|
<tr><td>Images other than PNG, JPEG, GIF, BMP, PSD, TGA</td><td>WIC (<code>Image_Decode/wic.jai</code>)</td><td>Refused</td><td>ImageIO, <code>CGImageSource</code> (phase 3)</td></tr>
|
||||||
|
<tr><td>Videos: import, frame for the thumbnail</td><td>FFmpeg (<code>modules/ffmpeg</code>, Windows bindings only)</td><td>Skipped</td><td>FFmpeg for macOS (phase 4)</td></tr>
|
||||||
|
<tr><td>Import: choosing a folder</td><td>Explorer's picker (<code>windows_choose_folder</code>)</td><td>Typed path only</td><td><code>NSOpenPanel</code> (phase 2)</td></tr>
|
||||||
|
<tr><td>Delete to the bin</td><td>Recycle Bin (<code>SHFileOperationW</code>)</td><td>A "deleted" folder beside the file</td><td><code>NSFileManager trashItemAtURL</code> (phase 2)</td></tr>
|
||||||
|
<tr><td>Temporary folder (open original, recordings)</td><td><code>%TEMP%</code></td><td><code>/tmp</code></td><td><code>NSTemporaryDirectory()</code> (phase 2)</td></tr>
|
||||||
|
<tr><td>Image to the clipboard</td><td><code>CF_DIB</code></td><td>None</td><td><code>NSPasteboard</code>, PNG (phase 2)</td></tr>
|
||||||
|
<tr><td>Models on the GPU</td><td>CUDA</td><td>CPU</td><td>CoreML execution provider (phase 5)</td></tr>
|
||||||
|
<tr><td>Screen capture stills</td><td>GDI overlay, <code>BitBlt</code></td><td>None</td><td>Cocoa overlay, ScreenCaptureKit screenshots (phase 6)</td></tr>
|
||||||
|
<tr><td>Screen recording, border, toolbar</td><td>DXGI or Windows.Graphics.Capture, GDI windows</td><td>None</td><td>ScreenCaptureKit stream, Cocoa panels (phase 7)</td></tr>
|
||||||
|
<tr><td>Sound</td><td>WASAPI</td><td>None</td><td>ScreenCaptureKit audio, plus its microphone or AVAudioEngine (phase 7)</td></tr>
|
||||||
|
<tr><td>Release</td><td><code>release.jai</code>: a zip with CUDA</td><td>None</td><td>A signed <code>.app</code> in a <code>.dmg</code> (phase 8)</td></tr>
|
||||||
|
</table></div>
|
||||||
|
<p>Already shared and fine as they are:</p>
|
||||||
|
<ul>
|
||||||
|
<li>The <code>.mcap</code> format, catalog, search, Stats.</li>
|
||||||
|
<li><code>.snagx</code> import, EXIF (<code>exif.jai</code> reads it itself).</li>
|
||||||
|
<li>The settings and hotkey files.</li>
|
||||||
|
<li>The encoder, remux and mixer in <code>modules/Screen_Record</code>. Their capture sources aren't shared.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="approach">3. Approach</h2>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Same interfaces, a second implementation.</strong> Each Windows-only piece already sits behind a function the app calls. The macOS version goes beside it, in its own file:
|
||||||
|
<ul>
|
||||||
|
<li><code>platform_screen_capture_select</code>, <code>platform_copy_image_to_clipboard</code>.</li>
|
||||||
|
<li><code>image_decode</code>'s WIC branch.</li>
|
||||||
|
<li><code>video_poster</code>.</li>
|
||||||
|
<li><code>Screen_Record</code>'s source: a third source beside <code>dxgi_source</code> and <code>wgc_source</code>.</li>
|
||||||
|
</ul>
|
||||||
|
The app's own files (<code>screen_capture.jai</code>, <code>screen_record.jai</code>, <code>import.jai</code>) shouldn't need to know which system they're on beyond what they do now.</li>
|
||||||
|
<li><strong>FFmpeg on macOS too</strong>, rather than AVFoundation. Video import, the thumbnail frame, the recording encoder and the remux are FFmpeg code already. On a Mac the encoder would be <code>h264_videotoolbox</code>, Apple's hardware encoder, put first in <code>RECORD_ENCODERS</code>. The cost is shipping FFmpeg's dylibs in the <code>.app</code>, and bindings for their macOS names.</li>
|
||||||
|
<li><strong>The Objective-C shim</strong> (<code>modules/Screen_Record/macos/capture.m</code>) holds ScreenCaptureKit and nothing else. It has a C interface:
|
||||||
|
<ul>
|
||||||
|
<li><code>mcap_sck_screenshot</code></li>
|
||||||
|
<li><code>mcap_sck_stream_start</code></li>
|
||||||
|
<li><code>mcap_sck_next_frame</code></li>
|
||||||
|
<li><code>mcap_sck_next_audio</code></li>
|
||||||
|
<li><code>mcap_sck_stream_stop</code></li>
|
||||||
|
</ul>
|
||||||
|
It is compiled by <code>first.jai</code> with clang when building on a Mac, and linked statically. AppKit work that is plain message sends (<code>NSOpenPanel</code>, <code>NSPasteboard</code>, the overlay and toolbar windows) stays in Jai through the Objective-C module the app already uses.</li>
|
||||||
|
<li><strong>The build is checked on every phase</strong> with <code>jai check_targets.jai - macos</code> and the test suites on the Mac. <code>jai check_targets.jai - linux</code> on Windows catches the shared files between Mac sessions.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="capture">4. Screen capture on macOS</h2>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Stills:</strong> the moment the key is pressed, each screen is copied:
|
||||||
|
<ul>
|
||||||
|
<li>Copied with <code>SCScreenshotManager captureImageWithFilter</code>, on macOS 14 and later. <code>CGWindowListCreateImage</code> is deprecated in macOS 15.</li>
|
||||||
|
<li>Shown frozen in a borderless <code>NSWindow</code> per screen, at the screen-saver level. A window can't span screens on macOS as the one Windows overlay does.</li>
|
||||||
|
<li>The drag, magnifier and keys work as on Windows. The overlay's drawing (darkened copy, bright selection, outline, magnifier) becomes Core Graphics in each window's <code>drawRect</code>, or an <code>NSImage</code> drawn by the shim.</li>
|
||||||
|
</ul></li>
|
||||||
|
<li><strong>Points and pixels:</strong> AppKit works in points. A Retina screen has 2 pixels a point, and screens can differ. The region is chosen in points and converted to each screen's pixels for the capture. The 240 × 180 minimum is in pixels, as on Windows.</li>
|
||||||
|
<li><strong>Recording:</strong> an <code>SCStream</code> for the region's display (<code>sourceRect</code>) delivers BGRA <code>CVPixelBuffer</code>s with timestamps on the host clock.
|
||||||
|
<ul>
|
||||||
|
<li>The shim hands them over as the pipeline's frames, and the encoder and the rest are unchanged.</li>
|
||||||
|
<li>The cursor: <code>showsCursor</code>.</li>
|
||||||
|
<li>Keeping mcap's own border and toolbar out: an <code>SCContentFilter</code> that excludes the app's windows. That's cleaner than Windows' window flag, and doesn't depend on it.</li>
|
||||||
|
</ul></li>
|
||||||
|
<li><strong>Sound:</strong>
|
||||||
|
<ul>
|
||||||
|
<li>System sound comes from the same stream (<code>capturesAudio</code>, macOS 13 on), as 48 kHz float.</li>
|
||||||
|
<li>The microphone comes from the stream too on macOS 15 (<code>captureMicrophone</code>), else from <code>AVAudioEngine</code>'s input.</li>
|
||||||
|
<li>Both go into the mixer as packets with host-clock times, as WASAPI's do.</li>
|
||||||
|
</ul></li>
|
||||||
|
<li><strong>The border and toolbar:</strong> non-activating <code>NSPanel</code>s at the floating level. The toolbar's buttons ask, and the app acts on its next frame, as on Windows.</li>
|
||||||
|
<li><strong>The window it was taken from:</strong> the frontmost app comes from <code>NSWorkspace</code>, and window titles from <code>CGWindowListCopyWindowInfo</code>. The titles need the Screen Recording permission, which capture needs anyway.</li>
|
||||||
|
<li><strong>Hotkeys:</strong> no PrintScreen on Mac keyboards. ⌘⇧3, 4 and 5 are the system's screenshot keys. See open question 3.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="permissions">5. Permissions and signing</h2>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Screen Recording</strong> (Privacy & Security) must be granted before any capture. The first capture asks; until it is granted, the overlay gets only the desktop, without other apps' windows. mcap will check (<code>CGPreflightScreenCaptureAccess</code>) and say what to do instead of showing an empty capture.</li>
|
||||||
|
<li><strong>Microphone</strong> permission is asked the first time the Mic button is turned on. <code>Info.plist</code> gains <code>NSMicrophoneUsageDescription</code>.</li>
|
||||||
|
<li><strong>macOS remembers these per code signature.</strong> The release <code>.app</code> is signed ad hoc today, and an ad hoc signature changes with every build, so each build would ask again, or silently stay refused. A stable signing identity solves it: an Apple Development certificate for builds, and a Developer ID for releases, notarized so Gatekeeper opens it. See open question 4.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="phases">6. Phases</h2>
|
||||||
|
<p>Each phase ends with something to run and check on the Mac, and is a commit. They are ordered so a usable mcap comes first: by the end of phase 4 the library browser, search, imports and OCR all work.</p>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 1: Build and run on a Mac</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>The type check:</strong> <code>jai check_targets.jai - macos</code>, fixing what it finds.</li>
|
||||||
|
<li><strong>The builds:</strong> <code>jai first.jai</code> and <code>- release</code>.</li>
|
||||||
|
<li><strong>A library folder for macOS:</strong> a default in place of <code>D:/McapIndex/mcap</code>, open question 5.</li>
|
||||||
|
<li><strong>The <code>.app</code> made whole:</strong> <code>libonnxruntime.dylib</code> in <code>Contents/Frameworks</code>, and the OCR, label and segmentation models in <code>Contents/Resources</code>, found from there.</li>
|
||||||
|
<li><strong>Tests:</strong> <code>run_tests -headless</code> on the Mac.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong></p>
|
||||||
|
<ul>
|
||||||
|
<li>Both builds succeed and the test suites pass on the Mac.</li>
|
||||||
|
<li><code>mcap.app</code>, started from Finder, shows a library copied from Windows: thumbnails, the Capture window, search, Stats.</li>
|
||||||
|
<li>An OCR scan reads captures on the CPU, and its time per capture is recorded.</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 2: The small platform gaps</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Choosing a folder:</strong> <code>NSOpenPanel</code> for Import's Browse button.</li>
|
||||||
|
<li><strong>Delete:</strong> into the Trash with <code>NSFileManager trashItemAtURL</code>, with "Move to Trash" in the delete dialog.</li>
|
||||||
|
<li><strong>Temporary files:</strong> <code>NSTemporaryDirectory()</code> for opening an original and for recordings.</li>
|
||||||
|
<li><strong>The clipboard:</strong> <code>platform_copy_image_to_clipboard</code> through <code>NSPasteboard</code>, as PNG and TIFF.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong> Browse picks a folder; a deleted capture is in the Trash and can be put back from there; Open plays a video's original; an image pasted from the clipboard (phase 6 is its first user, so a test fills it) pastes into Preview.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 3: Images through ImageIO</h3>
|
||||||
|
<p><code>Image_Decode/imageio.jai</code>: <code>CGImageSourceCreateWithData</code> and <code>CGImageSourceCreateImageAtIndex</code>, drawn into an RGBA bitmap context. It takes the formats WIC takes on Windows:</p>
|
||||||
|
<ul>
|
||||||
|
<li>HEIC and AVIF (AVIF from macOS 13).</li>
|
||||||
|
<li>WebP (from macOS 11).</li>
|
||||||
|
<li>TIFF.</li>
|
||||||
|
<li>The camera RAW formats.</li>
|
||||||
|
</ul>
|
||||||
|
<p>EXIF orientation stays <code>exif.jai</code>'s. ImageIO is asked for the image as stored, so it isn't turned twice.</p>
|
||||||
|
<p><strong>Done when:</strong> <code>image-probe</code> on the Mac decodes the same test images as on Windows, the same way up and the same size, and importing an iPhone photo folder makes the same <code>.mcap</code> files, except for the import date.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 4: Video through FFmpeg for macOS</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>FFmpeg itself:</strong> a shared FFmpeg 6 build for macOS arm64, with VideoToolbox and the native AAC encoder. Its dylibs and their SHA-256 sums go in <code>modules/ffmpeg/fetch.jai</code>, as onnxruntime's are.</li>
|
||||||
|
<li><strong>Bindings:</strong> <code>modules/ffmpeg/macos.jai</code>, from the same headers with the dylib names, and <code>#load</code>ed on macOS.</li>
|
||||||
|
<li><strong>The video code:</strong> <code>video_ffmpeg.jai</code> and the import's video path, no longer Windows only.</li>
|
||||||
|
<li><strong>The <code>.app</code>:</strong> the dylibs go in it.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong> importing a folder with MP4, MOV and HEVC iPhone videos gives the same <code>.mcap</code> files as on Windows, with thumbnails, and they play with Open.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 5: The models on the GPU, with CoreML</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>CoreML first:</strong> <code>session.jai</code> asks onnxruntime for the CoreML execution provider on macOS (<code>SessionOptionsAppendExecutionProvider</code> with <code>"CoreML"</code>, the ML Program format), and the CPU for what it can't take.</li>
|
||||||
|
<li><strong>Measured per model:</strong> PP-OCR detection and recognition, SigLIP 2, Mask2Former. Each is timed against the CPU and checked to give the same output as on Windows, within the tolerances the Windows probes already use.</li>
|
||||||
|
<li><strong>The status line</strong> says CoreML, as it says CUDA on Windows.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong> OCR, labels and segmentation give the same results as on Windows on the probes' test images, and each is faster than the CPU, or is left on the CPU with the reason recorded.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 6: Screen capture stills</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>The shim's screenshot</strong> (<code>mcap_sck_screenshot</code>, per display).</li>
|
||||||
|
<li><strong>The overlay:</strong> a window per screen, with the drag, magnifier, keys, and a click for the whole screen.</li>
|
||||||
|
<li><strong>The window it came from:</strong> its title and app.</li>
|
||||||
|
<li><strong>The clipboard</strong> (phase 2).</li>
|
||||||
|
<li><strong>The capture hotkey</strong> (open question 3).</li>
|
||||||
|
<li><strong>Permission:</strong> the check, and what mcap says when Screen Recording isn't granted.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong> on a Mac with a Retina screen and an external non-Retina one:</p>
|
||||||
|
<ul>
|
||||||
|
<li>The hotkey freezes both screens.</li>
|
||||||
|
<li>A region dragged on either is taken at the screen's own pixels, and is on the clipboard and in Screen Captures with its window's title and app.</li>
|
||||||
|
<li>With the permission refused, mcap says how to grant it.</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 7: Screen recording</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>The source:</strong> <code>modules/Screen_Record/sck_source.jai</code> over the shim's stream, as the third source. The encoder tries <code>h264_videotoolbox</code> first.</li>
|
||||||
|
<li><strong>Sound:</strong> system sound and the microphone, through the existing mixer.</li>
|
||||||
|
<li><strong>The overlay's video mode</strong>, with the 240 × 180 minimum.</li>
|
||||||
|
<li><strong>The border and toolbar</strong> as panels, kept out of the stream by the content filter. Pause, Discard, the sound buttons, and leftovers after a crash.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when</strong> the same checks pass as on Windows:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>record-probe</code> runs on the Mac, with <code>-sync-test</code>, <code>-pause</code> and <code>-mic-toggle</code>.</li>
|
||||||
|
<li>Neither border nor toolbar is in a recording.</li>
|
||||||
|
<li>A 10 s pause leaves no gap.</li>
|
||||||
|
<li>Beeps stay within about 25 ms of their flashes with no drift over 3 minutes.</li>
|
||||||
|
<li>A killed recording is offered at the next start.</li>
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="phase">
|
||||||
|
<h3>Phase 8: A signed release</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong><code>release.jai</code> for macOS:</strong> the <code>.app</code> with everything in it (SDL3, onnxruntime, FFmpeg, the models), signed with a Developer ID with the hardened runtime and the entitlements it needs.</li>
|
||||||
|
<li><strong>Notarized and stapled.</strong></li>
|
||||||
|
<li><strong>Packaged</strong> in a <code>.dmg</code>, with its SHA-256 printed, as the Windows zip's is.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong>Done when:</strong> the <code>.dmg</code>, downloaded on a second Mac, opens without a Gatekeeper warning, asks once for Screen Recording, and keeps that permission across an update to the next release.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="risks">7. Risks</h2>
|
||||||
|
<div class="risk"><strong>No Mac in this loop yet.</strong> Everything here can be written on Windows, but only built, run and checked on a Mac. Each phase needs a session there, yours or a remote one (open question 1). The Linux type check catches the shared files in between, but not the Objective-C or the macOS-only files.</div>
|
||||||
|
<div class="risk"><strong>CoreML and the models.</strong> CoreML takes some operators and not others. Mask2Former's attention and the panoptic post-processing may fall back to the CPU in parts, which can be slower than the CPU alone, as the data moves back and forth. Phase 5 measures each model, and keeps the CPU where CoreML doesn't win.</div>
|
||||||
|
<div class="risk"><strong>Permissions and signing.</strong> Without a stable signature, Screen Recording has to be granted again after every build, and a refused permission gives a capture of the empty desktop rather than an error. Development needs a certificate early, by phase 6.</div>
|
||||||
|
<div class="risk"><strong>FFmpeg licensing.</strong> The Windows FFmpeg is a GPL build (with libx264). On macOS an LGPL build is enough: VideoToolbox does H.264, and FFmpeg's own AAC encoder does sound. That avoids GPL obligations for the Mac release; the Windows release should be checked against the same question.</div>
|
||||||
|
<div class="risk"><strong>Multi-screen overlays.</strong> One window per screen, with Spaces, full-screen apps and screens at different scales, is where macOS differs most from Windows. The overlay must sit above full-screen apps (<code>NSWindowCollectionBehaviorCanJoinAllSpaces</code> and <code>FullScreenAuxiliary</code>).</div>
|
||||||
|
|
||||||
|
<h2 id="questions">8. Open questions</h2>
|
||||||
|
<ol>
|
||||||
|
<li><strong>The Mac:</strong> which Mac and macOS version will the work be checked on? Can I build and run there (SSH, or a remote session), or will you run the builds and send back the output?</li>
|
||||||
|
<li><strong>The oldest macOS:</strong> 14 Sonoma, for ScreenCaptureKit's screenshots, is proposed. 13 would need the deprecated <code>CGWindowListCreateImage</code> for stills; 15 would give the microphone in the stream itself.</li>
|
||||||
|
<li><strong>The keys:</strong> Mac keyboards have no PrintScreen, and ⌘⇧3/4/5 are the system's. Proposed: <strong>⌘⇧2</strong> for a still and <strong>⌃⌘⇧2</strong> to record, or F13 (PrintScreen on Apple's full keyboards) and ⌃F13, or your own.</li>
|
||||||
|
<li><strong>Signing:</strong> do you have, or want, an Apple Developer account (99 USD a year) for a stable development signature now and Developer ID notarization for releases? Without it, phases 6 and 7 work but ask for permission after every build.</li>
|
||||||
|
<li><strong>The library on a Mac:</strong> where should it default to? Proposed: <code>~/Pictures/mcap</code>. Should the library folder become a setting on both systems, in place of the hard-coded <code>D:/McapIndex/mcap</code>?</li>
|
||||||
|
<li><strong>The shim:</strong> is a small Objective-C file built with clang acceptable in the repo, or should ScreenCaptureKit be done from Jai alone, at more cost and risk?</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Loading…
Reference in New Issue
Block a user