Musa Mahmood c333d95da7 [0.0.7] README: how to use mcap, and the search bar

2026-10-08 13:21:14 +01:00

19 KiB

Raw Blame History

mcap

A browser for a library of screen captures: every .mcap file in a folder as a grid of thumbnails, searchable by the text in the images (OCR), by what they show (labels), by their window title, URL, app and annotations, and by meaning ("a circuit board"). Written in Jai with Dear ImGui, for Windows (macOS and Linux build, but OCR and labels are tested on Windows with an NVIDIA GPU).

The .mcap files come from Snagit's .snagx captures, converted by snagit-converter (D:\SnagitIndex\SnagitConverter); modules/Capture_File/README.md describes the format.

Getting started

Fetch what is not in git (each download's SHA-256 is checked):

cd modules/onnxruntime && jai fetch.jai           # onnxruntime 1.30.0, built for CUDA 13
cd modules/OCR         && jai fetch_models.jai    # the OCR models (~196 MB; `- mobile` for the light one only)
cd modules/Labels      && jai fetch_models.jai    # the label model, SigLIP 2 (~690 MB)

For the GPU (optional; without it everything runs on the CPU, several times slower): an NVIDIA driver, the CUDA Toolkit 13.x, and cuDNN 9.x for CUDA 13. cuDNN installed in its usual place (C:\Program Files\NVIDIA\CUDNN) is found without changing PATH. The OCR window says what, if anything, is missing.
Build and run: jai first.jai - release, then bin\mcap.exe. The build copies onnxruntime and the models beside the program.
The library is D:\SnagitIndex\mcap; -library FOLDER shows another folder.
Read the library: open the OCR window (button above the grid), check Labels too, and press Scan the library. Every capture is read and labelled once and the results are stored in its file, so this is needed only for new captures afterwards. On an RTX 5080 laptop, 8,600 captures took about 13 minutes with the mobile OCR model; labels add about 18 ms a capture, and the server OCR model is several times slower.

The main window

[ search box .......................... ]  8609 captures   Size [---o---]  OCR  Stats  Labels  Refresh
[ thin progress bar while a scan runs ]
+-------+ +-------+ +-------+ +-------+
| thumb | | thumb | | thumb | | thumb |   newest first
+-------+ +-------+ +-------+ +-------+

The grid shows every capture, newest first by capture date (by name while the catalog, what search needs from every file, is still loading: a line beside the count says so). Size sets the thumbnails' size. Click a thumbnail to open it in the Capture window.
Delete: hover a thumbnail and click the trash button in its corner. You are asked whether to move the file to the Recycle Bin or to delete it permanently, overwriting it with random bytes first; "Don't ask again" keeps the choice until the app restarts. Only the .mcap goes: the .snagx it was converted from is untouched.
Refresh looks at the folder again, after files were added or deleted outside the app.
OCR, Stats and Labels open their windows (below).
A thin bar under the search box shows a running scan; hover it for the details.

The Capture window

One capture at full size, with Left and Right stepping through the captures shown in the grid (in its order, so through a search's matches).

Fit: Window, Width, Height, or Off with the Zoom slider (and 100%).
OCR reads this capture now, and labels it when the label model is up (Read again when it has been read); Boxes shows the text lines found, outlined over the image (turned with the text for turned lines). Hover a box for its text and confidence. A capture read by several OCR models has a menu to choose whose lines show.
While searching, the lines that match are highlighted, with Matches: N and Previous / Next to scroll to each.
Labels: the capture's five best labels with their scores (dim when they come from an older label list).

The OCR window

Model: PP-OCRv5 mobile (light, about 85 ms a capture on the GPU) or server (more accurate, about 0.4 s). The choice is kept in the settings file (model under [ocr]).
Running on: the GPU or the CPU, and if the CPU, what to install for the GPU.
Scan the library reads every capture the model has not read yet and, with Labels too, labels every capture not yet labelled with the current list. While a search is active the button reads Scan the captures shown and covers only those. Each capture is stored as soon as it is read, so Pause, Cancel and quitting lose nothing; the next scan picks up where it stopped. Progress shows captures done, the rate, time elapsed and time left.
Erase OCR readings from the library (for testing) takes every OCR reading out of every file, after asking. A scan reads them again.
Below, the text of the capture open in the Capture window, line by line.

The Labels window

The label model's status, and how many captures are labelled, and with the current list.
The label list: labels.config in the app's data folder (%APPDATA%\mcap), with Open to edit it. Each line is a name and a prompt, the phrase the model compares with the image:
```
label: "Schematic"    "an electronic circuit schematic diagram"
label: "Code editor"  "a screenshot of a code editor showing source code"
```
A fuller description scores far better than a single word. Saved changes are picked up while the app runs; the next scan relabels every capture from the image vector stored in its file, without reading the images again, so changing the list is cheap.
Erase labels and vectors from the library (for testing), after asking.

The Stats window

What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR model: captures with text, lines, words, characters, confidence, words per capture with a breakdown, a table by year, the captures with the most words (click to open), each capture's best label counted (click one to search for it), and the commonest words (click one to search for it).

Searching

Type in the search box above the grid; the grid shows the matches as you type, with "N of M captures" beside the box. Clear the box to see everything again.

What a search matches

Every word must match, in any order: kicad footprint finds captures with both.
Matching ignores case and finds parts of words: foot matches "Footprints".
A plain word is looked for everywhere: the OCR text (every model's), the window title, the URL, the app's name, the annotations' text, and the labels.
"Quoted phrases" match as one, spaces and all: "symbol : footprint".

There is no OR and no NOT: to widen a search, take a word out.

Prefixes

A prefix limits a word, or a quoted phrase, to one field: title:kicad, title:"pcb editor".

Prefix	Looks in
(none)	everything below except `like:`
`ocr:`	the text read from the image
`title:`	the window title when it was captured
`url:`	the web page's address, for browser captures
`app:`	the app's name and version
`note:`	the text of the capture's annotations (Snagit text boxes and callouts)
`label:`	the capture's labels: its best label, and any scoring at least half as much
`like:`	the image's meaning (below)

A word with a colon that is not one of these stays an ordinary word, so http:// and C:\ work.

Searching by meaning: `like:`

like:"a circuit board under a microscope" ranks every labelled capture by how close its image is to the phrase, best first: no words need to appear in the image. The label model compares the phrase with the image vector each capture's file holds, so only labelled captures take part. The grid is then in that order instead of by date.

Describe what the image shows: like:"a graph with several colored lines" works better than like:graph.
Combine it with other words to filter the ranking: like:"a dark themed code editor" title:visual.
The phrase is quoted when it has spaces: like:"a terminal with an error".

Examples

Search	Finds
`digikey`	captures mentioning DigiKey anywhere: text, URL, title
`url:digikey`	DigiKey pages
`app:kicad`	captures of KiCad, by the app's name (more than `title:kicad`, as a window title need not say KiCad)
`ocr:"0402"`	images with "0402" in their text
`title:kicad footprint`	KiCad windows whose text, notes or labels mention a footprint
`label:schematic`	captures labelled Schematic
`label:"code review"`	captures labelled Code review
`note:todo`	captures with an annotation saying TODO
`like:"a photo of a soldering iron"`	captures that look like that, best first
`like:"a spreadsheet" app:excel`	Excel captures, the most spreadsheet-like first

Command line

Option	What it does
`-library FOLDER`	Shows the `.mcap` files in FOLDER instead of `D:\SnagitIndex\mcap`.
`-search QUERY`	Starts with QUERY in the search box.
`-scan`	Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed.
`-stats`	Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged.
`-erase ocr`, `-erase labels`, `-erase all`	Erases that from every file in the library, then exits (for testing).
`-allow-multiple-instances`	Runs beside another copy instead of bringing that one forward.
`run_tests`	Runs the test suites (debug builds); with `-headless`, exits with 0 if all passed.
`-headless`	No message boxes: a failure is logged and the app exits with 1.
`-version`	Prints the version.

-scan, -stats and -erase still open the window, to show their progress; they are meant for scripts and for trying things on a copy of the library: bin\mcap.exe -library D:\copy -scan.

Where things are kept

What	Where
OCR readings, labels, image vectors	inside each `.mcap` file (see `modules/Capture_File/README.md`)
The label list	`%APPDATA%\mcap\labels.config`
Settings (theme, hotkeys, OCR model)	`%APPDATA%\mcap\settings.config`
Logs	`logs\<version>\` beside the program

Nothing is written to D:\Snagit: the .mcap library is a converted copy.

How it works

src/app/viewer.jai: the grid and the Capture window. Previews and full images are decoded on worker threads and uploaded a few per frame; textures far from view are released past 1200. Each capture keeps its index for the app's life (Refresh and Delete mark captures removed and append new ones), as the workers, the scan and Stats refer to captures by index.
src/app/catalog.jai: what search needs from every file, read from the front of each .mcap (its text, OCR, labels and vector, not its images) on the workers at startup; the query parser and matching.
src/app/background.jai: searching and the Stats numbers run on a pool of threads, so typing never waits; the grid keeps the last results until the new ones are back, usually within a frame. like: phrases are turned into vectors there first.
src/app/ocr_ui.jai and labels_ui.jai: the engines, made on one background thread (OCR first, then labels); the scan, which decodes each image once and stores its OCR, labels and vector in one rewrite of its file; the erase; the windows.
modules/OCR, modules/Labels: the models, run with onnxruntime (modules/onnxruntime, which also holds the shared session and CUDA code); each has a README.
modules/Capture_File: reading and writing .mcap files.

The application template

mcap is built on a bare Dear ImGui (docking branch) application template, for Windows, macOS and Linux: the window, the frame loop and everything around them, with the app written into src/app/app_ui.jai.

What you get

Native rendering per platform. Win32 + Direct3D 11 on Windows; SDL3 + SDL_GPU (Metal on macOS, Vulkan on Linux) elsewhere. On Windows the window comes up on a WARP device in a few milliseconds while the hardware device is created on a thread, then moves across; a lost device is recovered.
Docking and multi-viewport ImGui, with a default layout (assets/data/default_imgui.ini) and a Reset Layout hotkey.
One instance. Launching the app again brings the running one forward, over a shared memory channel (a file lock decides the owner on macOS and Linux).
Tray / menu bar. Closing or minimizing hides the window to the tray; the tray icon brings it back or quits.
Hotkeys from a [hotkeys] section of the settings file, and global hotkeys ([global_hotkeys]: Windows and macOS) that work while another app is in front.
A settings file (settings.config) that is written with every key listed, reloaded as soon as it is saved, and merged rather than overwritten when the app saves to it.
Themes: built-ins, 36 shipped .theme files, and a theme editor that saves new ones.
Logging through a multi-producer, single-consumer queue to a per-version log file and an in-app Event Log, plus crash reports.
Debug panels: frame times, memory, renderer, fonts (Ctrl+Shift+D, Ctrl+Shift+1).
DPI awareness, live resize that keeps drawing, CJK font fallback from the system fonts.
Background shutdown: closing hands the remaining work to a thread while the window shows progress, so it never looks hung.
Asynchronous tests, compiled into debug builds and run with run_tests.

Building

Requires the Jai compiler, beta 0.2.030 or later.

jai first.jai                           # debug build
jai first.jai - release                 # release build
jai first.jai - full_release            # debug and release
jai first.jai - release compile_tests   # release build with the test suite

Binaries are written to bin/, with assets/ copied beside them.

Platform	Output	Notes
Windows	`bin/mcap-debug.exe`, `bin/mcap.exe`	The release build gets the icon and a manifest, and no console.
macOS	`bin/mcap-macos-debug`, `bin/mcap.app`	Apple silicon (arm64) only, with a native `jai`. The release build is an ad hoc signed `.app`.
Linux	`bin/mcap-linux-debug`, `bin/mcap-linux`	Needs SDL3 installed (`libSDL3.so`).

jai check_targets.jai - linux type-checks the Linux build from any OS without linking it. The macOS check only works on a Mac, since the Objective-C module runs code at compile time.

Run any build with run_tests to run the test suites (debug builds include them), -allow-multiple-instances to skip the single-instance check, or -version.

For unattended runs (scripts, CI, agents), add -headless: the app exits once the tests finish, with exit code 0 if every suite passed and 1 otherwise, and a failed assertion or runtime check is written to stderr, the log and the crash report and exits with 1 instead of opening a message box (runtime_support_headless in modules/Runtime_Support.jai):

bin/mcap-debug.exe run_tests -headless -allow-multiple-instances > test-output.txt

jai first.jai - no_crash_dialog [...] makes builds for automation: NO_CRASH_DIALOG makes them headless whatever their arguments, and everything they build goes to bin-headless/ instead of bin/, so an automated build never writes over the one you run by hand.

Starting a new app

jai fork.jai - <output_directory> <app_name> [-folder <name>] [-bundle-id <id>] [-version <x.y.z>] [-git] [-build]
jai fork.jai - ../my-cool-app "My Cool App" -git -build

This copies the template (without bin/, .build/, .git/ or fork.jai) and sets the names below in the copy. The folder name defaults to the app name in lowercase with spaces as - (my-cool-app), the bundle identifier to com.example.<folder without dashes>, and the version to 0.0.0. -git initializes a repository and commits the copy; -build makes a debug build. It also writes <app_name>.focus-config, a focus project with the build and run commands, which stays out of git since it holds this machine's paths.

Making it your app

Name it at the top of first.jai, or let fork.jai do it: APP_NAME (window title, tray, macOS bundle), APP_FOLDER_NAME (executable and data folder; one short lowercase word), APP_BUNDLE_IDENTIFIER and VERSION.
Replace assets/images/icon.png (any square size; it becomes the .exe, .app and window icon) and the tray icons tray_icon.ico (Windows) and tray_icon.png.
Write the app in src/app/app_ui.jai: app_ui_startup runs once, app_ui_frame_start every frame before input, app_main_window_ui draws, and app_ui_shutdown runs on the shutdown thread.

To add a hotkey, declare a Hotkey in src/ui/imgui_keyboard.jai, add it to hotkey_actions_build and app_check_key_combinations, bind it in both assets/data/default_hotkeys*.config files (the build fails if a default leaves an action unbound), and raise APP_USER_SETTINGS_VERSION in src/App.jai so existing settings files get the new default.

To add a setting, add it to App_User_Settings, Settings_Seen, settings_parse and app_save_user_settings_file in src/ui/imgui_theme_registry.jai.

Layout

first.jai               build: workspaces, build constants, Windows icon/manifest, macOS .app
fork.jai                copy the template to a new, named project
check_targets.jai       type-check another OS's build
src/App.jai             loads everything; global state (app, platform, perf)
src/app/                main loop, your UI, background shutdown
src/base/               logging, fonts, debug panels, platform layers (base/platform/)
src/ui/                 ImGui setup, hotkeys, themes and settings, toasts
src/graphics/DX11.jai   Direct3D 11 device, WARP-first startup, device-loss recovery
src/backends/           ImGui's SDL_GPU backend; src/freetype/: ImGui's FreeType loader
assets/                 fonts, icons, default hotkeys, default layout, themes
modules/                Basic (with arenas), Thread, Shared_Memory_Channel, File_Watcher,
                        Secure_Memory, XDG, sdl3, imgui-sdl, imgui-dx11-win32, imgui-source

Files at run time

File	Location
`settings.config`, `imgui.ini`, user themes	`%APPDATA%\<APP_FOLDER_NAME>` on Windows, `~/Library/Application Support/<APP_FOLDER_NAME>` on macOS, `~/.local/share/<APP_FOLDER_NAME>` on Linux
Logs	`logs/<version>/` next to the executable (next to the `.app` on macOS)
Shipped themes	`data/themes/` next to the executable

Platform notes

Global hotkeys are not implemented on Linux (Wayland has no API for them). A desktop shortcut that launches the executable has the same effect, since a second launch brings the running instance forward.
On XFCE the tray icon stays after the window is restored.
Secure_Memory (used by Basic's arenas to wipe memory) is built without libsodium; see the note at the top of modules/Secure_Memory/module.jai if your app keeps secrets in memory.

Uses https://github.com/overlord-systems/jai-imgui as a template.

19 KiB Raw Blame History