MusaMahmood/mcap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
91f6cf768f
mcap/README.md

626 lines
40 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# mcap
A browser for a library of screenshots and photos: every `.mcap` file in a folder (and its folders)
as a grid of thumbnails, searchable by the text in the images (OCR), by what they show (labels), by
their window title or file name, URL, app or camera and annotations, and by meaning ("a circuit board"). Written in [Jai](https://jai.community)
with [Dear ImGui](https://github.com/ocornut/imgui), for Windows (macOS and Linux build, but OCR
and labels are tested on Windows with an NVIDIA GPU).
The `.mcap` files come from the **Import** window, which takes any folder of images (PNG, JPEG, HEIC,
AVIF, WebP, GIF, BMP, TIFF, camera RAW), videos (MP4, MOV, MKV, WebM, AVI, WMV, MPEG) and Snagit
`.snagx` captures, or from `snagit-converter`
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. On
Windows, **PrintScreen** captures a region of the screen straight into the library, as Snagit does;
see [Screen capture](#screen-capture).
## Getting started
1. **Fetch what is not in git** (each download's SHA-256 is checked):
```
cd modules/onnxruntime && jai fetch.jai # onnxruntime 1.30.0, built for CUDA 13
cd modules/OCR && jai fetch_models.jai # the OCR models (~196 MB; `- mobile` for the light one only)
cd modules/Labels && jai fetch_models.jai # the label model, SigLIP 2 (~690 MB)
```
2. **For the GPU** (optional; without it everything runs on the CPU, several times slower): an
NVIDIA driver, the CUDA Toolkit 13.x, and cuDNN 9.x for CUDA 13. cuDNN installed in its usual
place (`C:\Program Files\NVIDIA\CUDNN`) is found without changing PATH. The OCR window says
what, if anything, is missing.
**On macOS**, labelling, OCR and segmentation run on the GPU through Core ML, from Core ML copies
of the models made once with `tools/export_siglip_coreml.py` (SigLIP 2's image model, about ten
times as fast as the CPU), `tools/export_ppocr_coreml.py` (PP-OCRv5's detector and recognizer,
about twice as fast; run it with `--model mobile` and `--model server`) and
`tools/export_mask2former_coreml.py` (Mask2Former, about four times as fast; after
`tools/export_mask2former.py`). Their first lines say how to set up the Python they need. Without them, each runs on the
CPU. The first scan after making them is slower: Core ML prepares each image size the first time it
meets it, and macOS keeps that for later runs.
3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime
and the models beside the program.
4. **The library** is `D:\McapIndex\mcap` on Windows and `~/Pictures/mcap` on macOS and Linux, a
folder for each place imported from (`SnagitLibraryPersonal`, `SnagitLibraryWork`, ...).
`folder:` under `[library]` in `settings.config` names another, read when mcap starts, and
`-library FOLDER` overrides both.
**Import** (button above the grid) brings in more; see [Importing](#importing).
5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**,
and press **Scan the library**. Every capture is read and labelled once and the results are
stored in its file, so this is needed only for new captures afterwards. On an RTX 5080 laptop,
8,600 captures took about 13 minutes with the mobile OCR model; labels add about 18 ms a capture,
and the server OCR model is several times slower.
## The main window
```
[ search box .......................... ] 8609 captures Size [---o---] OCR Stats Labels Refresh
[ thin progress bar while a scan runs ]
+-------+ +-------+ +-------+ +-------+
| thumb | | thumb | | thumb | | thumb | newest first
+-------+ +-------+ +-------+ +-------+
```
- **The grid** shows every capture, newest first by capture date (by name while the catalog,
what search needs from every file, is still loading: a line beside the count says so). **Size**
sets the thumbnails' size. Click a thumbnail to open it in the Capture window.
- **Delete:** hover a thumbnail and click the trash button in its corner. You are asked whether to
move the file to the Recycle Bin (the Trash on macOS, a `deleted` folder in the library on Linux)
or to delete it permanently, overwriting it with random bytes
first; "Don't ask again" keeps the choice until the app restarts. Only the `.mcap` goes: the
`.snagx` it was converted from is untouched.
- **Copy:** click a thumbnail to select it (it is outlined, and shown in the Capture window), then
**Ctrl-C** copies its image, as the Capture window shows it (annotations and all; a video's
frame), and **Ctrl-Shift-C** its full path. Neither acts while you type in the search box. They
are `copy_image` and `copy_path` in `[hotkeys]` in `settings.config` (**Cmd** for Ctrl on macOS).
Copying images works on Windows and macOS, not yet on Linux.
- **Ctrl-F** puts the keyboard in the search box, its text selected (`focus_search` in `[hotkeys]`).
- **Files added or deleted outside the app** (in Explorer, by another program) appear in the grid
or leave it on their own, folders added, renamed or deleted too: the library is watched.
**Refresh** looks at the whole folder again, should the watcher have missed something.
- **OCR**, **Stats** and **Labels** open their windows (below).
- A **thin bar** under the search box shows a running scan; hover it for the details.
### The Capture window
One capture at full size, with **Left** and **Right** stepping through the captures shown in the
grid (in its order, so through a search's matches).
- **Fit:** Window, Width, Height, or Off with the **Zoom** slider (and **100%**).
- **OCR** reads this capture now, and labels it when the label model is up (**Read again** when it
has been read); **Boxes** shows the text
lines found, outlined over the image (turned with the text for turned lines). Hover a box for its
text and confidence. A capture read by several OCR models has a menu to choose whose lines show.
- While searching, the lines that match are highlighted, with **Matches: N** and **Previous** /
**Next** to scroll to each.
- **Labels:** the capture's five best labels with their scores (dim when they come from an older
label list).
- **Segments**, for a segmented image:
- **Drawing:** what segmentation found is drawn over the image, each class in its own colour, the
same in every image. Each instance is a different shade, so two people side by side differ, and
edges are darker.
- **Controls:** an opacity slider, and a legend of the classes, largest first, with how many of
each. Untick a class to hide it; the choice holds as you step through captures. Right-click a
class to search `has:` for it.
- **Colours:** a class's own colour is a hue from a hash of its name. Click its square in the legend
to choose another, in every image and library. Right-click the square to go back to its own. The
choices are kept in `segment-colours.config` in the app's data folder.
- **Hover** the image for the segment under the mouse: its class, score and share of the image.
### The OCR window
- **Model:** PP-OCRv5 mobile (light, about 85 ms a capture on the GPU) or server (more accurate,
about 0.3 s when it runs well; see below). The choice is kept in the settings file (`model` under
`[ocr]`).
- **GPU memory** (fixed in 0.1.24): OCR used to fill the GPU, 9 GB with the mobile model on
gpu-bench's 30 images and past 16 GB with the server model, which Windows then paged into shared
system memory, running it at 1.5 to 11 s a capture. onnxruntime's arena keeps every block it takes
from CUDA, and OCR's inputs change size every run, so each larger image or wider batch of lines left
another block behind. OCR now runs the largest images and widest batches first, has the detector's
and recognizer's arenas give back what they hold after each stage of a call, and keeps its 4 worker
threads for the engine's life. The mobile model now peaks at 5.7 GB and drops to 2 GB between
calls, at 142 ms a capture against 126 before (gpu-bench, 2026-10-17). The server model has not been
measured again yet.
- **Running on:** the GPU or the CPU, and if the CPU, what to install for the GPU.
- **Scan the library** reads every capture the model has not read yet and, with **Labels too**,
labels every capture not yet labelled with the current list. While a search is active the button
reads **Scan the captures shown** and covers only those. Each capture is stored as soon as it is
read, so **Pause**, **Cancel** and quitting lose nothing; the next scan picks up where it stopped.
Progress shows captures done, the rate, time elapsed and time left.
- **Erase OCR readings from the library** (for testing) takes every OCR reading out of every file,
after asking. A scan reads them again.
- Below, the text of the capture open in the Capture window, line by line.
### The Labels window
- The label model's status, and how many captures are labelled, and with the current list.
- **The label list:** `labels.config` in the app's data folder (`%APPDATA%\mcap`), with **Open** to
edit it. Each line is a name and a prompt, the phrase the model compares with the image:
```
label: "Schematic" "an electronic circuit schematic diagram"
label: "Code editor" "a screenshot of a code editor showing source code"
```
A fuller description scores far better than a single word. Saved changes are picked up while the
app runs; the next scan relabels every capture from the image vector stored in its file, without
reading the images again, so changing the list is cheap.
- **Erase labels and vectors from the library** (for testing), after asking.
### The Segmentation window
**Segments** (above the grid) finds what is where in images: people, animals and things, each one
apart, and stuff such as sky, wall and grass (Mask2Former, `modules/Segment`, 133 COCO classes). It
works only in the library folders you choose, and never on videos.
- **Library folders:** a table of every folder in the library:
- **Segment:** whether scans read that folder's images. It starts off for every folder, so adding a
large archive never starts a long GPU job by itself.
- **Segmented:** how many of its images hold segments.
- **Erase:** takes that folder's segments out of its files, after asking.
The choice is kept in the library (`.library-folders`), not in the app's settings, so it goes
where the library goes.
- **Segment the folders set to segment** reads every image in them not yet segmented. Each is stored
as it is done, so a scan can stop and start again. It runs on the OCR thread, with the same
progress bar, Pause and Cancel. The model loads when the first scan starts and takes about 2.4 GB
of GPU memory; on an RTX 5080 laptop a scan does about 17 images a second.
- **Importing** into a folder set to segment segments the new images once they are in. The Import
window's "Segment the images in this folder" checkbox is the same setting.
- **Storage:** each image's segments are stored in its file: a short list (class, score, area, box)
for search, and a mask of about 5 KB for drawing. See `modules/Capture_File/README.md`.
### The Stats window
What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR
model: captures with text, lines, words, characters, confidence, words per capture with a breakdown,
a table by year, the captures with the most words (click to open), each capture's best label
counted (click one to search for it), and the commonest words (click one to search for it). When any
of them are segmented, it also shows **What is in the images**: how many images hold each class
(person, sky, dog ...). Click one to search `has:` for it.
## Screen capture
On Windows, **PrintScreen** captures any part of the screen, from any app, even while mcap is hidden
in the tray:
1. **The screens freeze** the moment it is pressed: everything on every screen is copied and shown
darkened over them, so menus, tooltips and video hold still while you choose.
2. **Drag a rectangle** over what you want. It shows at full brightness, outlined. A magnifier beside
the cursor shows the pixels under it, with the cursor's position, or the selection's size while
dragging. The arrow keys move the cursor a pixel at a time, ten with Shift.
3. **Let go** to take it. A **click** without dragging, or **Enter**, takes the whole screen the
cursor is on. **Escape**, or a right click, cancels; a right click while dragging starts the
selection again.
What is taken is on the clipboard at once, ready to paste. It is also saved into the library, in
the folder **Screen Captures**, as a PNG in an `.mcap` with the window it was taken from (the window
under the middle of the selection): its title as the window title and its program as the app, so
searches find it as they find Snagit's. mcap then comes to the front showing it in the Capture
window, and OCR and labels read it once they are free (and segmentation, if the folder is set to
segment).
The key is the `capture_region` line of `[global_hotkeys]` in the settings file. Only one program can
have PrintScreen: if Snagit is running it has it, and so does Windows while Settings >
Accessibility > Keyboard > "Use the Print screen key to open screen capture" is on. When mcap cannot
register the key, the Event Log says so; choose another, such as `Ctrl-Shift-S capture_region`.
### Recording the screen
**Ctrl+PrintScreen** records a region of one screen as a video:
1. **The screens freeze**, as for a still, with the outline and magnifier in red.
2. **Drag a region**, or click (or press Enter) for the whole screen under the cursor:
- The drag stays on the screen it began on: a recording is of one screen.
- A region smaller than **240 × 180** is outlined dashed, with the minimum in the label, and letting
go of it starts nothing; drag again.
- Escape or a right click cancels.
3. **Recording starts** at once, at up to 30 frames a second, cursor included. A frame is written
only when the screen changes, so a still screen costs almost nothing. **Sound:** what the
computer plays is recorded with it, and the microphone once you turn it on. The toolbar's
**System sound** and **Mic** buttons turn each on and off, mid-recording too, and their state is
kept for the next recording.
- **A red border** is drawn just outside the region.
- **A toolbar** beside it shows the time, the size and any frames dropped, with **System sound**
and **Mic** (each on or off), **Pause** (the time paused is left out of the video), **Stop**, and **Discard**, which asks
first.
- **Neither is in the recording.** Neither takes the focus from the window being recorded.
4. **Ctrl+PrintScreen again**, or Stop, stops it. The video goes into **Screen Captures** beside the stills,
as an H.264 MP4 in an `.mcap` with the window it was recorded from, and opens in the Capture window.
OCR and labels read a frame from it.
- **The screen** is taken with DXGI Desktop Duplication, or with Windows.Graphics.Capture where that
cannot run (a laptop with two GPUs, a rotated screen).
- **Encoding** uses NVENC where there is an NVIDIA GPU, else AMD's or Intel's encoder, Media
Foundation, or libx264.
- **Losing nothing:**
- A recording that stops by itself keeps what it has, and so does quitting while recording.
- When the screen is lost for a moment (a UAC prompt, the lock screen, a resolution change), the
recording carries on once it is back.
- A recording a crash cut short is offered at the next start: into the library, or deleted. It
keeps all but its last two seconds or so.
- **Settings:** `[video]` in the settings file:
- `fps` (30) and `cursor` (true).
- `capture`: `auto` (Desktop Duplication, else Windows.Graphics.Capture), `dxgi` or `wgc`.
- `system_audio` (true): what the computer plays, from the default output device. The toolbar's
System sound button sets it.
- `microphone` (false): the default input device. The toolbar's Mic button sets it.
- Sound is AAC at 160 kb/s. A device unplugged mid-recording is picked up again from the new default.
- **PrintScreen while recording** takes the whole screen under the cursor at once, without freezing
the screens, which would show in the recording.
- **The key** is `record_region` in `[global_hotkeys]`.
## Importing
**Import** (above the grid) brings a folder of screenshots, photos and videos into the library:
1. **Import from:** the folder, typed or chosen with **Browse...** (Windows and macOS). Everything under it is looked
at, its folders too (not hidden ones, such as `.thumbnails`).
2. **Into library folder:** the folder in the library it goes into, such as `SnagitLibraryWork` or
`Phone`. It starts as the name of the folder imported from, or the library folder that folder went
into last time, so importing the same place again adds to the same folder.
3. **Import.** The progress, the time left, and Pause and Cancel show while it runs; new captures
appear in the grid as they are written. With **Read text and labels** checked, the OCR scan then
reads and labels what came in.
What becomes of each file:
- **Images** (PNG, JPEG, HEIC and AVIF, WebP, GIF, BMP, PSD, TGA, TIFF, and camera RAW such as DNG,
CR2, CR3, NEF and ARW): an `.mcap` holding the file exactly as it was, a preview turned the right
way up (by its EXIF orientation), and where it came from: its file name (searched by `title:`), the
camera or phone (`app:`), and the date it was taken, from its EXIF, else a date in its name
(`IMG_20240512_101530`, `Screenshot 2024-05-12 at 10.15.30`), else the file's time. It is named
`2024-05-12_10-15-30_1a2b3c4d.mcap`, the date and the start of its hash. HEIC, AVIF, WebP, TIFF and RAW
are read by the system's own codecs: on Windows the HEIF, AV1, WebP and Raw Image Extensions from
the Microsoft Store, and on macOS ImageIO, which has them all built in. Linux reads only PNG, JPEG,
GIF, BMP, PSD and TGA.
- **Animated GIFs** are images; their thumbnail is marked **GIF**, and the Capture window says how
long they run and opens them in the default viewer.
- **Videos** (MP4, MOV, M4V, 3GP, MKV, WebM, AVI, WMV, MPEG, MTS and M2TS, FLV), read with FFmpeg
(`modules/ffmpeg`; Windows and macOS, not Linux): an `.mcap` holding the file exactly as it was, of any size (it is
copied a few MB at a time, never read whole), and a frame from it a tenth of the way in (at most 3
seconds), turned the right way up, which is what the grid, the Capture window, OCR and labels see.
The date is the phone's (Apple's creation date), else one in its name, else the video's own
creation time, else the file's time; the phone and its location are kept too. The thumbnail shows a
play mark and how long it runs; **Play in the default player** in the Capture window writes the
video out to the temporary folder and opens it. **Export...** (any capture) writes the file as it
was imported to a folder it asks for: one of the `folder:` lines under `[export]` in the settings
file (the first is the default; none: the Downloads folder), or **Custom...**, which can be added to
them. A file of that name already there, and different, is not overwritten.
- **Snagit `.snagx`:** the same `.mcap` as `snagit-converter mcap` makes, byte for byte, named as
the `.snagx` is.
- **Duplicates:** an image already in the library, in any of its folders, is not stored again; the
Import window lists each with where it is. Importing a folder a second time imports only what is
new.
- **Skipped:** sidecar files (`.aae`, `.xmp`, `.json`), a GoPro's thumbnails and low-resolution
copies (`.thm`, `.lrv`), videos on macOS and Linux, and anything that is not an image or a video,
counted by kind.
- **Failed:** files that cannot be read or decoded, listed with why.
The files imported from are only read: nothing there is changed, moved or deleted.
## Opening a file
`mcap.exe FILE`, with FILE first, opens one file: an `.mcap`, or any image, video or `.snagx` that
Import takes. This is what Explorer passes to the program a file type opens with (Open with > mcap).
- **mcap not running:** it starts as usual and shows the file.
- **An `.mcap` in the library** is selected and opened in the Capture window.
- **Anything else** opens in the **Opened File** window, with its details. If the library has it
already, in any folder (found by its hash in `.import-index`), the window says where, with
**Show**. If not, **Import** brings it into the library folder typed or chosen from the menu
beside the name. That's the Import window's import, of this one file, so the same naming,
duplicate check and reading of text and labels apply. Once in, it opens in the Capture window.
- **An `.mcap` from outside the library** is copied in as it is, under its own name. (A folder's
import skips `.mcap` files.)
- **mcap already running:** a second window opens in **view mode**: the file alone, with Fit and
Zoom, and the same Import.
- **Show in mcap** (for a file the library has) and **Open in mcap** (for any other) hand the file to
the running mcap, which comes forward showing it, and close the view.
- **Ctrl-C** copies the image, and **Ctrl-Shift-C** the path.
- **Escape**, Ctrl-W or the close button ends it.
- **What's left out:** the library isn't loaded or watched, and there are no models, OCR, labels or
segmentation. There's no tray icon and no PrintScreen. It writes no settings or window layout.
- **After an import here:** the running mcap picks the capture up through the library watcher. OCR
and labels read it at the next scan, or right away with **Show in mcap** and **OCR** in the
Capture window.
## Searching
Type in the search box above the grid; the grid shows the matches as you type, with "N of M
captures" beside the box. Clear the box to see everything again.
### What a search matches
- **Every word must match**, in any order: `kicad footprint` finds captures with both.
- **Matching ignores case and finds parts of words:** `foot` matches "Footprints".
- **A plain word is looked for everywhere:** the OCR text (every model's), the window title, the
URL, the app's name, the annotations' text, and the labels.
- **"Quoted phrases"** match as one, spaces and all: `"symbol : footprint"`.
There is no OR and no NOT: to widen a search, take a word out.
### Prefixes
A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:"pcb editor"`.
| Prefix | Looks in |
| --- | --- |
| *(none)* | everything below except `like:`, `folder:` and `has:` |
| `ocr:` | the text read from the image |
| `title:` | the window title when it was captured; for an imported image, its file name |
| `url:` | the web page's address, for browser captures |
| `app:` | the app's name and version; for a photo, the camera or phone |
| `note:` | the text of the capture's annotations (Snagit text boxes and callouts) |
| `label:` | the capture's labels: its best label, and any scoring at least half as much |
| `like:` | the image's meaning (below) |
| `folder:` | the library folder it is in: `folder:work` |
| `has:` | what segmentation found in the image (below): `has:dog` |
A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work.
### What is in the image: `has:`
`has:dog` finds images that segmentation (the Segmentation window) found a dog in. Only images in
folders set to segment, and already segmented, take part.
- **The whole class name:** `has:table` is a table, not a dining table; `has:"dining table"` is that.
- **The 133 COCO classes:** 80 kinds of thing (person, bicycle, car, dog, cat, chair, laptop, cell
phone, book ...) and 53 of stuff (sky, wall, grass, road, floor wood, ...). The Stats window lists
those in the images shown.
- **Combining:** `has:person has:dog` finds images with both; `has:cat label:"photo"` mixes it with
labels.
### Searching by meaning: `like:`
`like:"a circuit board under a microscope"` ranks every labelled capture by how close its image is
to the phrase, best first: no words need to appear in the image. The label model compares the phrase
with the image vector each capture's file holds, so only labelled captures take part. The grid is
then in that order instead of by date.
- Describe what the image shows: `like:"a graph with several colored lines"` works better than
`like:graph`.
- Combine it with other words to filter the ranking: `like:"a dark themed code editor" title:visual`.
- The phrase is quoted when it has spaces: `like:"a terminal with an error"`.
### Examples
| Search | Finds |
| --- | --- |
| `digikey` | captures mentioning DigiKey anywhere: text, URL, title |
| `url:digikey` | DigiKey pages |
| `app:kicad` | captures of KiCad, by the app's name (more than `title:kicad`, as a window title need not say KiCad) |
| `ocr:"0402"` | images with "0402" in their text |
| `title:kicad footprint` | KiCad windows whose text, notes or labels mention a footprint |
| `label:schematic` | captures labelled Schematic |
| `label:"code review"` | captures labelled Code review |
| `note:todo` | captures with an annotation saying TODO |
| `like:"a photo of a soldering iron"` | captures that look like that, best first |
| `like:"a spreadsheet" app:excel` | Excel captures, the most spreadsheet-like first |
| `has:dog` | images with a dog in them (segmented folders) |
| `has:person has:"cell phone"` | images with a person and a phone |
## Command line
| Option | What it does |
| --- | --- |
| `FILE` (first) | Opens FILE: an `.mcap`, image or video; in view mode when mcap is already running. See [Opening a file](#opening-a-file). |
| `-open-import DEST` | With FILE: imports it into the library folder DEST as its Import button does, then exits: 0 once the library has it (imported, or there already), else 1. |
| `-library FOLDER` | Shows the `.mcap` files in FOLDER (and its folders) instead of the library folder in `settings.config` (`[library] folder`). |
| `-import SOURCE DEST` | Imports SOURCE, a folder or a single file, into the library folder DEST, as the Import window does, then exits: 0, or 1 if any file failed. |
| `-search QUERY` | Starts with QUERY in the search box. |
| `-cold` | Loads the library as on a cold start: its files and the catalog cache are dropped from the system's file cache first (Windows and Linux; not macOS, which allows it only with `sudo purge`). The `[startup]` lines in the log time each step. With `-stats`, a startup benchmark: `bin\mcap.exe -cold -stats`. |
| `-no-catalog-cache` | Reads every capture from its file, as without the catalog cache (for measuring); the cache is still written. |
| `-catalog-threads N` | Loads the catalog on N threads instead of all the background group's (for measuring). |
| `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. |
| `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. |
| `-record SECONDS` | Records the screen under the cursor that long (no overlay), puts it in the library, then exits: 0, or 1 if it failed. `-record-region X Y W H` records that region instead. `-record-quit` quits while still recording, to check the recording is kept. `-record-ui-visible` leaves the border and toolbar in screenshots, to look at them. |
| `-segment` | Segments the images not yet segmented in the library folders set to segment (see the Segmentation window), once the library has loaded, then exits: 0, or 1 if any image failed. |
| `-erase ocr`, `-erase labels`, `-erase segments`, `-erase all` | Erases that from every file in the library, then exits (for testing). |
| `-allow-multiple-instances` | Runs beside another copy instead of bringing that one forward. |
| `run_tests` | Runs the test suites (debug builds); with `-headless`, exits with 0 if all passed. |
| `-headless` | No message boxes: a failure is logged and the app exits with 1. |
| `-version` | Prints the version. |
| `-h`, `--help` | Opens the Command Line window, this table in the app (also in the Debug Panel). With mcap already running, that one comes forward with it. |
`-import`, `-scan`, `-stats` and `-erase` still open the window, to show their progress; they are meant for
scripts and for trying things on a copy of the library: `bin\mcap.exe -library D:\copy -scan`.
## Where things are kept
| What | Where |
| --- | --- |
| OCR readings, labels, image vectors | inside each `.mcap` file (see `modules/Capture_File/README.md`) |
| The label list | `%APPDATA%\mcap\labels.config` |
| The segment colours chosen in the Capture window | `%APPDATA%\mcap\segment-colours.config` |
| Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` |
| Logs | `logs\<version>\` beside the program |
| The hash of every capture's image, for finding duplicates | `.import-index` in the library folder |
| The catalog cache: every capture's catalog entry, so a start reads one file instead of every capture (an entry is used while its file's size and modified time are unchanged; deleting it is safe) | `.catalog-cache` in the library folder |
| The folder each library folder was imported from | `.import-sources` in the library folder |
| Which library folders are segmented | `.library-folders` in the library folder |
Nothing is written to the folders imported from (`D:\Snagit`, a phone's backup): the library is a
converted copy.
## How it works
- `src/app/viewer.jai`: the grid and the Capture window. Previews and full images are decoded on
worker threads and uploaded a few per frame; textures far from view are released past 1200.
Each capture keeps its index for the app's life (Refresh and Delete mark captures removed and
append new ones), as the workers, the scan and Stats refer to captures by index.
- `src/app/catalog.jai`: what search needs from every file, read from the front of each `.mcap`
(its text, OCR, labels and vector, not its images) on the workers at startup; the query parser
and matching.
- `src/app/background.jai`: searching and the Stats numbers run on a pool of threads, so typing
never waits; the grid keeps the last results until the new ones are back, usually within a frame.
`like:` phrases are turned into vectors there first.
- `src/app/ocr_ui.jai` and `labels_ui.jai`: the engines, made on one background thread (OCR first,
then labels); the scan, which decodes each image once and stores its OCR, labels and vector in one
rewrite of its file; the erase; the windows.
- `modules/OCR`, `modules/Labels`: the models, run with onnxruntime (`modules/onnxruntime`, which
also holds the shared session and CUDA code); each has a README.
- `src/app/import.jai`: the Import window and `-import`. A thread lists the files, brings the
library index up to date (hashing the image in any `.mcap` it has not seen), then imports on
several threads; captures written are handed to the grid each frame.
- `modules/Image_Decode`: any image file's bytes to pixels, the right way up: stb_image for PNG,
JPEG, GIF, BMP, PSD and TGA, and for the rest Windows Imaging Component (`wic.jai`) or macOS's
ImageIO (`imageio.jai`); EXIF (from JPEG, PNG, WebP,
HEIF and TIFF); previews; and videos' frames and what their containers say, with FFmpeg
(`video.jai`), and how many frames a GIF has. `bin\image-probe FILE...` (`jai first.jai -
image_probe`) shows what it makes of files.
- `modules/Screen_Record`: a region of one screen recorded to H.264 in an MP4, with DXGI Desktop
Duplication, or Windows.Graphics.Capture where duplication cannot run (two GPUs, a rotated screen),
and FFmpeg (NVENC, else AMF, QSV, Media Foundation or libx264), for screen recording
(`video-capture-plan.html`). `bin\record-probe` (`jai first.jai - record_probe`) records with it
and says how it went; `-list` shows the screens, `-capture auto|dxgi|wgc` picks the way, `-check
FILE` reads a recording back.
- `modules/Snagx`: `.snagx` to `.mcap`, copied from SnagitConverter.
- `modules/Capture_File`: reading and writing `.mcap` files, with a video's original left on disk.
- `modules/Segment`: panoptic segmentation with Mask2Former, run with onnxruntime (see its README and
`segmentation-plan.html`). `src/app/segment_ui.jai` is the scan and the Segmentation window;
`src/app/library_folders.jai` the per-folder options.
- `modules/ffmpeg`: FFmpeg 6.0's bindings and DLLs (`modules/ffmpeg/windows`, in git; copied beside
the executable by the build), from the gyan.dev shared build, for reading videos.
# The application template
mcap is built on a bare Dear ImGui (docking branch) application template, for Windows, macOS and
Linux: the window, the frame loop and everything around them, with the app written into
`src/app/app_ui.jai`.
## What you get
- **Native rendering per platform.** Win32 + Direct3D 11 on Windows; SDL3 + SDL_GPU (Metal on
macOS, Vulkan on Linux) elsewhere. On Windows the window comes up on a WARP device in a few
milliseconds while the hardware device is created on a thread, then moves across; a lost
device is recovered.
- **Docking and multi-viewport ImGui**, with a default layout (`assets/data/default_imgui.ini`)
and a Reset Layout hotkey.
- **One instance.** Launching the app again brings the running one forward, over a shared
memory channel (a file lock decides the owner on macOS and Linux).
- **Tray / menu bar.** Closing or minimizing hides the window to the tray; the tray icon brings
it back or quits.
- **Hotkeys** from a `[hotkeys]` section of the settings file, and **global hotkeys**
(`[global_hotkeys]`: Windows and macOS) that work while another app is in front.
- **A settings file** (`settings.config`) that is written with every key listed, reloaded as
soon as it is saved, and merged rather than overwritten when the app saves to it.
- **Themes**: built-ins, 36 shipped `.theme` files, and a theme editor that saves new ones.
- **Logging** through a multi-producer, single-consumer queue to a per-version log file and an
in-app Event Log, plus crash reports.
- **Debug panels**: frame times, memory, renderer, fonts (Ctrl+Shift+D, Ctrl+Shift+1).
- **DPI awareness**, live resize that keeps drawing, CJK font fallback from the system fonts.
- **Background shutdown**: closing hands the remaining work to a thread while the window shows
progress, so it never looks hung.
- **Asynchronous tests**, compiled into debug builds and run with `run_tests`.
## Building
Requires the Jai compiler, beta 0.2.030 or later.
```
jai first.jai # debug build
jai first.jai - release # release build
jai first.jai - full_release # debug and release
jai first.jai - release compile_tests # release build with the test suite
```
Binaries are written to `bin/`, with `assets/` copied beside them.
| Platform | Output | Notes |
| --- | --- | --- |
| Windows | `bin/mcap-debug.exe`, `bin/mcap.exe` | The release build gets the icon and a manifest, and no console. |
| macOS | `bin/mcap-debug.app`, `bin/mcap.app` (and the bare `bin/mcap-macos-debug`, `bin/mcap-macos`) | Apple silicon (arm64) only, with a native `jai`; macOS 14 or later. Each `.app` holds onnxruntime and the models, and is signed with the self-signed "Ex4 Dev" identity when the keychain has it (made by Ex4's `scripts/macos_dev_identity.sh`), so macOS keeps its permissions across builds; otherwise ad hoc. |
| Linux | `bin/mcap-linux-debug`, `bin/mcap-linux` | Needs SDL3 installed (`libSDL3.so`). |
`jai check_targets.jai - linux` type-checks the Linux build from any OS without linking it (on the
host's CPU: from a Mac, Linux on arm64).
The macOS check only works on a Mac, since the Objective-C module runs code at compile time.
Run any build with `run_tests` to run the test suites (debug builds include them),
`-allow-multiple-instances` to skip the single-instance check, or `-version`.
For unattended runs (scripts, CI, agents), add `-headless`: the app exits once the tests finish,
with exit code 0 if every suite passed and 1 otherwise, and a failed assertion or runtime check
is written to stderr, the log and the crash report and exits with 1 instead of opening a message
box (`runtime_support_headless` in `modules/Runtime_Support.jai`):
```
bin/mcap-debug.exe run_tests -headless -allow-multiple-instances > test-output.txt
```
`jai first.jai - no_crash_dialog [...]` makes builds for automation: `NO_CRASH_DIALOG` makes them
headless whatever their arguments, and everything they build goes to `bin-headless/` instead of
`bin/`, so an automated build never writes over the one you run by hand.
## Starting a new app
```
jai fork.jai - <output_directory> <app_name> [-folder <name>] [-bundle-id <id>] [-version <x.y.z>] [-git] [-build]
jai fork.jai - ../my-cool-app "My Cool App" -git -build
```
This copies the template (without `bin/`, `.build/`, `.git/` or `fork.jai`) and sets the names
below in the copy. The folder name defaults to the app name in lowercase with spaces as `-`
(`my-cool-app`), the bundle identifier to `com.example.<folder without dashes>`, and the version
to `0.0.0`. `-git` initializes a repository and commits the copy; `-build` makes a debug build. It also
writes `<app_name>.focus-config`, a [focus](https://focus-editor.dev) project with the build and
run commands, which stays out of git since it holds this machine's paths.
## Making it your app
1. Name it at the top of `first.jai`, or let `fork.jai` do it: `APP_NAME` (window title, tray,
macOS bundle), `APP_FOLDER_NAME` (executable and data folder; one short lowercase word),
`APP_BUNDLE_IDENTIFIER` and `VERSION`.
2. Replace `assets/images/icon.png` (any square size; it becomes the `.exe`, `.app` and window
icon) and the tray icons `tray_icon.ico` (Windows) and `tray_icon.png`.
3. Write the app in `src/app/app_ui.jai`: `app_ui_startup` runs once, `app_ui_frame_start` every
frame before input, `app_main_window_ui` draws, and `app_ui_shutdown` runs on the shutdown
thread.
To add a hotkey, declare a `Hotkey` in `src/ui/imgui_keyboard.jai`, add it to
`hotkey_actions_build` and `app_check_key_combinations`, bind it in both
`assets/data/default_hotkeys*.config` files (the build fails if a default leaves an action
unbound), and raise `APP_USER_SETTINGS_VERSION` in `src/App.jai` so existing settings files get
the new default.
To add a setting, add it to `App_User_Settings`, `Settings_Seen`, `settings_parse` and
`app_save_user_settings_file` in `src/ui/imgui_theme_registry.jai`.
## Layout
```
first.jai build: workspaces, build constants, Windows icon/manifest, macOS .app
fork.jai copy the template to a new, named project
check_targets.jai type-check another OS's build
src/App.jai loads everything; global state (app, platform, perf)
src/app/ main loop, your UI, background shutdown
src/base/ logging, fonts, debug panels, platform layers (base/platform/)
src/ui/ ImGui setup, hotkeys, themes and settings, toasts
src/graphics/DX11.jai Direct3D 11 device, WARP-first startup, device-loss recovery
src/backends/ ImGui's SDL_GPU backend; src/freetype/: ImGui's FreeType loader
assets/ fonts, icons, default hotkeys, default layout, themes
modules/ Basic (with arenas), Thread, Shared_Memory_Channel, File_Watcher,
Secure_Memory, XDG, sdl3, imgui-sdl, imgui-dx11-win32, imgui-source
```
## Files at run time
| File | Location |
| --- | --- |
| `settings.config`, `imgui.ini`, user themes | `%APPDATA%\<APP_FOLDER_NAME>` on Windows, `~/Library/Application Support/<APP_FOLDER_NAME>` on macOS, `~/.local/share/<APP_FOLDER_NAME>` on Linux |
| Logs | `logs/<version>/` next to the executable (next to the `.app` on macOS) |
| Shipped themes | `data/themes/` next to the executable |
## Platform notes
- Global hotkeys are not implemented on Linux (Wayland has no API for them). A desktop
shortcut that launches the executable has the same effect, since a second launch brings the
running instance forward.
- On XFCE the tray icon stays after the window is restored.
- `Secure_Memory` (used by `Basic`'s arenas to wipe memory) is built without libsodium; see the
note at the top of `modules/Secure_Memory/module.jai` if your app keeps secrets in memory.
Uses https://github.com/overlord-systems/jai-imgui as a template.
Reference in New Issue View Git Blame Copy Permalink