MusaMahmood/mcap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

[0.0.7] README: how to use mcap, and the search bar

Browse Source
This commit is contained in:
Musa Mahmood 2026-10-08 13:21:14 +01:00
parent bedeaebe41
commit c333d95da7
4 changed files with 222 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

271
README.md
View File

@ -1,8 +1,224 @@
# mcap
A bare [Dear ImGui](https://github.com/ocornut/imgui) (docking branch) desktop application
written in [Jai](https://jai.community), for Windows, macOS and Linux: the window, the frame loop
and everything around them, ready for an app to be written into `src/app/app_ui.jai`.
A browser for a library of screen captures: every `.mcap` file in a folder as a grid of thumbnails,
searchable by the text in the images (OCR), by what they show (labels), by their window title, URL,
app and annotations, and by meaning ("a circuit board"). Written in [Jai](https://jai.community)
with [Dear ImGui](https://github.com/ocornut/imgui), for Windows (macOS and Linux build, but OCR
and labels are tested on Windows with an NVIDIA GPU).
The `.mcap` files come from Snagit's `.snagx` captures, converted by `snagit-converter`
(`D:\SnagitIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format.
## Getting started
1. **Fetch what is not in git** (each download's SHA-256 is checked):
```
cd modules/onnxruntime && jai fetch.jai # onnxruntime 1.30.0, built for CUDA 13
cd modules/OCR && jai fetch_models.jai # the OCR models (~196 MB; `- mobile` for the light one only)
cd modules/Labels && jai fetch_models.jai # the label model, SigLIP 2 (~690 MB)
```
2. **For the GPU** (optional; without it everything runs on the CPU, several times slower): an
NVIDIA driver, the CUDA Toolkit 13.x, and cuDNN 9.x for CUDA 13. cuDNN installed in its usual
place (`C:\Program Files\NVIDIA\CUDNN`) is found without changing PATH. The OCR window says
what, if anything, is missing.
3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime
and the models beside the program.
4. **The library** is `D:\SnagitIndex\mcap`; `-library FOLDER` shows another folder.
5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**,
and press **Scan the library**. Every capture is read and labelled once and the results are
stored in its file, so this is needed only for new captures afterwards. On an RTX 5080 laptop,
8,600 captures took about 13 minutes with the mobile OCR model; labels add about 18 ms a capture,
and the server OCR model is several times slower.
## The main window
```
[ search box .......................... ] 8609 captures Size [---o---] OCR Stats Labels Refresh
[ thin progress bar while a scan runs ]
+-------+ +-------+ +-------+ +-------+
| thumb | | thumb | | thumb | | thumb | newest first
+-------+ +-------+ +-------+ +-------+
```
- **The grid** shows every capture, newest first by capture date (by name while the catalog,
what search needs from every file, is still loading: a line beside the count says so). **Size**
sets the thumbnails' size. Click a thumbnail to open it in the Capture window.
- **Delete:** hover a thumbnail and click the trash button in its corner. You are asked whether to
move the file to the Recycle Bin or to delete it permanently, overwriting it with random bytes
first; "Don't ask again" keeps the choice until the app restarts. Only the `.mcap` goes: the
`.snagx` it was converted from is untouched.
- **Refresh** looks at the folder again, after files were added or deleted outside the app.
- **OCR**, **Stats** and **Labels** open their windows (below).
- A **thin bar** under the search box shows a running scan; hover it for the details.
### The Capture window
One capture at full size, with **Left** and **Right** stepping through the captures shown in the
grid (in its order, so through a search's matches).
- **Fit:** Window, Width, Height, or Off with the **Zoom** slider (and **100%**).
- **OCR** reads this capture now, and labels it when the label model is up (**Read again** when it
has been read); **Boxes** shows the text
lines found, outlined over the image (turned with the text for turned lines). Hover a box for its
text and confidence. A capture read by several OCR models has a menu to choose whose lines show.
- While searching, the lines that match are highlighted, with **Matches: N** and **Previous** /
**Next** to scroll to each.
- **Labels:** the capture's five best labels with their scores (dim when they come from an older
label list).
### The OCR window
- **Model:** PP-OCRv5 mobile (light, about 85 ms a capture on the GPU) or server (more accurate,
about 0.4 s). The choice is kept in the settings file (`model` under `[ocr]`).
- **Running on:** the GPU or the CPU, and if the CPU, what to install for the GPU.
- **Scan the library** reads every capture the model has not read yet and, with **Labels too**,
labels every capture not yet labelled with the current list. While a search is active the button
reads **Scan the captures shown** and covers only those. Each capture is stored as soon as it is
read, so **Pause**, **Cancel** and quitting lose nothing; the next scan picks up where it stopped.
Progress shows captures done, the rate, time elapsed and time left.
- **Erase OCR readings from the library** (for testing) takes every OCR reading out of every file,
after asking. A scan reads them again.
- Below, the text of the capture open in the Capture window, line by line.
### The Labels window
- The label model's status, and how many captures are labelled, and with the current list.
- **The label list:** `labels.config` in the app's data folder (`%APPDATA%\mcap`), with **Open** to
edit it. Each line is a name and a prompt, the phrase the model compares with the image:
```
label: "Schematic" "an electronic circuit schematic diagram"
label: "Code editor" "a screenshot of a code editor showing source code"
```
A fuller description scores far better than a single word. Saved changes are picked up while the
app runs; the next scan relabels every capture from the image vector stored in its file, without
reading the images again, so changing the list is cheap.
- **Erase labels and vectors from the library** (for testing), after asking.
### The Stats window
What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR
model: captures with text, lines, words, characters, confidence, words per capture with a breakdown,
a table by year, the captures with the most words (click to open), each capture's best label
counted (click one to search for it), and the commonest words (click one to search for it).
## Searching
Type in the search box above the grid; the grid shows the matches as you type, with "N of M
captures" beside the box. Clear the box to see everything again.
### What a search matches
- **Every word must match**, in any order: `kicad footprint` finds captures with both.
- **Matching ignores case and finds parts of words:** `foot` matches "Footprints".
- **A plain word is looked for everywhere:** the OCR text (every model's), the window title, the
URL, the app's name, the annotations' text, and the labels.
- **"Quoted phrases"** match as one, spaces and all: `"symbol : footprint"`.
There is no OR and no NOT: to widen a search, take a word out.
### Prefixes
A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:"pcb editor"`.
| Prefix | Looks in |
| --- | --- |
| *(none)* | everything below except `like:` |
| `ocr:` | the text read from the image |
| `title:` | the window title when it was captured |
| `url:` | the web page's address, for browser captures |
| `app:` | the app's name and version |
| `note:` | the text of the capture's annotations (Snagit text boxes and callouts) |
| `label:` | the capture's labels: its best label, and any scoring at least half as much |
| `like:` | the image's meaning (below) |
A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work.
### Searching by meaning: `like:`
`like:"a circuit board under a microscope"` ranks every labelled capture by how close its image is
to the phrase, best first: no words need to appear in the image. The label model compares the phrase
with the image vector each capture's file holds, so only labelled captures take part. The grid is
then in that order instead of by date.
- Describe what the image shows: `like:"a graph with several colored lines"` works better than
`like:graph`.
- Combine it with other words to filter the ranking: `like:"a dark themed code editor" title:visual`.
- The phrase is quoted when it has spaces: `like:"a terminal with an error"`.
### Examples
| Search | Finds |
| --- | --- |
| `digikey` | captures mentioning DigiKey anywhere: text, URL, title |
| `url:digikey` | DigiKey pages |
| `app:kicad` | captures of KiCad, by the app's name (more than `title:kicad`, as a window title need not say KiCad) |
| `ocr:"0402"` | images with "0402" in their text |
| `title:kicad footprint` | KiCad windows whose text, notes or labels mention a footprint |
| `label:schematic` | captures labelled Schematic |
| `label:"code review"` | captures labelled Code review |
| `note:todo` | captures with an annotation saying TODO |
| `like:"a photo of a soldering iron"` | captures that look like that, best first |
| `like:"a spreadsheet" app:excel` | Excel captures, the most spreadsheet-like first |
## Command line
| Option | What it does |
| --- | --- |
| `-library FOLDER` | Shows the `.mcap` files in FOLDER instead of `D:\SnagitIndex\mcap`. |
| `-search QUERY` | Starts with QUERY in the search box. |
| `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. |
| `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. |
| `-erase ocr`, `-erase labels`, `-erase all` | Erases that from every file in the library, then exits (for testing). |
| `-allow-multiple-instances` | Runs beside another copy instead of bringing that one forward. |
| `run_tests` | Runs the test suites (debug builds); with `-headless`, exits with 0 if all passed. |
| `-headless` | No message boxes: a failure is logged and the app exits with 1. |
| `-version` | Prints the version. |
`-scan`, `-stats` and `-erase` still open the window, to show their progress; they are meant for
scripts and for trying things on a copy of the library: `bin\mcap.exe -library D:\copy -scan`.
## Where things are kept
| What | Where |
| --- | --- |
| OCR readings, labels, image vectors | inside each `.mcap` file (see `modules/Capture_File/README.md`) |
| The label list | `%APPDATA%\mcap\labels.config` |
| Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` |
| Logs | `logs\<version>\` beside the program |
Nothing is written to `D:\Snagit`: the `.mcap` library is a converted copy.
## How it works
- `src/app/viewer.jai`: the grid and the Capture window. Previews and full images are decoded on
worker threads and uploaded a few per frame; textures far from view are released past 1200.
Each capture keeps its index for the app's life (Refresh and Delete mark captures removed and
append new ones), as the workers, the scan and Stats refer to captures by index.
- `src/app/catalog.jai`: what search needs from every file, read from the front of each `.mcap`
(its text, OCR, labels and vector, not its images) on the workers at startup; the query parser
and matching.
- `src/app/background.jai`: searching and the Stats numbers run on a pool of threads, so typing
never waits; the grid keeps the last results until the new ones are back, usually within a frame.
`like:` phrases are turned into vectors there first.
- `src/app/ocr_ui.jai` and `labels_ui.jai`: the engines, made on one background thread (OCR first,
then labels); the scan, which decodes each image once and stores its OCR, labels and vector in one
rewrite of its file; the erase; the windows.
- `modules/OCR`, `modules/Labels`: the models, run with onnxruntime (`modules/onnxruntime`, which
also holds the shared session and CUDA code); each has a README.
- `modules/Capture_File`: reading and writing `.mcap` files.
# The application template
mcap is built on a bare Dear ImGui (docking branch) application template, for Windows, macOS and
Linux: the window, the frame loop and everything around them, with the app written into
`src/app/app_ui.jai`.
## What you get
@ -29,55 +245,6 @@ and everything around them, ready for an app to be written into `src/app/app_ui.
progress, so it never looks hung.
- **Asynchronous tests**, compiled into debug builds and run with `run_tests`.
## The capture viewer
`src/app/viewer.jai`, `catalog.jai`, `ocr_ui.jai` and `labels_ui.jai`: every `.mcap` in
`D:\SnagitIndex\mcap` as a grid of previews, and one capture at full size in the Capture window.
- **Search** above the grid: every word must match, anywhere in the OCR text, window title, URL, app
name, annotation text or labels; `"a phrase"` matches as one; `ocr:`, `title:`, `url:`, `app:`,
`note:` and `label:` limit a word to one field. `like:"a circuit board"` searches by meaning: the
captures whose image is nearest the phrase, best first (the label model's text encoder makes the
phrase's vector; each labelled capture's image vector is in its file). In the Capture window the matching OCR lines are highlighted, with Next
and Previous. Matching runs on a pool of background threads (`app/background.jai`), as do the
Stats window's numbers, so typing never waits on them; the grid shows the last results until the
new ones are back, usually within a frame.
- **Order:** newest first by capture date, once the catalog (what search needs from every file) has
loaded; by name until then.
- **OCR** (`modules/OCR`): the OCR window shows whether it runs on the GPU and, if not, what to install,
and switches model (kept as `model` under `[ocr]` in the settings file). **Scan the library** reads
every capture shown that the model has not read yet, storing each reading in its file as it goes,
so it can pause, cancel and pick up again; with **Labels too** it labels them in the same pass.
For testing, the OCR window's **Erase OCR readings** and the Labels window's **Erase labels and
vectors** take them out of every file in the library again (after asking); `-erase ocr|labels|all`
does the same without the UI and exits. The
Capture window's **OCR** button reads one capture; a capture read by several models keeps each
reading, and the window can show any of them. `-library FOLDER` shows another folder of `.mcap` files;
`-scan` scans it with the chosen model once it has loaded and exits (0, or 1 if any capture
failed), for a scan without the UI, such as on a copy.
- **Delete:** hovering a thumbnail shows a trash button in its corner. The confirmation offers moving
the file to the Recycle Bin (on Linux and macOS, to a `deleted` folder in the library folder) or
deleting it permanently: overwritten with random bytes, flushed to the disk, renamed and deleted.
Copies elsewhere (the source `.snagx`, backups, synced versions) are untouched, and an SSD may keep
old blocks. The choice, and "don't ask again", last until the app restarts. Not while OCR is running.
- **Refresh** above the grid looks at the folder again: captures whose file was deleted leave the
grid, and new files are added (each capture keeps its index, as the workers, the OCR scan and Stats
refer to captures by index). By hand for now; File_Watcher can call `viewer_refresh` later.
- **Stats** above the grid: what OCR found in the captures shown, by the chosen model: captures with
text, lines, words, characters, confidence, words per capture, a year table, the captures with
the most words (click to open) and the commonest words (click to search). `-stats` logs the same
numbers once the catalog has loaded, and exits. `-search QUERY` starts with QUERY in the search box
(with `-stats`, the numbers are for its matches, and the first ten are logged). Stats also counts
each capture's best label (click one to search for it).
- **Labels** (`modules/Labels`, SigLIP 2 base at 384 px): each capture is scored against a label list,
`labels.config` in the app's data folder (`%APPDATA%\mcap`): a name and a prompt per label, a fuller
description scoring better than a word. A starter list is written the first time; edits are
picked up while the app runs. A scan stores each capture's five best labels and its image vector in
its file (`labl` and `embd` chunks, see `modules/Capture_File/README.md`), so a changed list is
scored again from the stored vectors without reading the images. The Labels window shows the list
and how much of the library is labelled; the Capture window shows the capture's labels. `label:`
matches a capture's best label, and any scoring at least half as much.
## Building
Requires the Jai compiler, beta 0.2.030 or later.

2
check_targets.jai
View File

@ -29,7 +29,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
GIT_BRANCH :: "check";
GIT_HASH :: "-";
VERSION :: "0.0.6";
VERSION :: "0.0.7";
BUILD_DATE :: "-";
DEBUG :: true;
COMPILE_TESTS :: true;

2
first.jai
View File

@ -8,7 +8,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap";
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
VERSION :: "0.0.6";
VERSION :: "0.0.7";
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
// builds never write over the ones you run by hand.
build_directory := "bin";

2
src/app/viewer.jai
View File

@ -752,7 +752,7 @@ viewer_ui :: () { using ImGui;
}
SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520));
if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes. \"a phrase\" ocr: title: url: app: note:", viewer.search_buffer.data, viewer.search_buffer.count) {
if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like:", viewer.search_buffer.data, viewer.search_buffer.count) {
viewer_search_changed();
}
SameLine();