diff --git a/README.md b/README.md index 1265f1d..73a1690 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,224 @@ # mcap -A bare [Dear ImGui](https://github.com/ocornut/imgui) (docking branch) desktop application -written in [Jai](https://jai.community), for Windows, macOS and Linux: the window, the frame loop -and everything around them, ready for an app to be written into `src/app/app_ui.jai`. +A browser for a library of screen captures: every `.mcap` file in a folder as a grid of thumbnails, +searchable by the text in the images (OCR), by what they show (labels), by their window title, URL, +app and annotations, and by meaning ("a circuit board"). Written in [Jai](https://jai.community) +with [Dear ImGui](https://github.com/ocornut/imgui), for Windows (macOS and Linux build, but OCR +and labels are tested on Windows with an NVIDIA GPU). + +The `.mcap` files come from Snagit's `.snagx` captures, converted by `snagit-converter` +(`D:\SnagitIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. + +## Getting started + +1. **Fetch what is not in git** (each download's SHA-256 is checked): + + ``` + cd modules/onnxruntime && jai fetch.jai # onnxruntime 1.30.0, built for CUDA 13 + cd modules/OCR && jai fetch_models.jai # the OCR models (~196 MB; `- mobile` for the light one only) + cd modules/Labels && jai fetch_models.jai # the label model, SigLIP 2 (~690 MB) + ``` + +2. **For the GPU** (optional; without it everything runs on the CPU, several times slower): an + NVIDIA driver, the CUDA Toolkit 13.x, and cuDNN 9.x for CUDA 13. cuDNN installed in its usual + place (`C:\Program Files\NVIDIA\CUDNN`) is found without changing PATH. The OCR window says + what, if anything, is missing. + +3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime + and the models beside the program. + +4. **The library** is `D:\SnagitIndex\mcap`; `-library FOLDER` shows another folder. + +5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**, + and press **Scan the library**. Every capture is read and labelled once and the results are + stored in its file, so this is needed only for new captures afterwards. On an RTX 5080 laptop, + 8,600 captures took about 13 minutes with the mobile OCR model; labels add about 18 ms a capture, + and the server OCR model is several times slower. + +## The main window + +``` +[ search box .......................... ] 8609 captures Size [---o---] OCR Stats Labels Refresh +[ thin progress bar while a scan runs ] ++-------+ +-------+ +-------+ +-------+ +| thumb | | thumb | | thumb | | thumb | newest first ++-------+ +-------+ +-------+ +-------+ +``` + +- **The grid** shows every capture, newest first by capture date (by name while the catalog, + what search needs from every file, is still loading: a line beside the count says so). **Size** + sets the thumbnails' size. Click a thumbnail to open it in the Capture window. +- **Delete:** hover a thumbnail and click the trash button in its corner. You are asked whether to + move the file to the Recycle Bin or to delete it permanently, overwriting it with random bytes + first; "Don't ask again" keeps the choice until the app restarts. Only the `.mcap` goes: the + `.snagx` it was converted from is untouched. +- **Refresh** looks at the folder again, after files were added or deleted outside the app. +- **OCR**, **Stats** and **Labels** open their windows (below). +- A **thin bar** under the search box shows a running scan; hover it for the details. + +### The Capture window + +One capture at full size, with **Left** and **Right** stepping through the captures shown in the +grid (in its order, so through a search's matches). + +- **Fit:** Window, Width, Height, or Off with the **Zoom** slider (and **100%**). +- **OCR** reads this capture now, and labels it when the label model is up (**Read again** when it + has been read); **Boxes** shows the text + lines found, outlined over the image (turned with the text for turned lines). Hover a box for its + text and confidence. A capture read by several OCR models has a menu to choose whose lines show. +- While searching, the lines that match are highlighted, with **Matches: N** and **Previous** / + **Next** to scroll to each. +- **Labels:** the capture's five best labels with their scores (dim when they come from an older + label list). + +### The OCR window + +- **Model:** PP-OCRv5 mobile (light, about 85 ms a capture on the GPU) or server (more accurate, + about 0.4 s). The choice is kept in the settings file (`model` under `[ocr]`). +- **Running on:** the GPU or the CPU, and if the CPU, what to install for the GPU. +- **Scan the library** reads every capture the model has not read yet and, with **Labels too**, + labels every capture not yet labelled with the current list. While a search is active the button + reads **Scan the captures shown** and covers only those. Each capture is stored as soon as it is + read, so **Pause**, **Cancel** and quitting lose nothing; the next scan picks up where it stopped. + Progress shows captures done, the rate, time elapsed and time left. +- **Erase OCR readings from the library** (for testing) takes every OCR reading out of every file, + after asking. A scan reads them again. +- Below, the text of the capture open in the Capture window, line by line. + +### The Labels window + +- The label model's status, and how many captures are labelled, and with the current list. +- **The label list:** `labels.config` in the app's data folder (`%APPDATA%\mcap`), with **Open** to + edit it. Each line is a name and a prompt, the phrase the model compares with the image: + + ``` + label: "Schematic" "an electronic circuit schematic diagram" + label: "Code editor" "a screenshot of a code editor showing source code" + ``` + + A fuller description scores far better than a single word. Saved changes are picked up while the + app runs; the next scan relabels every capture from the image vector stored in its file, without + reading the images again, so changing the list is cheap. +- **Erase labels and vectors from the library** (for testing), after asking. + +### The Stats window + +What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR +model: captures with text, lines, words, characters, confidence, words per capture with a breakdown, +a table by year, the captures with the most words (click to open), each capture's best label +counted (click one to search for it), and the commonest words (click one to search for it). + +## Searching + +Type in the search box above the grid; the grid shows the matches as you type, with "N of M +captures" beside the box. Clear the box to see everything again. + +### What a search matches + +- **Every word must match**, in any order: `kicad footprint` finds captures with both. +- **Matching ignores case and finds parts of words:** `foot` matches "Footprints". +- **A plain word is looked for everywhere:** the OCR text (every model's), the window title, the + URL, the app's name, the annotations' text, and the labels. +- **"Quoted phrases"** match as one, spaces and all: `"symbol : footprint"`. + +There is no OR and no NOT: to widen a search, take a word out. + +### Prefixes + +A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:"pcb editor"`. + +| Prefix | Looks in | +| --- | --- | +| *(none)* | everything below except `like:` | +| `ocr:` | the text read from the image | +| `title:` | the window title when it was captured | +| `url:` | the web page's address, for browser captures | +| `app:` | the app's name and version | +| `note:` | the text of the capture's annotations (Snagit text boxes and callouts) | +| `label:` | the capture's labels: its best label, and any scoring at least half as much | +| `like:` | the image's meaning (below) | + +A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work. + +### Searching by meaning: `like:` + +`like:"a circuit board under a microscope"` ranks every labelled capture by how close its image is +to the phrase, best first: no words need to appear in the image. The label model compares the phrase +with the image vector each capture's file holds, so only labelled captures take part. The grid is +then in that order instead of by date. + +- Describe what the image shows: `like:"a graph with several colored lines"` works better than + `like:graph`. +- Combine it with other words to filter the ranking: `like:"a dark themed code editor" title:visual`. +- The phrase is quoted when it has spaces: `like:"a terminal with an error"`. + +### Examples + +| Search | Finds | +| --- | --- | +| `digikey` | captures mentioning DigiKey anywhere: text, URL, title | +| `url:digikey` | DigiKey pages | +| `app:kicad` | captures of KiCad, by the app's name (more than `title:kicad`, as a window title need not say KiCad) | +| `ocr:"0402"` | images with "0402" in their text | +| `title:kicad footprint` | KiCad windows whose text, notes or labels mention a footprint | +| `label:schematic` | captures labelled Schematic | +| `label:"code review"` | captures labelled Code review | +| `note:todo` | captures with an annotation saying TODO | +| `like:"a photo of a soldering iron"` | captures that look like that, best first | +| `like:"a spreadsheet" app:excel` | Excel captures, the most spreadsheet-like first | + +## Command line + +| Option | What it does | +| --- | --- | +| `-library FOLDER` | Shows the `.mcap` files in FOLDER instead of `D:\SnagitIndex\mcap`. | +| `-search QUERY` | Starts with QUERY in the search box. | +| `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. | +| `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. | +| `-erase ocr`, `-erase labels`, `-erase all` | Erases that from every file in the library, then exits (for testing). | +| `-allow-multiple-instances` | Runs beside another copy instead of bringing that one forward. | +| `run_tests` | Runs the test suites (debug builds); with `-headless`, exits with 0 if all passed. | +| `-headless` | No message boxes: a failure is logged and the app exits with 1. | +| `-version` | Prints the version. | + +`-scan`, `-stats` and `-erase` still open the window, to show their progress; they are meant for +scripts and for trying things on a copy of the library: `bin\mcap.exe -library D:\copy -scan`. + +## Where things are kept + +| What | Where | +| --- | --- | +| OCR readings, labels, image vectors | inside each `.mcap` file (see `modules/Capture_File/README.md`) | +| The label list | `%APPDATA%\mcap\labels.config` | +| Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` | +| Logs | `logs\\` beside the program | + +Nothing is written to `D:\Snagit`: the `.mcap` library is a converted copy. + +## How it works + +- `src/app/viewer.jai`: the grid and the Capture window. Previews and full images are decoded on + worker threads and uploaded a few per frame; textures far from view are released past 1200. + Each capture keeps its index for the app's life (Refresh and Delete mark captures removed and + append new ones), as the workers, the scan and Stats refer to captures by index. +- `src/app/catalog.jai`: what search needs from every file, read from the front of each `.mcap` + (its text, OCR, labels and vector, not its images) on the workers at startup; the query parser + and matching. +- `src/app/background.jai`: searching and the Stats numbers run on a pool of threads, so typing + never waits; the grid keeps the last results until the new ones are back, usually within a frame. + `like:` phrases are turned into vectors there first. +- `src/app/ocr_ui.jai` and `labels_ui.jai`: the engines, made on one background thread (OCR first, + then labels); the scan, which decodes each image once and stores its OCR, labels and vector in one + rewrite of its file; the erase; the windows. +- `modules/OCR`, `modules/Labels`: the models, run with onnxruntime (`modules/onnxruntime`, which + also holds the shared session and CUDA code); each has a README. +- `modules/Capture_File`: reading and writing `.mcap` files. + +# The application template + +mcap is built on a bare Dear ImGui (docking branch) application template, for Windows, macOS and +Linux: the window, the frame loop and everything around them, with the app written into +`src/app/app_ui.jai`. ## What you get @@ -29,55 +245,6 @@ and everything around them, ready for an app to be written into `src/app/app_ui. progress, so it never looks hung. - **Asynchronous tests**, compiled into debug builds and run with `run_tests`. -## The capture viewer - -`src/app/viewer.jai`, `catalog.jai`, `ocr_ui.jai` and `labels_ui.jai`: every `.mcap` in -`D:\SnagitIndex\mcap` as a grid of previews, and one capture at full size in the Capture window. - -- **Search** above the grid: every word must match, anywhere in the OCR text, window title, URL, app - name, annotation text or labels; `"a phrase"` matches as one; `ocr:`, `title:`, `url:`, `app:`, - `note:` and `label:` limit a word to one field. `like:"a circuit board"` searches by meaning: the - captures whose image is nearest the phrase, best first (the label model's text encoder makes the - phrase's vector; each labelled capture's image vector is in its file). In the Capture window the matching OCR lines are highlighted, with Next - and Previous. Matching runs on a pool of background threads (`app/background.jai`), as do the - Stats window's numbers, so typing never waits on them; the grid shows the last results until the - new ones are back, usually within a frame. -- **Order:** newest first by capture date, once the catalog (what search needs from every file) has - loaded; by name until then. -- **OCR** (`modules/OCR`): the OCR window shows whether it runs on the GPU and, if not, what to install, - and switches model (kept as `model` under `[ocr]` in the settings file). **Scan the library** reads - every capture shown that the model has not read yet, storing each reading in its file as it goes, - so it can pause, cancel and pick up again; with **Labels too** it labels them in the same pass. - For testing, the OCR window's **Erase OCR readings** and the Labels window's **Erase labels and - vectors** take them out of every file in the library again (after asking); `-erase ocr|labels|all` - does the same without the UI and exits. The - Capture window's **OCR** button reads one capture; a capture read by several models keeps each - reading, and the window can show any of them. `-library FOLDER` shows another folder of `.mcap` files; - `-scan` scans it with the chosen model once it has loaded and exits (0, or 1 if any capture - failed), for a scan without the UI, such as on a copy. -- **Delete:** hovering a thumbnail shows a trash button in its corner. The confirmation offers moving - the file to the Recycle Bin (on Linux and macOS, to a `deleted` folder in the library folder) or - deleting it permanently: overwritten with random bytes, flushed to the disk, renamed and deleted. - Copies elsewhere (the source `.snagx`, backups, synced versions) are untouched, and an SSD may keep - old blocks. The choice, and "don't ask again", last until the app restarts. Not while OCR is running. -- **Refresh** above the grid looks at the folder again: captures whose file was deleted leave the - grid, and new files are added (each capture keeps its index, as the workers, the OCR scan and Stats - refer to captures by index). By hand for now; File_Watcher can call `viewer_refresh` later. -- **Stats** above the grid: what OCR found in the captures shown, by the chosen model: captures with - text, lines, words, characters, confidence, words per capture, a year table, the captures with - the most words (click to open) and the commonest words (click to search). `-stats` logs the same - numbers once the catalog has loaded, and exits. `-search QUERY` starts with QUERY in the search box - (with `-stats`, the numbers are for its matches, and the first ten are logged). Stats also counts - each capture's best label (click one to search for it). -- **Labels** (`modules/Labels`, SigLIP 2 base at 384 px): each capture is scored against a label list, - `labels.config` in the app's data folder (`%APPDATA%\mcap`): a name and a prompt per label, a fuller - description scoring better than a word. A starter list is written the first time; edits are - picked up while the app runs. A scan stores each capture's five best labels and its image vector in - its file (`labl` and `embd` chunks, see `modules/Capture_File/README.md`), so a changed list is - scored again from the stored vectors without reading the images. The Labels window shows the list - and how much of the library is labelled; the Capture window shows the capture's labels. `label:` - matches a capture's best label, and any scoring at least half as much. - ## Building Requires the Jai compiler, beta 0.2.030 or later. diff --git a/check_targets.jai b/check_targets.jai index 8ec9445..73acd04 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.0.6"; +VERSION :: "0.0.7"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 703ab45..e5fb0fc 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.0.6"; +VERSION :: "0.0.7"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; diff --git a/src/app/viewer.jai b/src/app/viewer.jai index e87d515..16a090b 100644 --- a/src/app/viewer.jai +++ b/src/app/viewer.jai @@ -752,7 +752,7 @@ viewer_ui :: () { using ImGui; } SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520)); - if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes. \"a phrase\" ocr: title: url: app: note:", viewer.search_buffer.data, viewer.search_buffer.count) { + if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like:", viewer.search_buffer.data, viewer.search_buffer.count) { viewer_search_changed(); } SameLine();