- video-capture-plan.html: recording a region of one screen to H.264 MP4 and into the library as a
video .mcap. DXGI Desktop Duplication first, on a device of its own on the screen's adapter;
Windows.Graphics.Capture when duplication cannot run (two GPUs, rotated screens, too many
duplications). Encoders tried in turn: NVENC, AMF, QSV, Media Foundation, libx264, from the
FFmpeg already shipped. Ctrl+PrintScreen opens the overlay in a video mode (a 240 x 180 minimum,
kept to one screen) and stops the recording. Five phases, the first a command-line probe that
answers the two-GPU question on the laptop.
- capture_region, a new [global_hotkeys] action, on PrintScreen by default (Windows only; none on
macOS). Existing settings files get the line appended on the next launch. When the key cannot be
registered, the log names Snagit and Windows' "Print screen opens screen capture" setting.
- app/screen_capture_win32.jai: every screen copied the moment the key fires (BitBlt with
CAPTUREBLT over the virtual desktop), shown darkened in a topmost GDI window in a modal loop of its
own, so it works with mcap hidden in the tray. Dragging shows the region bright and outlined; a
magnifier shows the pixels under the cursor with its position or the selection's size; arrow keys
move a pixel (ten with Shift). A click or Enter takes the whole screen under the cursor; Escape,
a right click or losing the focus cancels. Only the parts that change are redrawn.
- app/screen_capture.jai: the region goes on the clipboard (CF_DIB) at once, then a thread writes it
into the library's "Screen Captures" folder as a PNG .mcap: metadata.json with the window under
the region's middle (its title as WindowName, its program's description as AppName) and
capture.json (where on the desktop, the program's path, the hash). mcap then comes to the front
showing it, and OCR, labels and segmentation read it as they read an import.
- A "screen capture" group in the import tests: the file written, its pixels, preview and Capture
Text read back. README and the format's README describe it.
- jai release.jai builds the release build, then stages release/mcap-VERSION-windows-x64/ with
mcap.exe and its symbols, onnxruntime (CUDA provider), FFmpeg, the assets, the OCR, label and
segmentation models (each checked against its models.config), NVIDIA's CUDA 13 runtime, all of
cuDNN 9 and NVRTC, a README and third-party notices; zips it with Windows' tar; prints its
SHA-256. - no_cuda leaves the CUDA runtime out; - no_build packages bin/ as it is.
- cuDNN is copied whole: it loads sub-libraries by name (9.26's cudnn_engines_tensor_ir64_9.dll),
and one missing fails every run on the GPU.
- Checked by running the package with no CUDA or cuDNN on PATH: OCR, labels and segmentation on
CUDA, video import through its FFmpeg.
- release/ is ignored.
- The Capture window draws a segmented image's segments over it: each class in its colour, a shade
of it for each instance, edges darker; under the OCR boxes, at any fit and zoom. A viewer worker
reads the segs and mask chunks (never the images); the main thread makes the texture from the
mask when the capture, the opacity or the classes hidden change.
- Above the image: Segments on or off, the opacity, and a legend of the classes (largest first,
with counts): untick one to hide it, right click it to search has: for it. Hovering the image
shows the segment's class, score and share of the image.
- Colours: a class's own is a hue from its name's hash. Its legend swatch is a colour picker
(ImGui ColorEdit3) to choose another, kept in %APPDATA%\mcap\segment-colours.config for every
image and library; right click it to go back to its own.
- Tests: the overlay's pixels (shades, edges, hidden classes, opacity), reading a file's segments
for it, and chosen colours.
- This completes segmentation-plan.html's six phases.
- has: searches what segmentation found, by the whole class name: has:dog, has:"dining table"
(has:table is a table, not a dining table). Plain words do not match segments.
- The catalog keeps each capture's segments (class, score, area) and its classes between newlines,
so a has: term is one contains per capture.
- Stats: "What is in the images", how many of the segmented images hold each class, each one a
has: search; shown even when nothing has been read by OCR. -stats logs it.
- Tests: has: (whole names, quoted names, case, combinations, plain words not matching).
- .library-folders in the library (src/app/library_folders.jai): each library folder's options,
starting with segment: true. Every folder starts off; unknown keys are kept when it is rewritten.
- The Segmentation window (Segments, above the grid): each folder's Segment checkbox, how many of
its images are segmented, and Erase per folder; a button to segment the folders set to segment.
- The segmentation scan (src/app/segment_ui.jai) runs on the OCR thread as its own task, with the
OCR scan's progress, pause and cancel: images only (never videos), not yet segmented, in folders
set to segment. The engine loads with the first scan (2.4 GB of GPU memory). Each image's segs and
mask are stored in one rewrite and its catalog entry loaded again.
- Importing into a folder set to segment segments the new images; the Import window's checkbox is
the folder's setting.
- -segment segments and exits; -erase segments, and -erase all now includes segments.
- The catalog reads each capture's segs (class list hash, segment count). The exit overlay names the
segmentation scan and model. segment_mcap_chunks moved into modules/Segment, shared with
segment-probe. The quoting helpers of Capture_File's text formats are public.
- Tests: .library-folders read and written back (quoted names, unknown keys, versions).
- modules/Capture_File/segment_chunks.jai: a front "segs" chunk (text: engine, date, the image read
and its size, the class list's hash, and each segment's id, score, area, box and class) and a back
"mask" chunk (a segment id per pixel, run-length encoded with LEB128 lengths, longest side at most
1024), keyed by model. No new file version: chunks came in version 3.
- mcap_store takes back chunks too, so a model's summary and mask are stored in one rewrite.
- Tests: the text round trip, the runs' edge cases (one segment over everything, none, runs across
rows, multi-byte lengths, id 255), bad masks refused, and storing, replacing and erasing both
chunks in a file with other chunks kept.
- segment-probe takes .mcap files (the image each is shown as; videos skipped) and with -store stores
its segments in them. On 27 images a mask is 0.8 to 6.4 KB.
- The format README's "Segmentation chunks" section; the plan's phase 3 results.
- segmentation-plan.html: the plan for panoptic segmentation of images, per library folder, stored
in .mcap chunks (no file version bump), in six phases.
- tools/export_mask2former.py: exports facebook/mask2former-swin-base-coco-panoptic to ONNX (opset 17,
dynamic batch, 384 x 384 input) and checks it against PyTorch: on 25 images, all 180 segments
identical (IoU 1.000). Writes the class list with names for people ("sky-other-merged" is "sky"),
and PyTorch's own segmentations for segment-probe -check.
- modules/Segment: runs the model with onnxruntime (CUDA, else the CPU) and makes segments of its
outputs as transformers' post_process_panoptic_segmentation does: a segment id map (longest side at
most 1024) and each segment's class, score, area and bounds. Matches PyTorch on 99.8% of pixels,
every segment scored 0.6 or more found. 57 ms an image at batch 4 on an RTX 5080 laptop.
- tools/segment_probe.jai (jai first.jai - segment_probe): segments image files; -out draws them,
-check compares with PyTorch, -dump-input writes the model's input.
- modules/onnxruntime: ort_run_float_outputs, for a model with several outputs.
- first.jai copies the model to bin/segment-models. The model itself is not in git
(modules/Segment/models, 462 MB); tools/export_mask2former.py makes it.
- The exit overlay says which step of shutdown is running and what it waits on ("Stopping the OCR
scan: finishing the text lines it is reading.", the import, the viewer, the models), and how long
that step has taken once it passes 2 s. It said "Saving..." for all of it.
- OCR stops mid-batch: ocr_process takes a cancel flag (Ocr_Engine.cancel), checked by its workers
before each piece of work; a scan sets it to its own cancel. A cut-short batch stores nothing and
counts no failures. Closing during a scan took minutes when a batch held large images; now well
under a second.
- Once shutdown starts, the main thread stops running the app's frames (the viewer, search, import,
OCR, Stats) and draws only the overlay: background_frame read the search thread group while the
shutdown thread was tearing it down, an access violation on exit.
- Import brings in videos (MP4, MOV, MKV, WebM, AVI, WMV, MPEG, MTS/M2TS, FLV) on Windows, read with
FFmpeg: the file as it was in the original, a frame a tenth of the way in (at most 3 s), turned
the right way up, as the rendered image (what the grid, Capture window, OCR and labels see), and
import.json with Duration, FrameRate, VideoCodec, Rotation, PosterTime, Audio, and the camera,
date and location from the container. Elsewhere videos are skipped as "video (Windows only)".
- .mcap version 4: the original's size is a u64, so a video can be any size. Versions 1-3 still read.
mcap_read_without_original and mcap_write_streamed read and rewrite a file with its original left
on disk; mcap_store, mcap_erase, the import index and the import itself never hold a video in memory.
- Animated GIFs: Frames and Duration in import.json (gif_info).
- Grid: a play mark and duration on video thumbnails, GIF on animations. Capture window: what the
video is, and Play in the default player (written out to %TEMP%\mcap).
- modules/ffmpeg: FFmpeg 6.0 bindings and the five DLLs used (avformat, avcodec, avutil, swscale,
swresample), copied beside the executable by the build.
swscale converts into a buffer of its own allocator with padded rows: an exact-sized one let
its SIMD write past the end, crashing on frames of odd widths (1916, 1106 ...).
- image-probe shows videos and GIF frame counts; ocr-bench reads a video's frame.
- Tests: new video formats, display-matrix rotation, ISO 6709, video dates, GIF frames, and v4
serialize/parse, v3 compatibility, streamed writes, store/erase and extraction.
SnagitConverter's copy of Capture_File needs syncing to read version 4 files.
- Import window (and -import SOURCE DEST): every image under a folder (PNG, JPEG, HEIC, AVIF, WebP,
GIF, BMP, TIFF, camera RAW) becomes an .mcap holding the file as it was, a preview turned by its
EXIF orientation, and metadata.json and import.json Capture Text (date from EXIF, the file name or
its time; file name as title; camera as app). .snagx files become the same .mcap SnagitConverter
makes. Duplicates across the whole library are skipped by a hash index (.import-index); videos
and sidecars are skipped; source files are only read.
- modules/Image_Decode: stb_image, then WIC for everything else; EXIF; orientation; previews.
tools/image_probe.jai shows what it reads.
- modules/Snagx (and modules/jaison), copied from SnagitConverter.
- The library is a folder of folders: recursive scan, a folder filter, and a folder: search prefix.
- OCR, the viewer and ocr-bench decode any image format, turned the right way up.
- Rename D:\SnagitIndex to D:\McapIndex.
