bin/gpu-bench times every GPU model (OCR mobile and server, the label
image encoder, segmentation; optionally the CPU and the label text
encoder) over a fixed 30-image corpus in tools/gpu_bench/corpus. Every
corpus file is checked against corpus.txt's SHA-256 before anything
runs; each model's files and models.config entry are fingerprinted;
the GPU, driver, power limit, CUDA runtime, cuDNN and onnxruntime are
recorded, and NVML samples memory, load, power and clocks during the
timed passes. Results go to bin/gpu-bench-results as text files.
The GPU Bench button opens a window that runs gpu-bench in its own
process (no console) and lays results files from several machines side
by side against a baseline, marking a different corpus or model.
The workers ran gpu_workers (4) runs at once whatever the images: the server
model, four large images at a time, took 15 GB of a 16 GB GPU and spilled into
shared system memory, and a smaller GPU would fare far worse.
modules/OCR/budget.jai: each run reserves what it is expected to need, and
waits while the runs in flight or the GPU's free memory leave too little; one
run always goes. The need is measured when the engine is made, from the arena's
peak (AllocatorGetStats MaxInUse) over runs on two input sizes, so it holds for
any model and GPU: the mobile detector 6 MB + 137 MB a megapixel, the server's
4 MB + 1015, against 511 and 522 MB measured for a 2560 x 1440 tile. The limit
is the GPU's free memory less 10% (at least 512 MB), read at the start of each
ocr_process call (cudaMemGetInfo, ort_cuda_memory); the free memory is also read
as each run starts, since finished runs' blocks stay in the arena until the
stage ends.
gpu-bench, RTX 5080 laptop: the server model 335 ms a capture (854 before),
peak 13.1 GB (15.4), passes within 4%; the mobile model unchanged (140 ms,
5.7 GB), as its runs fit.
OCR filled the GPU: 9 GB for the mobile model on gpu-bench's 30 images, past
16 GB for the server model, which Windows then paged into shared memory (1.5 to
11 s a capture). onnxruntime's logged arena extensions showed why: the arena
keeps every block it takes from CUDA and never joins or frees them, and OCR's
inputs grow from run to run (recognizer batches went narrowest first; images in
any order across calls), so each larger run left another block behind: 5.7 GB
of recognizer blocks where the widest batch needs 0.9 GB.
- Detection runs the largest images first, recognition the widest batches.
- The detector's and recognizer's arenas shrink once each stage of an
ocr_process call is done, by a run on a small input with
memory.enable_memory_arena_shrinkage. Shrinking after every run cost 33 to
59% of the speed.
- The worker threads are made with the engine and kept, not made for every
stage of every call (onnxruntime keeps GPU state for each thread that runs a
session, microsoft/onnxruntime#33037); 6% faster.
- OCR's arenas grow by what is asked (kSameAsRequested).
- ort.log_level sets what the engines' onnxruntime environments log; INFO shows
every arena extension.
Mobile OCR on gpu-bench (RTX 5080 laptop, 4 workers): peak 5.7 GB over the
baseline, 2 GB between calls, 142 ms a capture, against 9.0 GB and 126 ms. The
text read differs by a line in 2,986 (batches are grouped differently). The
server model is not yet measured again.
bring_to_foreground in [global_hotkeys] was "none"; new settings files get Ctrl-Shift-M. A file
that already says "none" keeps it. macOS stays unbound.
hotkeys_missing_lines writes the shipped default of an action the file does not mention, but that
save records the hash of what it wrote, so the file was not reloaded and the action stayed unbound
until the file next changed. The bindings installed at startup and on reload now include those
defaults (hotkeys_with_missing_defaults), as in Ex4 1.47.2; a file of an older version still gets
"none". A logger test covers it.
The search box takes the keyboard on the next frame, its window brought to the front and its text
selected, so typing replaces the query. It is focus_search in [hotkeys] (Cmd-F on macOS), and it
acts while typing in another field, such as the Import window's. View mode has no search box.
mcap.exe FILE, with FILE first (as Explorer's Open with passes it), opens an .mcap, image, video or
.snagx (app/open_file.jai). With no mcap running, the app starts as usual: an .mcap in the library
opens in the Capture window, anything else in the Opened File window, which says where the library
has it already (by its hash in .import-index) or imports it into the library folder typed or chosen.
With mcap running, a second window opens in view mode: the file alone, with Import, and Open in mcap
/ Show in mcap, which hand it to the running copy (a new Open_File message) and close the view. The
library is not loaded or watched, no model loads, and it has no tray icon, no global hotkeys, and
writes no settings or layout.
Import takes a single file as its source too (and -import FILE DEST); an .mcap is copied in as it is,
while a folder's import still skips them. -open-import DEST imports the opened file as its Import
button does, then exits.
-h and --help open a Command Line window listing the options, also toggled from the Debug Panel; with
mcap running, that one comes forward with it (a Show_Help message).
Thumbnails: the "First preview shown" log line, put between creating a preview's texture and its
else, had every preview after the first marked failed. Once its texture was released (scrolled far
away, or the switch from WARP to the GPU at startup) it was never decoded again and showed "Cannot
read".
The README's "about 0.4 s" for the server model was measured on 2026-10-07 (e024b7f), and the
batching it ran with has not changed since. Measured now, the same build on the same 20 captures
runs at about 0.27 s a capture with 2.4 GB at peak, or about 1.5 s with 6.6 GB, with either
allocator; longer runs fare worse. Until that is fixed, the mobile model is the one to scan a large
library with.
The selected capture is the one outlined in the grid and shown in the Capture window. Its image is
copied as the Capture window shows it, decoded on a thread of its own and put on the clipboard as
screen captures are (Windows only so far); its path is copied as the system writes paths. Both are
actions in [hotkeys] (copy_image, copy_path), which do not act while typing in the search box.
Settings files at the current version get their default bindings at the next start; the comment
saying to raise APP_USER_SETTINGS_VERSION for a new action had it backwards.
The catalog is kept in .catalog-cache in the library, each entry with the size and modified time of
the file it was read from; a start takes the entries whose file is unchanged from it and reads only
the rest. It is saved once the catalog is complete and has changed (at most every 30 s) and at
shutdown. After a reboot the catalog was complete 145 ms after launch, against 3.5 s reading every
file.
The catalog loads on the background Thread_Group, every worker of it, instead of the four preview
workers. rpmalloc (modules/Default_Allocator, 2.0.1-jai.16) takes the place of Jai's single locked
heap: entries from the cache 380 -> 50 ms, and reading the files now scales with threads.
thread_idle is called where workers idle. check_targets.jai checks other OSes with Jai's own
allocator, as rpmalloc runs at compile time with the target's OS layer.
-cold drops the library from the system's file cache first; -no-catalog-cache and -catalog-threads N
are for measuring. [startup] log lines time the listing, the first frame, the cache, the first
preview and the complete catalog.
File_Watcher brought up to Ex4's (the pending index reset keeping its memory). Each changed .mcap is
looked at once settled: added when the grid lacks it, removed when its file is gone, read again when
it failed to load. A folder added, renamed or deleted, or lost events, compare the whole library as
Refresh does, quietly. Removals wait while the OCR thread is busy. viewer_add_capture returns the
index a capture already has, so an import or screen capture the watcher saw first is still scanned.
Off Windows, the deleted folder is no longer listed as part of the library.
- No model is made at startup (private memory there falls from about 4.3 GB to about 420 MB).
ocr_ui_models_ready makes the OCR engine, then the label engine and the list's set, on the OCR
thread when a reading, a scan (the button, -scan, or after a capture or an import) needs them;
the OCR and scan buttons wait for them, showing "Loading the models...". A like: search asks
for the label engine alone (labels_ui_want) and is worked out again once it is up.
- ocr_ui_unload_idle_models, each frame and also in the tray: an engine (OCR, labels or
segmentation) unused for MODELS_IDLE_UNLOAD_SECONDS (10 min) is destroyed by an UNLOAD task on
the OCR thread, never during a scan, a waiting read or scan, or while search jobs hold the label
engine. The label list's set stays, so loading again does not score the prompts again.
- Choosing another OCR model with none loaded only notes it; -segment no longer waits for the OCR
and label engines. Some 1.4 GB stays after an unload, the CUDA runtime it loaded.
The template's generic icon is replaced in icon.png, tray_icon.png and tray_icon.ico: the black, white-edged plus of the cross cursor shown when choosing a region with PrintScreen, on a plain light red circle.
- macos-plan.html: where mcap stands on macOS (built to compile there, never yet run in this
project; two build breaks fixed in 0.1.11), what is Windows only, and eight phases to parity:
build and run on a Mac; the small platform gaps (folder picker, Trash, temporary folder,
clipboard); images through ImageIO; video through FFmpeg for macOS; the models on the GPU with
CoreML; screen capture stills and recording through ScreenCaptureKit, behind a small
Objective-C shim; and a signed, notarized release. Permissions and signing, risks, and six open
questions: which Mac, the oldest macOS, the capture keys, signing, the library's place, the shim.
- src/backends/unix_sdl3_gpu3.jai (macOS and Linux) brought all of ImGui into the app's scope,
where ImGui.Image collided with modules/Image_Decode's Image struct: both builds have failed
since 0.0.8 brought in Image_Decode. It now uses ImGui except Image, with a note to keep that
when the file is generated again.
- app.record_region (-record-region) was a Windows RECT in the shared App struct, from 0.1.6;
it and its parsing are Windows only now.
- jai check_targets.jai - linux passes. The macOS check needs a Mac: macOS shares the same
backend file and App struct, so both breaks applied there too.
- System sound: on / off (red while on) turns what the computer plays on and off while
recording, as Mic does the microphone; its state is saved as system_audio in [video].
- modules/Screen_Record: the microphone's switch is now one for either source:
record_set_audio, record_audio_on and record_audio_failed take an Audio_Source_Kind, and the
audio thread opens and closes each as it is turned on and off.
- record-probe -system-toggle. With -sync-test, system sound on at 3 s and off at 7 s keeps only
the beeps at 3 and 5 s (8-12 ms from their flashes), and the track runs the whole recording.
- The toolbar's Mic: on / off (red while on) turns the microphone on and off while recording;
its state is saved as microphone in [video] for the next recording. A microphone that cannot
be opened turns it off again, with a toast saying why.
- modules/Screen_Record: record_set_microphone, record_microphone, record_microphone_failed. The
audio thread opens the microphone only while it is on (so Windows shows it in use only then)
and closes it when turned off. Record_Options.audio_track gives the MP4 a sound track whatever
is on at the start, silent while nothing is, as one cannot be added once the header is
written; the app always asks for it. The sources heard and the failures are kept as they
happen, not only at the start.
- record-probe -mic-toggle: the microphone on at 3 s and off at 7 s. With system sound and with
nothing else on, the track runs the whole recording without a break.
- modules/Screen_Record/audio_source.jai: WASAPI, in loopback from the default output device
(what the computer plays) and from the default input device (the microphone), asked for as
48 kHz stereo float, which Windows converts any device to; a device that will not convert is
taken as it is when it gives 48 kHz float. Packets carry the time they were heard. A device
lost (unplugged, default changed) is opened again from the new default once a second.
- An audio thread mixes them onto the video's own timeline: the same time 0, less the same
pauses (record_pause now keeps one account of time paused for both threads). A packet within
30 ms of where its source left off follows on exactly; a source that sends nothing is silence,
passed 200 ms late. Each 1024 samples every source has covered become an AAC frame.
- encoder.jai: an AAC stream (FFmpeg's aac, 160 kb/s) beside the video, written from the audio
thread; both threads' packets meet in the file under a lock. The remux keeps it.
- [video] system_audio (true) and microphone (false). capture.json: Audio, AudioSources. A toast
when a source asked for could not be opened.
- record-probe: -audio system|mic|both, the sound read back (length, level), and -sync-test,
which flashes a white square and beeps together every 2 s, then finds both in the file.
- Measured: 20 s, beeps 0-21 ms from their flashes, no drift; both sources mixed, 5-18 ms;
a 6 s pause cuts the same 6 s from sound and video; 3 min, 85 of 90 beeps within 3-24 ms and
-14 ms from first to last, with 5 outliers (53-147 ms) where two beeps went missing. Tests:
an MP4 with sound keeps its AAC stream through the remux into the library (30 checks).
- A recording killed mid-way had wrong timestamps in players: its fragmented MP4's header gives a
length of 0, and one cut short never gets the index a clean stop writes. modules/Screen_Record's
record_remux now copies every recording's packets, as they are, into a regular MP4 (one header
first with the length and every frame's index; faststart) before it goes into the library;
should that fail, the fragmented MP4 goes in as it was.
- Keyframes were every 60 frames, which, with frames written only on change, could span minutes
of time, and a crash could lose a fragment that long. One is now forced every 2 s of time
(forced-idr for NVENC and libx264).
- A crash's leftover no longer claims a region at 0, 0 in its capture.json.
- record-probe -remux IN OUT. Tests: a recording cut at its last fragment, as a crash cuts it,
goes in as a regular MP4 of the time before the cut.
- Checked on the recording that showed it: its MP4, cut mid-fragment, rewrites to 76 frames
(5.2 s) with positive frame durations. A 12 s still recording now fragments every ~2 s, and
the app killed 10 s into a recording leaves 8.1 s.
- src/app/screen_record_win32.jai: while recording, a red frame just outside the region (one
window shaped by a window region, clicks let through) and a toolbar beside it (below the
region at its right, else above, else inside a whole screen's top right): a blinking dot,
the time less time paused, the size, frames dropped and screen losses, and Pause/Resume,
Stop, and Discard, confirmed in the toolbar itself ("Yes, discard / No"). GDI windows that
never take the focus, kept out of every capture (WDA_EXCLUDEFROMCAPTURE). Buttons only ask;
screen_record_frame acts on the next frame. On the shutdown thread they are hidden, not
destroyed.
- modules/Screen_Record: record_pause; while paused the source is drained but nothing is sent,
and frames' times leave the pause out, so the video has no gap. Record_Stats.paused_seconds;
Record_Testing.LOSE_DUPLICATION loses Desktop Duplication every 3 s.
- Recordings left in %TEMP%/mcap by a crash are offered at startup: into the library (no window
known; size from its frames, date from its name), deleted, or later.
- [video] in settings.config: fps (1-120, 30), cursor (true), capture (auto, dxgi, wgc); written
into existing files on the next launch.
- The Win32 GDI bindings move to src/base/platform/win32_gdi_bindings.jai, shared by the overlay
and the toolbar.
- -record-region X Y W H and -record-ui-visible for -record; -pause SECONDS and -lose for
record-probe. capture.json's Paused.
- Checked: the toolbar is not in a whole-screen recording's frame; a 10 s pause leaves a 6.2 s
video of a 16 s run at a steady 30 fps (DXGI and WGC); 3 simulated losses in 10 s recover with
no gap. Tests: a leftover's date and its write (22 checks in the recording group); all pass.
- Fixed on the way: a recording's path repeated the temporary folder ("%1" is the first argument).
- record_region, a new [global_hotkeys] action on Ctrl-PrintScreen (Windows only; none on
macOS): starts a recording, or stops the one running.
- The overlay's video mode (screen_capture_win32.jai): red outline and magnifier; the drag held
to the screen it began on; under 240 x 180 (record_size_allowed, in Screen_Record) the outline
is dashed, the label gives the minimum, and letting go takes nothing; a click or Enter takes
the whole screen; no pixels copied. With immediate, no overlay at all: the screen under the
cursor, for PrintScreen while recording.
- src/app/screen_record.jai: records with modules/Screen_Record to %TEMP%/mcap at 30 fps; a
recording that stops by itself keeps what it has, as does quitting (the shutdown step stops it).
The MP4 goes into the library through the stills' jobs: a video .mcap in Screen Captures, the
MP4 its original, a frame of it its rendered image, metadata.json as a still's, capture.json
with Duration, FrameRate, VideoCodec, PosterTime, Encoder, Capture and DroppedFrames; the MP4 is
deleted once it is in. The catalog reads a video's keys from capture.json too.
- The capture thread makes itself aware of each monitor's DPI: ImGui makes only the main thread
aware, and Desktop Duplication refused the app's capture thread with DXGI_ERROR_UNSUPPORTED
(the fallback recorded with Windows.Graphics.Capture, and the log said so).
- -record SECONDS records the screen under the cursor, puts it in the library and exits;
-record-quit quits while recording instead. Both pass on a scratch library.
- Tests: a "screen recording" group (the size rule; an MP4 from the module's encoder written as a
recording reads back, and the catalog takes it as a video). The segment overlay test no longer
depends on the colours chosen in the app's own data folder.
- README, the format's README and the plan describe it.
- modules/Screen_Record/wgc_source.jai: the screen through Windows.Graphics.Capture, with
hand-written bindings for its WinRT interfaces (capture item interop, item, free-threaded frame
pool, frame, session 1-3, IDirect3DDxgiInterfaceAccess, IClosable). A small COM handler on
FrameArrived sets an event per frame; when Windows refuses it the pool is polled each
millisecond. The cursor is drawn by Windows; the yellow border is turned off where the system
allows (Windows 11). A screen that changes size makes the pool again. WinRT's own calls are
loaded from combase.dll at run time: runtimeobject.lib names no DLL for #library.
- capture_gpu.jai: what both sources share (the device on the screen's adapter, the region copy,
the staging ring), so each is only "the next image's region into a texture".
- Record_Options.capture: AUTO (Desktop Duplication, else Windows.Graphics.Capture when it says
fall_back), DXGI or WGC. Sources open on the capture thread, as WinRT objects belong to their
apartment; record_start waits for it to say how that went. Record_Options.testing simulates the
two-GPU failure, forces polling, or keeps the border, for record-probe.
- record-probe: -capture auto|dxgi|wgc, -simulate-unsupported, -poll, -border-test with
-keep-border as its control, -cursor-test; DPI awareness set already is no longer reported as
a failure.
- On the RTX 5080 laptop: auto falls back on the simulated failure; WGC records 4K at 60 fps with
nothing dropped; its cursor and the one cursor.jai draws for duplication are the same; the
border is off for an unpackaged app (0 yellow edge pixels, 14,214 with it kept);
WDA_EXCLUDEFROMCAPTURE is respected; polling holds 30 fps. The plan records the results.
- README: Screen_Record's line, which held a carriage return in place of "\r" since 0.1.3.
- modules/Screen_Record: a region of one screen to H.264 in a fragmented MP4. A capture thread
duplicates the screen (DuplicateOutput1, else DuplicateOutput) on a D3D11 device of its own on
the screen's adapter, copies the region on the GPU, reads it back through three staging
textures and draws the cursor in (colour, masked colour and monochrome shapes). An encoder
thread feeds FFmpeg: h264_nvenc, else h264_amf, h264_qsv, h264_mf, libx264, each tried in turn;
BGRA straight to NVENC, swscale to NV12 or YUV 4:2:0 (BT.709) for the rest. A frame is sent
when the screen or cursor changed and its turn on an fps schedule is due, with a repeat each
second of an unchanged screen; the encoder falling 8 frames behind drops the oldest. A lost
duplication (mode change, secure desktop) is made again and recording goes on. record_start
says fall_back when duplication cannot run (unsupported, too many, rotated screen). Regions
under 240 x 180 are refused, then sizes are rounded down to even; a process whose view of the
desktop is scaled is refused.
- tools/record_probe.jai (jai first.jai - record_probe): lists the screens, records a screen or
region and reports source, encoder, frames captured, sent, written and dropped, time per stage
and file size; -legacy, -cuda, -exclude-test, -animate, -snapshot DIR, -check FILE.
- On the RTX 5080 laptop: 60 s of 4K at 30 fps, 1802 frames, none dropped, 18 MB a minute; every
change kept at 60 fps; plain DuplicateOutput works when DPI aware; a CUDA context changes
nothing; WDA_EXCLUDEFROMCAPTURE is respected; QSV, Media Foundation and libx264 keep up too;
a recording killed mid-way still plays. The two-GPU failure was not reproduced: the screens are
on the NVIDIA GPU in this mode. The plan records the results.
- video-capture-plan.html: recording a region of one screen to H.264 MP4 and into the library as a
video .mcap. DXGI Desktop Duplication first, on a device of its own on the screen's adapter;
Windows.Graphics.Capture when duplication cannot run (two GPUs, rotated screens, too many
duplications). Encoders tried in turn: NVENC, AMF, QSV, Media Foundation, libx264, from the
FFmpeg already shipped. Ctrl+PrintScreen opens the overlay in a video mode (a 240 x 180 minimum,
kept to one screen) and stops the recording. Five phases, the first a command-line probe that
answers the two-GPU question on the laptop.
- capture_region, a new [global_hotkeys] action, on PrintScreen by default (Windows only; none on
macOS). Existing settings files get the line appended on the next launch. When the key cannot be
registered, the log names Snagit and Windows' "Print screen opens screen capture" setting.
- app/screen_capture_win32.jai: every screen copied the moment the key fires (BitBlt with
CAPTUREBLT over the virtual desktop), shown darkened in a topmost GDI window in a modal loop of its
own, so it works with mcap hidden in the tray. Dragging shows the region bright and outlined; a
magnifier shows the pixels under the cursor with its position or the selection's size; arrow keys
move a pixel (ten with Shift). A click or Enter takes the whole screen under the cursor; Escape,
a right click or losing the focus cancels. Only the parts that change are redrawn.
- app/screen_capture.jai: the region goes on the clipboard (CF_DIB) at once, then a thread writes it
into the library's "Screen Captures" folder as a PNG .mcap: metadata.json with the window under
the region's middle (its title as WindowName, its program's description as AppName) and
capture.json (where on the desktop, the program's path, the hash). mcap then comes to the front
showing it, and OCR, labels and segmentation read it as they read an import.
- A "screen capture" group in the import tests: the file written, its pixels, preview and Capture
Text read back. README and the format's README describe it.
- jai release.jai builds the release build, then stages release/mcap-VERSION-windows-x64/ with
mcap.exe and its symbols, onnxruntime (CUDA provider), FFmpeg, the assets, the OCR, label and
segmentation models (each checked against its models.config), NVIDIA's CUDA 13 runtime, all of
cuDNN 9 and NVRTC, a README and third-party notices; zips it with Windows' tar; prints its
SHA-256. - no_cuda leaves the CUDA runtime out; - no_build packages bin/ as it is.
- cuDNN is copied whole: it loads sub-libraries by name (9.26's cudnn_engines_tensor_ir64_9.dll),
and one missing fails every run on the GPU.
- Checked by running the package with no CUDA or cuDNN on PATH: OCR, labels and segmentation on
CUDA, video import through its FFmpeg.
- release/ is ignored.
- The Capture window draws a segmented image's segments over it: each class in its colour, a shade
of it for each instance, edges darker; under the OCR boxes, at any fit and zoom. A viewer worker
reads the segs and mask chunks (never the images); the main thread makes the texture from the
mask when the capture, the opacity or the classes hidden change.
- Above the image: Segments on or off, the opacity, and a legend of the classes (largest first,
with counts): untick one to hide it, right click it to search has: for it. Hovering the image
shows the segment's class, score and share of the image.
- Colours: a class's own is a hue from its name's hash. Its legend swatch is a colour picker
(ImGui ColorEdit3) to choose another, kept in %APPDATA%\mcap\segment-colours.config for every
image and library; right click it to go back to its own.
- Tests: the overlay's pixels (shades, edges, hidden classes, opacity), reading a file's segments
for it, and chosen colours.
- This completes segmentation-plan.html's six phases.
- has: searches what segmentation found, by the whole class name: has:dog, has:"dining table"
(has:table is a table, not a dining table). Plain words do not match segments.
- The catalog keeps each capture's segments (class, score, area) and its classes between newlines,
so a has: term is one contains per capture.
- Stats: "What is in the images", how many of the segmented images hold each class, each one a
has: search; shown even when nothing has been read by OCR. -stats logs it.
- Tests: has: (whole names, quoted names, case, combinations, plain words not matching).
- .library-folders in the library (src/app/library_folders.jai): each library folder's options,
starting with segment: true. Every folder starts off; unknown keys are kept when it is rewritten.
- The Segmentation window (Segments, above the grid): each folder's Segment checkbox, how many of
its images are segmented, and Erase per folder; a button to segment the folders set to segment.
- The segmentation scan (src/app/segment_ui.jai) runs on the OCR thread as its own task, with the
OCR scan's progress, pause and cancel: images only (never videos), not yet segmented, in folders
set to segment. The engine loads with the first scan (2.4 GB of GPU memory). Each image's segs and
mask are stored in one rewrite and its catalog entry loaded again.
- Importing into a folder set to segment segments the new images; the Import window's checkbox is
the folder's setting.
- -segment segments and exits; -erase segments, and -erase all now includes segments.
- The catalog reads each capture's segs (class list hash, segment count). The exit overlay names the
segmentation scan and model. segment_mcap_chunks moved into modules/Segment, shared with
segment-probe. The quoting helpers of Capture_File's text formats are public.
- Tests: .library-folders read and written back (quoted names, unknown keys, versions).
- modules/Capture_File/segment_chunks.jai: a front "segs" chunk (text: engine, date, the image read
and its size, the class list's hash, and each segment's id, score, area, box and class) and a back
"mask" chunk (a segment id per pixel, run-length encoded with LEB128 lengths, longest side at most
1024), keyed by model. No new file version: chunks came in version 3.
- mcap_store takes back chunks too, so a model's summary and mask are stored in one rewrite.
- Tests: the text round trip, the runs' edge cases (one segment over everything, none, runs across
rows, multi-byte lengths, id 255), bad masks refused, and storing, replacing and erasing both
chunks in a file with other chunks kept.
- segment-probe takes .mcap files (the image each is shown as; videos skipped) and with -store stores
its segments in them. On 27 images a mask is 0.8 to 6.4 KB.
- The format README's "Segmentation chunks" section; the plan's phase 3 results.
- segmentation-plan.html: the plan for panoptic segmentation of images, per library folder, stored
in .mcap chunks (no file version bump), in six phases.
- tools/export_mask2former.py: exports facebook/mask2former-swin-base-coco-panoptic to ONNX (opset 17,
dynamic batch, 384 x 384 input) and checks it against PyTorch: on 25 images, all 180 segments
identical (IoU 1.000). Writes the class list with names for people ("sky-other-merged" is "sky"),
and PyTorch's own segmentations for segment-probe -check.
- modules/Segment: runs the model with onnxruntime (CUDA, else the CPU) and makes segments of its
outputs as transformers' post_process_panoptic_segmentation does: a segment id map (longest side at
most 1024) and each segment's class, score, area and bounds. Matches PyTorch on 99.8% of pixels,
every segment scored 0.6 or more found. 57 ms an image at batch 4 on an RTX 5080 laptop.
- tools/segment_probe.jai (jai first.jai - segment_probe): segments image files; -out draws them,
-check compares with PyTorch, -dump-input writes the model's input.
- modules/onnxruntime: ort_run_float_outputs, for a model with several outputs.
- first.jai copies the model to bin/segment-models. The model itself is not in git
(modules/Segment/models, 462 MB); tools/export_mask2former.py makes it.
- The exit overlay says which step of shutdown is running and what it waits on ("Stopping the OCR
scan: finishing the text lines it is reading.", the import, the viewer, the models), and how long
that step has taken once it passes 2 s. It said "Saving..." for all of it.
- OCR stops mid-batch: ocr_process takes a cancel flag (Ocr_Engine.cancel), checked by its workers
before each piece of work; a scan sets it to its own cancel. A cut-short batch stores nothing and
counts no failures. Closing during a scan took minutes when a batch held large images; now well
under a second.
- Once shutdown starts, the main thread stops running the app's frames (the viewer, search, import,
OCR, Stats) and draws only the overlay: background_frame read the search thread group while the
shutdown thread was tearing it down, an access violation on exit.
- Import brings in videos (MP4, MOV, MKV, WebM, AVI, WMV, MPEG, MTS/M2TS, FLV) on Windows, read with
FFmpeg: the file as it was in the original, a frame a tenth of the way in (at most 3 s), turned
the right way up, as the rendered image (what the grid, Capture window, OCR and labels see), and
import.json with Duration, FrameRate, VideoCodec, Rotation, PosterTime, Audio, and the camera,
date and location from the container. Elsewhere videos are skipped as "video (Windows only)".
- .mcap version 4: the original's size is a u64, so a video can be any size. Versions 1-3 still read.
mcap_read_without_original and mcap_write_streamed read and rewrite a file with its original left
on disk; mcap_store, mcap_erase, the import index and the import itself never hold a video in memory.
- Animated GIFs: Frames and Duration in import.json (gif_info).
- Grid: a play mark and duration on video thumbnails, GIF on animations. Capture window: what the
video is, and Play in the default player (written out to %TEMP%\mcap).
- modules/ffmpeg: FFmpeg 6.0 bindings and the five DLLs used (avformat, avcodec, avutil, swscale,
swresample), copied beside the executable by the build.
swscale converts into a buffer of its own allocator with padded rows: an exact-sized one let
its SIMD write past the end, crashing on frames of odd widths (1916, 1106 ...).
- image-probe shows videos and GIF frame counts; ocr-bench reads a video's frame.
- Tests: new video formats, display-matrix rotation, ISO 6709, video dates, GIF frames, and v4
serialize/parse, v3 compatibility, streamed writes, store/erase and extraction.
SnagitConverter's copy of Capture_File needs syncing to read version 4 files.
- Import window (and -import SOURCE DEST): every image under a folder (PNG, JPEG, HEIC, AVIF, WebP,
GIF, BMP, TIFF, camera RAW) becomes an .mcap holding the file as it was, a preview turned by its
EXIF orientation, and metadata.json and import.json Capture Text (date from EXIF, the file name or
its time; file name as title; camera as app). .snagx files become the same .mcap SnagitConverter
makes. Duplicates across the whole library are skipped by a hash index (.import-index); videos
and sidecars are skipped; source files are only read.
- modules/Image_Decode: stb_image, then WIC for everything else; EXIF; orientation; previews.
tools/image_probe.jai shows what it reads.
- modules/Snagx (and modules/jaison), copied from SnagitConverter.
- The library is a folder of folders: recursive scan, a folder filter, and a folder: search prefix.
- OCR, the viewer and ocr-bench decode any image format, turned the right way up.
- Rename D:\SnagitIndex to D:\McapIndex.
