The workers ran gpu_workers (4) runs at once whatever the images: the server
model, four large images at a time, took 15 GB of a 16 GB GPU and spilled into
shared system memory, and a smaller GPU would fare far worse.
modules/OCR/budget.jai: each run reserves what it is expected to need, and
waits while the runs in flight or the GPU's free memory leave too little; one
run always goes. The need is measured when the engine is made, from the arena's
peak (AllocatorGetStats MaxInUse) over runs on two input sizes, so it holds for
any model and GPU: the mobile detector 6 MB + 137 MB a megapixel, the server's
4 MB + 1015, against 511 and 522 MB measured for a 2560 x 1440 tile. The limit
is the GPU's free memory less 10% (at least 512 MB), read at the start of each
ocr_process call (cudaMemGetInfo, ort_cuda_memory); the free memory is also read
as each run starts, since finished runs' blocks stay in the arena until the
stage ends.
gpu-bench, RTX 5080 laptop: the server model 335 ms a capture (854 before),
peak 13.1 GB (15.4), passes within 4%; the mobile model unchanged (140 ms,
5.7 GB), as its runs fit.
OCR filled the GPU: 9 GB for the mobile model on gpu-bench's 30 images, past
16 GB for the server model, which Windows then paged into shared memory (1.5 to
11 s a capture). onnxruntime's logged arena extensions showed why: the arena
keeps every block it takes from CUDA and never joins or frees them, and OCR's
inputs grow from run to run (recognizer batches went narrowest first; images in
any order across calls), so each larger run left another block behind: 5.7 GB
of recognizer blocks where the widest batch needs 0.9 GB.
- Detection runs the largest images first, recognition the widest batches.
- The detector's and recognizer's arenas shrink once each stage of an
ocr_process call is done, by a run on a small input with
memory.enable_memory_arena_shrinkage. Shrinking after every run cost 33 to
59% of the speed.
- The worker threads are made with the engine and kept, not made for every
stage of every call (onnxruntime keeps GPU state for each thread that runs a
session, microsoft/onnxruntime#33037); 6% faster.
- OCR's arenas grow by what is asked (kSameAsRequested).
- ort.log_level sets what the engines' onnxruntime environments log; INFO shows
every arena extension.
Mobile OCR on gpu-bench (RTX 5080 laptop, 4 workers): peak 5.7 GB over the
baseline, 2 GB between calls, 142 ms a capture, against 9.0 GB and 126 ms. The
text read differs by a line in 2,986 (batches are grouped differently). The
server model is not yet measured again.
- The exit overlay says which step of shutdown is running and what it waits on ("Stopping the OCR
scan: finishing the text lines it is reading.", the import, the viewer, the models), and how long
that step has taken once it passes 2 s. It said "Saving..." for all of it.
- OCR stops mid-batch: ocr_process takes a cancel flag (Ocr_Engine.cancel), checked by its workers
before each piece of work; a scan sets it to its own cancel. A cut-short batch stores nothing and
counts no failures. Closing during a scan took minutes when a batch held large images; now well
under a second.
- Once shutdown starts, the main thread stops running the app's frames (the viewer, search, import,
OCR, Stats) and draws only the overlay: background_frame read the search thread group while the
shutdown thread was tearing it down, an access violation on exit.
