OCR filled the GPU: 9 GB for the mobile model on gpu-bench's 30 images, past
16 GB for the server model, which Windows then paged into shared memory (1.5 to
11 s a capture). onnxruntime's logged arena extensions showed why: the arena
keeps every block it takes from CUDA and never joins or frees them, and OCR's
inputs grow from run to run (recognizer batches went narrowest first; images in
any order across calls), so each larger run left another block behind: 5.7 GB
of recognizer blocks where the widest batch needs 0.9 GB.
- Detection runs the largest images first, recognition the widest batches.
- The detector's and recognizer's arenas shrink once each stage of an
ocr_process call is done, by a run on a small input with
memory.enable_memory_arena_shrinkage. Shrinking after every run cost 33 to
59% of the speed.
- The worker threads are made with the engine and kept, not made for every
stage of every call (onnxruntime keeps GPU state for each thread that runs a
session, microsoft/onnxruntime#33037); 6% faster.
- OCR's arenas grow by what is asked (kSameAsRequested).
- ort.log_level sets what the engines' onnxruntime environments log; INFO shows
every arena extension.
Mobile OCR on gpu-bench (RTX 5080 laptop, 4 workers): peak 5.7 GB over the
baseline, 2 GB between calls, 142 ms a capture, against 9.0 GB and 126 ms. The
text read differs by a line in 2,986 (batches are grouped differently). The
server model is not yet measured again.
- The exit overlay says which step of shutdown is running and what it waits on ("Stopping the OCR
scan: finishing the text lines it is reading.", the import, the viewer, the models), and how long
that step has taken once it passes 2 s. It said "Saving..." for all of it.
- OCR stops mid-batch: ocr_process takes a cancel flag (Ocr_Engine.cancel), checked by its workers
before each piece of work; a scan sets it to its own cancel. A cut-short batch stores nothing and
counts no failures. Closing during a scan took minutes when a batch held large images; now well
under a second.
- Once shutdown starts, the main thread stops running the app's frames (the viewer, search, import,
OCR, Stats) and draws only the overlay: background_frame read the search thread group while the
shutdown thread was tearing it down, an access violation on exit.
