The workers ran gpu_workers (4) runs at once whatever the images: the server
model, four large images at a time, took 15 GB of a 16 GB GPU and spilled into
shared system memory, and a smaller GPU would fare far worse.
modules/OCR/budget.jai: each run reserves what it is expected to need, and
waits while the runs in flight or the GPU's free memory leave too little; one
run always goes. The need is measured when the engine is made, from the arena's
peak (AllocatorGetStats MaxInUse) over runs on two input sizes, so it holds for
any model and GPU: the mobile detector 6 MB + 137 MB a megapixel, the server's
4 MB + 1015, against 511 and 522 MB measured for a 2560 x 1440 tile. The limit
is the GPU's free memory less 10% (at least 512 MB), read at the start of each
ocr_process call (cudaMemGetInfo, ort_cuda_memory); the free memory is also read
as each run starts, since finished runs' blocks stay in the arena until the
stage ends.
gpu-bench, RTX 5080 laptop: the server model 335 ms a capture (854 before),
peak 13.1 GB (15.4), passes within 4%; the mobile model unchanged (140 ms,
5.7 GB), as its runs fit.
Musa Mahmood
dd7cf83ca6