mcap/modules/OCR

OCR

Text in images, with where each line is: a batch of RGBA images in, each image's text lines out, each with its corners, its box and a confidence. Lines at an angle, read downwards or upwards, or upside down are read too. It runs ONNX models through onnxruntime (modules/onnxruntime): on an NVIDIA GPU through CUDA when every library that needs is present, and on the CPU otherwise.

Setting up

The binaries and the models are downloaded, not committed:

cd modules/onnxruntime && jai fetch.jai           # onnxruntime 1.30.0, GPU build (CUDA 13); `- cpu` for the CPU-only build
cd modules/OCR         && jai fetch_models.jai    # both models (about 196 MB); `- mobile` for the light one only
jai generate.jai                                  # in modules/onnxruntime, only when changing onnxruntime versions

Each checks the SHA-256 of what it downloads. Building the app (jai first.jai) copies onnxruntime beside the executable and the models into bin/ocr-models.

Using it

#import "OCR";

engine, ok, error := ocr_create(models_folder, "ppocr-v5-mobile");   // prefer_gpu = true
in_use, explanation := ocr_gpu_status(engine);                      // why not the GPU, and what to install
results := ocr_process(engine, images);                             // [] Ocr_Image (RGBA) -> [] Ocr_Result
for results[0].lines  print("% at %,%: %\n", it.text, it.x, it.y, it.confidence);
ocr_destroy(engine);

ocr_process shares a batch out over worker threads: detection per image, then recognition of every line of every image in batches of similar width.

Turned text. Detection gives each line the smallest rectangle around it at any angle (rotating calipers over its convex hull), keeping a level one unless a turned one is clearly smaller; a level box takes in every pixel it touches, as recognition reads best. Recognition samples each rectangle along its own sides into a level crop. A box at least 1.5 times as tall as it is wide is taken as a line read downwards, and also read level, since a narrow word such as "1" is as tall. The orientation classifier marks lines that look upside down, and those are read both ways. Of each line's readings, the more confident is kept. Ocr_Line.corners go clockwise from the text's top left, so a turned line says which way it reads. The results come from the context allocator, or the one passed. ocr_stats gives the time spent in each stage.

Models

models.config lists them; ocr_create takes a name. Both are PaddleOCR's PP-OCRv5 (Apache 2.0), as converted to ONNX by RapidOCR, and read Chinese, English and Japanese:

Name	Files	On the 20 sample captures: RTX 5080 laptop / CPU
ppocr-v5-mobile	detector 4.8 MB, recognizer 16.6 MB	about 0.08 s / 0.26 s per capture
ppocr-v5-server	detector 88 MB, recognizer 85 MB	about 0.4 s / 2.4 s per capture

Both also use the text line orientation classifier ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx (1 MB, textline_ori: in models.config).

Another model with the same two-stage design is a new [model NAME] section; a different design needs its own detection or recognition code beside detect.jai and recognize.jai.

The GPU

onnxruntime's CUDA build needs, on PATH or beside the executable:

• the NVIDIA driver;
• the CUDA Toolkit 13.x (cudart64_13, cublas64_13, cublasLt64_13, cufft64_12), from developer.nvidia.com/cuda-downloads, whose installer puts its bin folder on PATH;
• cuDNN 9.x for CUDA 13 (cudnn64_9 and its cudnn_*64_9 companions), from developer.nvidia.com/cudnn-downloads. Its installer does not touch PATH; when cuDNN is not on it, the newest C:\Program Files\NVIDIA\CUDNN\v9.*\bin\13.*\x64 is added to the program's own PATH, so an install in the usual place needs nothing more.

Ocr_Options holds how it runs on the GPU. Its defaults came from measuring: onnxruntime's kNextPowerOfTwo arena, cuDNN's workspace capped, recognizer batches rounded to a width of 64, and 4 worker threads (with 8, the server model filled GPU memory and ran 5 times slower again). With the exact-size arena GPU memory filled up, as nearly every image and batch is a new size, and the server model ran 5 times slower. Several copies of the sessions, each on its own CUDA streams, made both models slower. GPU memory still peaks near full with the server model; that is for a later change.

ocr_create looks for each of these first (modules/onnxruntime/cuda.jai, shared with Labels) and says which are missing, rather than letting onnxruntime fail later and obscurely. If they are all there but CUDA still cannot start, it runs on the CPU and keeps onnxruntime's message.

Measuring

jai first.jai - release ocr_bench, then bin/ocr-bench MCAP_FOLDER: each model on the GPU (when available) and the CPU, the time per capture and per stage, and each capture's lines in bin/ocr-out/<model>-<provider>/. Like any command-line tool here it sets runtime_support_headless (modules/Runtime_Support.jai): a failure is printed and exits with code 1, with no message box to wait on.