mcap/modules/OCR/README.md

OCR

Text in images, with where each line is: a batch of RGBA images in, each image's text lines out, each with a box and a confidence. It runs ONNX models through onnxruntime (modules/onnxruntime): on an NVIDIA GPU through CUDA when every library that needs is present, and on the CPU otherwise.

Setting up

The binaries and the models are downloaded, not committed:

cd modules/onnxruntime && jai fetch.jai           # onnxruntime 1.30.0, GPU build (CUDA 13); `- cpu` for the CPU-only build
cd modules/OCR         && jai fetch_models.jai    # both models (about 195 MB); `- mobile` for the light one only
jai generate.jai                                  # in modules/onnxruntime, only when changing onnxruntime versions

Each checks the SHA-256 of what it downloads. Building the app (jai first.jai) copies onnxruntime beside the executable and the models into bin/ocr-models.

Using it

#import "OCR";

engine, ok, error := ocr_create(models_folder, "ppocr-v5-mobile");   // prefer_gpu = true
in_use, explanation := ocr_gpu_status(engine);                      // why not the GPU, and what to install
results := ocr_process(engine, images);                             // [] Ocr_Image (RGBA) -> [] Ocr_Result
for results[0].lines  print("% at %,%: %\n", it.text, it.x, it.y, it.confidence);
ocr_destroy(engine);

ocr_process shares a batch out over worker threads: detection per image, then recognition of every line of every image in batches of similar width. The results come from the context allocator, or the one passed. ocr_stats gives the time spent in each stage.

Models

models.config lists them; ocr_create takes a name. Both are PaddleOCR's PP-OCRv5 (Apache 2.0), as converted to ONNX by RapidOCR, and read Chinese, English and Japanese:

Name	Files	On the 20 sample captures, CPU
ppocr-v5-mobile	detector 4.8 MB, recognizer 16.6 MB	about 0.5 s per capture
ppocr-v5-server	detector 88 MB, recognizer 85 MB	about 2.5 s per capture

Another model with the same two-stage design is a new [model NAME] section; a different design needs its own detection or recognition code beside detect.jai and recognize.jai.

The GPU

onnxruntime's CUDA build needs, on PATH or beside the executable:

• the NVIDIA driver;
• the CUDA Toolkit 13.x (cudart64_13, cublas64_13, cublasLt64_13, cufft64_12), from developer.nvidia.com/cuda-downloads, whose installer puts its bin folder on PATH;
• cuDNN 9.x for CUDA 13 (cudnn64_9 and its cudnn_*64_9 companions), from developer.nvidia.com/cudnn-downloads, whose bin folder for CUDA 13 you add to PATH yourself.

ocr_create looks for each of these first (cuda_check.jai) and says which are missing, rather than letting onnxruntime fail later and obscurely. If they are all there but CUDA still cannot start, it runs on the CPU and keeps onnxruntime's message.

Measuring

jai first.jai - release ocr_bench, then bin/ocr-bench MCAP_FOLDER: each model on the GPU (when available) and the CPU, the time per capture and per stage, and each capture's lines in bin/ocr-out/<model>-<provider>/. Like any command-line tool here it sets runtime_support_headless (modules/Runtime_Support.jai): a failure is printed and exits with code 1, with no message box to wait on.