# OCR

Text in images, with where each line is: a batch of RGBA images in, each image's text lines out,
each with its corners, its box and a confidence. Lines at an angle, read downwards or upwards, or
upside down are read too. It runs ONNX models through onnxruntime
([`modules/onnxruntime`](../onnxruntime)): on an NVIDIA GPU through CUDA when every library that
needs is present, and on the CPU otherwise.

## Setting up

The binaries and the models are downloaded, not committed:

```
cd modules/onnxruntime && jai fetch.jai           # onnxruntime 1.30.0, GPU build (CUDA 13); `- cpu` for the CPU-only build
cd modules/OCR         && jai fetch_models.jai    # both models (about 196 MB); `- mobile` for the light one only
jai generate.jai                                  # in modules/onnxruntime, only when changing onnxruntime versions
```

Each checks the SHA-256 of what it downloads. Building the app (`jai first.jai`) copies onnxruntime
beside the executable and the models into `bin/ocr-models`.

## Using it

```jai
#import "OCR";

engine, ok, error := ocr_create(models_folder, "ppocr-v5-mobile");   // prefer_gpu = true
in_use, explanation := ocr_gpu_status(engine);                      // why not the GPU, and what to install
results := ocr_process(engine, images);                             // [] Ocr_Image (RGBA) -> [] Ocr_Result
for results[0].lines  print("% at %,%: %\n", it.text, it.x, it.y, it.confidence);
ocr_destroy(engine);
```

`ocr_process` shares a batch out over worker threads: detection per image, then recognition of
every line of every image in batches of similar width.

**Turned text.** Detection gives each line the smallest rectangle around it at any angle (rotating
calipers over its convex hull), keeping a level one unless a turned one is clearly smaller; a level
box takes in every pixel it touches, as recognition reads best. Recognition samples each rectangle
along its own sides into a level crop. A box at least 1.5 times as tall as it is wide is taken as a
line read downwards, and also read level, since a narrow word such as "1" is as tall. The
orientation classifier marks lines that look upside down, and those are read both ways. Of each
line's readings, the more confident is kept. `Ocr_Line.corners` go clockwise from the text's top
left, so a turned line says which way it reads. The results come from the context
allocator, or the one passed. `ocr_stats` gives the time spent in each stage.

## Models

`models.config` lists them; `ocr_create` takes a name. Both are PaddleOCR's PP-OCRv5 (Apache 2.0),
as converted to ONNX by RapidOCR, and read Chinese, English and Japanese:

| Name | Files | On the 20 sample captures: RTX 5080 laptop / CPU |
| --- | --- | --- |
| `ppocr-v5-mobile` | detector 4.8 MB, recognizer 16.6 MB | about 0.08 s / 0.26 s per capture |
| `ppocr-v5-server` | detector 88 MB, recognizer 85 MB | about 0.4 s / 2.4 s per capture |

Both also use the text line orientation classifier `ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx`
(1 MB, `textline_ori:` in `models.config`).

Another model with the same two-stage design is a new `[model NAME]` section; a different design
needs its own detection or recognition code beside `detect.jai` and `recognize.jai`.

## The GPU

onnxruntime's CUDA build needs, on PATH or beside the executable:

- the NVIDIA driver;
- the CUDA Toolkit 13.x (`cudart64_13`, `cublas64_13`, `cublasLt64_13`, `cufft64_12`), from
  [developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads), whose installer puts its bin folder on PATH;
- cuDNN 9.x for CUDA 13 (`cudnn64_9` and its `cudnn_*64_9` companions), from
  [developer.nvidia.com/cudnn-downloads](https://developer.nvidia.com/cudnn-downloads). Its installer does not touch PATH;
  when cuDNN is not on it, the newest `C:\Program Files\NVIDIA\CUDNN\v9.*\bin\13.*\x64` is added to the
  program's own PATH, so an install in the usual place needs nothing more.

`Ocr_Options` holds how it runs on the GPU. Its defaults came from measuring: onnxruntime's
`kNextPowerOfTwo` arena, cuDNN's workspace capped, recognizer batches rounded to a width of 64, and 4
worker threads (with 8, the server model filled GPU memory and ran 5 times slower again). With the exact-size arena GPU memory filled up, as nearly every image and batch is a
new size, and the server model ran 5 times slower. Several copies of the sessions, each on its own
CUDA streams, made both models slower. GPU memory still peaks near full with the server model; that
is for a later change.

`ocr_create` looks for each of these first (`cuda_check.jai`) and says which are missing, rather
than letting onnxruntime fail later and obscurely. If they are all there but CUDA still cannot
start, it runs on the CPU and keeps onnxruntime's message.

## Measuring

`jai first.jai - release ocr_bench`, then `bin/ocr-bench MCAP_FOLDER`: each model on the GPU (when
available) and the CPU, the time per capture and per stage, and each capture's lines in
`bin/ocr-out/<model>-<provider>/`. Like any command-line tool here it sets
`runtime_support_headless` (`modules/Runtime_Support.jai`): a failure is printed and exits with
code 1, with no message box to wait on.