93 lines
5.1 KiB
Markdown
93 lines
5.1 KiB
Markdown
# OCR
|
|
|
|
Text in images, with where each line is: a batch of RGBA images in, each image's text lines out,
|
|
each with its corners, its box and a confidence. Lines at an angle, read downwards or upwards, or
|
|
upside down are read too. It runs ONNX models through onnxruntime
|
|
([`modules/onnxruntime`](../onnxruntime)): on an NVIDIA GPU through CUDA when every library that
|
|
needs is present, and on the CPU otherwise.
|
|
|
|
## Setting up
|
|
|
|
The binaries and the models are downloaded, not committed:
|
|
|
|
```
|
|
cd modules/onnxruntime && jai fetch.jai # onnxruntime 1.30.0, GPU build (CUDA 13); `- cpu` for the CPU-only build
|
|
cd modules/OCR && jai fetch_models.jai # both models (about 196 MB); `- mobile` for the light one only
|
|
jai generate.jai # in modules/onnxruntime, only when changing onnxruntime versions
|
|
```
|
|
|
|
Each checks the SHA-256 of what it downloads. Building the app (`jai first.jai`) copies onnxruntime
|
|
beside the executable and the models into `bin/ocr-models`.
|
|
|
|
## Using it
|
|
|
|
```jai
|
|
#import "OCR";
|
|
|
|
engine, ok, error := ocr_create(models_folder, "ppocr-v5-mobile"); // prefer_gpu = true
|
|
in_use, explanation := ocr_gpu_status(engine); // why not the GPU, and what to install
|
|
results := ocr_process(engine, images); // [] Ocr_Image (RGBA) -> [] Ocr_Result
|
|
for results[0].lines print("% at %,%: %\n", it.text, it.x, it.y, it.confidence);
|
|
ocr_destroy(engine);
|
|
```
|
|
|
|
`ocr_process` shares a batch out over worker threads: detection per image, then recognition of
|
|
every line of every image in batches of similar width.
|
|
|
|
**Turned text.** Detection gives each line the smallest rectangle around it at any angle (rotating
|
|
calipers over its convex hull), keeping a level one unless a turned one is clearly smaller; a level
|
|
box takes in every pixel it touches, as recognition reads best. Recognition samples each rectangle
|
|
along its own sides into a level crop. A box at least 1.5 times as tall as it is wide is taken as a
|
|
line read downwards, and also read level, since a narrow word such as "1" is as tall. The
|
|
orientation classifier marks lines that look upside down, and those are read both ways. Of each
|
|
line's readings, the more confident is kept. `Ocr_Line.corners` go clockwise from the text's top
|
|
left, so a turned line says which way it reads. The results come from the context
|
|
allocator, or the one passed. `ocr_stats` gives the time spent in each stage.
|
|
|
|
## Models
|
|
|
|
`models.config` lists them; `ocr_create` takes a name. Both are PaddleOCR's PP-OCRv5 (Apache 2.0),
|
|
as converted to ONNX by RapidOCR, and read Chinese, English and Japanese:
|
|
|
|
| Name | Files | On the 20 sample captures: RTX 5080 laptop / CPU |
|
|
| --- | --- | --- |
|
|
| `ppocr-v5-mobile` | detector 4.8 MB, recognizer 16.6 MB | about 0.08 s / 0.26 s per capture |
|
|
| `ppocr-v5-server` | detector 88 MB, recognizer 85 MB | about 0.4 s / 2.4 s per capture |
|
|
|
|
Both also use the text line orientation classifier `ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx`
|
|
(1 MB, `textline_ori:` in `models.config`).
|
|
|
|
Another model with the same two-stage design is a new `[model NAME]` section; a different design
|
|
needs its own detection or recognition code beside `detect.jai` and `recognize.jai`.
|
|
|
|
## The GPU
|
|
|
|
onnxruntime's CUDA build needs, on PATH or beside the executable:
|
|
|
|
- the NVIDIA driver;
|
|
- the CUDA Toolkit 13.x (`cudart64_13`, `cublas64_13`, `cublasLt64_13`, `cufft64_12`), from
|
|
[developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads), whose installer puts its bin folder on PATH;
|
|
- cuDNN 9.x for CUDA 13 (`cudnn64_9` and its `cudnn_*64_9` companions), from
|
|
[developer.nvidia.com/cudnn-downloads](https://developer.nvidia.com/cudnn-downloads). Its installer does not touch PATH;
|
|
when cuDNN is not on it, the newest `C:\Program Files\NVIDIA\CUDNN\v9.*\bin\13.*\x64` is added to the
|
|
program's own PATH, so an install in the usual place needs nothing more.
|
|
|
|
`Ocr_Options` holds how it runs on the GPU. Its defaults came from measuring: onnxruntime's
|
|
`kNextPowerOfTwo` arena, cuDNN's workspace capped, recognizer batches rounded to a width of 64, and 4
|
|
worker threads (with 8, the server model filled GPU memory and ran 5 times slower again). With the exact-size arena GPU memory filled up, as nearly every image and batch is a
|
|
new size, and the server model ran 5 times slower. Several copies of the sessions, each on its own
|
|
CUDA streams, made both models slower. GPU memory still peaks near full with the server model; that
|
|
is for a later change.
|
|
|
|
`ocr_create` looks for each of these first (`modules/onnxruntime/cuda.jai`, shared with Labels) and says which are missing, rather
|
|
than letting onnxruntime fail later and obscurely. If they are all there but CUDA still cannot
|
|
start, it runs on the CPU and keeps onnxruntime's message.
|
|
|
|
## Measuring
|
|
|
|
`jai first.jai - release ocr_bench`, then `bin/ocr-bench MCAP_FOLDER`: each model on the GPU (when
|
|
available) and the CPU, the time per capture and per stage, and each capture's lines in
|
|
`bin/ocr-out/<model>-<provider>/`. Like any command-line tool here it sets
|
|
`runtime_support_headless` (`modules/Runtime_Support.jai`): a failure is printed and exits with
|
|
code 1, with no message box to wait on.
|