From ea9f055abc008eec75abe2901616abbeb5bbc2cc Mon Sep 17 00:00:00 2001 From: Musa Mahmood Date: Wed, 7 Oct 2026 21:51:18 +0100 Subject: [PATCH] [0.0.0] .mcap version 2: OCR stored in each capture, search, a library scan, sorting by date; turned text --- README.md | 19 + modules/Capture_File/README.md | 93 ++++ modules/Capture_File/module.jai | 65 ++- modules/Capture_File/ocr_text.jai | 209 +++++++++ modules/Capture_Text/module.jai | 746 ++++++++++++++++++++++++++++++ modules/OCR/README.md | 29 +- modules/OCR/cuda_check.jai | 82 +++- modules/OCR/detect.jai | 204 ++++++-- modules/OCR/fetch_models.jai | 3 +- modules/OCR/models.config | 4 + modules/OCR/models.jai | 43 ++ modules/OCR/module.jai | 99 +++- modules/OCR/recognize.jai | 110 +++-- src/App.jai | 7 + src/app/catalog.jai | 289 ++++++++++++ src/app/main.jai | 2 + src/app/ocr_ui.jai | 495 +++++++++++++++----- src/app/viewer.jai | 161 ++++++- src/test/catalog_tests.jai | 121 +++++ src/test/ocr_tests.jai | 213 +++++++++ src/test/testing.jai | 8 + src/ui/imgui_theme_registry.jai | 22 +- tools/ocr_bench.jai | 3 +- 23 files changed, 2776 insertions(+), 251 deletions(-) create mode 100644 modules/Capture_File/README.md create mode 100644 modules/Capture_File/ocr_text.jai create mode 100644 modules/Capture_Text/module.jai create mode 100644 src/app/catalog.jai create mode 100644 src/test/catalog_tests.jai create mode 100644 src/test/ocr_tests.jai diff --git a/README.md b/README.md index dbe5647..3ac2f39 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,25 @@ and everything around them, ready for an app to be written into `src/app/app_ui. progress, so it never looks hung. - **Asynchronous tests**, compiled into debug builds and run with `run_tests`. +## The capture viewer + +`src/app/viewer.jai`, `catalog.jai` and `ocr_ui.jai`: every `.mcap` in `D:\SnagitIndex\mcap` as a grid +of previews, and one capture at full size in the Capture window. + +- **Search** above the grid: every word must match, anywhere in the OCR text, window title, URL, app + name or annotation text; `"a phrase"` matches as one; `ocr:`, `title:`, `url:`, `app:` and `note:` + limit a word to one field. In the Capture window the matching OCR lines are highlighted, with Next + and Previous. +- **Order:** newest first by capture date, once the catalog (what search needs from every file) has + loaded; by name until then. +- **OCR** (`modules/OCR`): the OCR window shows whether it runs on the GPU and, if not, what to install, + and switches model (kept as `model` under `[ocr]` in the settings file). **Scan the library** reads + every capture shown that the model has not read yet, storing each reading in its file as it goes, + so it can pause, cancel and pick up again. `-library FOLDER` shows another folder of `.mcap` files; + `-scan` scans it with the chosen model once it has loaded and exits (0, or 1 if any capture + failed), for a scan without the UI, such as on a copy. The Capture window's **OCR** button reads one capture; + a capture read by several models keeps each reading, and the window can show any of them. + ## Building Requires the Jai compiler, beta 0.2.030 or later. diff --git a/modules/Capture_File/README.md b/modules/Capture_File/README.md new file mode 100644 index 0000000..634e0dc --- /dev/null +++ b/modules/Capture_File/README.md @@ -0,0 +1,93 @@ +# The .mcap file + +One capture in one file: a fixed magic, a file version stamp, the capture's data as +[Capture Text](capture_text.md), what OCR read in it, a small preview, then the images. There is no ZIP +and no compression around it; the images keep their own PNG compression. + +The parts a library browser needs (header, text, OCR and preview) come first, so it reads the start of +each file and never touches the images; a search, which needs no preview, reads less still. + +## Layout, version 2 + +All integers are little-endian. Sizes count bytes. + +| Field | Type | | +| --- | --- | --- | +| magic | 8 bytes | `89 4D 43 41 50 0D 0A 1A`, which is `\x89MCAP\r\n\x1A`. Says this is an .mcap, in every version. | +| file version | u32 | `2`. Version 1 files are still read; see below. | +| text size, text | u32, bytes | Capture Text, UTF-8: every JSON file of the capture, as `[[file]]` sections. | +| OCR size, OCR | u32, bytes | OCR text, UTF-8 (below): what each OCR model read. Size 0 when none has. | +| preview width, height | u16, u16 | In pixels. | +| preview size, preview | u32, bytes | PNG. Size 0 when there is none. | +| original name size, name | u16, bytes | The original image's name in the .snagx, such as `{GUID}.png`; the page's `CaptureBackgroundImage` names it too. | +| original size, original | u32, bytes | The capture as taken: PNG, exactly as it was in the .snagx. | +| rendered kind | u8 | 0: none. 1: the same as the original (stored once). 2: its own, below. | +| rendered size, rendered | u32, bytes | Only for kind 2. The capture with its annotations drawn in (Snagit's `thumbnail.png`): PNG, exactly as it was. | + +The file ends there: anything after the last part is a mistake. + +The magic borrows PNG's: the first byte has its high bit set, so a channel that strips it shows; `\r\n` +shows line endings being converted; and `\x1A` stops `type` on Windows. + +**Version 1** is the same without the OCR size and OCR. A reader takes both; anything written is +version 2, so a version 1 file becomes version 2 the first time OCR is stored in it. + +## The parts + +- **Text.** Everything the capture's JSON held: `index.json`, `metadata.json`, the page's `{GUID}.json`, + and `{GUID}.backup.json` when there is one. +- **OCR.** One section per OCR model, so a capture read by several models keeps every reading; + reading it again with a model replaces only that model's section. See *OCR text* below. +- **Preview.** The rendered image (or the original, when there is no rendered one) scaled so its longest + side is at most 320 pixels, as a PNG. An image already that small is used as it is, never scaled up. +- **Original and rendered.** A capture with no annotations has a rendered image identical to the original, + byte for byte; it is then stored once, as kind 1. +- **One page.** An .mcap holds one page, as every capture so far does; a .snagx with more is not converted. +- **Left out:** `{GUID}.backup.png`, Snagit's image from before an edit. Its `{GUID}.backup.json` is kept, + as plenty of captures have one without the image. + +## Versions + +A reader checks the magic, then the file version. It refuses a version above its own, never writing +over a file from a later version, and reads every version up to its own. Version 0 does not exist. + +## Converting + +`snagit-converter mcap` converts .snagx files (or extracted captures) to .mcap, and `snagit-converter +snagx` converts .mcap files back to a .snagx Snagit can open. See the README for what each checks. + +Back in a .snagx, the JSON is laid out as Snagit writes it, the images are the same bytes, and only the +backup image is missing. JSON numbers come back as values: `3.0` is written `3`. + +## OCR text + +In the style of the Ex4 text formats: + +``` +@version 2 + +[ocr ppocr-v5-mobile] +engine: "onnxruntime 1.30.0, CUDA" +image: rendered 1920 1080 +date: 2026-10-08T21:04:11Z +line: 412 96 640 18 412 96 1052 96 1052 114 412 114 0.993 "Assign Footprints - KiCad" +line: 20 140 220 14 20 140 240 140 240 154 20 154 0.981 "Symbol : Footprint Assignments" +line: 1700 300 20 160 1720 300 1720 460 1700 460 1700 300 0.952 "Read downwards" +``` + +| Key | Value | +| --- | --- | +| `[ocr NAME]` | A section: the model's name, as in its `models.config`. | +| `engine` | What ran it, quoted. | +| `image` | Which image was read (`rendered` or `original`) and its width and height; the lines' boxes are in its pixels. | +| `date` | When it was read, UTC. | +| `line` | A text line: x, y, width and height, then its four corners as x y pairs, its confidence (0 to 1), and its text, quoted with `""` for a quote. A control character in the text is written as a space. | + +The corners go clockwise from the text's top left: top left, top right, bottom right, bottom left, as +the text reads. Level text has the box's corners; a turned line's say which way it reads (the third +line above reads downwards, so its top left is the box's top right). x, y, width and height bound them. + +A line that cannot be read is skipped, as are keys this version does not know. **Version 1** OCR text +had no corners (`line: x y width height confidence "text"`); it is still read, with the box's corners. +The OCR text carries its own version, so this did not change the .mcap file version. `mcap_store_ocr` +(`modules/Capture_File/ocr_text.jai`) stores a section in a file. diff --git a/modules/Capture_File/module.jai b/modules/Capture_File/module.jai index 39b8f91..9de3899 100644 --- a/modules/Capture_File/module.jai +++ b/modules/Capture_File/module.jai @@ -1,10 +1,13 @@ // .mcap: one capture in one file. A fixed magic, a file version stamp, the capture's Capture Text, -// a small preview, then the images. See docs/mcap.md for the layout. +// its OCR text (version 2), a small preview, then the images. See docs/mcap.md for the layout. // // Reading gives views into the bytes it was given; nothing is copied. Images stay PNG. +// +// The main copy is mcap's (D:/jai_projects/active/mcap/modules/Capture_File); SnagitConverter has +// the same files. MCAP_MAGIC :: u8.[0x89, #char "M", #char "C", #char "A", #char "P", 0x0D, 0x0A, 0x1A]; -MCAP_VERSION :: 1; +MCAP_VERSION :: 2; // what is written; every version from 1 up is read MCAP_EXTENSION :: "mcap"; MCAP_PREVIEW_MAX_SIDE :: 320; // the preview's longest side, in pixels @@ -23,6 +26,7 @@ Mcap_Image :: struct { Mcap :: struct { version: u32 = MCAP_VERSION; // as read; always MCAP_VERSION when written text: string; // Capture Text + ocr: string; // OCR text (ocr_text.jai), one section per model; "" when none, and always in version 1 preview: Mcap_Image; original_name: string; // the original image's name in the .snagx, such as {GUID}.png original: [] u8; // PNG @@ -44,7 +48,7 @@ mcap_rendered :: (m: Mcap) -> [] u8 { // The file's bytes. A part too large for its size field fails the whole file rather than writing one // that cannot be read back. mcap_serialize :: (m: Mcap) -> Serializer, ok: bool, error: string { - size := MCAP_HEADER_SIZE + 4 + m.text.count + 8 + m.preview.png.count + 2 + m.original_name.count + 4 + m.original.count + 1; + size := MCAP_HEADER_SIZE + 4 + m.text.count + 4 + m.ocr.count + 8 + m.preview.png.count + 2 + m.original_name.count + 4 + m.original.count + 1; if m.rendered_kind == .OWN size += 4 + m.rendered.count; s := new_serializer(size); @@ -53,6 +57,7 @@ mcap_serialize :: (m: Mcap) -> Serializer, ok: bool, error: string { Add(*s, cast(u32) MCAP_VERSION); if !AddString(*s, m.text, u32) return s, false, "the text is too large for its size field"; + if !AddString(*s, m.ocr, u32) return s, false, "the OCR text is too large for its size field"; Add(*s, m.preview.width); Add(*s, m.preview.height); if !AddSizedArray(*s, m.preview.png, u32) return s, false, "the preview is too large for its size field"; @@ -95,10 +100,10 @@ mcap_parse :: (data: [] u8) -> Mcap, ok: bool, error: string { return m, true, ""; } -// Only the header, text and preview, as a library browser needs them: one read of the start of the -// file, and at most two more when the text or preview runs past it. The images are left unread and -// empty. The parts are views into a buffer from the context allocator. -mcap_read_front :: (path: string) -> Mcap, ok: bool, error: string { +// Only the front of the file, as a library browser needs it: the header, text, OCR text and, with +// preview, the preview. One read of the start of the file, and a few more only when a part runs past +// it. The images are left unread and empty. The parts are views into a buffer from the context allocator. +mcap_read_front :: (path: string, preview := true) -> Mcap, ok: bool, error: string { m: Mcap; file, opened := file_open(path, log_errors = false); if !opened return m, false, "cannot be opened"; @@ -107,17 +112,17 @@ mcap_read_front :: (path: string) -> Mcap, ok: bool, error: string { if !length_ok return m, false, "cannot be read"; wanted := min(length, MCAP_FRONT_READ_SIZE); - for attempt: 1..3 { + for attempt: 1..5 { // each further read reaches at least one more size field buffer := NewArray(wanted, u8, initialized = false); if !file_read(file, buffer.data, wanted) { array_free(buffer); return m, false, "cannot be read"; } d := to_deserializer(buffer); - ok, error := mcap_read_front_part(*d, *m); + ok, error := mcap_read_front_part(*d, *m, preview); if ok return m, true, ""; if d.ok || wanted == length { array_free(buffer); return m, false, error; } // not a matter of too few bytes // Short: the sizes read so far say how much is needed. - needed := mcap_front_size(buffer); + needed := mcap_front_size(buffer, preview); array_free(buffer); if needed <= wanted || needed > length return m, false, "the file ends early (it is truncated or damaged)"; wanted = needed; @@ -131,8 +136,8 @@ mcap_read_front :: (path: string) -> Mcap, ok: bool, error: string { MCAP_HEADER_SIZE :: 12; // magic and version MCAP_FRONT_READ_SIZE :: 64 * 1024; // holds the header, text and preview of nearly every capture -// Header, text and preview. -mcap_read_front_part :: (d: *Deserializer, m: *Mcap) -> ok: bool, error: string { +// Header, text, OCR text and, with preview, the preview. +mcap_read_front_part :: (d: *Deserializer, m: *Mcap, preview := true) -> ok: bool, error: string { magic := ReadArrayView(d, MCAP_MAGIC.count, u8); if !d.ok return false, "is not an .mcap file (it is shorter than the magic)"; for MCAP_MAGIC if magic[it_index] != it return false, "is not an .mcap file (its magic is wrong)"; @@ -142,6 +147,11 @@ mcap_read_front_part :: (d: *Deserializer, m: *Mcap) -> ok: bool, error: string if m.version > MCAP_VERSION return false, tprint("is file version %, but this build reads up to %; it was written by a later version", m.version, MCAP_VERSION); m.text = ReadSizedString(d, u32); + if m.version >= 2 m.ocr = ReadSizedString(d, u32); // version 1 has no OCR block + if !preview { + if !d.ok return false, "the file ends early (it is truncated or damaged)"; + return true, ""; + } m.preview.width = Read(d, u16); m.preview.height = Read(d, u16); m.preview.png = ReadSizedArray(d, u8, u32); @@ -149,18 +159,31 @@ mcap_read_front_part :: (d: *Deserializer, m: *Mcap) -> ok: bool, error: string return true, ""; } -// How many bytes the header, text and preview take, as far as the sizes in prefix tell: with only the -// text's size in it, enough to reach the preview's size; with that too, the whole front. -mcap_front_size :: (prefix: [] u8) -> s64 { +// How many bytes the front takes, as far as the size fields in prefix tell: the whole front when +// they are all in it, otherwise how far to read to reach the next size field. +mcap_front_size :: (prefix: [] u8, preview := true) -> s64 { d := to_deserializer(prefix); - d.cursor = MCAP_HEADER_SIZE; - text_size := cast(s64) Read(*d, u32); - preview_header := MCAP_HEADER_SIZE + 4 + text_size + 4; // where the preview's size is - if !d.ok || preview_header + 4 > prefix.count return preview_header + 4; - d.cursor = preview_header; - return preview_header + 4 + cast(s64) Read(*d, u32); + d.cursor = MCAP_MAGIC.count; + version := Read(*d, u32); + if !d.ok return MCAP_HEADER_SIZE; + position: s64 = MCAP_HEADER_SIZE; + skip_sized :: (prefix: [] u8, position: *s64) -> reached: bool { // a u32 size and what it counts + if position.* + 4 > prefix.count { position.* += 4; return false; } + d := to_deserializer(prefix); + d.cursor = position.*; + position.* += 4 + cast(s64) Read(*d, u32); + return true; + } + if !skip_sized(prefix, *position) return position; // the text + if version >= 2 && !skip_sized(prefix, *position) return position; // the OCR text + if !preview return position; + position += 4; // the preview's width and height + skip_sized(prefix, *position); // the preview + return position; } +#load "ocr_text.jai"; + #import "Basic"; #import "File"; #import "File_Utilities"; diff --git a/modules/Capture_File/ocr_text.jai b/modules/Capture_File/ocr_text.jai new file mode 100644 index 0000000..16addb4 --- /dev/null +++ b/modules/Capture_File/ocr_text.jai @@ -0,0 +1,209 @@ +// OCR text: the OCR block of an .mcap (version 2), in the style of the Ex4 text formats. One section +// per model, so a capture read by several models keeps every reading; reading it again with a model +// replaces only that model's section. +// +// @version 2 +// [ocr ppocr-v5-mobile] +// engine: "onnxruntime 1.30.0, CUDA" +// image: rendered 1920 1080 +// date: 2026-10-08T21:04:11Z +// line: 412 96 640 18 412 96 1052 96 1052 114 412 114 0.993 "Assign Footprints - KiCad" +// +// line: x y width height, the line's four corners as x y pairs, confidence, "text". All in the pixels +// of the image named by image: (rendered or original, at that size). x y width height bound the +// corners; the corners go clockwise from the text's top left, so a turned line says which way it +// reads. Text is quoted, with "" for a quote; a control character is written as a space. A line that +// cannot be read is skipped; keys this version does not know are skipped too. +// +// Version 1 had no corners (line: x y width height confidence "text"); it is still read, its corners +// taken from its box. + +MCAP_OCR_TEXT_VERSION :: 2; + +Mcap_Ocr_Line :: struct { + x, y, width, height: s32; + confidence: float; + text: string; + corners: [8] s32; // x0 y0 x1 y1 x2 y2 x3 y3: top left, top right, bottom right, bottom left of the text; all 0 means the box's +} + +// The line's corners: its own, or its box's when it has none. +mcap_ocr_corners :: (line: Mcap_Ocr_Line) -> [8] s32 { + for line.corners if it return line.corners; + x0, y0, x1, y1 := line.x, line.y, line.x + line.width, line.y + line.height; + return .[x0, y0, x1, y0, x1, y1, x0, y1]; +} + +Mcap_Ocr_Section :: struct { + model: string; // the OCR model's name, as in its models.config + engine: string; // what ran it, such as "onnxruntime 1.30.0, CUDA" + image: string; // "rendered" or "original" + image_width, image_height: s32; + date: string; // when it was read, UTC: 2026-10-08T21:04:11Z + lines: [..] Mcap_Ocr_Line; +} + +// The sections of an OCR block; strings and arrays from the context allocator. "" gives none. +mcap_ocr_parse :: (text: string) -> [..] Mcap_Ocr_Section, ok: bool, error: string { + sections: [..] Mcap_Ocr_Section; + if !text.count return sections, true, ""; + section: *Mcap_Ocr_Section; + version_seen := false; + version: s64; + for raw: split(text, "\n",, temp) { + line := trim(raw); + if !line.count || line[0] == #char "#" continue; + if begins_with(line, "@version") { + rest := trim(advance(line, 8)); + number, ok := string_to_int(rest); + if !ok || number < 1 || number > MCAP_OCR_TEXT_VERSION return sections, false, tprint("its OCR text is version %, but this build reads up to %", rest, MCAP_OCR_TEXT_VERSION); + version = number; + version_seen = true; + continue; + } + if !version_seen return sections, false, "its OCR text does not start with @version"; + + if begins_with(line, "[ocr ") && ends_with(line, "]") { + section = array_add(*sections); + section.* = .{}; + section.model = copy_string(trim(slice(line, 5, line.count - 6))); + continue; + } + colon := find_index_from_left(line, #char ":"); + if colon < 0 || !section continue; + key := trim(slice(line, 0, colon)); + value := trim(advance(line, colon + 1)); + if key == { + case "engine"; + quoted, _, ok := ocr_read_quoted(value); + if ok section.engine = copy_string(quoted); + case "image"; + parts := split(value, " ",, temp); + if parts.count == 3 { + section.image = copy_string(parts[0]); + section.image_width = cast(s32) string_to_int(parts[1]); + section.image_height = cast(s32) string_to_int(parts[2]); + } + case "date"; + section.date = copy_string(value); + case "line"; + ocr_line, ok := ocr_parse_line(value, version); + if ok array_add(*section.lines, ocr_line); + } + } + return sections, true, ""; +} + +// The OCR block for sections. Temp. +mcap_ocr_text :: (sections: [] Mcap_Ocr_Section) -> string { + if !sections.count return ""; + builder: String_Builder; + builder.allocator = temp; + print_to_builder(*builder, "@version %\n", MCAP_OCR_TEXT_VERSION); + for sections { + print_to_builder(*builder, "\n[ocr %]\n", it.model); + print_to_builder(*builder, "engine: %\n", ocr_quote(it.engine)); + print_to_builder(*builder, "image: % % %\n", it.image, it.image_width, it.image_height); + print_to_builder(*builder, "date: %\n", it.date); + for line: it.lines { + c := mcap_ocr_corners(line); + print_to_builder(*builder, "line: % % % % % % % % % % % % % %\n", line.x, line.y, line.width, line.height, + c[0], c[1], c[2], c[3], c[4], c[5], c[6], c[7], formatFloat(line.confidence, trailing_width = 3, zero_removal = .NO), ocr_quote(line.text)); + } + } + return builder_to_string(*builder,, temp); +} + +// Stores section in the .mcap at path, in place of any section by the same model, keeping the +// others, and upgrades the file to the current version. The file is rewritten beside itself and +// swapped in (mcap_write), so a failure leaves it as it was. +mcap_store_ocr :: (path: string, section: Mcap_Ocr_Section) -> ok: bool, error: string { + data, read_ok := read_entire_file(path, log_errors = false); + if !read_ok return false, "cannot be read"; + defer free(data); + m, parsed, parse_error := mcap_parse(cast([] u8) data); + if !parsed return false, parse_error; + + push_allocator(temp); + sections, ok, error := mcap_ocr_parse(m.ocr); + if !ok return false, error; + replaced := false; + for * sections if it.model == section.model { it.* = section; replaced = true; break; } + if !replaced array_add(*sections, section); + m.ocr = mcap_ocr_text(sections); + m.version = MCAP_VERSION; + + serializer, serialized, serialize_error := mcap_serialize(m); + if !serialized return false, serialize_error; + written, write_error := mcap_write(path, serializer); + return written, write_error; +} + +// The time now as the date: key writes it, UTC. Temp. +mcap_ocr_now :: () -> string { + t := to_calendar(current_time_consensus(), .UTC); + two :: (n: s64) -> string { return tprint("%", formatInt(n, minimum_digits = 2)); } + return tprint("%-%-%T%:%:%Z", t.year, two(t.month_starting_at_0 + 1), two(t.day_of_month_starting_at_0 + 1), two(t.hour), two(t.minute), two(t.second)); +} + +#scope_file + +// Quoted, a quote written twice, a control character as a space. Temp. +ocr_quote :: (text: string) -> string { + builder: String_Builder; + builder.allocator = temp; + append(*builder, #char "\""); + for 0..text.count - 1 { + c := text[it]; + if c == #char "\"" append(*builder, "\"\""); + else if c < 0x20 || c == 0x7F append(*builder, #char " "); + else append(*builder, c); + } + append(*builder, #char "\""); + return builder_to_string(*builder,, temp); +} + +// One quoted string ("" for a quote) at the start of s: its text (context allocator) and what follows. +ocr_read_quoted :: (s: string) -> text: string, rest: string, ok: bool { + if !s.count || s[0] != #char "\"" return "", s, false; + builder: String_Builder; + i := 1; + while i < s.count { + if s[i] == #char "\"" { + if i + 1 < s.count && s[i + 1] == #char "\"" { append(*builder, #char "\""); i += 2; continue; } + return builder_to_string(*builder), advance(s, i + 1), true; + } + append(*builder, s[i]); + i += 1; + } + free_buffers(*builder); + return "", s, false; +} + +// Version 2: x y width height x0 y0 x1 y1 x2 y2 x3 y3 confidence "text". Version 1: x y width height confidence "text". +ocr_parse_line :: (value: string, version: s64) -> Mcap_Ocr_Line, bool { + result: Mcap_Ocr_Line; + rest := value; + all_numbers: [13] float64; + numbers := array_view(all_numbers, 0, ifx version >= 2 then 13 else 5); + for * numbers { + rest = trim_left(rest); + space := find_index_from_left(rest, #char " "); + if space < 0 return result, false; + number, ok := string_to_float64(slice(rest, 0, space)); + if !ok return result, false; + it.* = number; + rest = advance(rest, space); + } + text, after, ok := ocr_read_quoted(trim_left(rest)); + if !ok || trim(after).count return result, false; + result.x, result.y = cast(s32) numbers[0], cast(s32) numbers[1]; + result.width, result.height = cast(s32) numbers[2], cast(s32) numbers[3]; + result.confidence = cast(float) numbers[numbers.count - 1]; + result.text = text; + if numbers.count == 13 for 0..7 result.corners[it] = cast(s32) numbers[4 + it]; + result.corners = mcap_ocr_corners(result); + return result, true; +} + +#import "String"; diff --git a/modules/Capture_Text/module.jai b/modules/Capture_Text/module.jai new file mode 100644 index 0000000..679ff8d --- /dev/null +++ b/modules/Capture_Text/module.jai @@ -0,0 +1,746 @@ +// Capture Text: the text format for a capture's structured data, the JSON files of a .snagx in +// the style of the Ex4 text formats. See docs/capture_text.md for the format itself. +// +// A Capture_Document allocates everything through the allocator it is given, and frees nothing on +// its own: give it an arena or a pool, and drop that to drop the document. + +CAPTURE_TEXT_VERSION :: 1; +CAPTURE_TEXT_EXTENSION :: "capture-text"; + +Capture_Value_Kind :: enum u8 { NULL; BOOLEAN; NUMBER; STRING; ARRAY; OBJECT; } + +Capture_Value :: struct { + kind: Capture_Value_Kind; + boolean: bool; + number: float64; + text: string; + items: [..] Capture_Value; // ARRAY + fields: [..] Capture_Field; // OBJECT, in the order they were added +} + +Capture_Field :: struct { + key: string; + value: Capture_Value; +} + +// One [[file "NAME"]] section: one of the capture's JSON files. +Capture_File :: struct { + name: string; + root: Capture_Value; // always an OBJECT +} + +Capture_Document :: struct { + allocator: Allocator; + files: [..] Capture_File; + comment: string; // written as # lines under @version; not read back + mistakes: s64; // lines skipped by the last parse +} + +capture_document_init :: (doc: *Capture_Document, allocator: Allocator) { + doc.* = .{}; + doc.allocator = allocator; + doc.files.allocator = allocator; +} + +capture_add_file :: (doc: *Capture_Document, name: string) -> *Capture_File { + file := array_add(*doc.files); + file.name = copy_string(name,, doc.allocator); + file.root = capture_object(doc); + return file; +} + +capture_find_file :: (doc: *Capture_Document, name: string) -> *Capture_File { + for * doc.files if it.name == name return it; + return null; +} + +// ---------------------------------------------------------------- values + +capture_null :: () -> Capture_Value { + return .{kind = .NULL}; +} + +capture_boolean :: (b: bool) -> Capture_Value { + return .{kind = .BOOLEAN, boolean = b}; +} + +capture_number :: (n: float64) -> Capture_Value { + return .{kind = .NUMBER, number = n}; +} + +capture_string :: (doc: *Capture_Document, text: string) -> Capture_Value { + return .{kind = .STRING, text = copy_string(text,, doc.allocator)}; +} + +capture_array :: (doc: *Capture_Document) -> Capture_Value { + result: Capture_Value; + result.kind = .ARRAY; + result.items.allocator = doc.allocator; + return result; +} + +capture_object :: (doc: *Capture_Document) -> Capture_Value { + result: Capture_Value; + result.kind = .OBJECT; + result.fields.allocator = doc.allocator; + return result; +} + +// Adds key to object without checking whether it is already there; the key is copied. +capture_object_add :: (doc: *Capture_Document, object: *Capture_Value, key: string, value: Capture_Value) -> *Capture_Value { + assert(object.kind == .OBJECT); + field := array_add(*object.fields); + field.key = copy_string(key,, doc.allocator); + field.value = value; + return *field.value; +} + +capture_object_find :: (object: *Capture_Value, key: string) -> *Capture_Value { + if object.kind != .OBJECT return null; + for * object.fields if it.key == key return *it.value; + return null; +} + +// Equal as JSON values: the order of an object's keys does not matter, the order of an array's +// elements does. Numbers are compared bit for bit, so -0 and 0 differ. +capture_values_equal :: (a: Capture_Value, b: Capture_Value) -> bool { + if a.kind != b.kind return false; + if #complete a.kind == { + case .NULL; return true; + case .BOOLEAN; return a.boolean == b.boolean; + case .NUMBER; return (cast(*u64) *a.number).* == (cast(*u64) *b.number).*; + case .STRING; return a.text == b.text; + case .ARRAY; + if a.items.count != b.items.count return false; + for a.items if !capture_values_equal(it, b.items[it_index]) return false; + return true; + case .OBJECT; + if a.fields.count != b.fields.count return false; + for a.fields { + other := capture_object_find(*b, it.key); + if !other || !capture_values_equal(it.value, other.*) return false; + } + return true; + } + return false; +} + +// ---------------------------------------------------------------- reading + +// Reads file_data into doc, which should be empty. A bad line, or a bad [[file]] with its lines, is +// logged and skipped, and counted in doc.mistakes. Returns false when the file was not read at all: +// no @version 1 before the first line that needs it. +capture_text_parse :: (doc: *Capture_Document, path: string, file_data: string) -> bool { + push_allocator(doc.allocator); + doc.mistakes = 0; + + data := file_data; + if data.count >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF data = advance(data, 3); + + state: Capture_Parse_State; + state.doc = doc; + state.path = path; + + // Every line stands alone: a value never continues onto the next line. + offset: s64; + line_number := 0; + while offset < data.count { + line_number += 1; + line, colon, valid := capture_scan_line(data, *offset); + if !valid { capture_mistake(*state, line_number, "Quote is unterminated. Ignored."); continue; } + body := trim(line); + if !body.count continue; + if colon >= 0 colon -= body.data - line.data; // from the start of body + + if body[0] == #char "@" { + key, value := capture_split_directive(body); + if key == "version" { + rest := value; + version, version_ok := parse_int(*rest, s64); + if !version_ok || trim(rest).count || version < 1 || version > CAPTURE_TEXT_VERSION { + log_error("% is version %, but this build reads version %. It was not read.", path, value, CAPTURE_TEXT_VERSION); + return false; + } + state.version_seen = true; + } + continue; + } + + if !state.version_seen { + log_error("% does not start with @version %. It was not read.", path, CAPTURE_TEXT_VERSION); + return false; + } + + if begins_with(body, "[[") { + capture_open_file(*state, line_number, body); + } else if body[0] == #char "[" { + capture_open_section(*state, line_number, body); + } else { + capture_key_line(*state, line_number, body, colon); + } + } + return true; +} + +// ---------------------------------------------------------------- writing + +CAPTURE_MAX_KEY_WIDTH :: 40; // keys longer than this do not push the other values further right + +// The document as its file holds it. Temp. +capture_text_write :: (doc: *Capture_Document) -> string { + builder: String_Builder; + builder.allocator = temp; + print_to_builder(*builder, "@version %\n", CAPTURE_TEXT_VERSION); + if doc.comment.count { + for split(doc.comment, "\n",, temp) print_to_builder(*builder, "# %\n", it); + } + + for file: doc.files { + print_to_builder(*builder, "\n[[file %]]\n", capture_quote(file.name)); + + lines: [..] Capture_Line; + lines.allocator = temp; + for file.root.fields { + if capture_is_section_array(it.value) continue; + capture_collect_lines(*lines, capture_key_text(it.key), it.value); + } + capture_write_block(*builder, lines); + + for field: file.root.fields { + if !capture_is_section_array(field.value) continue; + header := capture_key_text(field.key); + for item: field.value.items { + print_to_builder(*builder, "\n[%]\n", header); + array_reset_keeping_memory(*lines); + for item.fields capture_collect_lines(*lines, capture_key_text(it.key), it.value); + capture_write_block(*builder, lines); + } + } + } + return builder_to_string(*builder,, temp); +} + +// Always quoted, a quote written twice. Control characters cannot be quoted; see capture_string_parts. +capture_quote :: (text: string) -> string { + return tprint("\"%\"", replace(text, "\"", "\"\"",, temp)); +} + +// A key as one segment of a key path: bare when it can be, quoted otherwise. A key cannot hold +// control characters (see capture_key_is_writable). Temp. +capture_key_text :: (key: string) -> string { + if capture_is_bare_key(key) return key; + return capture_quote(key); +} + +// Whether a key can be written at all: a quoted key is one part, so it cannot hold the characters +// below space, or DEL, that a string writes as 0xNN bytes. +capture_key_is_writable :: (key: string) -> bool { + for 0..key.count - 1 if key[it] < 0x20 || key[it] == 0x7F return false; + return true; +} + +// The shortest text that reads back as exactly n. Temp. +capture_number_text :: (n: float64) -> string { + bits := (cast(*u64) *n).*; + if n == 0 return ifx bits >> 63 then "-0" else "0"; + // Whole numbers that a float64 holds exactly are written as integers: 100, not 100.0 or 1e+02. + if n > -9007199254740992.0 && n < 9007199254740992.0 && n == cast(float64) cast,trunc(s64) n { + return tprint("%", cast,trunc(s64) n); + } + + // Basic's SHORTEST mode keeps only six significant digits, so the fewest digits that read back as + // exactly n are found by trying: plain decimals for ordinary magnitudes, an exponent otherwise. + // 17 significant digits always suffice for a float64. + magnitude := ifx n < 0 then -n else n; + if magnitude >= 0.000001 && magnitude < 1000000000000000000000.0 { + for digits: 1..24 { + text := trim(tprint("%", formatFloat(n, trailing_width = digits, zero_removal = .YES))); + if capture_number_reads_back(text, bits) return text; + } + } + for digits: 0..16 { + text := trim(tprint("%", formatFloat(n, trailing_width = digits, mode = .SCIENTIFIC, zero_removal = .YES))); + if capture_number_reads_back(text, bits) return text; + } + return tprint("%", formatFloat(n, trailing_width = 16, mode = .SCIENTIFIC, zero_removal = .NO)); +} + +capture_number_reads_back :: (text: string, bits: u64) -> bool { + if capture_number_length(text) != text.count return false; + value, ok, rest := string_to_float64(text); + return ok && !rest.count && (cast(*u64) *value).* == bits; +} + +#scope_module + +// ---------------------------------------------------------------- reading, in detail + +Capture_Parse_State :: struct { + doc: *Capture_Document; + path: string; + version_seen: bool; + + file: *Capture_File; // the [[file]] being read; null before the first, or after a bad one + base: *Capture_Value; // the object key lines go into: the file's, or a [KEY] section's + skipping: bool; // after a bad header, until the next good one +} + +MAX_NESTING :: 64; + +capture_mistake :: (state: *Capture_Parse_State, line_number: s64, format: string, args: ..Any) { + log_error("%:%: %", state.path, line_number, tprint(format, ..args)); + state.doc.mistakes += 1; +} @PrintLike + +capture_open_file :: (state: *Capture_Parse_State, line_number: s64, body: string) { + state.file = null; + state.base = null; + state.skipping = true; + + if !ends_with(body, "]]") { capture_mistake(state, line_number, "Malformed header (expected `[[file \"NAME\"]]`); its lines are skipped."); return; } + inner := trim(slice(body, 2, body.count - 4)); + if !begins_with(inner, "file") || inner.count < 5 || !is_space(inner[4]) { + capture_mistake(state, line_number, "Unknown header % (expected `[[file \"NAME\"]]`); its lines are skipped.", body); + return; + } + name_text := trim(advance(inner, 4)); + name, rest, ok := capture_read_quoted(name_text); + if !ok || trim(rest).count { capture_mistake(state, line_number, "The file name in % must be one quoted string; its lines are skipped.", body); return; } + if capture_find_file(state.doc, name) { capture_mistake(state, line_number, "File \"%\" was already read; this section is skipped.", name); return; } + + state.file = capture_add_file(state.doc, name); + state.base = *state.file.root; + state.skipping = false; +} + +capture_open_section :: (state: *Capture_Parse_State, line_number: s64, body: string) { + state.base = null; + if !state.file { capture_mistake(state, line_number, "% comes before any [[file]] section; its lines are skipped.", body); state.skipping = true; return; } + state.skipping = true; + + if body[body.count - 1] != #char "]" { capture_mistake(state, line_number, "Malformed header (expected `[KEY]`); its lines are skipped."); return; } + segments, ok := capture_parse_path(trim(slice(body, 1, body.count - 2))); + if !ok { capture_mistake(state, line_number, "% is not a key path; its lines are skipped.", body); return; } + + array, error := capture_walk(state.doc, *state.file.root, segments, .ARRAY); + if !array { capture_mistake(state, line_number, "%: %; its lines are skipped.", body, error); return; } + + element := array_add(*array.items); + element.* = capture_object(state.doc); + state.base = element; + state.skipping = false; +} + +// colon: where in body the first : outside quotes is, or -1. +capture_key_line :: (state: *Capture_Parse_State, line_number: s64, body: string, colon: s64) { + if state.skipping return; + if !state.base { capture_mistake(state, line_number, "Key line before any [[file]] section. Ignored."); return; } + + if colon < 0 { capture_mistake(state, line_number, "Expected `KEY: VALUE`. Ignored."); return; } + key_text := trim(slice(body, 0, colon)); + value_text := trim(advance(body, colon + 1)); + + segments, path_ok := capture_parse_path(key_text); + if !path_ok { capture_mistake(state, line_number, "% is not a key path. Ignored.", key_text); return; } + + rest := value_text; + value, value_ok := capture_parse_value(state.doc, *rest, 0); + if !value_ok || trim(rest).count { capture_mistake(state, line_number, "% has no value it can read in `%`. Ignored.", key_text, value_text); return; } + + slot, error := capture_walk(state.doc, state.base, segments, value.kind, value_must_be_new = true); + if !slot { capture_mistake(state, line_number, "%: %. Ignored.", key_text, error); return; } + slot.* = value; +} + +Capture_Path_Segment :: struct { + key: string; + index := -1; // an array index when 0 or more +} + +// Key path: segments separated by dots, each a bare name, a quoted key, or digits for an array index. +capture_parse_path :: (text: string) -> [..] Capture_Path_Segment, bool { + segments: [..] Capture_Path_Segment; + segments.allocator = temp; + rest := text; + while true { + segment: Capture_Path_Segment; + if rest.count && rest[0] == #char "\"" { + key, after, ok := capture_read_quoted(rest); + if !ok return segments, false; + segment.key = key; + rest = after; + } else { + i := 0; + while i < rest.count && capture_is_bare_char(rest[i]) i += 1; + if i == 0 return segments, false; + word := slice(rest, 0, i); + rest = advance(rest, i); + if capture_is_digits(word) { + if word.count > 9 return segments, false; // no array is that long + index_text := word; + segment.index = parse_int(*index_text, s64); + } else { + segment.key = word; + } + } + array_add(*segments, segment); + if !rest.count return segments, true; + if rest[0] != #char "." return segments, false; + rest = advance(rest, 1); + } + return segments, false; +} + +// Follows segments down from node, making the objects and arrays the path needs, and returns the +// slot at the end: a new one for a key line, which must not have a value yet, or for a [KEY] header +// the array that is there or a new one. On failure returns null and why. +capture_walk :: (doc: *Capture_Document, node: *Capture_Value, segments: [] Capture_Path_Segment, final_kind: Capture_Value_Kind, value_must_be_new := false) -> *Capture_Value, string { + current := node; + for segment, segment_index: segments { + last := segment_index == segments.count - 1; + // What a new slot here has to be: what the next segment needs, or what is being put at the end. + kind := final_kind; + if !last kind = ifx segments[segment_index + 1].index >= 0 then Capture_Value_Kind.ARRAY else .OBJECT; + + child: *Capture_Value; + is_new := false; + if segment.index >= 0 { + if current.kind != .ARRAY return null, tprint("index % of something that is not an array", segment.index); + if segment.index > current.items.count return null, tprint("index % skips past the next free index %", segment.index, current.items.count); + if segment.index == current.items.count { + child = array_add(*current.items); + is_new = true; + } else { + child = *current.items[segment.index]; + } + } else { + if current.kind != .OBJECT return null, tprint("key \"%\" of something that is not an object", segment.key); + child = capture_object_find(current, segment.key); + if !child { + child = capture_object_add(doc, current, segment.key, .{}); + is_new = true; + } + } + + if is_new { + if kind == .ARRAY child.* = capture_array(doc); + else if kind == .OBJECT child.* = capture_object(doc); + } else if last && value_must_be_new { + return null, "already has a value"; + } else if child.kind != kind { + return null, tprint("already holds % where % is needed", capture_kind_name(child.kind), capture_kind_name(kind)); + } + current = child; + } + return current, ""; +} + +capture_kind_name :: (kind: Capture_Value_Kind) -> string { + if #complete kind == { + case .NULL; return "null"; + case .BOOLEAN; return "a boolean"; + case .NUMBER; return "a number"; + case .STRING; return "a string"; + case .ARRAY; return "an array"; + case .OBJECT; return "an object"; + } + return ""; +} + +capture_parse_value :: (doc: *Capture_Document, s: *string, depth: s64) -> Capture_Value, bool { + fail: Capture_Value; + s.* = trim_left(s.*); + if !s.count || depth > MAX_NESTING return fail, false; + + c := s.*[0]; + if c == #char "\"" { + first_part, rest, ok := capture_read_quoted(s.*); + if !ok return fail, false; + s.* = rest; + // Nearly every string is one part, copied straight into the document. Further parts are joined in a builder. + builder: String_Builder; + builder.allocator = temp; + parts := 1; + while true { + after := trim_left(s.*, " \t"); // spaces or tabs between parts + if after.count && after[0] == #char "\"" { + text, rest, ok := capture_read_quoted(after); + if !ok return fail, false; + if parts == 1 append(*builder, first_part); + append(*builder, text); + s.* = rest; + } else if after.count >= 4 && after[0] == #char "0" && after[1] == #char "x" { + high := capture_hex_digit(after[2]); + low := capture_hex_digit(after[3]); + if high < 0 || low < 0 return fail, false; + if after.count > 4 && !capture_ends_token(after[4]) return fail, false; + if parts == 1 append(*builder, first_part); + append(*builder, cast(u8) (high * 16 + low)); + s.* = advance(after, 4); + } else { + break; + } + parts += 1; + } + if parts == 1 return capture_string(doc, first_part), true; + return capture_string(doc, builder_to_string(*builder,, temp)), true; + } + + if c == #char "[" { + result := capture_array(doc); + s.* = trim_left(advance(s.*, 1)); + if s.count && s.*[0] == #char "]" { s.* = advance(s.*, 1); return result, true; } + while true { + item, ok := capture_parse_value(doc, s, depth + 1); + if !ok return fail, false; + array_add(*result.items, item); + s.* = trim_left(s.*); + if !s.count return fail, false; + if s.*[0] == #char "]" { s.* = advance(s.*, 1); return result, true; } + if s.*[0] != #char "," return fail, false; + s.* = advance(s.*, 1); + } + } + + if c == #char "{" { + rest := trim_left(advance(s.*, 1)); + if !rest.count || rest[0] != #char "}" return fail, false; // only {} is written inline + s.* = advance(rest, 1); + return capture_object(doc), true; + } + + if c == #char "-" || (c >= #char "0" && c <= #char "9") { + length := capture_number_length(s.*); + if length <= 0 || (length < s.count && !capture_ends_token(s.*[length])) return fail, false; + n, ok, remainder := string_to_float64(slice(s.*, 0, length)); + if !ok || remainder.count return fail, false; + s.* = advance(s.*, length); + return capture_number(n), true; + } + + for word: string.["null", "true", "false"] { + if !begins_with(s.*, word) continue; + if s.count > word.count && !capture_ends_token(s.*[word.count]) continue; + s.* = advance(s.*, word.count); + if word == "null" return capture_null(), true; + return capture_boolean(word == "true"), true; + } + return fail, false; +} + +// The length of a JSON number at the start of s: -?digits(.digits)?([eE][+-]?digits)?. 0 when there is none. +capture_number_length :: (s: string) -> s64 { + i := 0; + if i < s.count && s[i] == #char "-" i += 1; + digits_start := i; + while i < s.count && s[i] >= #char "0" && s[i] <= #char "9" i += 1; + if i == digits_start return 0; + if i < s.count && s[i] == #char "." { + i += 1; + fraction_start := i; + while i < s.count && s[i] >= #char "0" && s[i] <= #char "9" i += 1; + if i == fraction_start return 0; + } + if i < s.count && (s[i] == #char "e" || s[i] == #char "E") { + i += 1; + if i < s.count && (s[i] == #char "+" || s[i] == #char "-") i += 1; + exponent_start := i; + while i < s.count && s[i] >= #char "0" && s[i] <= #char "9" i += 1; + if i == exponent_start return 0; + } + return i; +} + +capture_ends_token :: (c: u8) -> bool { + return is_space(c) || c == #char "," || c == #char "]"; +} + +capture_hex_digit :: (c: u8) -> s64 { + if c >= #char "0" && c <= #char "9" return c - #char "0"; + if c >= #char "A" && c <= #char "F" return c - #char "A" + 10; + if c >= #char "a" && c <= #char "f" return c - #char "a" + 10; + return -1; +} + +// One quoted string ("" for a quote) at the start of s. Returns the text and what follows. The text is +// a slice of s when it holds no doubled quote, as nearly every string does; otherwise it is temp. +capture_read_quoted :: (s: string) -> text: string, rest: string, ok: bool { + if !s.count || s[0] != #char "\"" return "", s, false; + i := 1; + while i < s.count && s[i] != #char "\"" i += 1; + if i >= s.count return "", s, false; + if i + 1 >= s.count || s[i + 1] != #char "\"" return slice(s, 1, i - 1), advance(s, i + 1), true; + + // A doubled quote: the text has to be put together. + builder: String_Builder; + builder.allocator = temp; + run_start := 1; + while i < s.count { + if s[i] == #char "\"" { + append(*builder, slice(s, run_start, i - run_start)); + if i + 1 < s.count && s[i + 1] == #char "\"" { + append(*builder, #char "\""); + i += 2; + run_start = i; + continue; + } + return builder_to_string(*builder,, temp), advance(s, i + 1), true; + } + i += 1; + } + return "", s, false; +} + +// One line, from offset, in a single pass: the line without its comment or line ending, and where in it +// the first : outside quotes is (-1 when there is none). "#" opens a comment only outside quotes, +// and only at the start of the line or after whitespace. valid is false when the line ends inside quotes. +capture_scan_line :: (data: string, offset: *s64) -> line: string, colon: s64, valid: bool { + start := offset.*; + i := start; + colon := -1; + end := -1; // where a comment starts + while i < data.count { + c := data[i]; + if c == #char "\"" { + // Straight to the closing quote: strings are most of the bytes. A doubled quote closes and opens. + i += 1; + while i < data.count && data[i] != #char "\"" && data[i] != #char "\n" i += 1; + if i >= data.count || data[i] == #char "\n" { + while i < data.count && data[i] != #char "\n" i += 1; + offset.* = ifx i < data.count then i + 1 else data.count; + return "", -1, false; + } + } else if c == #char "\n" { + break; + } else if c == #char "#" && (i == start || is_space(data[i - 1])) { + end = i; + while i < data.count && data[i] != #char "\n" i += 1; + break; + } else if c == #char ":" && colon < 0 { + colon = i - start; + } + i += 1; + } + offset.* = ifx i < data.count then i + 1 else data.count; + if end < 0 { + end = i; + if end > start && data[end - 1] == #char "\r" end -= 1; // CRLF + } + return string.{end - start, data.data + start}, colon, true; +} + +capture_split_directive :: (body: string) -> key: string, value: string { + rest := advance(body, 1); // skip the @ + i := 0; + while i < rest.count && !is_space(rest[i]) i += 1; + return slice(rest, 0, i), trim(advance(rest, i)); +} + +capture_is_bare_char :: (c: u8) -> bool { + return (c >= #char "a" && c <= #char "z") || (c >= #char "A" && c <= #char "Z") || (c >= #char "0" && c <= #char "9") || c == #char "_" || c == #char "-"; +} + +capture_is_digits :: (s: string) -> bool { + if !s.count return false; + for 0..s.count - 1 if s[it] < #char "0" || s[it] > #char "9" return false; + return true; +} + +capture_is_bare_key :: (key: string) -> bool { + if !key.count || capture_is_digits(key) return false; + for 0..key.count - 1 if !capture_is_bare_char(key[it]) return false; + return true; +} + +// ---------------------------------------------------------------- writing, in detail + +Capture_Line :: struct { + path: string; + value: Capture_Value; +} + +// A root array of objects is written as [KEY] sections, one per object. +capture_is_section_array :: (value: Capture_Value) -> bool { + if value.kind != .ARRAY || !value.items.count return false; + for value.items if it.kind != .OBJECT return false; + return true; +} + +// An array that holds an object with keys, at any depth, cannot be written inline. +capture_needs_paths :: (value: Capture_Value) -> bool { + if value.kind == .OBJECT return value.fields.count > 0; + if value.kind == .ARRAY { + for value.items if capture_needs_paths(it) return true; + } + return false; +} + +capture_collect_lines :: (lines: *[..] Capture_Line, path: string, value: Capture_Value) { + if capture_needs_paths(value) { + if value.kind == .OBJECT { + for value.fields capture_collect_lines(lines, tprint("%.%", path, capture_key_text(it.key)), it.value); + } else { + for value.items capture_collect_lines(lines, tprint("%.%", path, it_index), it); + } + return; + } + array_add(lines, .{path, value}); +} + +// Key lines with their values lined up, each on one line however long. +capture_write_block :: (builder: *String_Builder, lines: [] Capture_Line) { + key_width := 0; + for lines if it.path.count + 1 <= CAPTURE_MAX_KEY_WIDTH key_width = max(key_width, it.path.count + 1); + + for lines { + head := tprint("%:", it.path); + append(builder, head); + for head.count..max(head.count, key_width) append(builder, #char " "); + capture_append_value(builder, it.value); + append(builder, #char "\n"); + } +} + +capture_append_value :: (builder: *String_Builder, value: Capture_Value) { + if #complete value.kind == { + case .NULL; append(builder, "null"); + case .BOOLEAN; append(builder, ifx value.boolean then "true" else "false"); + case .NUMBER; append(builder, capture_number_text(value.number)); + case .STRING; append(builder, capture_string_text(value.text)); + case .OBJECT; append(builder, "{}"); // only an empty one reaches here + case .ARRAY; + append(builder, #char "["); + for value.items { + if it_index > 0 append(builder, ", "); + capture_append_value(builder, it); + } + append(builder, #char "]"); + } +} + +// A string on one line: quoted, with control characters and DEL, which quotes cannot hold, written +// as 0xNN parts between quoted ones. Temp. +capture_string_text :: (text: string) -> string { + if !text.count return "\"\""; + builder: String_Builder; + builder.allocator = temp; + i := 0; + while i < text.count { + if i > 0 append(*builder, #char " "); + c := text[i]; + if c < 0x20 || c == 0x7F { + if i == 0 append(*builder, "\"\" "); // the first part is always quoted + print_to_builder(*builder, "0x%", formatInt(c, base = 16, minimum_digits = 2)); + i += 1; + continue; + } + end := i; + while end < text.count && text[end] >= 0x20 && text[end] != 0x7F end += 1; + append(*builder, capture_quote(slice(text, i, end - i))); + i = end; + } + return builder_to_string(*builder,, temp); +} + +#import "Basic"; +#import "String"; diff --git a/modules/OCR/README.md b/modules/OCR/README.md index e76cc3d..5f7406f 100644 --- a/modules/OCR/README.md +++ b/modules/OCR/README.md @@ -1,7 +1,8 @@ # OCR Text in images, with where each line is: a batch of RGBA images in, each image's text lines out, -each with a box and a confidence. It runs ONNX models through onnxruntime +each with its corners, its box and a confidence. Lines at an angle, read downwards or upwards, or +upside down are read too. It runs ONNX models through onnxruntime ([`modules/onnxruntime`](../onnxruntime)): on an NVIDIA GPU through CUDA when every library that needs is present, and on the CPU otherwise. @@ -11,7 +12,7 @@ The binaries and the models are downloaded, not committed: ``` cd modules/onnxruntime && jai fetch.jai # onnxruntime 1.30.0, GPU build (CUDA 13); `- cpu` for the CPU-only build -cd modules/OCR && jai fetch_models.jai # both models (about 195 MB); `- mobile` for the light one only +cd modules/OCR && jai fetch_models.jai # both models (about 196 MB); `- mobile` for the light one only jai generate.jai # in modules/onnxruntime, only when changing onnxruntime versions ``` @@ -31,7 +32,16 @@ ocr_destroy(engine); ``` `ocr_process` shares a batch out over worker threads: detection per image, then recognition of -every line of every image in batches of similar width. The results come from the context +every line of every image in batches of similar width. + +**Turned text.** Detection gives each line the smallest rectangle around it at any angle (rotating +calipers over its convex hull), keeping a level one unless a turned one is clearly smaller; a level +box takes in every pixel it touches, as recognition reads best. Recognition samples each rectangle +along its own sides into a level crop. A box at least 1.5 times as tall as it is wide is taken as a +line read downwards, and also read level, since a narrow word such as "1" is as tall. The +orientation classifier marks lines that look upside down, and those are read both ways. Of each +line's readings, the more confident is kept. `Ocr_Line.corners` go clockwise from the text's top +left, so a turned line says which way it reads. The results come from the context allocator, or the one passed. `ocr_stats` gives the time spent in each stage. ## Models @@ -39,10 +49,13 @@ allocator, or the one passed. `ocr_stats` gives the time spent in each stage. `models.config` lists them; `ocr_create` takes a name. Both are PaddleOCR's PP-OCRv5 (Apache 2.0), as converted to ONNX by RapidOCR, and read Chinese, English and Japanese: -| Name | Files | On the 20 sample captures, CPU | +| Name | Files | On the 20 sample captures: RTX 5080 laptop / CPU | | --- | --- | --- | -| `ppocr-v5-mobile` | detector 4.8 MB, recognizer 16.6 MB | about 0.5 s per capture | -| `ppocr-v5-server` | detector 88 MB, recognizer 85 MB | about 2.5 s per capture | +| `ppocr-v5-mobile` | detector 4.8 MB, recognizer 16.6 MB | 0.13 s / 0.5 to 0.65 s per capture | +| `ppocr-v5-server` | detector 88 MB, recognizer 85 MB | 2.3 s / 2.5 s per capture (0.36 s on the GPU with `rec_batch: 4`) | + +Both also use the text line orientation classifier `ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx` +(1 MB, `textline_ori:` in `models.config`). Another model with the same two-stage design is a new `[model NAME]` section; a different design needs its own detection or recognition code beside `detect.jai` and `recognize.jai`. @@ -55,7 +68,9 @@ onnxruntime's CUDA build needs, on PATH or beside the executable: - the CUDA Toolkit 13.x (`cudart64_13`, `cublas64_13`, `cublasLt64_13`, `cufft64_12`), from [developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads), whose installer puts its bin folder on PATH; - cuDNN 9.x for CUDA 13 (`cudnn64_9` and its `cudnn_*64_9` companions), from - [developer.nvidia.com/cudnn-downloads](https://developer.nvidia.com/cudnn-downloads), whose bin folder for CUDA 13 you add to PATH yourself. + [developer.nvidia.com/cudnn-downloads](https://developer.nvidia.com/cudnn-downloads). Its installer does not touch PATH; + when cuDNN is not on it, the newest `C:\Program Files\NVIDIA\CUDNN\v9.*\bin\13.*\x64` is added to the + program's own PATH, so an install in the usual place needs nothing more. `ocr_create` looks for each of these first (`cuda_check.jai`) and says which are missing, rather than letting onnxruntime fail later and obscurely. If they are all there but CUDA still cannot diff --git a/modules/OCR/cuda_check.jai b/modules/OCR/cuda_check.jai index 8b47288..b0823bc 100644 --- a/modules/OCR/cuda_check.jai +++ b/modules/OCR/cuda_check.jai @@ -2,6 +2,10 @@ // The provider needs the NVIDIA driver, parts of the CUDA 13 toolkit and cuDNN 9 for CUDA 13; it // fails late and obscurely without them, so each library is looked for first, by the name the // provider loads it by (from onnxruntime_providers_cuda.dll's imports and the names it loads). +// +// cuDNN's Windows installer does not put its DLLs on PATH. When they are not found, the newest +// cuDNN 9 for CUDA 13 under Program Files\NVIDIA\CUDNN is added to this process's PATH, so the +// provider finds it without anyone editing PATH. Cuda_Library :: struct { name: string; @@ -53,6 +57,8 @@ ocr_cuda_check :: () -> available: bool, explanation: string { return false, tprint("OCR runs on the CPU: the onnxruntime beside the program is the CPU-only build (% is not there).\nTo use an NVIDIA GPU, run `jai fetch.jai` in modules/onnxruntime (without `cpu`) and rebuild.", CUDA_PROVIDER_LIBRARY); } + #if OS == .WINDOWS ocr_add_cudnn_to_path(); + missing_driver, missing_toolkit, missing_cudnn: [..] string; missing_driver.allocator, missing_toolkit.allocator, missing_cudnn.allocator = temp, temp, temp; for CUDA_LIBRARIES { @@ -87,8 +93,8 @@ ocr_cuda_check :: () -> available: bool, explanation: string { print_to_builder(*builder, "\n%. cuDNN 9.x for CUDA 13, which has % that are missing.\n", step, join(..missing_cudnn, ", ",, temp)); append(*builder, " https://developer.nvidia.com/cudnn-downloads\n"); #if OS == .WINDOWS { - append(*builder, " Its installer does not change PATH: add the folder with these DLLs to it yourself,\n"); - append(*builder, " such as C:\\Program Files\\NVIDIA\\CUDNN\\v9.x\\bin\\13.x\\x64.\n"); + append(*builder, " Its installer does not change PATH. Installed in its usual place, C:\\Program Files\\NVIDIA\\CUDNN,\n"); + append(*builder, " it is found there; elsewhere, add the folder with these DLLs to PATH yourself.\n"); } else { append(*builder, " Then make sure its lib folder is on LD_LIBRARY_PATH.\n"); } @@ -115,3 +121,75 @@ ocr_library_loads :: (name: string) -> bool { return true; } } + +#if OS == .WINDOWS { + // When cuDNN does not load, puts the newest Program Files\NVIDIA\CUDNN\v9.*\bin\13.*\x64 (or + // bin\13.*, as older releases lay it out) holding cudnn64_9.dll at the front of this process's PATH. + ocr_add_cudnn_to_path :: () { + if ocr_library_loads("cudnn64_9.dll") return; + program_files := ocr_environment_variable("ProgramW6432"); + if !program_files program_files = ocr_environment_variable("ProgramFiles"); + if !program_files return; + + versions := ocr_subfolders_newest_first(tprint("%\\NVIDIA\\CUDNN", program_files), "v9."); + for version: versions { + for cuda: ocr_subfolders_newest_first(tprint("%\\bin", version), "13.") { + for folder: string.[tprint("%\\x64", cuda), cuda] { + if !file_exists(tprint("%\\cudnn64_9.dll", folder)) continue; + found := replace(folder, "/", "\\",, temp); + path := ocr_environment_variable("PATH"); + OcrSetEnvironmentVariableW(utf8_to_wide("PATH",, temp), utf8_to_wide(tprint("%;%", found, path),, temp)); + log("[OCR] cuDNN is not on PATH; using the one in %.", found); + return; + } + } + } + } + + // The folders in parent whose names start with prefix, the highest version first ("v9.26" above + // "v9.3"). Temp. + ocr_subfolders_newest_first :: (parent: string, prefix: string) -> [..] string { + Found :: struct { names: [..] string; prefix: string; } + found: Found; + found.names.allocator = temp; + found.prefix = prefix; + visit_files(parent, recursive = false, *found, (info: *File_Visit_Info, found: *Found) { + if begins_with(info.short_name, found.prefix) array_add(*found.names, copy_string(info.full_name,, temp)); + }, visit_files = false, visit_directories = true); + quick_sort(found.names, (a: string, b: string) -> s64 { return ocr_compare_versions(path_filename(b), path_filename(a)); }); + return found.names; + } + + // Compares the numbers in two names one by one: "v9.26" > "v9.3". + ocr_compare_versions :: (a: string, b: string) -> s64 { + next_number :: (s: *string) -> s64, found: bool { + while s.count && (s.*[0] < #char "0" || s.*[0] > #char "9") advance(s); + if !s.count return 0, false; + n: s64; + while s.count && s.*[0] >= #char "0" && s.*[0] <= #char "9" { n = n * 10 + s.*[0] - #char "0"; advance(s); } + return n, true; + } + x, y := a, b; + while true { + p, p_found := next_number(*x); + q, q_found := next_number(*y); + if !p_found || !q_found return ifx p_found then 1 else ifx q_found then -1 else 0; + if p != q return ifx p > q then 1 else -1; + } + return 0; + } + + // Temp; "" when it is not set. + ocr_environment_variable :: (name: string) -> string { + wide_name := utf8_to_wide(name,, temp); + size := OcrGetEnvironmentVariableW(wide_name, null, 0); + if !size return ""; + buffer := NewArray(size, u16,, temp); + written := OcrGetEnvironmentVariableW(wide_name, buffer.data, size); + if !written || written >= size return ""; + return wide_to_utf8(buffer.data, cast(s32) written,, temp); + } + + OcrGetEnvironmentVariableW :: (name: *u16, buffer: *u16, size: u32) -> u32 #foreign kernel32 "GetEnvironmentVariableW"; + OcrSetEnvironmentVariableW :: (name: *u16, value: *u16) -> s32 #foreign kernel32 "SetEnvironmentVariableW"; +} diff --git a/modules/OCR/detect.jai b/modules/OCR/detect.jai index 25eaaa8..b5fb184 100644 --- a/modules/OCR/detect.jai +++ b/modules/OCR/detect.jai @@ -1,19 +1,26 @@ // Detection: where the text lines are. The detector (a DB network) gives each pixel a probability -// of being text. Pixels above det_threshold are grouped into connected regions, each kept region -// becomes a box, and the box is grown by the DB unclip distance, since the network is trained to -// mark a shrunken core of each line. +// of being text. Pixels above det_threshold are grouped into connected regions, and each kept region +// becomes the smallest rectangle around it, at whatever angle, grown by the DB unclip distance, since +// the network is trained to mark a shrunken core of each line. // // Images wider than det_max_side are scaled down for detection. Images taller than det_tile_height // (after scaling) are detected in overlapping strips, so a long scrolling capture keeps its text at // full size; each box is kept by the strip whose own band holds its centre. -// -// Version 1 finds horizontal text only, as screenshots have: boxes are axis-aligned. Detected_Box :: struct { - x0, y0, x1, y1: s32; // in the image, x1 and y1 exclusive + // In the image's pixels, clockwise: the text's top left, top right, bottom right, bottom left, as + // far as its shape tells. A box at least TURN_RATIO times as tall as it is wide is taken as a line + // read downwards, so its first corner is its top right. Recognition may still turn it by 180°. + corners: [4] Vector2; + turned: bool; // taken as a line read downwards: started from its top right score: float; } +TURN_RATIO :: 1.5; + +// Screenshots are mostly level: a region keeps a level box unless a turned one is this much smaller. +LEVEL_PREFERENCE :: 1.2; // a long line tilted by 1° already has a level box ~30% bigger + DET_TILE_OVERLAP : s32 : 128; // px of each strip shared with the next, after scaling // The image's text boxes, from the default allocator. @@ -84,48 +91,180 @@ ocr_detect_tile :: (engine: *Ocr_Engine, image: Ocr_Image, scale: float, scaled_ } regions := ocr_find_regions(out.data, w, valid_w, valid_h, model.det_threshold); - defer array_free(regions,, context.default_allocator); + defer { + for regions array_free(it.outline); + array_free(regions,, context.default_allocator); + } for regions { if it.pixels == 0 continue; score := cast(float) (it.probability_sum / it.pixels); if score < model.box_threshold continue; - bw := it.x1 - it.x0; - bh := it.y1 - it.y0; - if min(bw, bh) < 3 continue; // specks + + rect := ocr_min_area_rect(it.outline); + rect_w := rect.u_max - rect.u_min; + rect_h := rect.v_max - rect.v_min; + if min(rect_w, rect_h) < 3 continue; // specks // Grown as DB's unclip does: by area * ratio / perimeter on every side. - distance := (cast(float) bw * bh * model.unclip_ratio) / (2.0 * (bw + bh)); - gx0 := it.x0 - distance; - gy0 := it.y0 - distance; - gx1 := it.x1 + distance; - gy1 := it.y1 + distance; + distance := (rect_w * rect_h * model.unclip_ratio) / (2.0 * (rect_w + rect_h)); + rect.u_min -= distance; rect.u_max += distance; + rect.v_min -= distance; rect.v_max += distance; + corners := ocr_rect_corners(rect); - centre_y := tile_top + (it.y0 + it.y1) / 2.0; + centre_y := tile_top + (corners[0].y + corners[2].y) / 2; if centre_y < cast(float) band_top || centre_y >= cast(float) band_bottom continue; // the neighbouring strip's box: Detected_Box; - box.x0 = clamp(cast(s32) (gx0 * inverse), 0, image.width); - box.x1 = clamp(cast(s32) (gx1 * inverse + 0.999), 0, image.width); - box.y0 = clamp(cast(s32) ((gy0 + tile_top) * inverse), 0, image.height); - box.y1 = clamp(cast(s32) ((gy1 + tile_top) * inverse + 0.999), 0, image.height); box.score = score; - if box.x1 - box.x0 >= 2 && box.y1 - box.y0 >= 2 array_add(boxes, box); + for corners { + box.corners[it_index].x = clamp(it.x * inverse, 0, cast(float) image.width); + box.corners[it_index].y = clamp((it.y + tile_top) * inverse, 0, cast(float) image.height); + } + if rect.u.x == 1 && rect.u.y == 0 { + // A level box takes in every pixel it touches: the recognizers read these crops best. + x0, y0, x1, y1 := ocr_corner_bounds(box.corners, image.width, image.height); + box.corners = .[.{xx x0, xx y0}, .{xx x1, xx y0}, .{xx x1, xx y1}, .{xx x0, xx y1}]; + } + box.corners, box.turned = ocr_reading_order(box.corners); + x0, y0, x1, y1 := ocr_corner_bounds(box.corners, image.width, image.height); + if x1 - x0 >= 2 && y1 - y0 >= 2 array_add(boxes, box); } return true, ""; } +// The pixels the corners cover, clipped to the image: x0 and y0 inclusive, x1 and y1 exclusive. +ocr_corner_bounds :: (corners: [4] Vector2, image_width: s32, image_height: s32) -> x0: s32, y0: s32, x1: s32, y1: s32 { + low := corners[0]; + high := corners[0]; + for corners { + low.x = min(low.x, it.x); low.y = min(low.y, it.y); + high.x = max(high.x, it.x); high.y = max(high.y, it.y); + } + return clamp(cast(s32) floor(low.x), 0, image_width), clamp(cast(s32) floor(low.y), 0, image_height), + clamp(cast(s32) ceil(high.x), 0, image_width), clamp(cast(s32) ceil(high.y), 0, image_height); +} + +// The same rectangle's corners started from another one, keeping them clockwise: by 1, turned a +// quarter (what was the right side becomes the top), by 2, upside down. +ocr_corners_turned :: (corners: [4] Vector2, quarters: s64) -> [4] Vector2 { + result: [4] Vector2; + for 0..3 result[it] = corners[(it + quarters) % 4]; + return result; +} + +// A rectangle as ranges along two perpendicular unit axes: u, and v = u turned a quarter clockwise +// on the screen (y down). +Rotated_Rect :: struct { + u: Vector2; + u_min, u_max, v_min, v_max: float; +} + +ocr_rect_corners :: (rect: Rotated_Rect) -> [4] Vector2 { + u := rect.u; + v := Vector2.{-u.y, u.x}; + return .[u * rect.u_min + v * rect.v_min, u * rect.u_max + v * rect.v_min, + u * rect.u_max + v * rect.v_max, u * rect.u_min + v * rect.v_max]; +} + +// The corners clockwise from the text's top left, as PaddleOCR orders them: of the two leftmost, the +// higher is the top left; of the two rightmost, the higher is the top right. A box at least +// TURN_RATIO times as tall as it is wide is then started from its top right, so a line read +// downwards comes out level. +ocr_reading_order :: (corners: [4] Vector2) -> [4] Vector2, turned: bool { + sorted := corners; + quick_sort(sorted, (a: Vector2, b: Vector2) -> s64 { return ifx a.x < b.x then -1 else ifx a.x > b.x then 1 else 0; }); + top_left, bottom_left := sorted[0], sorted[1]; + if bottom_left.y < top_left.y swap(*top_left, *bottom_left); + top_right, bottom_right := sorted[2], sorted[3]; + if bottom_right.y < top_right.y swap(*top_right, *bottom_right); + result := Vector2.[top_left, top_right, bottom_right, bottom_left]; + + width := length(result[1] - result[0]); + height := length(result[3] - result[0]); + if height >= width * TURN_RATIO return ocr_corners_turned(result, 1), true; + return result, false; +} + +// The smallest rectangle holding the points, at any angle, found by rotating calipers: one side of +// it lies along an edge of the points' convex hull. A level rectangle is kept unless a turned one is +// smaller by more than LEVEL_PREFERENCE. +ocr_min_area_rect :: (points: [] Vector2) -> Rotated_Rect { + level: Rotated_Rect; + level.u = .{1, 0}; + level.u_min, level.v_min = FLOAT32_MAX, FLOAT32_MAX; + level.u_max, level.v_max = -FLOAT32_MAX, -FLOAT32_MAX; + for points { + level.u_min = min(level.u_min, it.x); level.u_max = max(level.u_max, it.x); + level.v_min = min(level.v_min, it.y); level.v_max = max(level.v_max, it.y); + } + if points.count < 3 return level; + level_area := (level.u_max - level.u_min) * (level.v_max - level.v_min); + + hull := ocr_convex_hull(points); + best := level; + best_area := level_area; + for i: 0..hull.count - 1 { + edge := hull[(i + 1) % hull.count] - hull[i]; + edge_length := length(edge); + if edge_length < 0.0001 continue; + rect: Rotated_Rect; + rect.u = edge / edge_length; + v := Vector2.{-rect.u.y, rect.u.x}; + rect.u_min, rect.v_min = FLOAT32_MAX, FLOAT32_MAX; + rect.u_max, rect.v_max = -FLOAT32_MAX, -FLOAT32_MAX; + for hull { + a := dot(it, rect.u); + b := dot(it, v); + rect.u_min = min(rect.u_min, a); rect.u_max = max(rect.u_max, a); + rect.v_min = min(rect.v_min, b); rect.v_max = max(rect.v_max, b); + } + area := (rect.u_max - rect.u_min) * (rect.v_max - rect.v_min); + if area < best_area { best = rect; best_area = area; } + } + if level_area <= best_area * LEVEL_PREFERENCE return level; + return best; +} + +// The convex hull of the points, counter-clockwise (Andrew's monotone chain). Temp. +ocr_convex_hull :: (points: [] Vector2) -> [] Vector2 { + sorted := array_copy(points,, temp); + quick_sort(sorted, (a: Vector2, b: Vector2) -> s64 { + if a.x != b.x return ifx a.x < b.x then -1 else 1; + return ifx a.y < b.y then -1 else ifx a.y > b.y then 1 else 0; + }); + cross :: (o: Vector2, a: Vector2, b: Vector2) -> float { return (a.x - o.x) * (b.y - o.y) - (a.y - o.y) * (b.x - o.x); } + + hull := NewArray(sorted.count * 2, Vector2,, temp); + k := 0; + for sorted { // lower hull + while k >= 2 && cross(hull[k - 2], hull[k - 1], it) <= 0 k -= 1; + hull[k] = it; k += 1; + } + lower := k + 1; + for < i: 0..sorted.count - 2 { // upper hull + p := sorted[i]; + while k >= lower && cross(hull[k - 2], hull[k - 1], p) <= 0 k -= 1; + hull[k] = p; k += 1; + } + hull.count = max(k - 1, 1); // the last point is the first again + return hull; +} + align_up_32 :: (n: s32) -> s32 { return (n + 31) & cast(s32) ~31; } Region :: struct { - x0, y0, x1, y1: s32; // x1 and y1 exclusive pixels: s64; probability_sum: float64; + // The corners of the first and last pixel in each of its rows: what its convex hull needs. + outline: [..] Vector2; // default allocator + row: s32 = -1; // the row being scanned, and its first and last pixel + row_first, row_last: s32; } // Groups the pixels above threshold into 8-connected regions: one pass assigning provisional labels -// and recording which touch (union-find), then one summing each region's extent and probability. +// and recording which touch (union-find), then one summing each region's probability and outline. ocr_find_regions :: (probability: *float, stride: s32, width: s32, height: s32, threshold: float) -> [..] Region { regions: [..] Region; regions.allocator = context.default_allocator; @@ -149,6 +288,14 @@ ocr_find_regions :: (probability: *float, stride: s32, width: s32, height: s32, else if rb < ra parent[ra] = rb; } + flush_row :: (r: *Region) { + if r.row < 0 return; + top, bottom := cast(float) r.row, cast(float) r.row + 1; + array_add(*r.outline, .{cast(float) r.row_first, top}, .{cast(float) r.row_first, bottom}, + .{cast(float) r.row_last + 1, top}, .{cast(float) r.row_last + 1, bottom}); + r.row = -1; + } + for y: 0..height - 1 { for x: 0..width - 1 { if probability[cast(s64) y * stride + x] <= threshold continue; @@ -184,16 +331,21 @@ ocr_find_regions :: (probability: *float, stride: s32, width: s32, height: s32, index := region_of[root] - 1; if index < 0 { region := array_add(*regions); - region.* = .{x0 = x, y0 = y, x1 = x + 1, y1 = y + 1}; + region.* = .{}; + region.outline.allocator = context.default_allocator; index = cast(s32) regions.count - 1; region_of[root] = index + 1; } r := *regions[index]; - r.x0 = min(r.x0, x); r.x1 = max(r.x1, x + 1); - r.y0 = min(r.y0, y); r.y1 = max(r.y1, y + 1); + if r.row != y { + flush_row(r); + r.row, r.row_first = y, x; + } + r.row_last = x; // rows are scanned left to right r.pixels += 1; r.probability_sum += probability[cast(s64) y * stride + x]; } } + for * regions flush_row(it); return regions; } diff --git a/modules/OCR/fetch_models.jai b/modules/OCR/fetch_models.jai index 9fc38d8..6c068dd 100644 --- a/modules/OCR/fetch_models.jai +++ b/modules/OCR/fetch_models.jai @@ -2,7 +2,7 @@ // are not in git; run this after cloning. Files already there with the right hash are skipped. // // jai fetch_models.jai every model -// jai fetch_models.jai - mobile only the files ppocr-v5-mobile needs (about 21 MB) +// jai fetch_models.jai - mobile only the files ppocr-v5-mobile needs (about 22 MB) // // From RapidOCR's conversions of PaddleOCR's PP-OCRv5 (both Apache 2.0): // https://www.modelscope.cn/models/RapidAI/RapidOCR @@ -15,6 +15,7 @@ FILES :: Model_File.[ .{"ch_PP-OCRv5_det_mobile.onnx", "onnx/PP-OCRv5/det/ch_PP-OCRv5_det_mobile.onnx", "4d97c44a20d30a81aad087d6a396b08f786c4635742afc391f6621f5c6ae78ae", false}, .{"ch_PP-OCRv5_rec_mobile.onnx", "onnx/PP-OCRv5/rec/ch_PP-OCRv5_rec_mobile.onnx", "5825fc7ebf84ae7a412be049820b4d86d77620f204a041697b0494669b1742c5", false}, .{"ppocrv5_dict.txt", "paddle/PP-OCRv5/rec/ch_PP-OCRv5_rec_mobile/ppocrv5_dict.txt", "d1979e9f794c464c0d2e0b70a7fe14dd978e9dc644c0e71f14158cdf8342af1b", false}, + .{"ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx", "onnx/PP-OCRv5/cls/ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx", "54379ae5174d026780215fc748a7f31910dee36818e63d49e17dc598ecc82df7", false}, .{"ch_PP-OCRv5_det_server.onnx", "onnx/PP-OCRv5/det/ch_PP-OCRv5_det_server.onnx", "0f8846b1d4bba223a2a2f9d9b44022fbc22cc019051a602b41a7fda9667e4cad", true}, .{"ch_PP-OCRv5_rec_server.onnx", "onnx/PP-OCRv5/rec/ch_PP-OCRv5_rec_server.onnx", "e09385400eaaaef34ceff54aeb7c4f0f1fe014c27fa8b9905d4709b65746562a", true}, ]; diff --git a/modules/OCR/models.config b/modules/OCR/models.config index 174ddb7..661c901 100644 --- a/modules/OCR/models.config +++ b/modules/OCR/models.config @@ -5,6 +5,8 @@ # # title shown to people # detector, recognizer, dictionary file names, quoted +# textline_ori optional: a text line orientation classifier's file name, quoted; lines it +# finds upside down are read both ways and the more confident reading kept # det_max_side wider images are scaled down to this width for detection (px) # det_tile_height taller images are detected in overlapping strips this high (px, after scaling) # det_threshold a pixel is text when its probability is above this @@ -18,6 +20,7 @@ title: "PP-OCRv5 mobile (light)" detector: "ch_PP-OCRv5_det_mobile.onnx" recognizer: "ch_PP-OCRv5_rec_mobile.onnx" dictionary: "ppocrv5_dict.txt" +textline_ori: "ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx" det_max_side: 2560 det_tile_height: 1536 det_threshold: 0.3 @@ -31,6 +34,7 @@ title: "PP-OCRv5 server (heavy)" detector: "ch_PP-OCRv5_det_server.onnx" recognizer: "ch_PP-OCRv5_rec_server.onnx" dictionary: "ppocrv5_dict.txt" +textline_ori: "ch_PP-LCNet_x0_25_textline_ori_cls_mobile.onnx" det_max_side: 2560 det_tile_height: 1536 det_threshold: 0.3 diff --git a/modules/OCR/models.jai b/modules/OCR/models.jai index 6684414..080122a 100644 --- a/modules/OCR/models.jai +++ b/modules/OCR/models.jai @@ -5,6 +5,7 @@ Ocr_Model_Info :: struct { name: string; title: string; detector, recognizer, dictionary: string; // file names in the models folder + textline_ori: string; // the orientation classifier's file name; "" for none det_max_side: s32 = 2560; det_tile_height: s32 = 1536; det_threshold: float = 0.3; @@ -53,6 +54,7 @@ ocr_list_models :: (models_folder: string) -> [] Ocr_Model_Info, error: string { case "detector"; model.detector = copy_string(value); case "recognizer"; model.recognizer = copy_string(value); case "dictionary"; model.dictionary = copy_string(value); + case "textline_ori"; model.textline_ori = copy_string(value); case "det_max_side"; if number_ok model.det_max_side = cast(s32) number; case "det_tile_height"; if number_ok model.det_tile_height = cast(s32) number; case "det_threshold"; if number_ok model.det_threshold = number; @@ -82,6 +84,35 @@ ocr_strip_comment :: (line: string) -> string { Ocr_Session :: struct { session: *OrtSession; input_name, output_name: *u8; // heap C strings + input_shape: [4] s64; // as the model declares it: -1 for a size it leaves open + input_rank: s64; +} + +// The model's sessions on the GPU or the CPU: detector, recognizer and, when it has one, orientation +// classifier. On failure, the reason; the sessions made so far are left for ocr_release_sessions. +ocr_create_sessions :: (engine: *Ocr_Engine, models_folder: string, gpu: bool, intra_op_threads: s32) -> ok: bool, error: string { + model := *engine.model; + ok: bool; + error: string; + engine.det, ok, error = ocr_create_session(engine.env, tprint("%/%", models_folder, model.detector), gpu, intra_op_threads); + if !ok return false, error; + engine.rec, ok, error = ocr_create_session(engine.env, tprint("%/%", models_folder, model.recognizer), gpu, intra_op_threads); + if !ok return false, error; + if model.textline_ori { + engine.orientation, ok, error = ocr_create_session(engine.env, tprint("%/%", models_folder, model.textline_ori), gpu, intra_op_threads); + if !ok return false, error; + // PP-LCNet text line orientation takes 160 by 80; a model that says otherwise is taken at its word. + shape := engine.orientation.input_shape; + engine.orientation_height = ifx engine.orientation.input_rank == 4 && shape[2] > 0 then cast(s32) shape[2] else 80; + engine.orientation_width = ifx engine.orientation.input_rank == 4 && shape[3] > 0 then cast(s32) shape[3] else 160; + } + return true, ""; +} + +ocr_release_sessions :: (engine: *Ocr_Engine) { + ocr_release_session(*engine.det); + ocr_release_session(*engine.rec); + ocr_release_session(*engine.orientation); } // A session for the model file; with CUDA when gpu is set. On failure, the reason. @@ -124,6 +155,18 @@ ocr_create_session :: (env: *OrtEnv, path: string, gpu: bool, intra_op_threads: ort_check(api.SessionGetOutputName(result.session, 0, allocator, *name)); result.output_name = to_c_string(to_string(name),, context.default_allocator); ort_check(api.AllocatorFree(allocator, name)); + + input_type: *OrtTypeInfo; + if ort_check(api.SessionGetInputTypeInfo(result.session, 0, *input_type)) { + tensor_info: *OrtTensorTypeAndShapeInfo; // belongs to input_type + if ort_check(api.CastTypeInfoToTensorInfo(input_type, *tensor_info)) && tensor_info { + rank: u64; + ort_check(api.GetDimensionsCount(tensor_info, *rank)); + result.input_rank = min(cast(s64) rank, result.input_shape.count); + ort_check(api.GetDimensions(tensor_info, result.input_shape.data, xx result.input_rank)); + } + api.ReleaseTypeInfo(input_type); + } return result, true, ""; } diff --git a/modules/OCR/module.jai b/modules/OCR/module.jai index 98042e2..61944ff 100644 --- a/modules/OCR/module.jai +++ b/modules/OCR/module.jai @@ -8,8 +8,9 @@ // with onnxruntime (modules/onnxruntime), on an NVIDIA GPU through CUDA when it can, else on the // CPU. ocr_gpu_status says why the GPU is not in use, and what to install so it can be. // -// Version 1 runs PP-OCRv5's two-stage pipeline: detection (detect.jai), then recognition -// (recognize.jai). Text is expected horizontal, as in screenshots. +// It runs PP-OCRv5's pipeline: detection (detect.jai) finds each line as a rectangle at any angle, +// then recognition (recognize.jai) reads it level. When the model lists an orientation classifier, +// lines that look upside down are read both ways; lines read downwards or upwards are read too. Ocr_Provider :: enum { CPU; CUDA; } @@ -20,9 +21,10 @@ Ocr_Image :: struct { } Ocr_Line :: struct { - x, y, width, height: s32; // in the image + x, y, width, height: s32; // in the image: the bounds of corners text: string; confidence: float; // 0 to 1: the mean probability of its characters + corners: [4] Vector2; // in the image's pixels: the text's top left, top right, bottom right, bottom left } Ocr_Result :: struct { @@ -38,6 +40,8 @@ Ocr_Engine :: struct { env: *OrtEnv; memory_info: *OrtMemoryInfo; det, rec: Ocr_Session; + orientation: Ocr_Session; // the text line orientation classifier; none when its session is null + orientation_width, orientation_height: s32; // its input size dictionary: [] string; // class n is dictionary[n - 1] workers: s32; // threads ocr_process uses stats: Ocr_Stats; // since creation, or the last ocr_reset_stats @@ -49,6 +53,8 @@ Ocr_Stats :: struct { lines: s64; detect_us: s64; // preparing images, the detector, and finding the boxes recognize_us: s64; // cropping lines, the recognizer, and decoding the text + orientation_us: s64; // cropping lines and the orientation classifier + turned: s64; // lines that read better the other way up, or level as found } ocr_stats :: (engine: *Ocr_Engine) -> Ocr_Stats { return engine.stats; } @@ -84,8 +90,6 @@ ocr_create :: (models_folder: string, model_name: string, prefer_gpu := true) -> ort_check(api.CreateCpuMemoryInfo(.OrtArenaAllocator, .OrtMemTypeDefault, *engine.memory_info)); cores := cast(s32) max(get_number_of_processors(.ALL_LOGICAL), 1); // efficiency cores included - detector_path := tprint("%/%", models_folder, model.detector); - recognizer_path := tprint("%/%", models_folder, model.recognizer); use_gpu := false; if prefer_gpu { @@ -99,16 +103,12 @@ ocr_create :: (models_folder: string, model_name: string, prefer_gpu := true) -> if use_gpu { // A few CPU threads prepare images and decode results while the GPU runs the networks. engine.workers = 4; - det_ok, rec_ok: bool; - det_error, rec_error: string; - engine.det, det_ok, det_error = ocr_create_session(engine.env, detector_path, true, 2); - if det_ok engine.rec, rec_ok, rec_error = ocr_create_session(engine.env, recognizer_path, true, 2); - if det_ok && rec_ok { + sessions_ok, sessions_error := ocr_create_sessions(engine, models_folder, true, 2); + if sessions_ok { engine.provider = .CUDA; } else { - ocr_release_session(*engine.det); - ocr_release_session(*engine.rec); - engine.gpu_explanation = sprint("OCR runs on the CPU: the CUDA libraries were found, but onnxruntime could not start on the GPU:\n%", ifx det_ok then rec_error else det_error); + ocr_release_sessions(engine); + engine.gpu_explanation = sprint("OCR runs on the CPU: the CUDA libraries were found, but onnxruntime could not start on the GPU:\n%", sessions_error); use_gpu = false; } } @@ -116,12 +116,8 @@ ocr_create :: (models_folder: string, model_name: string, prefer_gpu := true) -> // Several images at once, each network run spread over the rest of the cores. engine.workers = clamp(cores / 4, 1, 8); threads := max(cores / engine.workers, 1); - det_ok, rec_ok: bool; - det_error, rec_error: string; - engine.det, det_ok, det_error = ocr_create_session(engine.env, detector_path, false, threads); - if !det_ok { ocr_destroy(engine); return null, false, det_error; } - engine.rec, rec_ok, rec_error = ocr_create_session(engine.env, recognizer_path, false, threads); - if !rec_ok { ocr_destroy(engine); return null, false, rec_error; } + sessions_ok, sessions_error := ocr_create_sessions(engine, models_folder, false, threads); + if !sessions_ok { ocr_destroy(engine); return null, false, sessions_error; } engine.provider = .CPU; } return engine, true, ""; @@ -138,8 +134,7 @@ ocr_gpu_status :: (engine: *Ocr_Engine) -> in_use: bool, explanation: string { ocr_destroy :: (engine: *Ocr_Engine) { if !engine return; - ocr_release_session(*engine.det); - ocr_release_session(*engine.rec); + ocr_release_sessions(engine); if engine.memory_info ort.api.ReleaseMemoryInfo(engine.memory_info); if engine.env ort.api.ReleaseEnv(engine.env); free(engine.gpu_explanation,, context.default_allocator); @@ -174,10 +169,49 @@ ocr_process :: (engine: *Ocr_Engine, images: [] Ocr_Image, allocator := context. for jobs free(it.text,, context.default_allocator); array_free(jobs); } + rec_height := engine.model.rec_height; + turned: [..] bool; // per first job: whether detection turned its box a quarter + turned.allocator = context.default_allocator; + defer array_free(turned); for work: per_image { if !work.ok continue; - for work.boxes array_add(*jobs, .{image = work.image, box = it, width = ocr_line_width(it, engine.model.rec_height)}); + for work.boxes { + array_add(*jobs, .{image = work.image, corners = it.corners, width = ocr_line_width(it.corners, rec_height)}); + array_add(*turned, it.turned); + } } + + // Other ways to read a line, recognized beside it, the more confident reading kept: upside down, + // where the orientation classifier finds it more likely than not; and level as it was found, for a + // box detection turned a quarter, since a narrow word such as "1" is as tall as a line read downwards. + first_count := jobs.count; + upside_down := NewArray(first_count, bool,, context.default_allocator); + defer array_free(upside_down,, context.default_allocator); + Orientation_Context :: struct { jobs: [] Line_Job; upside_down: [] bool; size: s64; failed: bool; error: string; } + classify := Orientation_Context.{jobs = jobs, upside_down = upside_down, size = max(engine.model.rec_batch, 1)}; + if engine.orientation.session && first_count { + run_on_workers(engine, (first_count + classify.size - 1) / classify.size, (engine: *Ocr_Engine, index: s64, data: *void) { + c := cast(*Orientation_Context) data; + first := index * c.size; + count := min(c.size, c.jobs.count - first); + batch := NewArray(count, *Line_Job,, temp); + for * batch it.* = *c.jobs[first + it_index]; + start := current_time_monotonic(); + ok, error := ocr_classify_batch(engine, batch, array_view(c.upside_down, first, count)); + atomic_add(*engine.stats.orientation_us, to_microseconds(current_time_monotonic() - start)); + if !ok { c.failed = true; c.error = copy_string(error,, context.default_allocator); } // the last error wins + }, *classify); + } + defer free(classify.error,, context.default_allocator); + for n: 0..first_count - 1 { + job := jobs[n]; + if upside_down[n] array_add(*jobs, .{image = job.image, corners = ocr_corners_turned(job.corners, 2), width = job.width, primary = n}); + if turned[n] { + level := ocr_corners_turned(job.corners, 3); + array_add(*jobs, .{image = job.image, corners = level, width = ocr_line_width(level, rec_height), primary = n}); + } + } + order := NewArray(jobs.count, *Line_Job,, context.default_allocator); defer array_free(order,, context.default_allocator); for * jobs order[it_index] = it; @@ -196,6 +230,20 @@ ocr_process :: (engine: *Ocr_Engine, images: [] Ocr_Image, allocator := context. atomic_add(*engine.stats.recognize_us, to_microseconds(current_time_monotonic() - start)); if !ok { b.failed = true; b.error = copy_string(error,, context.default_allocator); } // the last error wins; any is enough to say why }, *batches); + if classify.failed && !batches.failed { + batches.failed = true; + batches.error = copy_string(classify.error,, context.default_allocator); + } + + // The more confident of each line's readings takes the first job's place. + for * other: array_view(jobs, first_count, jobs.count - first_count) { + first := *jobs[other.primary]; + if other.confidence > first.confidence { + swap(*first.text, *other.text); + first.corners, first.confidence = other.corners, other.confidence; + engine.stats.turned += 1; + } + } // Each image's lines, in reading order. for work, image_index: per_image { @@ -213,10 +261,11 @@ ocr_process :: (engine: *Ocr_Engine, images: [] Ocr_Image, allocator := context. } lines: [..] Ocr_Line; lines.allocator = allocator; - for job: jobs { + for job: array_view(jobs, 0, first_count) { if job.image != work.image || !job.text.count continue; - array_add(*lines, .{x = job.box.x0, y = job.box.y0, width = job.box.x1 - job.box.x0, height = job.box.y1 - job.box.y0, - text = copy_string(job.text,, allocator), confidence = job.confidence}); + x0, y0, x1, y1 := ocr_corner_bounds(job.corners, job.image.width, job.image.height); + array_add(*lines, .{x = x0, y = y0, width = x1 - x0, height = y1 - y0, text = copy_string(job.text,, allocator), + confidence = job.confidence, corners = job.corners}); } // Rows by the middle of each line, then left to right within a row. quick_sort(lines, (a: Ocr_Line, b: Ocr_Line) -> s64 { diff --git a/modules/OCR/recognize.jai b/modules/OCR/recognize.jai index 1cfef2d..dc27f4d 100644 --- a/modules/OCR/recognize.jai +++ b/modules/OCR/recognize.jai @@ -1,13 +1,18 @@ -// Recognition: the text of each detected line. Each box is cropped from the image at full size, -// scaled to rec_height keeping its shape, and normalized to -1..1 in BGR order, as PaddleOCR's +// Recognition: the text of each detected line. Each box is sampled along its own sides into a level +// crop rec_height high, keeping its shape, and normalized to -1..1 in BGR order, as PaddleOCR's // recognizers expect. Lines of similar width are batched, padded to the widest in the batch. // The output is a probability per character class per step; greedy CTC decoding takes the best // class at each step, merges repeats and drops blanks. +// +// A box's corners only say which way its text runs up to a half turn. The orientation classifier, +// when the model has one, says which lines look upside down; such a line is read both ways, as is a +// box taken as a line read downwards, and the more confident reading is kept. Line_Job :: struct { image: *Ocr_Image; - box: Detected_Box; + corners: [4] Vector2; // as Detected_Box's width: s32; // after scaling to rec_height + primary := -1; // for another way of reading a line: the index of the line's first job // Filled in by recognition: text: string; // default allocator confidence: float; @@ -24,38 +29,8 @@ ocr_recognize_batch :: (engine: *Ocr_Engine, jobs: [] *Line_Job) -> ok: bool, er plane := cast(s64) height * width; input := NewArray(plane * 3 * jobs.count, float, initialized = true,, context.default_allocator); // 0 is the padding defer array_free(input,, context.default_allocator); - for job, n: jobs { - base := input.data + plane * 3 * n; - box := job.box; - bw := box.x1 - box.x0; - bh := box.y1 - box.y0; - image := job.image; - // Bilinear: the crops are mostly enlarged (small UI text up to 48 px), where nearest would be blocky. - x_scale := cast(float) bw / job.width; - y_scale := cast(float) bh / height; - for y: 0..height - 1 { - sy := (y + 0.5) * y_scale - 0.5 + box.y0; - y0 := clamp(cast(s32) floor(sy), 0, image.height - 1); - y1 := min(y0 + 1, image.height - 1); - fy := clamp(sy - y0, 0, 1); - row0 := image.pixels + cast(s64) y0 * image.stride; - row1 := image.pixels + cast(s64) y1 * image.stride; - for x: 0..job.width - 1 { - sx := (x + 0.5) * x_scale - 0.5 + box.x0; - x0 := clamp(cast(s32) floor(sx), 0, image.width - 1); - x1 := min(x0 + 1, image.width - 1); - fx := clamp(sx - x0, 0, 1); - i := cast(s64) y * width + x; - for c: 0..2 { - channel := 2 - c; // B, G, R from RGBA - top := row0[x0 * 4 + channel] * (1 - fx) + row0[x1 * 4 + channel] * fx; - bottom := row1[x0 * 4 + channel] * (1 - fx) + row1[x1 * 4 + channel] * fx; - value := top * (1 - fy) + bottom * fy; - base[i + plane * c] = (value / 255.0 - 0.5) / 0.5; - } - } - } + ocr_sample_crop(job.image, job.corners, job.width, height, input.data + plane * 3 * n, width, plane, REC_MEAN, REC_STD); } shape := s64.[jobs.count, 3, height, width]; @@ -95,11 +70,68 @@ ocr_recognize_batch :: (engine: *Ocr_Engine, jobs: [] *Line_Job) -> ok: bool, er return true, ""; } -// The width a box is recognized at: rec_height high, keeping its shape. -ocr_line_width :: (box: Detected_Box, rec_height: s32) -> s32 { - bw := box.x1 - box.x0; - bh := max(box.y1 - box.y0, 1); - return clamp(cast(s32) (cast(float) bw * rec_height / bh + 0.5), 1, REC_MAX_WIDTH); +// The width corners are recognized at: rec_height high, keeping their shape. +ocr_line_width :: (corners: [4] Vector2, rec_height: s32) -> s32 { + w := length(corners[1] - corners[0]); + h := max(length(corners[3] - corners[0]), 1); + return clamp(cast(s32) (w * rec_height / h + 0.5), 1, REC_MAX_WIDTH); } REC_MAX_WIDTH :: 4096; // a line longer than this, at rec_height, is squeezed to fit + +// The normalization the PP-OCR recognizers were trained with. +REC_MEAN :: float.[0.5, 0.5, 0.5]; +REC_STD :: float.[0.5, 0.5, 0.5]; + +// Samples the rectangle the corners make (clockwise from its top left, at any angle) into a level +// crop width by height, bilinearly: crops are mostly enlarged (small UI text up to 48 px), where +// nearest would be blocky. Written planar in BGR order at dest, row_stride floats from one row to the +// next and plane from one channel to the next, as (value / 255 - mean) / deviation. +ocr_sample_crop :: (image: *Ocr_Image, corners: [4] Vector2, width: s32, height: s32, dest: *float, row_stride: s64, plane: s64, mean: [3] float, deviation: [3] float) { + step_x := (corners[1] - corners[0]) / cast(float) width; // one crop pixel along the line + step_y := (corners[3] - corners[0]) / cast(float) height; // and across it + // Pixel centres: an image pixel's centre is at +0.5. + origin := corners[0] + step_x * 0.5 + step_y * 0.5 - Vector2.{0.5, 0.5}; + for y: 0..height - 1 { + row_start := origin + step_y * cast(float) y; + for x: 0..width - 1 { + s := row_start + step_x * cast(float) x; + x0 := clamp(cast(s32) floor(s.x), 0, image.width - 1); + y0 := clamp(cast(s32) floor(s.y), 0, image.height - 1); + x1 := min(x0 + 1, image.width - 1); + y1 := min(y0 + 1, image.height - 1); + fx := clamp(s.x - x0, 0, 1); + fy := clamp(s.y - y0, 0, 1); + row0 := image.pixels + cast(s64) y0 * image.stride; + row1 := image.pixels + cast(s64) y1 * image.stride; + i := cast(s64) y * row_stride + x; + for c: 0..2 { + channel := 2 - c; // B, G, R from RGBA + top := row0[x0 * 4 + channel] * (1 - fx) + row0[x1 * 4 + channel] * fx; + bottom := row1[x0 * 4 + channel] * (1 - fx) + row1[x1 * 4 + channel] * fx; + value := top * (1 - fy) + bottom * fy; + dest[i + plane * c] = (value / 255.0 - mean[c]) / deviation[c]; + } + } + } +} + +// The orientation classifier (PP-LCNet text line orientation) on one batch: sets upside_down[n] when +// it finds jobs[n] more likely upside down than not. Its input is a fixed size, the line stretched to +// fit, normalized as the detector's input is. +ocr_classify_batch :: (engine: *Ocr_Engine, jobs: [] *Line_Job, upside_down: [] bool) -> ok: bool, error: string { + if !jobs.count return true, ""; + width, height := engine.orientation_width, engine.orientation_height; + plane := cast(s64) width * height; + input := NewArray(plane * 3 * jobs.count, float,, context.default_allocator); + defer array_free(input,, context.default_allocator); + for job, n: jobs ocr_sample_crop(job.image, job.corners, width, height, input.data + plane * 3 * n, width, plane, DET_MEAN, DET_STD); + + shape := s64.[jobs.count, 3, height, width]; + out, ok, error := ocr_run(*engine.orientation, engine.memory_info, input, shape); + if !ok return false, error; + defer ocr_release_output(*out); + if out.dim_count != 2 || out.dims[0] != jobs.count || out.dims[1] != 2 return false, tprint("The orientation classifier gave an output of shape %, not [%, 2].", array_view(out.dims, 0, out.dim_count), jobs.count); + for 0..jobs.count - 1 upside_down[it] = out.data[it * 2 + 1] > out.data[it * 2]; // classes: 0°, 180° + return true, ""; +} diff --git a/src/App.jai b/src/App.jai index f82961b..06cddb6 100644 --- a/src/App.jai +++ b/src/App.jai @@ -9,12 +9,15 @@ USING_WINDOWS :: OS == .WINDOWS; #if COMPILE_TESTS { #load "test/testing.jai"; #load "test/logger_tests.jai"; + #load "test/catalog_tests.jai"; + #load "test/ocr_tests.jai"; } #load "app/main.jai"; // entry point #load "app/app_ui.jai"; #load "app/viewer.jai"; #load "app/ocr_ui.jai"; +#load "app/catalog.jai"; #load "app/exit_shutdown.jai"; #load "base/circle_buffer.jai"; @@ -50,6 +53,8 @@ Shared_Channel :: #import "Shared_Memory_Channel"; #import "stb_image_resize"; #import "Capture_File"; // the .mcap format, copied from D:\SnagitIndex\SnagitConverter #import "OCR"; +#import "Capture_Text"; +#import "Base64"; #import "onnxruntime"; // Basic's arenas import Secure_Memory at its defaults; this import adds its test suite. @@ -167,6 +172,8 @@ App :: struct { run_tests: bool; headless: bool; // -headless (or a no_crash_dialog build): no message boxes on failures, and run_tests exits when done exit_code: s32; // the process's exit code when the main loop ends + library_folder: string; // -library FOLDER: the .mcap folder the viewer shows, in place of VIEWER_FOLDER + scan_and_exit: bool; // -scan: scan the library with the chosen OCR model once it has loaded, then exit (0, or 1 if any capture failed) last_window_restore_time: Apollo_Time; imgui_context: *ImGui.ImGuiContext; diff --git a/src/app/catalog.jai b/src/app/catalog.jai new file mode 100644 index 0000000..0cb76c3 --- /dev/null +++ b/src/app/catalog.jai @@ -0,0 +1,289 @@ +// The catalog: what search and sorting need from every capture, loaded once from the front of each +// file (its Capture Text and OCR text, not its images) on the viewer's worker threads. +// +// Entries are made on the workers and handed over through viewer.catalog_done; only the main thread +// sets or frees viewer.captures[i].catalog, and only it reads them, so no entry is freed while in use. + +Catalog_Entry :: struct { + date: string; // CaptureDate, "2026-10-07 08:11:15" (sorts as text); "" when unknown + title, url, app: string; // WindowName, WebURL, AppName + notes: string; // the annotations' text, one per line + ocr: [..] Mcap_Ocr_Section; // one per model that read it + + // Lowercased copies, for matching without case. + find_title, find_url, find_app, find_notes, find_ocr: string; +} + +// Everything the entry holds is from the default allocator. +catalog_entry_free :: (entry: *Catalog_Entry) { + if !entry return; + push_allocator(context.default_allocator); + free(entry.date); free(entry.title); free(entry.url); free(entry.app); free(entry.notes); + for entry.ocr { + free(it.model); free(it.engine); free(it.image); free(it.date); + for line: it.lines free(line.text); + array_free(it.lines); + } + array_free(entry.ocr); + free(entry.find_title); free(entry.find_url); free(entry.find_app); free(entry.find_notes); free(entry.find_ocr); + free(entry); +} + +// The entry for the .mcap at path, or null with why. On any thread. +catalog_load :: (path: string) -> *Catalog_Entry, error: string { + front: Mcap; + ok: bool; + error: string; + { + push_allocator(temp); + front, ok, error = mcap_read_front(path, preview = false); + } + if !ok return null, error; + + push_allocator(context.default_allocator); + entry := New(Catalog_Entry); + + // The metadata and the annotations, from the Capture Text. + doc: Capture_Document; + capture_document_init(*doc, temp); + if capture_text_parse(*doc, path, front.text) { + metadata := capture_find_file(*doc, "metadata.json"); + if metadata { + entry.date = catalog_string(metadata.root, "CaptureDate"); + entry.title = catalog_string(metadata.root, "WindowName"); + entry.url = catalog_string(metadata.root, "WebURL"); + entry.app = catalog_string(metadata.root, "AppName"); + } + notes: String_Builder; + notes.allocator = temp; + for file: doc.files { + if file.name == "index.json" || file.name == "metadata.json" || ends_with(file.name, ".backup.json") continue; + objects := capture_object_find(*file.root, "CaptureObjects"); + if !objects || objects.kind != .ARRAY continue; + for object: objects.items { + encoded := capture_object_find(*object, "RTFEncodedText"); + if !encoded || encoded.kind != .STRING || !encoded.text.count continue; + rtf, decoded := base64_decode(encoded.text,, temp); + if !decoded continue; + text := rtf_to_text(rtf); + if !text.count || text == "?" continue; // "?": an empty text box + if builder_string_length(*notes) append(*notes, "\n"); + append(*notes, text); + } + } + entry.notes = builder_to_string(*notes); + } + + sections, ocr_ok := mcap_ocr_parse(front.ocr); + if ocr_ok entry.ocr = sections; + + entry.find_title = to_lower_copy(entry.title); + entry.find_url = to_lower_copy(entry.url); + entry.find_app = to_lower_copy(entry.app); + entry.find_notes = to_lower_copy(entry.notes); + all_ocr: String_Builder; + all_ocr.allocator = temp; + for entry.ocr for line: it.lines { append(*all_ocr, line.text); append(*all_ocr, "\n"); } + entry.find_ocr = to_lower_copy(builder_to_string(*all_ocr,, temp)); + return entry, ""; +} + +catalog_section :: (entry: *Catalog_Entry, model: string) -> *Mcap_Ocr_Section { + if !entry return null; + for * entry.ocr if it.model == model return it; + return null; +} + +// ---------------------------------------------------------------- search + +Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; } + +Search_Term :: struct { + field: Search_Field; + text: string; // lowercased +} + +// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note:. Temp. +search_parse :: (query: string) -> [..] Search_Term { + terms: [..] Search_Term; + terms.allocator = temp; + rest := trim(query); + while rest.count { + term: Search_Term; + colon := find_index_from_left(rest, #char ":"); + space := find_index_from_left(rest, #char " "); + if colon > 0 && (space < 0 || colon < space) { + prefix := to_lower_copy(slice(rest, 0, colon),, temp); + field := Search_Field.ANY; + if prefix == { + case "ocr"; field = .OCR; + case "title"; field = .TITLE; + case "url"; field = .URL; + case "app"; field = .APP; + case "note"; field = .NOTE; + } + if field != .ANY { term.field = field; rest = advance(rest, colon + 1); } // "http:..." and the like stay words + } + if rest.count && rest[0] == #char "\"" { + close := find_index_from_left(advance(rest, 1), #char "\""); + if close < 0 { term.text = advance(rest, 1); rest = ""; } + else { term.text = slice(rest, 1, close); rest = advance(rest, close + 2); } + } else { + space = find_index_from_left(rest, #char " "); + if space < 0 { term.text = rest; rest = ""; } + else { term.text = slice(rest, 0, space); rest = advance(rest, space); } + } + term.text = to_lower_copy(term.text,, temp); + if term.text.count array_add(*terms, term); + rest = trim_left(rest); + } + return terms; +} + +// Whether entry holds every term. +search_matches :: (entry: *Catalog_Entry, terms: [] Search_Term) -> bool { + if !terms.count return true; + if !entry return false; + for terms { + found: bool; + if it.field == { + case .ANY; found = contains(entry.find_ocr, it.text) || contains(entry.find_title, it.text) || contains(entry.find_url, it.text) || + contains(entry.find_app, it.text) || contains(entry.find_notes, it.text); + case .OCR; found = contains(entry.find_ocr, it.text); + case .TITLE; found = contains(entry.find_title, it.text); + case .URL; found = contains(entry.find_url, it.text); + case .APP; found = contains(entry.find_app, it.text); + case .NOTE; found = contains(entry.find_notes, it.text); + } + if !found return false; + } + return true; +} + +// Whether an OCR line holds one of the terms that can be in OCR text: the ones it should be shown for. +search_line_matches :: (text: string, terms: [] Search_Term) -> bool { + lowered := to_lower_copy(text,, temp); + for terms if (it.field == .ANY || it.field == .OCR) && contains(lowered, it.text) return true; + return false; +} + +// ---------------------------------------------------------------- RTF + +// An annotation's RTF reduced to its text: control words and their groups dropped, \par and \line as +// line breaks, \uN and \'hh as the characters they stand for. Temp. +rtf_to_text :: (rtf: string) -> string { + out: String_Builder; + out.allocator = temp; + skip_depth := -1; // the group depth at which skipping began; -1 when not skipping + depth := 0; + unicode_skip := 0; // the fallback characters left to drop after a \uN + i := 0; + while i < rtf.count { + c := rtf[i]; + if c == #char "{" { depth += 1; i += 1; continue; } + if c == #char "}" { + if skip_depth == depth skip_depth = -1; + depth -= 1; i += 1; continue; + } + if c == #char "\r" || c == #char "\n" { i += 1; continue; } + if c != #char "\\" { + if skip_depth < 0 { + if unicode_skip > 0 unicode_skip -= 1; + else append(*out, c); + } + i += 1; + continue; + } + + // A control: \word[-123][ ], \'hh, or \ followed by one character. + i += 1; + if i >= rtf.count break; + c = rtf[i]; + if c == #char "'" && i + 2 < rtf.count { + byte := rtf_hex(rtf[i + 1]) * 16 + rtf_hex(rtf[i + 2]); + i += 3; + if skip_depth >= 0 continue; + if unicode_skip > 0 { unicode_skip -= 1; continue; } + rtf_append_code_point(*out, cp1252_to_unicode(cast(u8) byte)); + continue; + } + if !is_alpha(c) { + i += 1; + if skip_depth >= 0 continue; + if c == #char "*" skip_depth = depth; // \* starts a destination this reader does not know + else if c == #char "\\" || c == #char "{" || c == #char "}" append(*out, c); + else if c == #char "~" append(*out, " "); + continue; + } + word_start := i; + while i < rtf.count && is_alpha(rtf[i]) i += 1; + word := slice(rtf, word_start, i - word_start); + number_start := i; + if i < rtf.count && rtf[i] == #char "-" i += 1; + while i < rtf.count && is_digit(rtf[i]) i += 1; + number_text := slice(rtf, number_start, i - number_start); + if i < rtf.count && rtf[i] == #char " " i += 1; // the space ending a control word is part of it + if skip_depth >= 0 continue; + + if word == { + case "fonttbl"; #through; + case "colortbl"; #through; + case "stylesheet"; #through; + case "info"; #through; + case "pict"; #through; + case "header"; #through; + case "footer"; #through; + case "generator"; + skip_depth = depth; + case "par"; #through; + case "line"; + append(*out, "\n"); + case "tab"; + append(*out, "\t"); + case "u"; + value, ok := string_to_int(number_text); + if ok { + if value < 0 value += 65536; + rtf_append_code_point(*out, cast(u32) value); + unicode_skip = 1; + } + } + } + return trim(builder_to_string(*out,, temp)); +} + +#scope_file + +catalog_string :: (object: Capture_Value, key: string) -> string { + value := capture_object_find(*object, key); + if !value || value.kind != .STRING return ""; + return copy_string(value.text); +} + +rtf_hex :: (c: u8) -> s64 { + if c >= #char "0" && c <= #char "9" return c - #char "0"; + if c >= #char "a" && c <= #char "f" return c - #char "a" + 10; + if c >= #char "A" && c <= #char "F" return c - #char "A" + 10; + return 0; +} + +// Windows-1252, which Snagit's RTF uses for \'hh: Latin-1 except 0x80 to 0x9F. +cp1252_to_unicode :: (byte: u8) -> u32 { + HIGH :: u32.[0x20AC, 0x81, 0x201A, 0x0192, 0x201E, 0x2026, 0x2020, 0x2021, 0x02C6, 0x2030, 0x0160, 0x2039, 0x0152, 0x8D, 0x017D, 0x8F, + 0x90, 0x2018, 0x2019, 0x201C, 0x201D, 0x2022, 0x2013, 0x2014, 0x02DC, 0x2122, 0x0161, 0x203A, 0x0153, 0x9D, 0x017E, 0x0178]; + if byte >= 0x80 && byte <= 0x9F return HIGH[byte - 0x80]; + return byte; +} + +rtf_append_code_point :: (out: *String_Builder, code_point: u32) { + if code_point < 0x80 { + append(out, cast(u8) code_point); + } else if code_point < 0x800 { + append(out, cast(u8) (0xC0 | (code_point >> 6))); + append(out, cast(u8) (0x80 | (code_point & 0x3F))); + } else { + append(out, cast(u8) (0xE0 | (code_point >> 12))); + append(out, cast(u8) (0x80 | ((code_point >> 6) & 0x3F))); + append(out, cast(u8) (0x80 | (code_point & 0x3F))); + } +} diff --git a/src/app/main.jai b/src/app/main.jai index 7693405..95c6411 100644 --- a/src/app/main.jai +++ b/src/app/main.jai @@ -17,6 +17,8 @@ main :: () { app.headless = array_find(app.args, "-headless") || NO_CRASH_DIALOG; runtime_support_headless = app.headless; app.allow_multiple_instances = array_find(app.args, "-allow-multiple-instances"); + for app.args if it == "-library" && it_index + 1 < app.args.count app.library_folder = app.args[it_index + 1]; + app.scan_and_exit = array_find(app.args, "-scan"); defer report_memory_leaks(); diff --git a/src/app/ocr_ui.jai b/src/app/ocr_ui.jai index 5f4911f..63e620c 100644 --- a/src/app/ocr_ui.jai +++ b/src/app/ocr_ui.jai @@ -1,65 +1,127 @@ -// OCR in the app: an engine made on a background thread at startup (modules/OCR), an OCR window that -// says whether it runs on the GPU and, if not, what to install, and the OCR button in the Capture -// window, which reads the capture shown and draws its lines over it. +// OCR in the app (modules/OCR): an engine made on a background thread, the OCR window (whether it runs +// on the GPU and, if not, what to install; the model, kept in the settings file; the library scan), +// and in the Capture window the OCR button, the lines read drawn over the image, and the search's +// matches among them. // -// Engines take a moment to make and OCR takes a moment to run, so both happen on one background -// thread; the UI only looks at what it has finished. +// Every reading is stored in its .mcap (mcap_store_ocr: one section per model) and its catalog entry +// is loaded again, so the overlay, the text shown and search all come from what is stored. +// +// Making an engine, reading one capture and the scan all run on one background thread, one at a time; +// the UI only looks at what it has finished, and at the scan's progress counters. -OCR_WINDOW :: "OCR"; -OCR_DEFAULT_MODEL :: "ppocr-v5-mobile"; +OCR_WINDOW :: "OCR"; +OCR_DEFAULT_MODEL :: "ppocr-v5-mobile"; +OCR_SCAN_BATCH :: 8; // captures per ocr_process call in a scan +OCR_STORE_ATTEMPTS :: 4; // a file the viewer has open at that moment is tried again, 250 ms apart ocr_ui: struct { models: [] Ocr_Model_Info; models_error: string; - model_index: s32; // the model chosen in the window + model_index: s32; // the model chosen in the OCR window // Owned by the main thread while thread_busy is false; by the background thread while it is true. engine: *Ocr_Engine; - engine_model: string; // the model engine runs engine_error: string; // why the last engine could not be made thread: Thread; thread_started: bool; - thread_busy: bool; // atomic: set by the main thread, cleared by the background one + thread_busy: bool; // set by the main thread, cleared by the background one task: Ocr_Task; - // The last capture read, for the Capture window. - lines: [] Ocr_Line; - lines_index: s32 = -1; // the capture they are for - lines_error: string; - lines_seconds: float64; + read_error: string; // why the last single read failed + read_index: s32 = -1; + read_seconds: float64; + + scan: Ocr_Scan; + scan_requested: bool; // -scan has started its scan + + // The Capture window. + overlay_model: string; // the section shown over the image; "" for the chosen model's show_boxes := true; + match_cursor: s64 = -1; // which match Next and Previous last went to + scroll_to_match: bool; window_open := false; toast_shown := false; } -Ocr_Task_Kind :: enum { NONE; CREATE; READ; } +Ocr_Task_Kind :: enum { NONE; CREATE; READ; SCAN; } Ocr_Task :: struct { kind: Ocr_Task_Kind; - model: string; - path: string; // READ: the .mcap - index: s32; // READ: its capture index + model: string; // CREATE + path: string; // READ + index: s32; // READ } +// The library scan. The counters and flags are shared with the background thread. +Ocr_Scan :: struct { + items: [..] Scan_Item; // the captures it reads, in display order + model: string; + total: s64; + done: s64; // read and stored, or failed + failed: s64; + paused: bool; + cancel: bool; + running: bool; + start: Apollo_Time; + paused_seconds: float64; // time spent paused, left out of the rate + pause_start: Apollo_Time; +} + +Scan_Item :: struct { index: s32; path: string; } + ocr_models_folder :: () -> string { return tprint("%ocr-models", path_strip_filename(get_path_of_running_executable())); } +// ---------------------------------------------------------------- startup and shutdown + ocr_ui_startup :: () { push_allocator(context.default_allocator); models, error := ocr_list_models(ocr_models_folder()); ocr_ui.models = models; if error ocr_ui.models_error = copy_string(error); - for ocr_ui.models if it.name == OCR_DEFAULT_MODEL ocr_ui.model_index = cast(s32) it_index; + ocr_ui.model_index = ocr_model_index(app_user_settings.ocr_model); if ocr_ui.models.count ocr_ui_start(.{kind = .CREATE, model = ocr_ui.models[ocr_ui.model_index].name}); } ocr_ui_shutdown :: () { - if ocr_ui.thread_started thread_is_done(*ocr_ui.thread, -1); // finishes what it is doing + atomic_swap(*ocr_ui.scan.cancel, true); // a scan stops after the capture it is on + atomic_swap(*ocr_ui.scan.paused, false); + if ocr_ui.thread_started thread_is_done(*ocr_ui.thread, -1); ocr_destroy(ocr_ui.engine); } +// The model the settings name, or the default one, or the first; with a log line when the settings +// name one that is not in models.config. +ocr_model_index :: (name: string) -> s32 { + wanted := ifx name.count then name else OCR_DEFAULT_MODEL; + for ocr_ui.models if it.name == wanted return cast(s32) it_index; + if name.count log_error("[OCR] The settings name model %, which models.config does not have; using %.", name, OCR_DEFAULT_MODEL); + for ocr_ui.models if it.name == OCR_DEFAULT_MODEL return cast(s32) it_index; + return 0; +} + +// The settings file named another model (edited by hand): switch to it. +ocr_ui_settings_changed :: () { + if !ocr_ui.models.count return; + index := ocr_model_index(app_user_settings.ocr_model); + if index == ocr_ui.model_index return; + ocr_ui_choose_model(index, save = false); +} + +ocr_ui_choose_model :: (index: s32, save := true) { + if ocr_ui.thread_busy return; + ocr_ui.model_index = index; + ocr_ui.toast_shown = true; // said once already, or not needed + ocr_ui_start(.{kind = .CREATE, model = ocr_ui.models[index].name}); + if save { + free(app_user_settings.ocr_model,, context.default_allocator); + app_user_settings.ocr_model = copy_string(ocr_ui.models[index].name,, context.default_allocator); + app_save_user_settings_file(); + } +} + // Starts task on the background thread; false while it is still busy with the last one. ocr_ui_start :: (task: Ocr_Task) -> bool { if ocr_ui.thread_busy return false; @@ -76,6 +138,8 @@ ocr_ui_start :: (task: Ocr_Task) -> bool { return true; } +// ---------------------------------------------------------------- the background thread + ocr_ui_thread_proc :: (thread: *Thread) -> s64 { context.allocator = context.default_allocator; task := ocr_ui.task; @@ -85,18 +149,35 @@ ocr_ui_thread_proc :: (thread: *Thread) -> s64 { ocr_ui.engine = null; engine, ok, error := ocr_create(ocr_models_folder(), task.model); ocr_ui.engine = engine; - ocr_ui.engine_model = task.model; + free(ocr_ui.engine_error); ocr_ui.engine_error = ifx ok then "" else copy_string(error); if ok log("[OCR] % on %.%", task.model, engine.provider, ifx engine.gpu_explanation then tprint(" %", engine.gpu_explanation) else ""); else log_error("[OCR] Could not start %: %", task.model, error); case .READ; start := current_time_monotonic(); - lines, error := ocr_ui_read_capture(task.path); - ocr_ui.lines = lines; - ocr_ui.lines_error = error; - ocr_ui.lines_index = task.index; - ocr_ui.lines_seconds = to_float64_seconds(current_time_monotonic() - start); + items := Scan_Item.[.{task.index, task.path}]; + errors := ocr_read_and_store(items); + free(ocr_ui.read_error); + ocr_ui.read_error = ifx errors[0] then errors[0] else ""; + ocr_ui.read_index = task.index; + ocr_ui.read_seconds = to_float64_seconds(current_time_monotonic() - start); + + case .SCAN; + scan := *ocr_ui.scan; + position := 0; + while position < scan.items.count { + while scan.paused && !scan.cancel sleep_milliseconds(100); + if scan.cancel break; + count := min(OCR_SCAN_BATCH, scan.items.count - position); + errors := ocr_read_and_store(array_view(scan.items, position, count)); + for errors if it { atomic_add(*scan.failed, 1); log_error("[OCR scan] % %", scan.items[position + it_index].path, it); free(it); } + atomic_add(*scan.done, count); + position += count; + reset_temporary_storage(); + } + log("[OCR scan] % of % captures read with % (% failed)%.", scan.done, scan.total, scan.model, scan.failed, ifx scan.cancel then ", cancelled" else ""); + atomic_swap(*scan.running, false); } free(task.model); free(task.path); @@ -104,41 +185,170 @@ ocr_ui_thread_proc :: (thread: *Thread) -> s64 { return 0; } -// The lines in a capture's image as the Capture window shows it: the rendered one, annotations and -// all, or the original when there is none. Heap. -ocr_ui_read_capture :: (path: string) -> [] Ocr_Line, error: string { - if !ocr_ui.engine return .[], copy_string("No OCR engine is running."); - data, read_ok := read_entire_file(path, log_errors = false,, temp); - if !read_ok return .[], sprint("Cannot read %.", path); - m, ok, error := mcap_parse(cast([] u8) data); - if !ok return .[], copy_string(error); - png := mcap_rendered(m); - if !png.count png = m.original; - w, h, channels: s32; - pixels := stbi_load_from_memory(png.data, cast(s32) png.count, *w, *h, *channels, 4); - if !pixels return .[], copy_string("Its image cannot be decoded."); - defer stbi_image_free(pixels); - images := Ocr_Image.[.{pixels, w, h, w * 4}]; - results := ocr_process(ocr_ui.engine, images); - defer array_free(results); - if results[0].failed return .[], results[0].error; - return results[0].lines, ""; +// Reads items with the engine, stores each reading in its file, and hands each new catalog entry to +// the viewer. An error message (heap) for each item that failed, "" for the others. Background thread. +ocr_read_and_store :: (items: [] Scan_Item) -> [] string { + errors := NewArray(items.count, string); + if !ocr_ui.engine { for * errors it.* = copy_string("No OCR engine is running."); return errors; } + + // The image the Capture window shows: the rendered one, annotations and all, or the original. + Prepared :: struct { pixels: *u8; width, height: s32; image: string; } + prepared := NewArray(items.count, Prepared,, temp); + images: [..] Ocr_Image; + images.allocator = temp; + image_of: [..] s64; // which item each image is + image_of.allocator = temp; + for item, n: items { + data, read_ok := read_entire_file(item.path, log_errors = false,, temp); + if !read_ok { errors[n] = copy_string("cannot be read"); continue; } + m, ok, error := mcap_parse(cast([] u8) data); + if !ok { errors[n] = copy_string(error); continue; } + png := mcap_rendered(m); + kind := "rendered"; + if !png.count { png = m.original; kind = "original"; } + w, h, channels: s32; + pixels := stbi_load_from_memory(png.data, cast(s32) png.count, *w, *h, *channels, 4); + if !pixels { errors[n] = copy_string("its image cannot be decoded"); continue; } + prepared[n] = .{pixels, w, h, kind}; + array_add(*images, .{pixels, w, h, w * 4}); + array_add(*image_of, n); + } + defer for prepared if it.pixels stbi_image_free(it.pixels); + + results := ocr_process(ocr_ui.engine, images,, temp); + engine_text := tprint("onnxruntime %, %", ort.version, ifx ocr_ui.engine.provider == .CUDA then "CUDA" else "CPU"); + for result, r: results { + n := image_of[r]; + if result.failed { errors[n] = copy_string(result.error); continue; } + + section: Mcap_Ocr_Section; + section.model = ocr_ui.engine.model.name; + section.engine = engine_text; + section.image = prepared[n].image; + section.image_width, section.image_height = prepared[n].width, prepared[n].height; + section.date = mcap_ocr_now(); + section.lines.allocator = temp; + for result.lines { + line := array_add(*section.lines); + line.* = .{x = it.x, y = it.y, width = it.width, height = it.height, confidence = it.confidence, text = it.text}; + for corner: it.corners { + line.corners[it_index * 2] = cast(s32) round(corner.x); + line.corners[it_index * 2 + 1] = cast(s32) round(corner.y); + } + } + + stored: bool; + error: string; + for 1..OCR_STORE_ATTEMPTS { + stored, error = mcap_store_ocr(items[n].path, section); + if stored break; + sleep_milliseconds(250); + } + if !stored { errors[n] = copy_string(error); continue; } + + entry, load_error := catalog_load(items[n].path); + if !entry { errors[n] = sprint("stored, but could not be read back: %", load_error); continue; } + lock(*viewer.mutex); + array_add(*viewer.catalog_updates, .{items[n].index, entry}); + unlock(*viewer.mutex); + } + return errors; } -ocr_ui_free_lines :: () { - for ocr_ui.lines free(it.text); - array_free(ocr_ui.lines); - free(ocr_ui.lines_error); - ocr_ui.lines = .[]; - ocr_ui.lines_error = ""; - ocr_ui.lines_index = -1; +// ---------------------------------------------------------------- the scan + +// -scan: starts the scan once the engine and the catalog are ready, and ends the app when it is done. +ocr_ui_scan_and_exit :: (busy: bool) { + finish :: (code: s32, format: string, args: ..Any) { + if code log_error(format, ..args); + else log(format, ..args); + app.exit_code = code; + app.should_exit = true; + app.scan_and_exit = false; + } + if ocr_ui.models_error { finish(1, "[OCR scan] %", ocr_ui.models_error); return; } + if viewer.scan_error { finish(1, "[OCR scan] %", viewer.scan_error); return; } + if !ocr_ui.scan_requested { + if busy || !ocr_ui.engine || viewer.catalog_loaded < viewer.captures.count return; + ocr_ui.scan_requested = true; + ocr_scan_start(); + return; + } + if busy || ocr_ui.scan.running return; + finish(ifx ocr_ui.scan.failed then cast(s32) 1 else 0, "[OCR scan] -scan is done: % read, % failed.", ocr_ui.scan.done, ocr_ui.scan.failed); } -// ---------------------------------------------------------------- drawing +ocr_scan_start :: () { + if ocr_ui.thread_busy || !ocr_ui.engine return; + scan := *ocr_ui.scan; + for scan.items free(it.path); + array_reset(*scan.items); + free(scan.model); + scan.* = .{}; + scan.items.allocator = context.default_allocator; + scan.model = copy_string(ocr_ui.engine.model.name,, context.default_allocator); + // Every capture without a reading by this model, in the order the grid shows them. + for viewer.visible { + capture := *viewer.captures[it]; + if capture.catalog && catalog_section(capture.catalog, scan.model) continue; + array_add(*scan.items, .{it, copy_string(capture.path,, context.default_allocator)}); + } + scan.total = scan.items.count; + if !scan.total { toast_show(.Success, "Every capture shown has been read with %", scan.model); return; } + scan.running = true; + scan.start = current_time_monotonic(); + ocr_ui_start(.{kind = .SCAN}); +} + +ocr_scan_ui :: () { using ImGui; + scan := *ocr_ui.scan; + busy := ocr_ui.thread_busy; + catalog_ready := viewer.catalog_loaded >= viewer.captures.count; + + if !scan.running { + can_scan := ocr_ui.engine && !busy && catalog_ready; + if !can_scan BeginDisabled(); + label := ifx viewer.search_terms.count then "Scan the captures shown" else "Scan the library"; + if Button(temp_c_string(label)) ocr_scan_start(); + if !can_scan EndDisabled(); + SameLine(); + if !catalog_ready TextColored(app_theme.palette.dim, "(once the catalog has loaded)"); + else TextColored(app_theme.palette.dim, "Reads every capture without a reading by this model. Each is stored as it is read, so a scan can stop and start again."); + if scan.total && scan.done { + TextColored(app_theme.palette.dim, "Last scan: % of % read with %, % failed.", scan.done, scan.total, scan.model, scan.failed); + } + return; + } + + done, total, failed := scan.done, scan.total, scan.failed; + elapsed := to_float64_seconds(current_time_monotonic() - scan.start) - scan.paused_seconds; + if scan.paused elapsed -= to_float64_seconds(current_time_monotonic() - scan.pause_start); + rate := ifx elapsed > 0 then done / elapsed else 0; + left := ifx rate > 0 then (total - done) / rate else 0; + ProgressBar(cast(float) done / max(total, 1), .{-1, 0}, temp_c_string(tprint("% / %", done, total))); + Text("% captures/s, % left, % failed, on %", FF(rate, 1), ocr_duration_text(left), failed, ifx ocr_ui.engine && ocr_ui.engine.provider == .CUDA then "the GPU" else "the CPU"); + if scan.paused { + if Button("Resume") { scan.paused_seconds += to_float64_seconds(current_time_monotonic() - scan.pause_start); atomic_swap(*scan.paused, false); } + } else { + if Button("Pause") { scan.pause_start = current_time_monotonic(); atomic_swap(*scan.paused, true); } + } + SameLine(); + if Button("Cancel") { atomic_swap(*scan.cancel, true); atomic_swap(*scan.paused, false); } +} + +ocr_duration_text :: (seconds: float64) -> string { + s := cast(s64) seconds; + if s >= 3600 return tprint("% h % min", s / 3600, (s % 3600) / 60); + if s >= 60 return tprint("% min % s", s / 60, s % 60); + return tprint("% s", s); +} + +// ---------------------------------------------------------------- the OCR window // Each frame: the toast once the engine is up on the CPU though a GPU could run it, and the OCR window. ocr_ui_frame :: () { using ImGui; busy := ocr_ui.thread_busy; + if app.scan_and_exit ocr_ui_scan_and_exit(busy); if !busy && ocr_ui.engine && !ocr_ui.toast_shown { ocr_ui.toast_shown = true; in_use, explanation := ocr_gpu_status(ocr_ui.engine); @@ -149,46 +359,46 @@ ocr_ui_frame :: () { using ImGui; } if !ocr_ui.window_open return; - SetNextWindowSize(.{640, 360}, .FirstUseEver); + SetNextWindowSize(.{680, 420}, .FirstUseEver); if Begin(OCR_WINDOW, *ocr_ui.window_open) { if ocr_ui.models_error { TextColored(app_theme.palette.warning, "%", ocr_ui.models_error); TextWrapped("Run `jai fetch_models.jai` in modules/OCR and `jai fetch.jai` in modules/onnxruntime, then rebuild."); } else { - // The model: changing it makes a new engine on the background thread. + // The model: changing it makes a new engine, and is kept in the settings file. preview := ifx ocr_ui.models.count then ocr_ui.models[ocr_ui.model_index].title else ""; + if busy BeginDisabled(); SetNextItemWidth(320); if BeginCombo("Model", temp_c_string(preview)) { for ocr_ui.models { - if Selectable(temp_c_string(it.title), it_index == ocr_ui.model_index) && it_index != ocr_ui.model_index && !busy { - ocr_ui.model_index = cast(s32) it_index; - ocr_ui.toast_shown = true; // said once already, or not needed - ocr_ui_free_lines(); - ocr_ui_start(.{kind = .CREATE, model = it.name}); - } + if Selectable(temp_c_string(it.title), it_index == ocr_ui.model_index) && it_index != ocr_ui.model_index ocr_ui_choose_model(cast(s32) it_index); } EndCombo(); } + if busy EndDisabled(); if busy && ocr_ui.task.kind == .CREATE { TextColored(app_theme.palette.dim, "Starting % ...", ocr_ui.task.model); } else if ocr_ui.engine_error { TextColored(app_theme.palette.warning, "OCR could not start:"); TextWrapped("%", ocr_ui.engine_error); - } else if ocr_ui.engine && !busy { + } else if ocr_ui.engine { in_use, explanation := ocr_gpu_status(ocr_ui.engine); Text("Running on: %", ifx in_use then "the GPU (CUDA)" else "the CPU"); SameLine(); TextColored(app_theme.palette.dim, "onnxruntime %, % worker threads", ort.version, ocr_ui.engine.workers); + Separator(); + ocr_scan_ui(); if explanation { Separator(); text_with_links(explanation); } } - // The text of the capture last read, in reading order. - if !busy && ocr_ui.lines_index >= 0 && ocr_ui.lines_index < viewer.captures.count { + // The text of the capture shown in the Capture window, as the overlay's model read it. + section := ocr_overlay_section(); + if section && viewer.full_index >= 0 { Separator(); - Text("Text of %:", viewer.captures[ocr_ui.lines_index].name); + Text("Text of % (% , % lines):", viewer.captures[viewer.full_index].name, section.model, section.lines.count); BeginChild("ocr text"); - ocr_ui_capture_text(ocr_ui.lines_index); + for section.lines TextUnformatted(it.text); EndChild(); } } @@ -196,45 +406,6 @@ ocr_ui_frame :: () { using ImGui; End(); } -// The OCR row of the Capture window, above its image: the button, and how the last read went. -ocr_ui_capture_controls :: (index: s32, path: string) { using ImGui; - busy := ocr_ui.thread_busy; - can_read := ocr_ui.engine && !busy; - if !can_read BeginDisabled(); - if SmallButton("OCR") { - ocr_ui_free_lines(); - ocr_ui_start(.{kind = .READ, path = path, index = index}); - } - if !can_read EndDisabled(); - SameLine(); - if busy && ocr_ui.task.kind == .READ && ocr_ui.task.index == index { - TextColored(app_theme.palette.dim, "Reading..."); - } else if !busy && ocr_ui.lines_index == index { - if ocr_ui.lines_error TextColored(app_theme.palette.warning, "%", ocr_ui.lines_error); - else TextColored(app_theme.palette.dim, "% lines in % s", ocr_ui.lines.count, FF(ocr_ui.lines_seconds, 2)); - SameLine(); - Checkbox("Boxes", *ocr_ui.show_boxes); - } else if !ocr_ui.engine { - TextColored(app_theme.palette.dim, "(OCR is not running; see the OCR window)"); - } - SameLine(); - if SmallButton("OCR window") ocr_ui.window_open = true; -} - -// The lines read from this capture, as boxes over its image, drawn at image_min with scale; and their -// text in a list below the image area. -ocr_ui_capture_overlay :: (index: s32, image_min: ImGui.ImVec2, scale: float) { using ImGui; - if ocr_ui.thread_busy || ocr_ui.lines_index != index || !ocr_ui.show_boxes return; - draw := GetWindowDrawList(); - colour := GetColorU32(.PlotHistogram); - for ocr_ui.lines { - p0 := ImVec2.{image_min.x + it.x * scale, image_min.y + it.y * scale}; - p1 := ImVec2.{p0.x + it.width * scale, p0.y + it.height * scale}; - draw.AddRect(draw, p0, p1, colour, 0, 1.5); - if IsMouseHoveringRect(p0, p1) SetTooltip("% (%)", it.text, FF(it.confidence, 2)); - } -} - // Text a line at a time, each https:// address in it drawn as a link that opens in the browser. text_with_links :: (text: string) { using ImGui; for line: split(text, "\n",, temp) { @@ -250,7 +421,113 @@ text_with_links :: (text: string) { using ImGui; } } -ocr_ui_capture_text :: (index: s32) { using ImGui; - if ocr_ui.thread_busy || ocr_ui.lines_index != index || !ocr_ui.lines.count return; - for ocr_ui.lines TextUnformatted(it.text); +// ---------------------------------------------------------------- the Capture window + +// The reading shown over the capture in the Capture window: the overlay's choice, else the chosen +// model's, else the first there is. +ocr_overlay_section :: () -> *Mcap_Ocr_Section { + if viewer.full_index < 0 || viewer.full_index >= viewer.captures.count return null; + entry := viewer.captures[viewer.full_index].catalog; + if !entry || !entry.ocr.count return null; + if ocr_ui.overlay_model.count { + section := catalog_section(entry, ocr_ui.overlay_model); + if section return section; + } + if ocr_ui.models.count { + section := catalog_section(entry, ocr_ui.models[ocr_ui.model_index].name); + if section return section; + } + return *entry.ocr[0]; +} + +// The OCR row of the Capture window, above its image. +ocr_ui_capture_controls :: (index: s32, path: string) { using ImGui; + busy := ocr_ui.thread_busy; + entry := viewer.captures[index].catalog; + model := ifx ocr_ui.models.count then ocr_ui.models[ocr_ui.model_index].name else ""; + has_reading := entry && catalog_section(entry, model) != null; + + can_read := ocr_ui.engine && !busy; + if !can_read BeginDisabled(); + if SmallButton(temp_c_string(ifx has_reading then "Read again" else "OCR")) ocr_ui_start(.{kind = .READ, path = path, index = index}); + if !can_read EndDisabled(); + SameLine(); + if busy && ocr_ui.task.kind == .READ && ocr_ui.task.index == index { + TextColored(app_theme.palette.dim, "Reading..."); + SameLine(); + } else if !busy && ocr_ui.read_index == index { + if ocr_ui.read_error { TextColored(app_theme.palette.warning, "%", ocr_ui.read_error); SameLine(); } + else { TextColored(app_theme.palette.dim, "read in % s", FF(ocr_ui.read_seconds, 2)); SameLine(); } + } + + section := ocr_overlay_section(); + if section { + // A capture read by several models: which reading to show. + if entry.ocr.count > 1 { + SetNextItemWidth(170); + if BeginCombo("##reading", temp_c_string(section.model)) { + for entry.ocr if Selectable(temp_c_string(it.model), it.model == section.model) { + free(ocr_ui.overlay_model,, context.default_allocator); + ocr_ui.overlay_model = copy_string(it.model,, context.default_allocator); + } + EndCombo(); + } + SameLine(); + } + Checkbox("Boxes", *ocr_ui.show_boxes); + if viewer.search_terms.count { + matches := 0; + for section.lines if search_line_matches(it.text, viewer.search_terms) matches += 1; + SameLine(); + Text("Matches: %", matches); + if matches { + SameLine(); if SmallButton("Previous") { ocr_ui.match_cursor = (ocr_ui.match_cursor - 1 + matches) % matches; ocr_ui.scroll_to_match = true; } + SameLine(); if SmallButton("Next") { ocr_ui.match_cursor = (ocr_ui.match_cursor + 1) % matches; ocr_ui.scroll_to_match = true; } + } + } + SameLine(); + } else if !ocr_ui.engine { + TextColored(app_theme.palette.dim, "(OCR is not running; see the OCR window)"); + SameLine(); + } + if SmallButton("OCR window") ocr_ui.window_open = true; +} + +// The reading's lines as outlines over the image (turned with the text, for a turned line), drawn at +// image_min with zoom (the image's size as shown over its captured size): those matching the search +// highlighted, the others dimmer. Inside the image's scrolling child, which Next and Previous scroll. +ocr_ui_capture_overlay :: (index: s32, image_min: ImGui.ImVec2, zoom: float) { using ImGui; + section := ocr_overlay_section(); + if !section || !ocr_ui.show_boxes return; + // The section's pixels may be another image's (the original, where the rendered one differs). + scale_x := zoom * ifx section.image_width then cast(float) viewer.full_source_width / section.image_width else 1; + scale_y := zoom * ifx section.image_height then cast(float) viewer.full_source_height / section.image_height else 1; + + draw := GetWindowDrawList(); + searching := viewer.search_terms.count > 0; + plain := GetColorU32(.PlotHistogram, ifx searching then 0.35 else 1.0); + matching := GetColorU32(.PlotLinesHovered); + match_number := 0; + for section.lines { + p0 := ImVec2.{image_min.x + it.x * scale_x, image_min.y + it.y * scale_y}; + p1 := ImVec2.{p0.x + it.width * scale_x, p0.y + it.height * scale_y}; + c := mcap_ocr_corners(it); + q: [4] ImVec2; + for * q it.* = .{image_min.x + c[it_index * 2] * scale_x, image_min.y + c[it_index * 2 + 1] * scale_y}; + is_match := searching && search_line_matches(it.text, viewer.search_terms); + if is_match { + draw.AddQuadFilled(draw, q[0], q[1], q[2], q[3], GetColorU32(.PlotLinesHovered, 0.25)); + draw.AddQuad(draw, q[0], q[1], q[2], q[3], matching, 2.5); + if ocr_ui.scroll_to_match && match_number == ocr_ui.match_cursor { + // Into view, a third of the way down. + SetScrollX(it.x * scale_x - GetWindowWidth() / 3); + SetScrollY(it.y * scale_y - GetWindowHeight() / 3); + ocr_ui.scroll_to_match = false; + } + match_number += 1; + } else { + draw.AddQuad(draw, q[0], q[1], q[2], q[3], plain, 1.5); + } + if IsMouseHoveringRect(p0, p1) SetTooltip("% (%)", it.text, FF(it.confidence, 2)); + } } diff --git a/src/app/viewer.jai b/src/app/viewer.jai index cf6efc8..c343464 100644 --- a/src/app/viewer.jai +++ b/src/app/viewer.jai @@ -2,8 +2,16 @@ // its own. Scrolling never waits on the disk: previews are read and decoded on worker threads, // those in view first and then the rows around them, and are uploaded a few per frame. Textures // for previews far out of view are released once there are more than VIEWER_MAX_TEXTURES. +// +// When the workers have no previews to decode they load the catalog (app/catalog.jai), what search +// and sorting need from each capture. The grid shows viewer.visible: the captures in display order, +// newest first once the catalog is complete, and only those matching the search. -VIEWER_FOLDER :: "D:/SnagitIndex/mcap"; // hardcoded for now +VIEWER_FOLDER :: "D:/SnagitIndex/mcap"; // hardcoded for now; -library FOLDER shows another + +viewer_folder :: () -> string { + return ifx app.library_folder then app.library_folder else VIEWER_FOLDER; +} VIEWER_WORKER_COUNT :: 4; VIEWER_MAX_TEXTURES :: 1200; // previews on the GPU at once, about 200 KB each VIEWER_PREFETCH_ROWS :: 8; // most rows decoded ahead of the view, above and below @@ -36,6 +44,13 @@ Viewer_Capture :: struct { last_wanted_frame: u64; // last in the rows viewer_request_previews asked for failed: bool; in_flight: bool; // under viewer.mutex: queued for a worker, or being decoded + catalog: *Catalog_Entry; // main thread only; null until loaded +} + +// A catalog entry made off the main thread, for the main thread to put in place. +Catalog_Update :: struct { + index: s32; + entry: *Catalog_Entry; // null when it could not be loaded } // How the full-size window sizes its image. @@ -78,6 +93,17 @@ viewer: struct { quit: bool; workers: [VIEWER_WORKER_COUNT] Thread; workers_started: s64; + catalog_next: s64; // the next capture whose catalog entry a worker loads + catalog_updates: [..] Catalog_Update; // made, waiting for the main thread + + // Display order and search; main thread only. + visible: [..] s32; // indices into captures, as the grid shows them + visible_dirty := true; + visible_rebuilt_at: Apollo_Time; + catalog_loaded: s64; // entries put in place, or found unreadable + sorted_by_date: bool; + search_buffer: [256] u8; + search_terms: [..] Search_Term; // The full view; main thread only. full_open: bool; @@ -98,12 +124,15 @@ viewer_startup :: () { viewer.with_texture.allocator = context.default_allocator; viewer.queue.allocator = context.default_allocator; viewer.done.allocator = context.default_allocator; + viewer.catalog_updates.allocator = context.default_allocator; + viewer.visible.allocator = context.default_allocator; + viewer.search_terms.allocator = context.default_allocator; init(*viewer.mutex, "viewer"); init(*viewer.wake); start := current_time_monotonic(); - files := file_list(VIEWER_FOLDER,, temp); + files := file_list(viewer_folder(),, temp); for files { name := path_filename(it); if !ends_with_nocase(name, tprint(".%", MCAP_EXTENSION)) continue; @@ -113,8 +142,8 @@ viewer_startup :: () { } // Newest first, roughly: dated names (2026-...) sort after the older {GUID} ones. quick_sort(viewer.captures, (a: Viewer_Capture, b: Viewer_Capture) -> s64 { return compare(b.name, a.name); }); - if !viewer.captures.count viewer.scan_error = sprint("No .% files in %.", MCAP_EXTENSION, VIEWER_FOLDER); - log("[viewer] Found % captures in % in % ms.", viewer.captures.count, VIEWER_FOLDER, FF(to_float64_seconds(current_time_monotonic() - start) * 1000.0, 1)); + if !viewer.captures.count viewer.scan_error = sprint("No .% files in %.", MCAP_EXTENSION, viewer_folder()); + log("[viewer] Found % captures in % in % ms.", viewer.captures.count, viewer_folder(), FF(to_float64_seconds(current_time_monotonic() - start) * 1000.0, 1)); for * viewer.workers { // Reading a front takes 64 KB of temporary storage, and a full image several MB. @@ -144,6 +173,7 @@ viewer_worker :: (thread: *Thread) -> s64 { while true { index: s32 = -1; full: bool; + catalog: bool; request: u64; path: string; @@ -157,11 +187,26 @@ viewer_worker :: (thread: *Thread) -> s64 { } else if viewer.queue_next < viewer.queue.count { index = viewer.queue[viewer.queue_next]; viewer.queue_next += 1; + } else if viewer.catalog_next < viewer.captures.count { + // Nothing to show is waiting: load the next catalog entry. + catalog = true; + index = cast(s32) viewer.catalog_next; + viewer.catalog_next += 1; } if index >= 0 path = viewer.captures[index].path; // the array no longer grows after startup unlock(*viewer.mutex); if index < 0 break; // nothing to do: wait to be woken + if catalog { + entry, error := catalog_load(path); + if !entry log_error("[catalog] % %", path, error); + reset_temporary_storage(); + lock(*viewer.mutex); + array_add(*viewer.catalog_updates, .{index, entry}); + unlock(*viewer.mutex); + continue; + } + result := ifx full then viewer_decode_full(path) else viewer_decode_preview(path); result.index = index; @@ -318,6 +363,8 @@ viewer_frame_gpu_work :: () { } viewer_evict(); + viewer_apply_catalog_updates(); + if viewer.visible_dirty viewer_rebuild_visible(); } // When a preview was last in view or wanted ahead of it. A prefetched preview has not been drawn @@ -326,6 +373,69 @@ viewer_last_used :: (capture: Viewer_Capture) -> u64 { return max(capture.last_drawn_frame, capture.last_wanted_frame); } +// Catalog entries made by the workers (or by an OCR scan) go in place here, on the main thread, +// which is the only one that reads them; a replaced entry is freed now, before anything draws it. +viewer_apply_catalog_updates :: () { + lock(*viewer.mutex); + updates := array_copy(viewer.catalog_updates,, temp); + array_reset_keeping_memory(*viewer.catalog_updates); + more_to_load := viewer.catalog_next < viewer.captures.count; + unlock(*viewer.mutex); + + for updates { + capture := *viewer.captures[it.index]; + if !capture.catalog viewer.catalog_loaded += 1; + if it.entry || !capture.catalog { + catalog_entry_free(capture.catalog); + capture.catalog = it.entry; + } + } + if updates.count && viewer.search_terms.count viewer.visible_dirty = true; + if !viewer.sorted_by_date && viewer.catalog_loaded >= viewer.captures.count { + viewer.sorted_by_date = true; + viewer.visible_dirty = true; + } + // The workers sleep when there is nothing to show; wake them while there is catalog to load. + if more_to_load signal(*viewer.wake); +} + +// The captures in display order: by date, newest first, once every date is known (by name until +// then), and only those matching the search. +viewer_rebuild_visible :: () { + // While the catalog loads under a search, matches arrive every frame: a few rebuilds a second do. + if viewer.search_terms.count && !viewer.sorted_by_date && to_float64_seconds(current_time_monotonic() - viewer.visible_rebuilt_at) < 0.25 return; + viewer.visible_dirty = false; + viewer.visible_rebuilt_at = current_time_monotonic(); + + array_reset_keeping_memory(*viewer.visible); + for viewer.captures { + if viewer.search_terms.count && !search_matches(it.catalog, viewer.search_terms) continue; + array_add(*viewer.visible, cast(s32) it_index); + } + if viewer.sorted_by_date { + quick_sort(viewer.visible, (a: s32, b: s32) -> s64 { + ca, cb := *viewer.captures[a], *viewer.captures[b]; + return viewer_display_order(ifx ca.catalog then ca.catalog.date else "", ca.name, ifx cb.catalog then cb.catalog.date else "", cb.name); + }); + } +} + +// The grid's order: newest CaptureDate first, undated captures last, ties by name, newest first. +viewer_display_order :: (date_a: string, name_a: string, date_b: string, name_b: string) -> s64 { + if date_a != date_b { + if !date_a.count return 1; + if !date_b.count return -1; + return compare(date_b, date_a); + } + return compare(name_b, name_a); +} + +// Where index is in the display order, or -1. +viewer_visible_position :: (index: s32) -> s64 { + for viewer.visible if it == index return it_index; + return -1; +} + // Past VIEWER_MAX_TEXTURES, the previews used longest ago are released, never one in view or // wanted around it as of the last frame. viewer_evict :: () { @@ -350,8 +460,8 @@ viewer_evict :: () { // What the workers should decode next: the rows in view, then the rows around them, nearest first. viewer_request_previews :: (first_row: s64, last_row: s64, columns: s64) { - if !viewer.captures.count || columns <= 0 return; - rows := (viewer.captures.count + columns - 1) / columns; + if !viewer.visible.count || columns <= 0 return; + rows := (viewer.visible.count + columns - 1) / columns; // Rows ahead only as far as the textures they need leave half of VIEWER_MAX_TEXTURES spare, so // what is wanted always fits and nothing wanted is ever released to make room. @@ -363,8 +473,9 @@ viewer_request_previews :: (first_row: s64, last_row: s64, columns: s64) { wanted.allocator = temp; add_row :: (wanted: *[..] s32, row: s64, columns: s64) { for column: 0..columns - 1 { - index := row * columns + column; - if index >= viewer.captures.count break; + position := row * columns + column; + if position >= viewer.visible.count break; + index := viewer.visible[position]; capture := *viewer.captures[index]; capture.last_wanted_frame = viewer.frame; // kept by viewer_evict, texture or not if !capture.texture.handle && !capture.failed array_add(wanted, cast(s32) index); @@ -413,11 +524,25 @@ viewer_ui :: () { using ImGui; if viewer.scan_error { TextColored(app_theme.palette.warning, "%", viewer.scan_error); return; } - loaded := viewer.with_texture.count; - Text("% captures in %", viewer.captures.count, VIEWER_FOLDER); + SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520)); + if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes. \"a phrase\" ocr: title: url: app: note:", viewer.search_buffer.data, viewer.search_buffer.count) { + query := to_string(viewer.search_buffer.data); + terms := search_parse(query); + for viewer.search_terms free(it.text); + array_reset_keeping_memory(*viewer.search_terms); + for terms array_add(*viewer.search_terms, .{it.field, copy_string(it.text)}); + viewer.visible_dirty = true; + viewer.visible_rebuilt_at = .{}; // at once, not throttled + } SameLine(); - TextColored(app_theme.palette.dim, "(% previews loaded)", loaded); - SetNextItemWidth(200); + if viewer.search_terms.count Text("% of % captures", viewer.visible.count, viewer.captures.count); + else Text("% captures", viewer.captures.count); + if viewer.catalog_loaded < viewer.captures.count { + SameLine(); + TextColored(app_theme.palette.dim, "(loading the catalog: % of %; sorted by name until then)", viewer.catalog_loaded, viewer.captures.count); + } + SameLine(); + SetNextItemWidth(160); SliderFloat("Size", *viewer.thumb_size, 64, MCAP_PREVIEW_MAX_SIDE, "%.0f px"); BeginChild("grid"); @@ -433,7 +558,7 @@ viewer_grid :: () { using ImGui; label := GetTextLineHeight(); available := GetContentRegionAvail().x; columns := max(1, cast(s64) ((available + style.ItemSpacing.x) / (cell + style.ItemSpacing.x))); - rows := (viewer.captures.count + columns - 1) / columns; + rows := (viewer.visible.count + columns - 1) / columns; row_height := cell + label + style.ItemSpacing.y; draw := GetWindowDrawList(); @@ -452,8 +577,9 @@ viewer_grid :: () { using ImGui; last_row = max(last_row, cast(s64) clipper.DisplayEnd - 1); for row: clipper.DisplayStart..clipper.DisplayEnd - 1 { for column: 0..columns - 1 { - index := row * columns + column; - if index >= viewer.captures.count break; + position := row * columns + column; + if position >= viewer.visible.count break; + index := viewer.visible[position]; capture := *viewer.captures[index]; if column > 0 SameLine(); @@ -530,8 +656,9 @@ viewer_full_window :: () { using ImGui; // Left and Right step through the captures in grid order while this window has focus. if IsWindowFocused(.RootAndChildWindows) { - if IsKeyPressed(.LeftArrow) && viewer.full_index > 0 viewer_request_full(viewer.full_index - 1); - if IsKeyPressed(.RightArrow) && viewer.full_index < viewer.captures.count - 1 viewer_request_full(viewer.full_index + 1); + position := viewer_visible_position(viewer.full_index); + if IsKeyPressed(.LeftArrow) && position > 0 viewer_request_full(viewer.visible[position - 1]); + if IsKeyPressed(.RightArrow) && position >= 0 && position < viewer.visible.count - 1 viewer_request_full(viewer.visible[position + 1]); } if viewer.full_failed { diff --git a/src/test/catalog_tests.jai b/src/test/catalog_tests.jai new file mode 100644 index 0000000..d09ffc6 --- /dev/null +++ b/src/test/catalog_tests.jai @@ -0,0 +1,121 @@ +// The catalog's pieces on their own (app/catalog.jai, app/viewer.jai): the search query's parsing and +// matching, the RTF stripper on Snagit's annotations, and the grid's order by date. + +#scope_file + +checks_run := 0; +checks_failed := 0; + +MAX_GROUPS :: 8; +group_names : [MAX_GROUPS] string; +group_checks : [MAX_GROUPS] int; +group_fails : [MAX_GROUPS] int; +group_count := 0; + +output_builder: *String_Builder; + +emit :: (format_string: string, args: ..Any) { + if output_builder print_to_builder(output_builder, format_string, ..args); + else print(format_string, ..args); +} @PrintLike + +group :: (name: string) { + assert(group_count < MAX_GROUPS, "raise MAX_GROUPS"); + group_names[group_count] = name; + group_count += 1; + emit("\n-- % --\n", name); +} + +expect :: (condition: bool, format: string, args: ..Any, loc := #caller_location) { + checks_run += 1; + if group_count group_checks[group_count-1] += 1; + if condition return; + + checks_failed += 1; + if group_count group_fails[group_count-1] += 1; + emit(" FAIL [%:%] %\n", group_names[max(group_count-1, 0)], loc.line_number, tprint(format, ..args)); +} + +report :: () -> bool { + emit("\n"); + for 0..group_count-1 { + if group_fails[it] emit(" % -- % checks, % FAILED\n", group_names[it], group_checks[it], group_fails[it]); + else emit(" % -- % checks\n", group_names[it], group_checks[it]); + } + if checks_failed emit("\n% OF % CATALOG CHECKS FAILED\n", checks_failed, checks_run); + else emit("\nALL CATALOG TESTS PASSED (% checks over % groups)\n", checks_run, group_count); + return checks_failed == 0; +} + +query_tests :: () { + group("query"); + terms := search_parse(" KiCad \"Footprint Assignments\" url:digikey note:late http://x.y "); + expect(terms.count == 5, "five terms (got %)", terms.count); + if terms.count != 5 return; + expect(terms[0].field == .ANY && terms[0].text == "kicad", "a word, lowercased: %", terms[0]); + expect(terms[1].field == .ANY && terms[1].text == "footprint assignments", "a phrase: %", terms[1]); + expect(terms[2].field == .URL && terms[2].text == "digikey", "a field prefix: %", terms[2]); + expect(terms[3].field == .NOTE && terms[3].text == "late", "note: %", terms[3]); + expect(terms[4].field == .ANY && terms[4].text == "http://x.y", "an unknown prefix stays part of the word: %", terms[4]); + + phrase := search_parse("title:\"two words\""); + expect(phrase.count == 1 && phrase[0].field == .TITLE && phrase[0].text == "two words", "a prefixed phrase: %", phrase); + expect(search_parse(" ").count == 0, "blank is no terms"); +} + +match_tests :: () { + group("matching"); + entry: Catalog_Entry; + entry.find_title = "assign footprints - kicad"; + entry.find_url = "https://www.digikey.com/x"; + entry.find_app = "kicad eda 7.0.2"; + entry.find_notes = "late"; + entry.find_ocr = "0.1uf : capacitor_smd:c_0201_0603metric\nc1 -\n"; + + expect(search_matches(*entry, search_parse("capacitor kicad")), "words from different fields all match"); + expect(!search_matches(*entry, search_parse("capacitor resistor")), "every word must match"); + expect(search_matches(*entry, search_parse("url:digikey")), "a URL word"); + expect(!search_matches(*entry, search_parse("title:digikey")), "a prefix limits the word to its field"); + expect(search_matches(*entry, search_parse("ocr:\"c1 -\"")), "an OCR phrase"); + expect(search_matches(*entry, search_parse("note:LATE")), "matching ignores case"); + expect(search_matches(*entry, search_parse("")), "no terms match everything"); + expect(!search_matches(null, search_parse("x")), "an entry not loaded matches no search"); + + expect(search_line_matches("0.1uF : Capacitor_SMD", search_parse("capacitor")), "a line holding a word"); + expect(!search_line_matches("0.1uF : Capacitor_SMD", search_parse("title:capacitor")), "a title word is not an OCR line's"); +} + +rtf_tests :: () { + group("rtf"); + // A Snagit text annotation, as stored (base64 RTF), whose text is "late". + encoded :: "e1xydGYxXGFuc2lcYW5zaWNwZzEyNTJcZGVmZjBcZGVmbGFuZzEwMzN7XGZvbnR0Ymx7XGYwXGZzd2lzc1xmcHJxMlxmY2hhcnNldDAgQXJpYWw7fX0NCntcY29sb3J0YmwgO1xyZWQyNTJcZ3JlZW42NlxibHVlNjY7fQ0KXHZpZXdraW5kNFx1YzFccGFyZFxxY1x0eDcyMFxjZjFcZjBcZnMyNCBsYXRlXHBhcg0KfQ0K"; + rtf, decoded := base64_decode(encoded,, temp); + expect(decoded, "the sample decodes"); + text := rtf_to_text(rtf); + expect(text == "late", "a Snagit annotation reads as its text: \"%\"", text); + + text = rtf_to_text("{\\rtf1{\\fonttbl{\\f0 Arial;}}{\\*\\generator Riched20;}one\\par two\\tab three caf\\'e9 \\u8364? \\{x\\}}"); + expect(text == "one\ntwo\tthree caf\xC3\xA9 \xE2\x82\xAC {x}", "breaks, tabs, \\'hh, \\uN and escaped braces: \"%\"", text); + expect(rtf_to_text("{\\rtf1\\ansi\\'93quoted\\'94}") == "\xE2\x80\x9Cquoted\xE2\x80\x9D", "Windows-1252 quotes"); +} + +order_tests :: () { + group("order"); + expect(viewer_display_order("2026-10-07 08:00:00", "a", "2025-01-01 00:00:00", "b") < 0, "newer first"); + expect(viewer_display_order("", "a", "2025-01-01 00:00:00", "b") > 0, "undated after dated"); + expect(viewer_display_order("2025-01-01 00:00:00", "a", "", "b") < 0, "dated before undated"); + expect(viewer_display_order("2025-01-01 00:00:00", "b", "2025-01-01 00:00:00", "a") < 0, "the same date: by name, newest first"); +} + +#scope_export + +run_catalog_tests :: (builder: *String_Builder = null) -> bool { + output_builder = builder; + emit("=== catalog test suite ===\n"); + push_allocator(context.default_allocator); + query_tests(); + match_tests(); + rtf_tests(); + order_tests(); + return report(); +} diff --git a/src/test/ocr_tests.jai b/src/test/ocr_tests.jai new file mode 100644 index 0000000..25e7aec --- /dev/null +++ b/src/test/ocr_tests.jai @@ -0,0 +1,213 @@ +// OCR on turned text (modules/OCR): two lines drawn with Windows' GDI, read level, then turned by 15°, +// 90° (read downwards), 180° and 270° (read upwards). Each turn must give back the same text, and the +// line's corners must run the way the text does. Needs the models beside the program (the build copies +// them); without them, or off Windows, the suite says so and passes. + +#scope_file + +checks_run := 0; +checks_failed := 0; + +MAX_GROUPS :: 8; +group_names : [MAX_GROUPS] string; +group_checks : [MAX_GROUPS] int; +group_fails : [MAX_GROUPS] int; +group_count := 0; + +output_builder: *String_Builder; + +emit :: (format_string: string, args: ..Any) { + if output_builder print_to_builder(output_builder, format_string, ..args); + else print(format_string, ..args); +} @PrintLike + +group :: (name: string) { + assert(group_count < MAX_GROUPS, "raise MAX_GROUPS"); + group_names[group_count] = name; + group_count += 1; + emit("\n-- % --\n", name); +} + +expect :: (condition: bool, format: string, args: ..Any, loc := #caller_location) { + checks_run += 1; + if group_count group_checks[group_count-1] += 1; + if condition return; + + checks_failed += 1; + if group_count group_fails[group_count-1] += 1; + emit(" FAIL [%:%] %\n", group_names[max(group_count-1, 0)], loc.line_number, tprint(format, ..args)); +} + +report :: () -> bool { + emit("\n"); + for 0..group_count-1 { + if group_fails[it] emit(" % -- % checks, % FAILED\n", group_names[it], group_checks[it], group_fails[it]); + else emit(" % -- % checks\n", group_names[it], group_checks[it]); + } + if checks_failed emit("\n% OF % OCR CHECKS FAILED\n", checks_failed, checks_run); + else emit("\nALL OCR TESTS PASSED (% checks over % groups)\n", checks_run, group_count); + return checks_failed == 0; +} + +LINES :: string.["Assign Footprints to Symbols", "Capacitor 100nF 0402"]; +TURNS :: float.[0, 15, 90, 180, 270]; // degrees, clockwise on the screen; 0 is the text as drawn + +// Lowercase, without spaces: recognition may split or join words differently once turned. +squeezed :: (s: string) -> string { + builder: String_Builder; + builder.allocator = temp; + for 0..s.count - 1 if s[it] != #char " " append(*builder, to_lower(s[it])); + return builder_to_string(*builder,, temp); +} + +// The line found holding wanted, or null. +find_line :: (lines: [] Ocr_Line, wanted: string) -> *Ocr_Line { + for * lines if contains(squeezed(it.text), squeezed(wanted)) return it; + return null; +} + +// The image turned clockwise by degrees about its centre onto a white canvas just big enough. Temp. +turned_image :: (source: Ocr_Image, degrees: float) -> Ocr_Image { + radians := degrees * PI / 180; + c, s := cos(radians), sin(radians); + if abs(c) < 0.000001 c = 0; + if abs(s) < 0.000001 s = 0; + result: Ocr_Image; + result.width = cast(s32) ceil(abs(source.width * c) + abs(source.height * s)); + result.height = cast(s32) ceil(abs(source.width * s) + abs(source.height * c)); + result.stride = result.width * 4; + result.pixels = alloc(cast(s64) result.stride * result.height,, temp); + source_centre := Vector2.{source.width / 2.0, source.height / 2.0}; + centre := Vector2.{result.width / 2.0, result.height / 2.0}; + for y: 0..result.height - 1 for x: 0..result.width - 1 { + d := Vector2.{x + 0.5, y + 0.5} - centre; + p := Vector2.{c * d.x + s * d.y, -s * d.x + c * d.y} + source_centre - Vector2.{0.5, 0.5}; // turned back + out := result.pixels + cast(s64) y * result.stride + x * 4; + x0, y0 := cast(s32) floor(p.x), cast(s32) floor(p.y); + fx, fy := p.x - x0, p.y - y0; + for channel: 0..3 { + sample :: (image: Ocr_Image, x: s32, y: s32, channel: s64) -> float { + if x < 0 || y < 0 || x >= image.width || y >= image.height return 255; + return cast(float) (image.pixels + cast(s64) y * image.stride + x * 4)[channel]; + } + top := sample(source, x0, y0, channel) * (1 - fx) + sample(source, x0 + 1, y0, channel) * fx; + bottom := sample(source, x0, y0 + 1, channel) * (1 - fx) + sample(source, x0 + 1, y0 + 1, channel) * fx; + out[channel] = cast(u8) clamp(top * (1 - fy) + bottom * fy + 0.5, 0, 255); + } + } + return result; +} + +#if OS == .WINDOWS { + test_gdi :: #library,system "Gdi32"; + TestCreateFontW :: (height: s32, width: s32, escapement: s32, orientation: s32, weight: s32, italic: u32, underline: u32, strike_out: u32, char_set: u32, out_precision: u32, clip_precision: u32, quality: u32, pitch_and_family: u32, face_name: *u16) -> *void #foreign test_gdi "CreateFontW"; + TestCreateDIBSection :: (hdc: HDC, info: *BITMAPINFO, usage: u32, bits: **void, section: *void, offset: u32) -> *void #foreign test_gdi "CreateDIBSection"; + TestSetTextColor :: (hdc: HDC, color: u32) -> u32 #foreign test_gdi "SetTextColor"; + TestSetBkColor :: (hdc: HDC, color: u32) -> u32 #foreign test_gdi "SetBkColor"; + TestTextOutW :: (hdc: HDC, x: s32, y: s32, text: *u16, count: s32) -> s32 #foreign test_gdi "TextOutW"; + TestGdiFlush :: () -> s32 #foreign test_gdi "GdiFlush"; + ANTIALIASED_QUALITY :: 4; + + // LINES, black on white, in Segoe UI. Temp. + drawn_lines :: () -> Ocr_Image, ok: bool { + image: Ocr_Image; + image.width, image.height = 720, 170; + image.stride = image.width * 4; + + dc := CreateCompatibleDC(null); + if !dc return image, false; + defer DeleteDC(dc); + info: BITMAPINFO; + info.bmiHeader = .{biSize = size_of(BITMAPINFOHEADER), biWidth = image.width, biHeight = -image.height, biPlanes = 1, biBitCount = 32}; // top-down + bits: *void; + bitmap := TestCreateDIBSection(dc, *info, 0, *bits, null, 0); + if !bitmap || !bits return image, false; + defer DeleteObject(bitmap); + font := TestCreateFontW(-34, 0, 0, 0, 400, 0, 0, 0, 1, 0, 0, ANTIALIASED_QUALITY, 0, utf8_to_wide("Segoe UI",, temp)); + if !font return image, false; + defer DeleteObject(font); + + old_bitmap := SelectObject(dc, bitmap); + old_font := SelectObject(dc, font); + memset(bits, 0xFF, cast(s64) image.stride * image.height); + TestSetTextColor(dc, 0x000000); + TestSetBkColor(dc, 0xFFFFFF); + for LINES { + wide, _, bytes := utf8_to_wide(it,, temp); + TestTextOutW(dc, 40, cast(s32) (30 + it_index * 64), wide, bytes / 2); + } + TestGdiFlush(); + + // BGRA to RGBA. + image.pixels = alloc(cast(s64) image.stride * image.height,, temp); + source := cast(*u8) bits; + for i: 0..cast(s64) image.width * image.height - 1 { + image.pixels[i * 4] = source[i * 4 + 2]; + image.pixels[i * 4 + 1] = source[i * 4 + 1]; + image.pixels[i * 4 + 2] = source[i * 4]; + image.pixels[i * 4 + 3] = 255; + } + SelectObject(dc, old_font); + SelectObject(dc, old_bitmap); + return image, true; + } +} + +turn_tests :: () { + #if OS != .WINDOWS { + emit(" (skipped: the test text is drawn with GDI, on Windows only)\n"); + return; + } else { + group("turned text"); + folder := ocr_models_folder(); + if !file_exists(tprint("%/models.config", folder)) { + emit(" (skipped: no OCR models in %)\n", folder); + return; + } + engine, created, error := ocr_create(folder, OCR_DEFAULT_MODEL); + expect(created, "the OCR engine starts: %", error); + if !created return; + defer ocr_destroy(engine); + expect(engine.orientation.session != null, "% has an orientation classifier", OCR_DEFAULT_MODEL); + emit(" running on %; classifier input % x %\n", ifx engine.provider == .CUDA then "CUDA" else "the CPU", engine.orientation_width, engine.orientation_height); + + level, drawn := drawn_lines(); + expect(drawn, "the test text is drawn"); + if !drawn return; + + for degrees: TURNS { + image := ifx degrees == 0 then level else turned_image(level, degrees); + results := ocr_process(engine, Ocr_Image.[image],, temp); + expect(results.count == 1 && !results[0].failed, "% degrees: OCR runs (%)", degrees, ifx results.count then results[0].error else ""); + if results.count != 1 || results[0].failed continue; + lines := results[0].lines; + + radians := degrees * PI / 180; + direction := Vector2.{cos(radians), sin(radians)}; // the way turned text reads + for wanted: LINES { + line := find_line(lines, wanted); + if !line { + found: [..] string; + found.allocator = temp; + for lines array_add(*found, it.text); + expect(false, "% degrees: \"%\" is read (found %)", degrees, wanted, found); + continue; + } + along := line.corners[1] - line.corners[0]; + along_length := length(along); + cosine := ifx along_length > 0 then dot(along, direction) / along_length else 0; + expect(cosine > 0.95, "% degrees: \"%\" has corners that read its way (cosine %, corners %)", degrees, wanted, cosine, line.corners); + } + } + } +} + +#scope_export + +run_ocr_tests :: (builder: *String_Builder = null) -> bool { + output_builder = builder; + emit("=== OCR test suite ===\n"); + push_allocator(context.default_allocator); + turn_tests(); + return report(); +} diff --git a/src/test/testing.jai b/src/test/testing.jai index 32933d8..0e826f9 100644 --- a/src/test/testing.jai +++ b/src/test/testing.jai @@ -38,6 +38,8 @@ Test_Type :: enum s32 { fixed_arena_suite; secure_memory_suite; logger_suite; + catalog_suite; + ocr_suite; } testing_started :: () -> bool { @@ -78,6 +80,8 @@ run_all_tests :: () { add_suite_test(.fixed_arena_suite, "fixed_arena"); add_suite_test(.secure_memory_suite, "secure_memory"); add_suite_test(.logger_suite, "logger"); + add_suite_test(.catalog_suite, "catalog"); + add_suite_test(.ocr_suite, "ocr"); } // One work item per suite. Each suite keeps its tallies in file-scope globals, so the whole @@ -182,6 +186,10 @@ test_thread_group_proc :: (group: *Thread_Group, thread: *Thread, work: *void) - task.pass = run_secure_memory_tests(task.output); case .logger_suite; task.pass = run_logger_tests(task.output); + case .catalog_suite; + task.pass = run_catalog_tests(task.output); + case .ocr_suite; + task.pass = run_ocr_tests(task.output); } return .CONTINUE; diff --git a/src/ui/imgui_theme_registry.jai b/src/ui/imgui_theme_registry.jai index 0ed468b..703d2ff 100644 --- a/src/ui/imgui_theme_registry.jai +++ b/src/ui/imgui_theme_registry.jai @@ -292,6 +292,7 @@ app_theme_set_status :: (message: string, is_error: bool) { app_user_settings: App_User_Settings; App_User_Settings :: struct { theme: string; // "" means the default theme; heap-allocated + ocr_model: string; // [ocr] model: the OCR model's name in models.config; "" means the default; heap-allocated // Of the monitor the window first opens on. window_width_fraction: float = 0.80; @@ -313,6 +314,7 @@ App_User_Settings :: struct { settings_free :: (s: *App_User_Settings) { free(s.theme,, context.default_allocator); + free(s.ocr_model,, context.default_allocator); array_free(s.hotkey_bindings); array_free(s.global_hotkey_bindings); } @@ -368,11 +370,11 @@ app_load_user_settings_file :: () -> write_back: bool { if hotkeys_missing.count || global_missing.count return true; #if OS == .LINUX seen.global_hotkeys = true; // not written there: Linux has no global hotkeys yet - return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys); + return !(seen.theme && seen.width && seen.height && seen.font_size && seen.font_scale && seen.event_log && seen.hotkeys && seen.global_hotkeys && seen.ocr_model); } Settings_Seen :: struct { - theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys: bool; + theme, width, height, font_size, font_scale, event_log, hotkeys, global_hotkeys, ocr_model: bool; } // Of the file's bytes as last loaded or written, so a watcher event for our own save, or for an @@ -382,6 +384,7 @@ settings_file_hash: u64; // The settings the app itself changes, as it holds them now. Settings_App_Values :: struct { theme: string; + ocr_model: string; font_size_base: float; font_scale_main: float; } @@ -395,6 +398,7 @@ settings_app_values :: () -> Settings_App_Values { entry := app_theme_current(); values.theme = ifx entry then entry.name else app_user_settings.theme; if !values.theme.count values.theme = APP_DEFAULT_THEME_NAME; + values.ocr_model = ifx app_user_settings.ocr_model.count then app_user_settings.ocr_model else OCR_DEFAULT_MODEL; values.font_size_base = app_user_settings.font_size_base; values.font_scale_main = app_user_settings.font_scale_main; @@ -410,8 +414,10 @@ settings_app_values :: () -> Settings_App_Values { settings_note_synced :: () { free(settings_synced.theme,, context.default_allocator); + free(settings_synced.ocr_model,, context.default_allocator); settings_synced = settings_app_values(); settings_synced.theme = copy_string(settings_synced.theme,, context.default_allocator); + settings_synced.ocr_model = copy_string(settings_synced.ocr_model,, context.default_allocator); } // Parses the file's contents into out, replacing its theme and hotkey bindings. ok is false when @@ -483,7 +489,11 @@ settings_parse :: (path: string, file_data: string, out: *App_User_Settings) -> key, value, split_ok := theme_split_key(body); if !split_ok continue; - if section == "[ui]" && key == "theme" { + if section == "[ocr]" && key == "model" { + free(out.ocr_model,, context.default_allocator); + out.ocr_model = copy_string(settings_unquote(value),, context.default_allocator); + seen.ocr_model = true; + } else if section == "[ui]" && key == "theme" { free(out.theme,, context.default_allocator); out.theme = copy_string(settings_unquote(value),, context.default_allocator); seen.theme = true; @@ -592,6 +602,7 @@ app_save_user_settings_file :: () -> bool { values := settings_app_values(); theme := values.theme; + ocr_model := values.ocr_model; font_size_base := values.font_size_base; font_scale_main := values.font_scale_main; @@ -621,6 +632,7 @@ app_save_user_settings_file :: () -> bool { } if disk_ok { if seen.theme && values.theme == settings_synced.theme && on_disk.theme.count theme = on_disk.theme; + if seen.ocr_model && values.ocr_model == settings_synced.ocr_model && on_disk.ocr_model.count ocr_model = on_disk.ocr_model; if seen.font_size && values.font_size_base == settings_synced.font_size_base font_size_base = on_disk.font_size_base; if seen.font_scale && values.font_scale_main == settings_synced.font_scale_main font_scale_main = on_disk.font_scale_main; if seen.width app_user_settings.window_width_fraction = on_disk.window_width_fraction; @@ -643,6 +655,9 @@ app_save_user_settings_file :: () -> bool { append(*builder, "# Text size is size_base * scale_main * the monitor's DPI scale. The font size hotkeys change scale_main.\n"); print_to_builder(*builder, "size_base: %\n", FF(font_size_base, 1)); print_to_builder(*builder, "scale_main: %\n\n", FF(font_scale_main, 2)); + append(*builder, "[ocr]\n"); + append(*builder, "# The OCR model, by its name in ocr-models/models.config: ppocr-v5-mobile (light) or ppocr-v5-server (heavy).\n"); + print_to_builder(*builder, "model: %\n\n", settings_quote_if_needed(ocr_model)); append(*builder, "[logging]\n"); append(*builder, "# Entries the Event Log keeps in memory. Past it the oldest are dropped; the log file keeps every line.\n"); print_to_builder(*builder, "event_log_capacity: %\n", app_user_settings.event_log_capacity); @@ -766,6 +781,7 @@ app_reload_user_settings_if_changed :: () { } if fresh.event_log_capacity != old.event_log_capacity logger_set_event_log_capacity(fresh.event_log_capacity); + if fresh.ocr_model != old.ocr_model ocr_ui_settings_changed(); // app/ocr_ui.jai bindings: [] Hotkey_Binding = hotkey_defaults(); if seen.hotkeys bindings = app_user_settings.hotkey_bindings; diff --git a/tools/ocr_bench.jai b/tools/ocr_bench.jai index 7a94597..40d4054 100644 --- a/tools/ocr_bench.jai +++ b/tools/ocr_bench.jai @@ -124,7 +124,8 @@ main :: () { print(" first image alone: % s (warm-up)\n", FF(first_seconds, 3)); print(" all % images: % s, % ms per image, % megapixels/s, % failed\n", images.count, FF(seconds, 2), FF(1000 * seconds / images.count, 1), FF(megapixels / seconds, 2), failures); print(" found: % lines, % bytes of text\n", lines, characters); - print(" stage time (summed over threads): detection % s, recognition % s\n", FF(stats.detect_us / 1_000_000.0, 2), FF(stats.recognize_us / 1_000_000.0, 2)); + print(" stage time (summed over threads): detection % s, orientation % s, recognition % s\n", FF(stats.detect_us / 1_000_000.0, 2), FF(stats.orientation_us / 1_000_000.0, 2), FF(stats.recognize_us / 1_000_000.0, 2)); + print(" lines read better turned: %\n", stats.turned); print(" text in %\n\n", text_folder); ocr_destroy(engine); }