diff --git a/README.md b/README.md index d6be635..4524484 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,9 @@ works only in the library folders you choose, and never on videos. What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR model: captures with text, lines, words, characters, confidence, words per capture with a breakdown, a table by year, the captures with the most words (click to open), each capture's best label -counted (click one to search for it), and the commonest words (click one to search for it). +counted (click one to search for it), and the commonest words (click one to search for it). When any +of them are segmented, it also shows **What is in the images**: how many images hold each class +(person, sky, dog ...). Click one to search `has:` for it. ## Importing @@ -201,7 +203,7 @@ A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title: | Prefix | Looks in | | --- | --- | -| *(none)* | everything below except `like:` | +| *(none)* | everything below except `like:`, `folder:` and `has:` | | `ocr:` | the text read from the image | | `title:` | the window title when it was captured; for an imported image, its file name | | `url:` | the web page's address, for browser captures | @@ -210,9 +212,22 @@ A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title: | `label:` | the capture's labels: its best label, and any scoring at least half as much | | `like:` | the image's meaning (below) | | `folder:` | the library folder it is in: `folder:work` | +| `has:` | what segmentation found in the image (below): `has:dog` | A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work. +### What is in the image: `has:` + +`has:dog` finds images that segmentation (the Segmentation window) found a dog in. Only images in +folders set to segment, and already segmented, take part. + +- **The whole class name:** `has:table` is a table, not a dining table; `has:"dining table"` is that. +- **The 133 COCO classes:** 80 kinds of thing (person, bicycle, car, dog, cat, chair, laptop, cell + phone, book ...) and 53 of stuff (sky, wall, grass, road, floor wood, ...). The Stats window lists + those in the images shown. +- **Combining:** `has:person has:dog` finds images with both; `has:cat label:"photo"` mixes it with + labels. + ### Searching by meaning: `like:` `like:"a circuit board under a microscope"` ranks every labelled capture by how close its image is @@ -239,6 +254,8 @@ then in that order instead of by date. | `note:todo` | captures with an annotation saying TODO | | `like:"a photo of a soldering iron"` | captures that look like that, best first | | `like:"a spreadsheet" app:excel` | Excel captures, the most spreadsheet-like first | +| `has:dog` | images with a dog in them (segmented folders) | +| `has:person has:"cell phone"` | images with a person and a phone | ## Command line diff --git a/check_targets.jai b/check_targets.jai index 1aeb110..02c04b7 100644 --- a/check_targets.jai +++ b/check_targets.jai @@ -29,7 +29,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; GIT_BRANCH :: "check"; GIT_HASH :: "-"; -VERSION :: "0.0.13"; +VERSION :: "0.0.14"; BUILD_DATE :: "-"; DEBUG :: true; COMPILE_TESTS :: true; diff --git a/first.jai b/first.jai index 6b31404..b64ebd6 100644 --- a/first.jai +++ b/first.jai @@ -8,7 +8,7 @@ APP_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; -VERSION :: "0.0.13"; +VERSION :: "0.0.14"; // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // builds never write over the ones you run by hand. build_directory := "bin"; diff --git a/segmentation-plan.html b/segmentation-plan.html index cc183e4..8d468b0 100644 --- a/segmentation-plan.html +++ b/segmentation-plan.html @@ -78,7 +78,7 @@
  • Decided Images only, and only in the library folders you opt in. The setting is per folder, kept in a file in the library root, so it travels with the library.
  • Decided No .mcap version bump. Results go in two new chunk kinds: a small segs summary at the front for search, and a mask at the back for drawing.
  • It is built in six phases. The first proves the ONNX export matches PyTorch before anything else depends on it.
    • -
  • Progress Phases 1 to 4 are done (2026-10-10). The model is exported and checked, and modules/Segment runs it in mcap at about 57 ms per image on the RTX 5080. Results are stored in .mcap files at about 5 KB per image. Library folders opt in through .library-folders, and the Segmentation window and -segment scan them. Next is phase 5: has: search and Stats.
    • +
  • Progress Phases 1 to 5 are done (2026-10-10). The model is exported and checked, and modules/Segment runs it in mcap at about 57 ms per image on the RTX 5080. Results are stored in .mcap files at about 5 KB per image. Library folders opt in through .library-folders, and the Segmentation window and -segment scan them. has: search and Stats' "What is in the images" are in (phase 5). Next is phase 6: the overlay in the Capture window.
  • Later Video, by treating sampled frames as images, is shelved.
    • @@ -399,8 +399,27 @@ then runs: (id, length) pairs over the whole map, row by row, wrapping a
    -

    Phase 5: Catalog, has: and Stats

    +

    Phase 5: Catalog, has: and Stats Done

    Done when: has:person finds the right captures in the scratch library, catalog tests cover the new field, and Stats shows the top classes, each clickable.

    +
    Result (2026-10-10). The test library is phase 4's: 18 segmented images. + +

    As built:

    + +

    Not yet checked by eye: the Stats section's layout.

    +

    Left for later: counts and sizes (has:person:3+, has:dog>20%).

    diff --git a/src/app/catalog.jai b/src/app/catalog.jai index 12c356b..75e8c55 100644 --- a/src/app/catalog.jai +++ b/src/app/catalog.jai @@ -23,9 +23,18 @@ Catalog_Entry :: struct { // The segmentation model's (SEGMENT_MODEL, app/segment_ui.jai) segs chunk, when it has one. segment_classes: string; // its class list's hash; "" when not segmented segment_count: s32; + segments: [..] Catalog_Segment; // largest first // Lowercased copies, for matching without case. find_title, find_url, find_app, find_notes, find_ocr, find_labels, find_folder: string; + find_segments: string; // each class in it once, between newlines: "|person|dog|" with | a newline, for has: +} + +// A segment of the capture's image (its segs chunk): what, how sure, and how much of the image. +Catalog_Segment :: struct { + class: string; // "person", "wall" + score: float; + area: float; // a fraction of the image } Catalog_Media :: enum u8 { @@ -50,6 +59,9 @@ catalog_entry_free :: (entry: *Catalog_Entry) { free(entry.label_list); free(entry.video_codec); free(entry.segment_classes); + for entry.segments free(it.class); + array_free(entry.segments); + free(entry.find_segments); array_free(entry.embedding); free(entry.find_title); free(entry.find_url); free(entry.find_app); free(entry.find_notes); free(entry.find_ocr); free(entry.find_labels); free(entry.find_folder); free(entry); @@ -132,6 +144,15 @@ catalog_load :: (path: string) -> *Catalog_Entry, error: string { if segments_ok { entry.segment_classes = copy_string(segments.classes); entry.segment_count = cast(s32) segments.segments.count; + entry.segments.allocator = context.default_allocator; + // Each class once, lowercased, between newlines, for has: (search_matches). + find := "\n"; + for segments.segments { + array_add(*entry.segments, .{class = copy_string(it.class), score = it.score, area = it.area}); + lower := to_lower_copy(it.class,, temp); + if !contains(find, tprint("\n%\n", lower)) find = tprint("%%\n", find, lower); + } + entry.find_segments = copy_string(find); } } @@ -172,14 +193,15 @@ catalog_section :: (entry: *Catalog_Entry, model: string) -> *Mcap_Ocr_Section { // LIKE is by meaning: the captures whose image is nearest the phrase, ranked by the label model // (app/background.jai does the ranking); here it only asks for a labelled capture. -Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; FOLDER; } +// HAS is a segment's class, the whole name: has:dog, has:"traffic light" (app/segment_ui.jai). +Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; FOLDER; HAS; } Search_Term :: struct { field: Search_Field; text: string; // lowercased } -// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like: folder:. Temp. +// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like: folder: has:. Temp. search_parse :: (query: string) -> [..] Search_Term { terms: [..] Search_Term; terms.allocator = temp; @@ -200,6 +222,7 @@ search_parse :: (query: string) -> [..] Search_Term { case "label"; field = .LABEL; case "like"; field = .LIKE; case "folder"; field = .FOLDER; + case "has"; field = .HAS; } if field != .ANY { term.field = field; rest = advance(rest, colon + 1); } // "http:..." and the like stay words } @@ -213,6 +236,8 @@ search_parse :: (query: string) -> [..] Search_Term { else { term.text = slice(rest, 0, space); rest = advance(rest, space); } } term.text = to_lower_copy(term.text,, temp); + // has: matches a whole class name, as find_segments holds them: between newlines. + if term.field == .HAS && term.text.count term.text = tprint("\n%\n", trim(term.text)); if term.text.count array_add(*terms, term); rest = trim_left(rest); } @@ -236,6 +261,7 @@ search_matches :: (entry: *Catalog_Entry, terms: [] Search_Term) -> bool { case .LABEL; found = contains(entry.find_labels, it.text); case .LIKE; found = entry.embedding.count > 0; case .FOLDER; found = contains(entry.find_folder, it.text); + case .HAS; found = contains(entry.find_segments, it.text); // it.text is the class between newlines: a whole name } if !found return false; } diff --git a/src/app/stats_ui.jai b/src/app/stats_ui.jai index c1b2f55..24cdfdf 100644 --- a/src/app/stats_ui.jai +++ b/src/app/stats_ui.jai @@ -6,6 +6,7 @@ STATS_WINDOW :: "Statistics"; STATS_TOP_CAPTURES :: 10; STATS_TOP_WORDS :: 40; +STATS_TOP_CLASSES :: 40; // segment classes listed STATS_LOW_CONFIDENCE :: 0.8; // Words per capture, in these ranges: 0, 1-10, 11-50, 51-200, 201-1000, more. @@ -30,6 +31,8 @@ Stats_Result :: struct { common_words: [..] Stats_Word; // commonest first labelled: s64; // captures with labels (app/labels_ui.jai) top_labels: [..] Stats_Word; // how many captures have each label as their best, most first + segmented: s64; // captures with segments (app/segment_ui.jai) + top_classes: [..] Stats_Word; // how many captures hold each segment class, most first } stats_ui: struct { @@ -64,6 +67,8 @@ stats_result_free :: (s: *Stats_Result) { for s.common_words free(it.word); for s.top_labels free(it.word); array_free(s.top_labels); + for s.top_classes free(it.word); + array_free(s.top_classes); array_free(s.years); array_free(s.most_words); array_free(s.common_words); @@ -131,6 +136,10 @@ stats_log :: () { labels.allocator = temp; for s.top_labels print_to_builder(*labels, "% %, ", it.word, it.count); log("[stats] % labelled; best labels: %", s.labelled, builder_to_string(*labels,, temp)); + classes: String_Builder; + classes.allocator = temp; + for s.top_classes print_to_builder(*classes, "% %, ", it.word, it.count); + log("[stats] % segmented; images holding each class: %", s.segmented, builder_to_string(*classes,, temp)); if viewer.search_terms.count { shown: String_Builder; shown.allocator = temp; @@ -155,6 +164,27 @@ stats_window_contents :: () { using ImGui; if s.read < s.captures { TextColored(app_theme.palette.dim, "% have not been read with this model; the OCR window's scan reads them.", stats_number(s.captures - s.read)); } + + // What segmentation found: shown whether or not OCR has read anything. + if s.segmented { + SeparatorText("What is in the images"); + TextColored(app_theme.palette.dim, "How many of the % segmented images hold each thing (%). Click one to search for it.", stats_number(s.segmented), SEGMENT_MODEL); + if BeginTable("##classes", 3, .SizingFixedFit) { + TableSetupColumn("class"); + TableSetupColumn("images"); + TableSetupColumn("bar", .WidthStretch); + for s.top_classes { + if it_index >= STATS_TOP_CLASSES break; + TableNextRow(); + TableNextColumn(); + if SmallButton(temp_c_string(tprint("%##class%", it.word, it_index))) viewer_set_search(tprint("has:\"%\"", it.word)); + TableNextColumn(); TextUnformatted(stats_number(it.count)); + TableNextColumn(); ProgressBar(cast(float) it.count / max(s.segmented, 1), .{-FLOAT32_MIN, 0}, temp_c_string(stats_percent(it.count, s.segmented))); + } + EndTable(); + } + } + if !s.read { TextColored(app_theme.palette.dim, "No readings yet."); return; } SeparatorText("Text found"); @@ -266,8 +296,11 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result { s.most_words.allocator = context.default_allocator; s.common_words.allocator = context.default_allocator; s.top_labels.allocator = context.default_allocator; + s.top_classes.allocator = context.default_allocator; top: Table(string, s64); // keys borrow the entries' label names top.allocator = temp; + classes: Table(string, s64); // keys borrow the entries' segment classes + classes.allocator = temp; per_capture: [..] Stats_Capture; per_capture.allocator = temp; @@ -289,6 +322,17 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result { count := find_or_add(*top, entry.labels[0].name); count.* += 1; } + if entry && entry.segment_classes { + s.segmented += 1; + // Each class once per capture, however many of it there are. + for segment, n: entry.segments { + seen := false; + for 0..n - 1 if entry.segments[it].class == segment.class { seen = true; break; } + if seen continue; + count := find_or_add(*classes, segment.class); + count.* += 1; + } + } section := catalog_section(entry, model); if !section continue; @@ -337,6 +381,8 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result { for count, name: top array_add(*s.top_labels, .{copy_string(name,, context.default_allocator), count}); quick_sort(s.top_labels, (a: Stats_Word, b: Stats_Word) -> s64 { return ifx a.count != b.count then b.count - a.count else compare(a.word, b.word); }); + for count, name: classes array_add(*s.top_classes, .{copy_string(name,, context.default_allocator), count}); + quick_sort(s.top_classes, (a: Stats_Word, b: Stats_Word) -> s64 { return ifx a.count != b.count then b.count - a.count else compare(a.word, b.word); }); s.compute_ms = to_float64_seconds(current_time_monotonic() - start) * 1000; return s; diff --git a/src/app/viewer.jai b/src/app/viewer.jai index 96212a9..7e7debb 100644 --- a/src/app/viewer.jai +++ b/src/app/viewer.jai @@ -826,7 +826,7 @@ viewer_ui :: () { using ImGui; } SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520)); - if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like: folder:", viewer.search_buffer.data, viewer.search_buffer.count) { + if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like: folder: has:", viewer.search_buffer.data, viewer.search_buffer.count) { viewer_search_changed(); } SameLine(); diff --git a/src/test/catalog_tests.jai b/src/test/catalog_tests.jai index ec33330..5503738 100644 --- a/src/test/catalog_tests.jai +++ b/src/test/catalog_tests.jai @@ -95,6 +95,21 @@ match_tests :: () { expect(search_matches(*entry, search_parse("like:anything")), "like: takes a labelled capture (the ranking is the background's)"); entry.embedding = .[]; + // has: a segment's class, the whole name (find_segments, as catalog_load makes it). + entry.find_segments = "\nperson\ndog\ndining table\n"; + expect(search_matches(*entry, search_parse("has:dog")), "has: a class"); + expect(search_matches(*entry, search_parse("has:DOG has:person")), "has: twice, ignoring case"); + expect(search_matches(*entry, search_parse("has:\"dining table\"")), "has: a quoted name with a space"); + expect(!search_matches(*entry, search_parse("has:table")), "has: is the whole name: table is not dining table"); + expect(!search_matches(*entry, search_parse("has:do")), "nor part of a name"); + expect(!search_matches(*entry, search_parse("has:cat")), "a class it does not hold"); + expect(!search_matches(*entry, search_parse("dog")), "a plain word does not match segments"); + expect(search_matches(*entry, search_parse("has:dog kicad")), "has: with other words"); + terms := search_parse("has:dog"); + expect(terms.count == 1 && terms[0].field == .HAS, "the has: prefix is read"); + entry.find_segments = ""; + expect(!search_matches(*entry, search_parse("has:dog")), "nothing segmented: no has: matches"); + expect(search_line_matches("0.1uF : Capacitor_SMD", search_parse("capacitor")), "a line holding a word"); expect(!search_line_matches("0.1uF : Capacitor_SMD", search_parse("title:capacitor")), "a title word is not an OCR line's"); }