[0.0.14] Segmentation phase 5: has: search, and what is in the images in Stats
- has: searches what segmentation found, by the whole class name: has:dog, has:"dining table" (has:table is a table, not a dining table). Plain words do not match segments. - The catalog keeps each capture's segments (class, score, area) and its classes between newlines, so a has: term is one contains per capture. - Stats: "What is in the images", how many of the segmented images hold each class, each one a has: search; shown even when nothing has been read by OCR. -stats logs it. - Tests: has: (whole names, quoted names, case, combinations, plain words not matching).
This commit is contained in:
parent
3e6e733a9d
commit
3c2c07bff6
21
README.md
21
README.md
@ -133,7 +133,9 @@ works only in the library folders you choose, and never on videos.
|
||||
What OCR found in the captures shown (the whole library, or a search's matches), by the chosen OCR
|
||||
model: captures with text, lines, words, characters, confidence, words per capture with a breakdown,
|
||||
a table by year, the captures with the most words (click to open), each capture's best label
|
||||
counted (click one to search for it), and the commonest words (click one to search for it).
|
||||
counted (click one to search for it), and the commonest words (click one to search for it). When any
|
||||
of them are segmented, it also shows **What is in the images**: how many images hold each class
|
||||
(person, sky, dog ...). Click one to search `has:` for it.
|
||||
|
||||
## Importing
|
||||
|
||||
@ -201,7 +203,7 @@ A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:
|
||||
|
||||
| Prefix | Looks in |
|
||||
| --- | --- |
|
||||
| *(none)* | everything below except `like:` |
|
||||
| *(none)* | everything below except `like:`, `folder:` and `has:` |
|
||||
| `ocr:` | the text read from the image |
|
||||
| `title:` | the window title when it was captured; for an imported image, its file name |
|
||||
| `url:` | the web page's address, for browser captures |
|
||||
@ -210,9 +212,22 @@ A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:
|
||||
| `label:` | the capture's labels: its best label, and any scoring at least half as much |
|
||||
| `like:` | the image's meaning (below) |
|
||||
| `folder:` | the library folder it is in: `folder:work` |
|
||||
| `has:` | what segmentation found in the image (below): `has:dog` |
|
||||
|
||||
A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work.
|
||||
|
||||
### What is in the image: `has:`
|
||||
|
||||
`has:dog` finds images that segmentation (the Segmentation window) found a dog in. Only images in
|
||||
folders set to segment, and already segmented, take part.
|
||||
|
||||
- **The whole class name:** `has:table` is a table, not a dining table; `has:"dining table"` is that.
|
||||
- **The 133 COCO classes:** 80 kinds of thing (person, bicycle, car, dog, cat, chair, laptop, cell
|
||||
phone, book ...) and 53 of stuff (sky, wall, grass, road, floor wood, ...). The Stats window lists
|
||||
those in the images shown.
|
||||
- **Combining:** `has:person has:dog` finds images with both; `has:cat label:"photo"` mixes it with
|
||||
labels.
|
||||
|
||||
### Searching by meaning: `like:`
|
||||
|
||||
`like:"a circuit board under a microscope"` ranks every labelled capture by how close its image is
|
||||
@ -239,6 +254,8 @@ then in that order instead of by date.
|
||||
| `note:todo` | captures with an annotation saying TODO |
|
||||
| `like:"a photo of a soldering iron"` | captures that look like that, best first |
|
||||
| `like:"a spreadsheet" app:excel` | Excel captures, the most spreadsheet-like first |
|
||||
| `has:dog` | images with a dog in them (segmented folders) |
|
||||
| `has:person has:"cell phone"` | images with a person and a phone |
|
||||
|
||||
## Command line
|
||||
|
||||
|
||||
@ -29,7 +29,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
GIT_BRANCH :: "check";
|
||||
GIT_HASH :: "-";
|
||||
VERSION :: "0.0.13";
|
||||
VERSION :: "0.0.14";
|
||||
BUILD_DATE :: "-";
|
||||
DEBUG :: true;
|
||||
COMPILE_TESTS :: true;
|
||||
|
||||
@ -8,7 +8,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
|
||||
|
||||
VERSION :: "0.0.13";
|
||||
VERSION :: "0.0.14";
|
||||
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
|
||||
// builds never write over the ones you run by hand.
|
||||
build_directory := "bin";
|
||||
|
||||
@ -78,7 +78,7 @@
|
||||
<li><span class="tag decided">Decided</span> <strong>Images only</strong>, and only in the <strong>library folders you opt in</strong>. The setting is per folder, kept in a file in the library root, so it travels with the library.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>No .mcap version bump.</strong> Results go in two new chunk kinds: a small <code>segs</code> summary at the front for search, and a <code>mask</code> at the back for drawing.</li>
|
||||
<li>It is built in six phases. The first proves the ONNX export matches PyTorch before anything else depends on it.</li>
|
||||
<li><span class="tag decided">Progress</span> <strong>Phases 1 to 4 are done</strong> (2026-10-10). The model is exported and checked, and <code>modules/Segment</code> runs it in mcap at about 57 ms per image on the RTX 5080. Results are stored in .mcap files at about 5 KB per image. Library folders opt in through <code>.library-folders</code>, and the Segmentation window and <code>-segment</code> scan them. Next is phase 5: <code>has:</code> search and Stats.</li>
|
||||
<li><span class="tag decided">Progress</span> <strong>Phases 1 to 5 are done</strong> (2026-10-10). The model is exported and checked, and <code>modules/Segment</code> runs it in mcap at about 57 ms per image on the RTX 5080. Results are stored in .mcap files at about 5 KB per image. Library folders opt in through <code>.library-folders</code>, and the Segmentation window and <code>-segment</code> scan them. <code>has:</code> search and Stats' "What is in the images" are in (phase 5). Next is phase 6: the overlay in the Capture window.</li>
|
||||
<li><span class="tag later">Later</span> Video, by treating sampled frames as images, is shelved.</li>
|
||||
</ul>
|
||||
</div>
|
||||
@ -399,8 +399,27 @@ then runs: (id, length) pairs over the whole map, row by row, wrapping a
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 5: Catalog, <code>has:</code> and Stats</h3>
|
||||
<h3>Phase 5: Catalog, <code>has:</code> and Stats <span class="tag decided">Done</span></h3>
|
||||
<p><strong>Done when:</strong> <code>has:person</code> finds the right captures in the scratch library, catalog tests cover the new field, and Stats shows the top classes, each clickable.</p>
|
||||
<div class="note"><strong>Result (2026-10-10).</strong> The test library is phase 4's: 18 segmented images.
|
||||
<ul>
|
||||
<li><strong><code>has:</code> searches</strong> (headless <code>-stats</code> with each search):
|
||||
<ul>
|
||||
<li><code>has:person</code>: 9 images.</li>
|
||||
<li><code>has:cat</code>: 3, the COCO cats photo and the 2 GIFs.</li>
|
||||
<li><code>has:table</code>: 1, the only plain "table", not the "dining table" images.</li>
|
||||
<li>A plain <code>person</code>: none, since plain words don't search segments.</li>
|
||||
</ul></li>
|
||||
<li><strong>Stats</strong> lists person 9, wall 7, sky 4, tree 4, cat 3 and so on, each class counted once per image.</li>
|
||||
<li><strong>Tests:</strong> 10 new catalog checks cover <code>has:</code>: whole names, quoted names with spaces, case, combinations, and a plain word not matching.</li>
|
||||
</ul>
|
||||
<p>As built:</p>
|
||||
<ul>
|
||||
<li><strong>The catalog</strong> keeps each segment's class, score and area, and a lookup string of the classes between newlines. <code>search_parse</code> wraps a <code>has:</code> term in newlines once, so matching a whole name is one <code>contains</code> per capture.</li>
|
||||
<li><strong>The Stats section</strong> "What is in the images" shows even when nothing has been OCR'd.</li>
|
||||
</ul>
|
||||
<p><strong>Not yet checked by eye:</strong> the Stats section's layout.</p>
|
||||
<p><strong>Left for later:</strong> counts and sizes (<code>has:person:3+</code>, <code>has:dog>20%</code>).</p></div>
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
|
||||
@ -23,9 +23,18 @@ Catalog_Entry :: struct {
|
||||
// The segmentation model's (SEGMENT_MODEL, app/segment_ui.jai) segs chunk, when it has one.
|
||||
segment_classes: string; // its class list's hash; "" when not segmented
|
||||
segment_count: s32;
|
||||
segments: [..] Catalog_Segment; // largest first
|
||||
|
||||
// Lowercased copies, for matching without case.
|
||||
find_title, find_url, find_app, find_notes, find_ocr, find_labels, find_folder: string;
|
||||
find_segments: string; // each class in it once, between newlines: "|person|dog|" with | a newline, for has:
|
||||
}
|
||||
|
||||
// A segment of the capture's image (its segs chunk): what, how sure, and how much of the image.
|
||||
Catalog_Segment :: struct {
|
||||
class: string; // "person", "wall"
|
||||
score: float;
|
||||
area: float; // a fraction of the image
|
||||
}
|
||||
|
||||
Catalog_Media :: enum u8 {
|
||||
@ -50,6 +59,9 @@ catalog_entry_free :: (entry: *Catalog_Entry) {
|
||||
free(entry.label_list);
|
||||
free(entry.video_codec);
|
||||
free(entry.segment_classes);
|
||||
for entry.segments free(it.class);
|
||||
array_free(entry.segments);
|
||||
free(entry.find_segments);
|
||||
array_free(entry.embedding);
|
||||
free(entry.find_title); free(entry.find_url); free(entry.find_app); free(entry.find_notes); free(entry.find_ocr); free(entry.find_labels); free(entry.find_folder);
|
||||
free(entry);
|
||||
@ -132,6 +144,15 @@ catalog_load :: (path: string) -> *Catalog_Entry, error: string {
|
||||
if segments_ok {
|
||||
entry.segment_classes = copy_string(segments.classes);
|
||||
entry.segment_count = cast(s32) segments.segments.count;
|
||||
entry.segments.allocator = context.default_allocator;
|
||||
// Each class once, lowercased, between newlines, for has: (search_matches).
|
||||
find := "\n";
|
||||
for segments.segments {
|
||||
array_add(*entry.segments, .{class = copy_string(it.class), score = it.score, area = it.area});
|
||||
lower := to_lower_copy(it.class,, temp);
|
||||
if !contains(find, tprint("\n%\n", lower)) find = tprint("%%\n", find, lower);
|
||||
}
|
||||
entry.find_segments = copy_string(find);
|
||||
}
|
||||
}
|
||||
|
||||
@ -172,14 +193,15 @@ catalog_section :: (entry: *Catalog_Entry, model: string) -> *Mcap_Ocr_Section {
|
||||
|
||||
// LIKE is by meaning: the captures whose image is nearest the phrase, ranked by the label model
|
||||
// (app/background.jai does the ranking); here it only asks for a labelled capture.
|
||||
Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; FOLDER; }
|
||||
// HAS is a segment's class, the whole name: has:dog, has:"traffic light" (app/segment_ui.jai).
|
||||
Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; FOLDER; HAS; }
|
||||
|
||||
Search_Term :: struct {
|
||||
field: Search_Field;
|
||||
text: string; // lowercased
|
||||
}
|
||||
|
||||
// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like: folder:. Temp.
|
||||
// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like: folder: has:. Temp.
|
||||
search_parse :: (query: string) -> [..] Search_Term {
|
||||
terms: [..] Search_Term;
|
||||
terms.allocator = temp;
|
||||
@ -200,6 +222,7 @@ search_parse :: (query: string) -> [..] Search_Term {
|
||||
case "label"; field = .LABEL;
|
||||
case "like"; field = .LIKE;
|
||||
case "folder"; field = .FOLDER;
|
||||
case "has"; field = .HAS;
|
||||
}
|
||||
if field != .ANY { term.field = field; rest = advance(rest, colon + 1); } // "http:..." and the like stay words
|
||||
}
|
||||
@ -213,6 +236,8 @@ search_parse :: (query: string) -> [..] Search_Term {
|
||||
else { term.text = slice(rest, 0, space); rest = advance(rest, space); }
|
||||
}
|
||||
term.text = to_lower_copy(term.text,, temp);
|
||||
// has: matches a whole class name, as find_segments holds them: between newlines.
|
||||
if term.field == .HAS && term.text.count term.text = tprint("\n%\n", trim(term.text));
|
||||
if term.text.count array_add(*terms, term);
|
||||
rest = trim_left(rest);
|
||||
}
|
||||
@ -236,6 +261,7 @@ search_matches :: (entry: *Catalog_Entry, terms: [] Search_Term) -> bool {
|
||||
case .LABEL; found = contains(entry.find_labels, it.text);
|
||||
case .LIKE; found = entry.embedding.count > 0;
|
||||
case .FOLDER; found = contains(entry.find_folder, it.text);
|
||||
case .HAS; found = contains(entry.find_segments, it.text); // it.text is the class between newlines: a whole name
|
||||
}
|
||||
if !found return false;
|
||||
}
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
STATS_WINDOW :: "Statistics";
|
||||
STATS_TOP_CAPTURES :: 10;
|
||||
STATS_TOP_WORDS :: 40;
|
||||
STATS_TOP_CLASSES :: 40; // segment classes listed
|
||||
STATS_LOW_CONFIDENCE :: 0.8;
|
||||
|
||||
// Words per capture, in these ranges: 0, 1-10, 11-50, 51-200, 201-1000, more.
|
||||
@ -30,6 +31,8 @@ Stats_Result :: struct {
|
||||
common_words: [..] Stats_Word; // commonest first
|
||||
labelled: s64; // captures with labels (app/labels_ui.jai)
|
||||
top_labels: [..] Stats_Word; // how many captures have each label as their best, most first
|
||||
segmented: s64; // captures with segments (app/segment_ui.jai)
|
||||
top_classes: [..] Stats_Word; // how many captures hold each segment class, most first
|
||||
}
|
||||
|
||||
stats_ui: struct {
|
||||
@ -64,6 +67,8 @@ stats_result_free :: (s: *Stats_Result) {
|
||||
for s.common_words free(it.word);
|
||||
for s.top_labels free(it.word);
|
||||
array_free(s.top_labels);
|
||||
for s.top_classes free(it.word);
|
||||
array_free(s.top_classes);
|
||||
array_free(s.years);
|
||||
array_free(s.most_words);
|
||||
array_free(s.common_words);
|
||||
@ -131,6 +136,10 @@ stats_log :: () {
|
||||
labels.allocator = temp;
|
||||
for s.top_labels print_to_builder(*labels, "% %, ", it.word, it.count);
|
||||
log("[stats] % labelled; best labels: %", s.labelled, builder_to_string(*labels,, temp));
|
||||
classes: String_Builder;
|
||||
classes.allocator = temp;
|
||||
for s.top_classes print_to_builder(*classes, "% %, ", it.word, it.count);
|
||||
log("[stats] % segmented; images holding each class: %", s.segmented, builder_to_string(*classes,, temp));
|
||||
if viewer.search_terms.count {
|
||||
shown: String_Builder;
|
||||
shown.allocator = temp;
|
||||
@ -155,6 +164,27 @@ stats_window_contents :: () { using ImGui;
|
||||
if s.read < s.captures {
|
||||
TextColored(app_theme.palette.dim, "% have not been read with this model; the OCR window's scan reads them.", stats_number(s.captures - s.read));
|
||||
}
|
||||
|
||||
// What segmentation found: shown whether or not OCR has read anything.
|
||||
if s.segmented {
|
||||
SeparatorText("What is in the images");
|
||||
TextColored(app_theme.palette.dim, "How many of the % segmented images hold each thing (%). Click one to search for it.", stats_number(s.segmented), SEGMENT_MODEL);
|
||||
if BeginTable("##classes", 3, .SizingFixedFit) {
|
||||
TableSetupColumn("class");
|
||||
TableSetupColumn("images");
|
||||
TableSetupColumn("bar", .WidthStretch);
|
||||
for s.top_classes {
|
||||
if it_index >= STATS_TOP_CLASSES break;
|
||||
TableNextRow();
|
||||
TableNextColumn();
|
||||
if SmallButton(temp_c_string(tprint("%##class%", it.word, it_index))) viewer_set_search(tprint("has:\"%\"", it.word));
|
||||
TableNextColumn(); TextUnformatted(stats_number(it.count));
|
||||
TableNextColumn(); ProgressBar(cast(float) it.count / max(s.segmented, 1), .{-FLOAT32_MIN, 0}, temp_c_string(stats_percent(it.count, s.segmented)));
|
||||
}
|
||||
EndTable();
|
||||
}
|
||||
}
|
||||
|
||||
if !s.read { TextColored(app_theme.palette.dim, "No readings yet."); return; }
|
||||
|
||||
SeparatorText("Text found");
|
||||
@ -266,8 +296,11 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result {
|
||||
s.most_words.allocator = context.default_allocator;
|
||||
s.common_words.allocator = context.default_allocator;
|
||||
s.top_labels.allocator = context.default_allocator;
|
||||
s.top_classes.allocator = context.default_allocator;
|
||||
top: Table(string, s64); // keys borrow the entries' label names
|
||||
top.allocator = temp;
|
||||
classes: Table(string, s64); // keys borrow the entries' segment classes
|
||||
classes.allocator = temp;
|
||||
|
||||
per_capture: [..] Stats_Capture;
|
||||
per_capture.allocator = temp;
|
||||
@ -289,6 +322,17 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result {
|
||||
count := find_or_add(*top, entry.labels[0].name);
|
||||
count.* += 1;
|
||||
}
|
||||
if entry && entry.segment_classes {
|
||||
s.segmented += 1;
|
||||
// Each class once per capture, however many of it there are.
|
||||
for segment, n: entry.segments {
|
||||
seen := false;
|
||||
for 0..n - 1 if entry.segments[it].class == segment.class { seen = true; break; }
|
||||
if seen continue;
|
||||
count := find_or_add(*classes, segment.class);
|
||||
count.* += 1;
|
||||
}
|
||||
}
|
||||
|
||||
section := catalog_section(entry, model);
|
||||
if !section continue;
|
||||
@ -337,6 +381,8 @@ stats_compute :: (items: [] Search_Item, model: string) -> *Stats_Result {
|
||||
|
||||
for count, name: top array_add(*s.top_labels, .{copy_string(name,, context.default_allocator), count});
|
||||
quick_sort(s.top_labels, (a: Stats_Word, b: Stats_Word) -> s64 { return ifx a.count != b.count then b.count - a.count else compare(a.word, b.word); });
|
||||
for count, name: classes array_add(*s.top_classes, .{copy_string(name,, context.default_allocator), count});
|
||||
quick_sort(s.top_classes, (a: Stats_Word, b: Stats_Word) -> s64 { return ifx a.count != b.count then b.count - a.count else compare(a.word, b.word); });
|
||||
|
||||
s.compute_ms = to_float64_seconds(current_time_monotonic() - start) * 1000;
|
||||
return s;
|
||||
|
||||
@ -826,7 +826,7 @@ viewer_ui :: () { using ImGui;
|
||||
}
|
||||
|
||||
SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520));
|
||||
if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like: folder:", viewer.search_buffer.data, viewer.search_buffer.count) {
|
||||
if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like: folder: has:", viewer.search_buffer.data, viewer.search_buffer.count) {
|
||||
viewer_search_changed();
|
||||
}
|
||||
SameLine();
|
||||
|
||||
@ -95,6 +95,21 @@ match_tests :: () {
|
||||
expect(search_matches(*entry, search_parse("like:anything")), "like: takes a labelled capture (the ranking is the background's)");
|
||||
entry.embedding = .[];
|
||||
|
||||
// has: a segment's class, the whole name (find_segments, as catalog_load makes it).
|
||||
entry.find_segments = "\nperson\ndog\ndining table\n";
|
||||
expect(search_matches(*entry, search_parse("has:dog")), "has: a class");
|
||||
expect(search_matches(*entry, search_parse("has:DOG has:person")), "has: twice, ignoring case");
|
||||
expect(search_matches(*entry, search_parse("has:\"dining table\"")), "has: a quoted name with a space");
|
||||
expect(!search_matches(*entry, search_parse("has:table")), "has: is the whole name: table is not dining table");
|
||||
expect(!search_matches(*entry, search_parse("has:do")), "nor part of a name");
|
||||
expect(!search_matches(*entry, search_parse("has:cat")), "a class it does not hold");
|
||||
expect(!search_matches(*entry, search_parse("dog")), "a plain word does not match segments");
|
||||
expect(search_matches(*entry, search_parse("has:dog kicad")), "has: with other words");
|
||||
terms := search_parse("has:dog");
|
||||
expect(terms.count == 1 && terms[0].field == .HAS, "the has: prefix is read");
|
||||
entry.find_segments = "";
|
||||
expect(!search_matches(*entry, search_parse("has:dog")), "nothing segmented: no has: matches");
|
||||
|
||||
expect(search_line_matches("0.1uF : Capacitor_SMD", search_parse("capacitor")), "a line holding a word");
|
||||
expect(!search_line_matches("0.1uF : Capacitor_SMD", search_parse("title:capacitor")), "a title word is not an OCR line's");
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user