[0.0.15] Segmentation phase 6: the segments overlay in the Capture window, colours to choose
- The Capture window draws a segmented image's segments over it: each class in its colour, a shade of it for each instance, edges darker; under the OCR boxes, at any fit and zoom. A viewer worker reads the segs and mask chunks (never the images); the main thread makes the texture from the mask when the capture, the opacity or the classes hidden change. - Above the image: Segments on or off, the opacity, and a legend of the classes (largest first, with counts): untick one to hide it, right click it to search has: for it. Hovering the image shows the segment's class, score and share of the image. - Colours: a class's own is a hue from its name's hash. Its legend swatch is a colour picker (ImGui ColorEdit3) to choose another, kept in %APPDATA%\mcap\segment-colours.config for every image and library; right click it to go back to its own. - Tests: the overlay's pixels (shades, edges, hidden classes, opacity), reading a file's segments for it, and chosen colours. - This completes segmentation-plan.html's six phases.
This commit is contained in:
parent
3c2c07bff6
commit
56588ff5e3
12
README.md
12
README.md
@ -74,6 +74,17 @@ grid (in its order, so through a search's matches).
|
||||
**Next** to scroll to each.
|
||||
- **Labels:** the capture's five best labels with their scores (dim when they come from an older
|
||||
label list).
|
||||
- **Segments**, for a segmented image:
|
||||
- **Drawing:** what segmentation found is drawn over the image, each class in its own colour, the
|
||||
same in every image. Each instance is a different shade, so two people side by side differ, and
|
||||
edges are darker.
|
||||
- **Controls:** an opacity slider, and a legend of the classes, largest first, with how many of
|
||||
each. Untick a class to hide it; the choice holds as you step through captures. Right-click a
|
||||
class to search `has:` for it.
|
||||
- **Colours:** a class's own colour is a hue from a hash of its name. Click its square in the legend
|
||||
to choose another, in every image and library. Right-click the square to go back to its own. The
|
||||
choices are kept in `segment-colours.config` in the app's data folder.
|
||||
- **Hover** the image for the segment under the mouse: its class, score and share of the image.
|
||||
|
||||
### The OCR window
|
||||
|
||||
@ -282,6 +293,7 @@ scripts and for trying things on a copy of the library: `bin\mcap.exe -library D
|
||||
| --- | --- |
|
||||
| OCR readings, labels, image vectors | inside each `.mcap` file (see `modules/Capture_File/README.md`) |
|
||||
| The label list | `%APPDATA%\mcap\labels.config` |
|
||||
| The segment colours chosen in the Capture window | `%APPDATA%\mcap\segment-colours.config` |
|
||||
| Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` |
|
||||
| Logs | `logs\<version>\` beside the program |
|
||||
| The hash of every capture's image, for finding duplicates | `.import-index` in the library folder |
|
||||
|
||||
@ -29,7 +29,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
GIT_BRANCH :: "check";
|
||||
GIT_HASH :: "-";
|
||||
VERSION :: "0.0.14";
|
||||
VERSION :: "0.0.15";
|
||||
BUILD_DATE :: "-";
|
||||
DEBUG :: true;
|
||||
COMPILE_TESTS :: true;
|
||||
|
||||
@ -8,7 +8,7 @@ APP_NAME :: "mcap";
|
||||
APP_FOLDER_NAME :: "mcap";
|
||||
APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
|
||||
|
||||
VERSION :: "0.0.14";
|
||||
VERSION :: "0.0.15";
|
||||
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
|
||||
// builds never write over the ones you run by hand.
|
||||
build_directory := "bin";
|
||||
|
||||
@ -68,7 +68,7 @@
|
||||
|
||||
<header>
|
||||
<h1>Semantic segmentation in mcap</h1>
|
||||
<p>Planning document · 2026-10-10 · for mcap after 0.0.10</p>
|
||||
<p>Planning document · 2026-10-10 · for mcap after 0.0.10 · <strong>all six phases built, mcap 0.0.15</strong></p>
|
||||
</header>
|
||||
|
||||
<div class="summary">
|
||||
@ -78,7 +78,7 @@
|
||||
<li><span class="tag decided">Decided</span> <strong>Images only</strong>, and only in the <strong>library folders you opt in</strong>. The setting is per folder, kept in a file in the library root, so it travels with the library.</li>
|
||||
<li><span class="tag decided">Decided</span> <strong>No .mcap version bump.</strong> Results go in two new chunk kinds: a small <code>segs</code> summary at the front for search, and a <code>mask</code> at the back for drawing.</li>
|
||||
<li>It is built in six phases. The first proves the ONNX export matches PyTorch before anything else depends on it.</li>
|
||||
<li><span class="tag decided">Progress</span> <strong>Phases 1 to 5 are done</strong> (2026-10-10). The model is exported and checked, and <code>modules/Segment</code> runs it in mcap at about 57 ms per image on the RTX 5080. Results are stored in .mcap files at about 5 KB per image. Library folders opt in through <code>.library-folders</code>, and the Segmentation window and <code>-segment</code> scan them. <code>has:</code> search and Stats' "What is in the images" are in (phase 5). Next is phase 6: the overlay in the Capture window.</li>
|
||||
<li><span class="tag decided">Done</span> <strong>All six phases are built</strong> (2026-10-10, mcap 0.0.11 to 0.0.15). See <a href="#state">Where it stands</a> for what was built, the commits, and what is left.</li>
|
||||
<li><span class="tag later">Later</span> Video, by treating sampled frames as images, is shelved.</li>
|
||||
</ul>
|
||||
</div>
|
||||
@ -93,6 +93,7 @@
|
||||
<li><a href="#search">Catalog, search, Stats</a></li>
|
||||
<li><a href="#overlay">Capture window overlay</a></li>
|
||||
<li><a href="#phases">Phases</a></li>
|
||||
<li><a href="#state">Where it stands</a></li>
|
||||
<li><a href="#questions">Open questions</a></li>
|
||||
<li><a href="#later">Later: video frames</a></li>
|
||||
</ol>
|
||||
@ -423,14 +424,80 @@ then runs: (id, length) pairs over the whole map, row by row, wrapping a
|
||||
</div>
|
||||
|
||||
<div class="phase">
|
||||
<h3>Phase 6: The overlay</h3>
|
||||
<h3>Phase 6: The overlay <span class="tag decided">Done</span></h3>
|
||||
<p>The mask texture on a worker, the legend, hiding classes, opacity and hover.</p>
|
||||
<p><strong>Done when:</strong> masks line up with the image at every fit and zoom, including on Snagit captures (rendered image) and on photos turned by their EXIF orientation.</p>
|
||||
<div class="note"><strong>Built (2026-10-10); waiting on a check by eye.</strong>
|
||||
<ul>
|
||||
<li><strong>Reading:</strong> a viewer worker reads the capture's <code>segs</code> and <code>mask</code>, not its images. It reads them again when the capture's catalog entry changes: stepped to, or segmented again.</li>
|
||||
<li><strong>The texture:</strong> the main thread makes it at the mask's size, and remakes it only when the capture, the opacity or the hidden classes change.</li>
|
||||
<li><strong>Colours:</strong> fixed by the class name's hash, a shade per instance (the plan's note from phase 2), with darker, stronger edges.</li>
|
||||
<li><strong>The row above the image:</strong> a Segments toggle, an opacity slider, and a legend that wraps across lines. Each class can be hidden, and right-clicking one searches <code>has:</code>.</li>
|
||||
<li><strong>Hover:</strong> the tooltip shows the class, score and share of the image.</li>
|
||||
<li><strong>Placement:</strong> the overlay covers the image as drawn, whatever the fit or zoom. It sits under the OCR boxes.</li>
|
||||
<li><strong>Tests:</strong> 12 checks on the pixels (shades, edges, hidden classes, opacity) and on reading a file's segments.</li>
|
||||
</ul>
|
||||
<p><strong>Checked by eye (2026-10-10):</strong> it looks right.</p>
|
||||
<p><strong>Then, colours you choose:</strong></p>
|
||||
<ul>
|
||||
<li><strong>The picker:</strong> each legend swatch is an ImGui colour picker (<code>ColorEdit3</code>, a hue wheel). The colour chosen becomes the class's in every image and library, and its instances shade from it.</li>
|
||||
<li><strong>Reset:</strong> right-clicking the swatch takes the class back to its own colour, from its name's hash.</li>
|
||||
<li><strong>Where they're kept:</strong> <code>%APPDATA%\mcap\segment-colours.config</code>, one line per class, such as <code>colour: "person" #e05a47</code>. Colours are a way of looking, not part of a library. The file is written once the picker closes, not on every frame of a drag.</li>
|
||||
<li><strong>Tests:</strong> 5 more checks: a chosen colour, its shades, other classes unchanged, reset.</li>
|
||||
</ul></div>
|
||||
</div>
|
||||
|
||||
<h2 id="state">Where it stands (2026-10-10)</h2>
|
||||
<p>All six phases are built and in <code>master</code>. You've checked them in the app: <code>has:person</code> and <code>has:cat</code> work, and the overlay with its colour picker looks right.</p>
|
||||
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<tr><th>Commit</th><th>Version</th><th>Phase</th></tr>
|
||||
<tr><td><code>cb21b73</code></td><td>0.0.11</td><td>1–2: the ONNX export and its check; <code>modules/Segment</code> and <code>segment-probe</code></td></tr>
|
||||
<tr><td><code>8b66021</code></td><td>0.0.12</td><td>3: the <code>segs</code> and <code>mask</code> chunks, <code>mcap_store</code> with back chunks</td></tr>
|
||||
<tr><td><code>3e6e733</code></td><td>0.0.13</td><td>4: <code>.library-folders</code>, the segmentation scan and window, <code>-segment</code>, <code>-erase segments</code></td></tr>
|
||||
<tr><td><code>3c2c07b</code></td><td>0.0.14</td><td>5: <code>has:</code> search; "What is in the images" in Stats</td></tr>
|
||||
<tr><td><code>cf3cecf</code></td><td>0.0.15</td><td>6: the overlay in the Capture window, colours chosen in its legend</td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>What you can do now</h3>
|
||||
<ul>
|
||||
<li><strong>Choose folders:</strong> in <strong>Segments</strong> above the grid, tick the library folders to segment. They are kept in the library's <code>.library-folders</code>. Importing into a ticked folder segments the new images.</li>
|
||||
<li><strong>Scan:</strong> segment those folders' images, with pause and cancel, from the window or with <code>-segment</code>. It runs at about 17 images a second on the RTX 5080 laptop and stores about 5 KB per image.</li>
|
||||
<li><strong>Search:</strong> <code>has:dog</code>, <code>has:"dining table"</code>, combined with any other search term.</li>
|
||||
<li><strong>Stats:</strong> how many images hold each class, each one a <code>has:</code> search when clicked.</li>
|
||||
<li><strong>Look:</strong> the Capture window draws the segments over the image. It has opacity, a legend to hide classes or search them, colours you can choose (kept in <code>%APPDATA%\mcap\segment-colours.config</code>), and the class under the mouse.</li>
|
||||
<li><strong>Erase:</strong> per folder in the window, or the whole library with <code>-erase segments</code>.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Tests</h3>
|
||||
<ul>
|
||||
<li><strong>Unit tests:</strong> 55 checks in the import suite cover the chunks (24), <code>.library-folders</code> (14), and the overlay and colours (17). 10 in the catalog suite cover <code>has:</code>.</li>
|
||||
<li><strong><code>segment-probe -check</code>:</strong> mcap's segmentation matches PyTorch's on 99.8% of pixels over 25 images.</li>
|
||||
<li><strong>Scratch libraries:</strong> the scan, cancel, erase and search were run end to end.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Left to do</h3>
|
||||
<ul>
|
||||
<li><strong>Push.</strong> The repo has no remote yet; adding one is all that's needed to push.</li>
|
||||
<li><strong>Getting the model.</strong> <code>fetch_models.jai</code> for <code>modules/Segment</code> waits on where the 462 MB .onnx is hosted (open question 2). Until then, <code>tools/export_mask2former.py</code> makes it, and a fresh clone shows "the model is not there" in the Segmentation window.</li>
|
||||
<li><strong>SnagitConverter.</strong> Its copy of <code>Capture_File</code> predates file version 4 (0.0.9), so it refuses files with a video's 64-bit size. The segment chunks themselves need no change: it keeps chunks it does not know.</li>
|
||||
<li><strong>Untested:</strong> writing <code>segment-colours.config</code> and reading it back. The tests set colours in memory only, so they never touch the real data folder.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Ideas for later</h3>
|
||||
<ul>
|
||||
<li><strong>Counts and sizes in <code>has:</code>:</strong> <code>has:person:3+</code>, <code>has:dog>20%</code>. The front chunk already has what they need.</li>
|
||||
<li><strong>Faster scans:</strong> making segments (30 ms) takes longer than the model (21 ms). Spreading it over threads would roughly double throughput, if large libraries make the scan's speed matter.</li>
|
||||
<li><strong>The score threshold</strong> is 0.5, transformers' default. Raise <code>score_threshold</code> in <code>models.config</code> toward 0.7 if weak segments make <code>has:</code> noisy on your photo libraries.</li>
|
||||
<li><strong>A warning for screenshot folders</strong> (open question 4), and <strong>erasing other models' masks</strong> after a model change (open question 5).</li>
|
||||
<li><strong>Video frames</strong> (section 10), shelved.</li>
|
||||
</ul>
|
||||
|
||||
<h2 id="questions">9. Open questions</h2>
|
||||
<ol>
|
||||
<li><strong>COCO panoptic or ADE20K?</strong> COCO counts things (people, pets, cars); ADE20K knows more indoor objects and scene parts, but has no instances. The default is COCO. Revisit after phase 1 by trying both on a sample of your photo libraries.</li>
|
||||
<li><strong>COCO panoptic or ADE20K?</strong> COCO counts things (people, pets, cars); ADE20K knows more indoor objects and scene parts, but has no instances. The default is COCO. <span class="tag decided">Built with COCO</span> Revisit when your photo libraries are in, by trying both on a sample; results are keyed by model, so the two can sit side by side.</li>
|
||||
<li><strong>Where to host the exported .onnx</strong> for <code>fetch_models.jai</code>: a private Hugging Face repo, a release asset, or committing it as the FFmpeg DLLs are. At about 400 MB for Swin-B, committing it is probably too much.</li>
|
||||
<li><strong>The mask resolution cap.</strong> <span class="tag decided">Answered</span> 1024 on the long side costs 0.8 to 6.4 KB per photo (phase 3), so there is no reason to go lower.</li>
|
||||
<li><strong>Snagit and screenshot folders.</strong> They stay off by default, and COCO classes mean little on UI. Turning one on is allowed, but the Segmentation window could warn that most of the folder is screenshots, judged from <code>Format</code> and <code>AppName</code>.</li>
|
||||
|
||||
@ -26,6 +26,20 @@ segment_ui: struct {
|
||||
// The window's counts per library folder, worked out again when the catalog changes.
|
||||
counted_generation: s64 = -1;
|
||||
counts: [..] Segment_Folder_Count;
|
||||
|
||||
overlay_state: Segment_Overlay_State; // the Capture window's overlay; main thread
|
||||
}
|
||||
|
||||
Segment_Overlay_State :: struct {
|
||||
show := true;
|
||||
opacity: float = 0.45;
|
||||
hidden: [..] string; // classes not drawn, by name, for every capture; heap
|
||||
colours: [..] Segment_Colour; // classes given their own colour in the legend; kept in SEGMENT_COLOURS_FILE
|
||||
overlay: *Segment_Overlay; // the capture's shown, or being replaced
|
||||
entry: *Catalog_Entry; // the catalog entry it was asked for: a new one is read again
|
||||
loading: bool;
|
||||
texture: Gpu_Texture;
|
||||
dirty: bool; // the texture is out of date
|
||||
}
|
||||
|
||||
Segment_Folder_Count :: struct {
|
||||
@ -35,6 +49,68 @@ Segment_Folder_Count :: struct {
|
||||
catalogued: bool; // every capture in it has its catalog entry
|
||||
}
|
||||
|
||||
// A class's colour, chosen in the Capture window's legend. Heap.
|
||||
Segment_Colour :: struct {
|
||||
class: string;
|
||||
rgb: [3] u8;
|
||||
}
|
||||
|
||||
// The colours chosen, one line each: colour: "person" #e05a47. In the app's data folder, beside the
|
||||
// label list: a way of looking, so the same in every library.
|
||||
SEGMENT_COLOURS_FILE :: "segment-colours.config";
|
||||
|
||||
segment_colours_path :: () -> string {
|
||||
return tprint("%/%", app_program_data_dir(), SEGMENT_COLOURS_FILE);
|
||||
}
|
||||
|
||||
segment_colours_load :: () {
|
||||
o := *segment_ui.overlay_state;
|
||||
o.colours.allocator = context.default_allocator;
|
||||
text, ok := read_entire_file(segment_colours_path(), log_errors = false,, temp);
|
||||
if !ok return;
|
||||
for raw: split(text, "\n",, temp) {
|
||||
line := trim(trim_right(raw, "\r"));
|
||||
if !begins_with(line, "colour:") continue; // @version 1, comments, and keys a later build adds
|
||||
class, rest, class_ok := mcap_read_quoted(trim(advance(line, 7)),, temp);
|
||||
rest = trim(rest);
|
||||
if !class_ok || rest.count != 7 || rest[0] != #char "#" continue;
|
||||
value, value_ok := string_to_int(advance(rest, 1), 16, u32);
|
||||
if !value_ok continue;
|
||||
segment_colour_set(class, .[cast(u8) (value >> 16), cast(u8) ((value >> 8) & 0xFF), cast(u8) (value & 0xFF)], save = false);
|
||||
}
|
||||
}
|
||||
|
||||
segment_colours_save :: () {
|
||||
builder: String_Builder;
|
||||
builder.allocator = temp;
|
||||
append(*builder, "@version 1\n# The colours of segment classes in the Capture window, chosen in its legend. A class not\n# listed has its own colour, from its name.\n");
|
||||
for segment_ui.overlay_state.colours {
|
||||
print_to_builder(*builder, "colour: % #%\n", mcap_quote(it.class), formatInt((cast(u32) it.rgb[0] << 16) | (cast(u32) it.rgb[1] << 8) | it.rgb[2], base = 16, minimum_digits = 6));
|
||||
}
|
||||
if !write_entire_file(segment_colours_path(), *builder) log_error("[Segment] Could not write %.", segment_colours_path());
|
||||
}
|
||||
|
||||
// Gives class its own colour; or, with default, takes it back to the one from its name.
|
||||
segment_colour_set :: (class: string, rgb: [3] u8, default := false, save := true) {
|
||||
o := *segment_ui.overlay_state;
|
||||
for * o.colours {
|
||||
if it.class != class continue;
|
||||
if default {
|
||||
free(it.class,, context.default_allocator);
|
||||
array_ordered_remove_by_index(*o.colours, it_index);
|
||||
} else {
|
||||
it.rgb = rgb;
|
||||
}
|
||||
if save segment_colours_save();
|
||||
o.dirty = true;
|
||||
return;
|
||||
}
|
||||
if default return;
|
||||
array_add(*o.colours, .{copy_string(class,, context.default_allocator), rgb});
|
||||
if save segment_colours_save();
|
||||
o.dirty = true;
|
||||
}
|
||||
|
||||
segment_models_folder :: () -> string {
|
||||
return tprint("%segment-models", path_strip_filename(get_path_of_running_executable()));
|
||||
}
|
||||
@ -43,11 +119,20 @@ segment_models_folder :: () -> string {
|
||||
|
||||
segment_ui_startup :: () {
|
||||
segment_ui.counts.allocator = context.default_allocator;
|
||||
segment_ui.overlay_state.hidden.allocator = context.default_allocator;
|
||||
segment_colours_load();
|
||||
library_folders_load();
|
||||
}
|
||||
|
||||
// After ocr_ui_shutdown has stopped the OCR thread.
|
||||
segment_ui_shutdown :: () {
|
||||
o := *segment_ui.overlay_state; // the viewer's workers have stopped; the texture goes with the device
|
||||
segment_overlay_free(o.overlay);
|
||||
segment_overlay_free(viewer.segments_done);
|
||||
for o.hidden free(it,, context.default_allocator);
|
||||
array_free(o.hidden);
|
||||
for o.colours free(it.class,, context.default_allocator);
|
||||
array_free(o.colours);
|
||||
segment_destroy(segment_ui.engine);
|
||||
segment_ui.engine = null;
|
||||
free(segment_ui.engine_error,, context.default_allocator);
|
||||
@ -350,6 +435,279 @@ segment_update_counts :: () {
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the Capture window's overlay
|
||||
//
|
||||
// The segments of the capture shown, over its image: each segment in its class's colour (a shade of
|
||||
// it for each instance, so two people side by side differ), its edges darker, with the class under
|
||||
// the mouse in a tooltip. A viewer worker reads the segs and mask chunks (never the images); the
|
||||
// main thread makes the texture from the mask, again only when the capture, the opacity or the
|
||||
// classes hidden change.
|
||||
|
||||
// A capture's segments as read for the overlay. Heap, with everything in it.
|
||||
Segment_Overlay :: struct {
|
||||
index: s32 = -1;
|
||||
error: string; // why there is none
|
||||
width, height: s32; // the mask's
|
||||
map: [] u8; // a segment id per pixel; 0: none
|
||||
segments: [] Mcap_Segment; // largest first
|
||||
by_id: [256] s16; // into segments, for each id; -1: none
|
||||
}
|
||||
|
||||
// On a viewer worker: the capture's segments by SEGMENT_MODEL, or why not.
|
||||
segment_overlay_read :: (path: string) -> *Segment_Overlay {
|
||||
overlay := New(Segment_Overlay,, context.default_allocator);
|
||||
for * overlay.by_id it.* = -1;
|
||||
m, original, ok, error := mcap_read_without_original(path,, temp);
|
||||
if !ok { overlay.error = copy_string(error,, context.default_allocator); return overlay; }
|
||||
segs := mcap_find_chunk(m.front_chunks, mcap_tag(MCAP_SEGMENTS_TAG), SEGMENT_MODEL);
|
||||
mask := mcap_find_chunk(m.back_chunks, mcap_tag(MCAP_MASK_TAG), SEGMENT_MODEL);
|
||||
if !segs || !mask { overlay.error = copy_string("it has not been segmented",, context.default_allocator); return overlay; }
|
||||
segments, segments_ok, segments_error := mcap_segments_parse(cast(string) segs.data,, temp);
|
||||
map, map_ok, map_error := mcap_mask_parse(mask.data,, temp);
|
||||
if !segments_ok || !map_ok { overlay.error = sprint("its segments cannot be read: %", ifx segments_error then segments_error else map_error,, context.default_allocator); return overlay; }
|
||||
|
||||
push_allocator(context.default_allocator);
|
||||
overlay.width, overlay.height = map.width, map.height;
|
||||
overlay.map = array_copy(map.map);
|
||||
overlay.segments = NewArray(segments.segments.count, Mcap_Segment);
|
||||
for segments.segments {
|
||||
overlay.segments[it_index] = it;
|
||||
overlay.segments[it_index].class = copy_string(it.class);
|
||||
overlay.by_id[it.id] = cast(s16) it_index;
|
||||
}
|
||||
return overlay;
|
||||
}
|
||||
|
||||
segment_overlay_free :: (overlay: *Segment_Overlay) {
|
||||
if !overlay return;
|
||||
push_allocator(context.default_allocator);
|
||||
free(overlay.error);
|
||||
array_free(overlay.map);
|
||||
for overlay.segments free(it.class);
|
||||
array_free(overlay.segments);
|
||||
free(overlay);
|
||||
}
|
||||
|
||||
// Main thread, each frame: takes what the workers read, asks for the capture shown, and makes the
|
||||
// texture when it is out of date.
|
||||
segment_overlay_frame :: () {
|
||||
o := *segment_ui.overlay_state;
|
||||
lock(*viewer.mutex);
|
||||
done := viewer.segments_done;
|
||||
arrived := viewer.segments_done_request != 0 && viewer.segments_done_request == viewer.segments_request;
|
||||
viewer.segments_done = null;
|
||||
viewer.segments_done_request = 0;
|
||||
unlock(*viewer.mutex);
|
||||
if arrived {
|
||||
segment_overlay_free(o.overlay);
|
||||
o.overlay = done;
|
||||
o.loading = false;
|
||||
o.dirty = true;
|
||||
} else {
|
||||
segment_overlay_free(done);
|
||||
}
|
||||
|
||||
// Nothing shown: let it all go.
|
||||
index := viewer.full_index;
|
||||
entry: *Catalog_Entry;
|
||||
if viewer.full_open && index >= 0 && index < viewer.captures.count && !viewer.captures[index].removed entry = viewer.captures[index].catalog;
|
||||
wanted := o.show && entry && entry.segment_classes.count > 0;
|
||||
if !wanted {
|
||||
gpu_texture_release(*o.texture);
|
||||
if !viewer.full_open { segment_overlay_free(o.overlay); o.overlay = null; o.entry = null; o.loading = false; }
|
||||
return;
|
||||
}
|
||||
|
||||
// The capture shown, read again when its catalog entry is new (stepped to, or segmented again).
|
||||
if o.entry != entry {
|
||||
o.entry = entry;
|
||||
o.loading = true;
|
||||
lock(*viewer.mutex);
|
||||
viewer.segments_wanted = index;
|
||||
viewer.segments_request += 1;
|
||||
unlock(*viewer.mutex);
|
||||
signal(*viewer.wake);
|
||||
}
|
||||
if o.dirty && o.overlay && o.overlay.index == index {
|
||||
o.dirty = false;
|
||||
segment_overlay_build();
|
||||
}
|
||||
}
|
||||
|
||||
// The device went (graphics/DX11.jai): the texture with it; made again from the mask held.
|
||||
segment_overlay_device_lost :: () {
|
||||
o := *segment_ui.overlay_state;
|
||||
gpu_texture_release(*o.texture);
|
||||
o.dirty = true;
|
||||
}
|
||||
|
||||
// The row above the image: the toggle, the opacity, and the classes in it, each to hide or show
|
||||
// (right click to search for it).
|
||||
segment_overlay_controls :: (index: s32) { using ImGui;
|
||||
if index < 0 || index >= viewer.captures.count return;
|
||||
entry := viewer.captures[index].catalog;
|
||||
if !entry || !entry.segment_classes return;
|
||||
o := *segment_ui.overlay_state;
|
||||
|
||||
if Checkbox("Segments", *o.show) o.dirty = true;
|
||||
if IsItemHovered() SetTooltip("What segmentation found in this image, drawn over it (%)", SEGMENT_MODEL);
|
||||
if !o.show { SameLine(); TextColored(app_theme.palette.dim, "% found", entry.segment_count); return; }
|
||||
SameLine();
|
||||
SetNextItemWidth(140);
|
||||
percent := o.opacity * 100;
|
||||
if SliderFloat("##opacity", *percent, 10, 90, "opacity %.0f") { o.opacity = percent / 100; o.dirty = true; }
|
||||
if o.loading { SameLine(); TextColored(app_theme.palette.dim, "Loading..."); }
|
||||
if o.overlay && o.overlay.index == index && o.overlay.error { SameLine(); TextColored(app_theme.palette.warning, "%", o.overlay.error); }
|
||||
|
||||
// The legend: each class once, largest first, with its colour and how many of it.
|
||||
right_edge := GetCursorScreenPos().x + GetContentRegionAvail().x;
|
||||
first := true;
|
||||
for segment, n: entry.segments {
|
||||
seen := false;
|
||||
for 0..n - 1 if entry.segments[it].class == segment.class { seen = true; break; }
|
||||
if seen continue;
|
||||
count := 0;
|
||||
for entry.segments if it.class == segment.class count += 1;
|
||||
label := ifx count > 1 then tprint("% x%##legend%", segment.class, count, n) else tprint("%##legend%", segment.class, n);
|
||||
width := GetFrameHeight() + GetStyle().ItemInnerSpacing.x + GetFrameHeight() + GetStyle().ItemInnerSpacing.x + CalcTextSize(temp_c_string(label), null, true).x;
|
||||
if !first {
|
||||
SameLine();
|
||||
if GetCursorScreenPos().x + width > right_edge NewLine();
|
||||
}
|
||||
first = false;
|
||||
|
||||
// Its colour: click to choose another (kept for every capture), right click to go back to its own.
|
||||
PushID(temp_c_string(segment.class));
|
||||
base := segment_class_base_colour(segment.class);
|
||||
rgb := float.[base[0] / 255.0, base[1] / 255.0, base[2] / 255.0];
|
||||
if ColorEdit3("##colour", *rgb, .NoInputs | .NoLabel | .NoAlpha | .NoOptions | .PickerHueWheel) {
|
||||
segment_colour_set(segment.class, .[cast(u8) (rgb[0] * 255 + 0.5), cast(u8) (rgb[1] * 255 + 0.5), cast(u8) (rgb[2] * 255 + 0.5)], save = false);
|
||||
}
|
||||
if IsItemDeactivatedAfterEdit() segment_colours_save(); // once the picker closes, not every frame of a drag
|
||||
chosen := false;
|
||||
for o.colours if it.class == segment.class chosen = true;
|
||||
if BeginPopupContextItem("##reset") {
|
||||
if MenuItem("Back to the default colour", null, false, chosen) segment_colour_set(segment.class, .[0, 0, 0], default = true);
|
||||
EndPopup();
|
||||
}
|
||||
PopID();
|
||||
SameLine(0, GetStyle().ItemInnerSpacing.x);
|
||||
shown := !array_find(o.hidden, segment.class);
|
||||
if Checkbox(temp_c_string(label), *shown) {
|
||||
if shown { found, at_hidden := array_find(o.hidden, segment.class); if found { free(o.hidden[at_hidden],, context.default_allocator); array_unordered_remove_by_index(*o.hidden, at_hidden); } }
|
||||
else array_add(*o.hidden, copy_string(segment.class,, context.default_allocator));
|
||||
o.dirty = true;
|
||||
}
|
||||
if IsItemClicked(.Right) viewer_set_search(tprint("has:\"%\"", segment.class));
|
||||
if IsItemHovered() SetTooltip("Shows or hides %; right click to search has:\"%\"", segment.class, segment.class);
|
||||
}
|
||||
}
|
||||
|
||||
// Over the image drawn from image_min to image_max: the overlay's texture, and the segment under the mouse.
|
||||
segment_overlay_draw :: (index: s32, image_min: ImGui.ImVec2, image_max: ImGui.ImVec2) { using ImGui;
|
||||
o := *segment_ui.overlay_state;
|
||||
overlay := o.overlay;
|
||||
if !o.show || !overlay || overlay.index != index || !o.texture.handle return;
|
||||
draw := GetWindowDrawList();
|
||||
draw.AddImage(draw, gpu_texture_ref(o.texture), image_min, image_max);
|
||||
|
||||
if !IsItemHovered() return; // the image
|
||||
mouse := GetMousePos();
|
||||
w, h := image_max.x - image_min.x, image_max.y - image_min.y;
|
||||
if w <= 0 || h <= 0 return;
|
||||
x := cast(s32) ((mouse.x - image_min.x) / w * overlay.width);
|
||||
y := cast(s32) ((mouse.y - image_min.y) / h * overlay.height);
|
||||
if x < 0 || y < 0 || x >= overlay.width || y >= overlay.height return;
|
||||
id := overlay.map[cast(s64) y * overlay.width + x];
|
||||
at := overlay.by_id[id];
|
||||
if !id || at < 0 return;
|
||||
segment := overlay.segments[at];
|
||||
if array_find(o.hidden, segment.class) return;
|
||||
SetTooltip("% score % % of the image", segment.class, FF(segment.score, 2), join(tprint("%", FF(segment.area * 100, 1)), "%",, temp));
|
||||
}
|
||||
|
||||
// A class's colour (the same in every capture), a shade of it for its instance'th segment in a
|
||||
// capture: the next ones darker, then round again.
|
||||
segment_class_colour :: (class: string, instance: s64) -> [3] u8 {
|
||||
base := segment_class_base_colour(class);
|
||||
shade := 1 - 0.18 * cast(float) (instance % 4);
|
||||
return .[cast(u8) (base[0] * shade), cast(u8) (base[1] * shade), cast(u8) (base[2] * shade)];
|
||||
}
|
||||
|
||||
// The colour chosen for class in the legend; else its own, from a hash of its name.
|
||||
segment_class_base_colour :: (class: string) -> [3] u8 {
|
||||
for segment_ui.overlay_state.colours if it.class == class return it.rgb;
|
||||
return segment_class_default_colour(class);
|
||||
}
|
||||
|
||||
// A hue from a hash of the name, at a fixed saturation and brightness.
|
||||
segment_class_default_colour :: (class: string) -> [3] u8 {
|
||||
hash := fnv1a_hash(class.data, class.count);
|
||||
hue := cast(float) (hash % 360) / 60.0;
|
||||
saturation := 0.65;
|
||||
value := 0.95;
|
||||
sector := cast(s32) hue;
|
||||
f := hue - sector;
|
||||
p := value * (1 - saturation);
|
||||
q := value * (1 - saturation * f);
|
||||
t := value * (1 - saturation * (1 - f));
|
||||
rgb: [3] float;
|
||||
if sector == {
|
||||
case 0; rgb = .[value, t, p];
|
||||
case 1; rgb = .[q, value, p];
|
||||
case 2; rgb = .[p, value, t];
|
||||
case 3; rgb = .[p, q, value];
|
||||
case 4; rgb = .[t, p, value];
|
||||
case; rgb = .[value, p, q];
|
||||
}
|
||||
return .[cast(u8) (rgb[0] * 255), cast(u8) (rgb[1] * 255), cast(u8) (rgb[2] * 255)];
|
||||
}
|
||||
|
||||
// The overlay as RGBA pixels at the mask's size (temp): each pixel in its segment's colour at the
|
||||
// opacity, its edges darker and stronger; hidden classes and no segment left clear.
|
||||
segment_overlay_pixels :: (overlay: *Segment_Overlay, hidden: [] string, opacity: float) -> [] u8 {
|
||||
w, h := overlay.width, overlay.height;
|
||||
fill, edge: [256] [4] u8; // by segment id
|
||||
alpha := cast(u8) clamp(opacity * 255, 0, 255);
|
||||
edge_alpha := cast(u8) clamp(opacity * 255 * 1.6 + 40, 0, 255);
|
||||
for segment, n: overlay.segments {
|
||||
if array_find(hidden, segment.class) continue;
|
||||
instance := 0;
|
||||
for 0..n - 1 if overlay.segments[it].class == segment.class instance += 1;
|
||||
c := segment_class_colour(segment.class, instance);
|
||||
fill[segment.id] = .[c[0], c[1], c[2], alpha];
|
||||
edge[segment.id] = .[c[0] / 3, c[1] / 3, c[2] / 3, edge_alpha];
|
||||
}
|
||||
pixels := NewArray(cast(s64) w * h * 4, u8,, temp);
|
||||
for y: 0..h - 1 {
|
||||
row := cast(s64) y * w;
|
||||
for x: 0..w - 1 {
|
||||
i := row + x;
|
||||
id := overlay.map[i];
|
||||
if !id continue;
|
||||
on_edge := (x > 0 && overlay.map[i - 1] != id) || (x + 1 < w && overlay.map[i + 1] != id) ||
|
||||
(y > 0 && overlay.map[i - w] != id) || (y + 1 < h && overlay.map[i + w] != id);
|
||||
colour := ifx on_edge then edge[id] else fill[id];
|
||||
memcpy(pixels.data + i * 4, colour.data, 4);
|
||||
}
|
||||
}
|
||||
return pixels;
|
||||
}
|
||||
|
||||
#scope_file
|
||||
|
||||
// The texture, from the mask held.
|
||||
segment_overlay_build :: () {
|
||||
o := *segment_ui.overlay_state;
|
||||
gpu_texture_release(*o.texture);
|
||||
overlay := o.overlay;
|
||||
if !overlay || !overlay.map.count return;
|
||||
pixels := segment_overlay_pixels(overlay, o.hidden, o.opacity);
|
||||
o.texture = gpu_texture_create(pixels.data, overlay.width, overlay.height);
|
||||
}
|
||||
|
||||
#scope_export
|
||||
|
||||
// The confirmation for erasing one folder's segments, as a modal popup; at the top level, each frame.
|
||||
segment_erase_popup :: () { using ImGui;
|
||||
POPUP :: "Erase segments?";
|
||||
|
||||
@ -100,6 +100,11 @@ viewer: struct {
|
||||
full_taken: u64; // the request a worker last took
|
||||
full_done: Decoded;
|
||||
full_done_request: u64; // 0 while there is none
|
||||
segments_wanted: s32 = -1; // the capture whose segments the overlay wants (app/segment_ui.jai)
|
||||
segments_request: u64;
|
||||
segments_taken: u64;
|
||||
segments_done: *Segment_Overlay; // read, waiting for the main thread
|
||||
segments_done_request: u64;
|
||||
quit: bool;
|
||||
workers: [VIEWER_WORKER_COUNT] Thread;
|
||||
workers_started: s64;
|
||||
@ -207,6 +212,7 @@ viewer_worker :: (thread: *Thread) -> s64 {
|
||||
request: u64;
|
||||
path: string;
|
||||
|
||||
segments: bool;
|
||||
lock(*viewer.mutex);
|
||||
if viewer.quit { unlock(*viewer.mutex); return 0; }
|
||||
if viewer.full_request != viewer.full_taken {
|
||||
@ -214,6 +220,12 @@ viewer_worker :: (thread: *Thread) -> s64 {
|
||||
request = viewer.full_request;
|
||||
viewer.full_taken = request;
|
||||
index = viewer.full_wanted;
|
||||
} else if viewer.segments_request != viewer.segments_taken {
|
||||
// The Capture window's segments overlay (app/segment_ui.jai).
|
||||
segments = true;
|
||||
request = viewer.segments_request;
|
||||
viewer.segments_taken = request;
|
||||
index = viewer.segments_wanted;
|
||||
} else if viewer.queue_next < viewer.queue.count {
|
||||
index = viewer.queue[viewer.queue_next];
|
||||
viewer.queue_next += 1;
|
||||
@ -227,6 +239,22 @@ viewer_worker :: (thread: *Thread) -> s64 {
|
||||
unlock(*viewer.mutex);
|
||||
if index < 0 break; // nothing to do: wait to be woken
|
||||
|
||||
if segments {
|
||||
overlay := segment_overlay_read(path);
|
||||
overlay.index = index;
|
||||
reset_temporary_storage();
|
||||
lock(*viewer.mutex);
|
||||
if request == viewer.segments_request { // still the one wanted
|
||||
segment_overlay_free(viewer.segments_done);
|
||||
viewer.segments_done = overlay;
|
||||
viewer.segments_done_request = request;
|
||||
} else {
|
||||
segment_overlay_free(overlay);
|
||||
}
|
||||
unlock(*viewer.mutex);
|
||||
continue;
|
||||
}
|
||||
|
||||
if catalog {
|
||||
entry, error := catalog_load(path);
|
||||
if !entry log_error("[catalog] % %", path, error);
|
||||
@ -348,6 +376,7 @@ viewer_frame_gpu_work :: () {
|
||||
gpu_texture_release(*viewer.full_texture);
|
||||
viewer_request_full(viewer.full_index);
|
||||
}
|
||||
segment_overlay_device_lost(); // made again from the mask it holds
|
||||
}
|
||||
|
||||
// The full-size image, when one has arrived.
|
||||
@ -396,6 +425,7 @@ viewer_frame_gpu_work :: () {
|
||||
viewer_evict();
|
||||
viewer_apply_catalog_updates();
|
||||
viewer_open_frame();
|
||||
segment_overlay_frame(); // the Capture window's segments: what arrived, and its texture (app/segment_ui.jai)
|
||||
background_frame(); // takes back search results, and starts a search when visible_dirty (app/background.jai)
|
||||
}
|
||||
|
||||
@ -1138,6 +1168,7 @@ viewer_full_window :: () { using ImGui;
|
||||
}
|
||||
ocr_ui_capture_controls(viewer.full_index, capture.path); // app/ocr_ui.jai
|
||||
labels_ui_capture_line(viewer.full_index); // app/labels_ui.jai
|
||||
segment_overlay_controls(viewer.full_index); // app/segment_ui.jai
|
||||
|
||||
// Left and Right step through the captures in grid order while this window has focus.
|
||||
if IsWindowFocused(.RootAndChildWindows) {
|
||||
@ -1171,6 +1202,8 @@ viewer_full_window :: () { using ImGui;
|
||||
}
|
||||
zoom := clamp(viewer.full_zoom, VIEWER_MIN_ZOOM, VIEWER_MAX_ZOOM) / 100;
|
||||
Image(gpu_texture_ref(viewer.full_texture), .{source_w * zoom, source_h * zoom});
|
||||
// The segments under the OCR boxes: both cover the image as shown, whatever its size.
|
||||
segment_overlay_draw(viewer.full_index, GetItemRectMin(), GetItemRectMax());
|
||||
// OCR boxes are in the image's own pixels, which is the captured size.
|
||||
ocr_ui_capture_overlay(viewer.full_index, GetItemRectMin(), zoom);
|
||||
EndChild();
|
||||
|
||||
@ -521,6 +521,85 @@ END;
|
||||
expect(!library_folders.folders.count, "no file: no options");
|
||||
}
|
||||
|
||||
// The Capture window's segments overlay (app/segment_ui.jai): the pixels it draws, and reading a file's
|
||||
// segments for it.
|
||||
segment_overlay_tests :: () {
|
||||
group("segment overlay");
|
||||
// A 4 x 3 mask: two people (ids 1 and 2) side by side on the left, a dog (3) on the right, a gap.
|
||||
map := u8.[1, 1, 2, 3,
|
||||
1, 1, 2, 3,
|
||||
0, 0, 0, 3];
|
||||
overlay: Segment_Overlay;
|
||||
overlay.width, overlay.height = 4, 3;
|
||||
overlay.map = map;
|
||||
segments := Mcap_Segment.[.{id = 1, class = "person"}, .{id = 2, class = "person"}, .{id = 3, class = "dog"}];
|
||||
overlay.segments = segments;
|
||||
pixel :: (pixels: [] u8, x: s64, y: s64) -> [4] u8 {
|
||||
p: [4] u8;
|
||||
memcpy(p.data, pixels.data + (y * 4 + x) * 4, 4);
|
||||
return p;
|
||||
}
|
||||
|
||||
pixels := segment_overlay_pixels(*overlay, .[], 0.5);
|
||||
expect(pixels.count == 4 * 3 * 4, "RGBA at the mask's size");
|
||||
expect(pixel(pixels, 0, 2)[3] == 0, "no segment: clear");
|
||||
a, b := pixel(pixels, 0, 0), pixel(pixels, 2, 0);
|
||||
expect(a[3] > 0 && b[3] > 0, "segments drawn");
|
||||
expect(a[0] != b[0] || a[1] != b[1] || a[2] != b[2], "two people side by side differ in shade");
|
||||
inner, rim := pixel(pixels, 0, 0), pixel(pixels, 1, 0); // the first person's left column, and its edge with the second
|
||||
expect(rim[3] > inner[3] && rim[0] <= inner[0], "edges stronger and darker than the inside: % and %", rim, inner);
|
||||
same :: (a: [3] u8, b: [3] u8) -> bool { return a[0] == b[0] && a[1] == b[1] && a[2] == b[2]; }
|
||||
dog := segment_class_colour("dog", 0);
|
||||
expect(same(segment_class_colour("dog", 0), dog) && !same(segment_class_colour("person", 0), dog), "a class's colour is its own, the same each time");
|
||||
|
||||
// A colour chosen in the legend: the class's base, its instances shades of it; then back to its own.
|
||||
own := segment_class_colour("dog", 0);
|
||||
segment_colour_set("dog", .[200, 100, 50], save = false);
|
||||
expect(same(segment_class_colour("dog", 0), .[200, 100, 50]), "a chosen colour is the class's");
|
||||
second := segment_class_colour("dog", 1);
|
||||
expect(second[0] < 200 && second[0] > 100 && second[1] < 100, "its next instance a darker shade of it: %", second);
|
||||
expect(same(segment_class_colour("person", 0), segment_class_default_colour("person")), "other classes keep their own");
|
||||
segment_colour_set("dog", .[0, 0, 0], default = true, save = false);
|
||||
expect(same(segment_class_colour("dog", 0), own), "back to its own colour");
|
||||
expect(!segment_ui.overlay_state.colours.count, "and nothing chosen is left");
|
||||
|
||||
hidden := segment_overlay_pixels(*overlay, .["person"], 0.5);
|
||||
expect(pixel(hidden, 0, 0)[3] == 0 && pixel(hidden, 2, 1)[3] == 0, "a hidden class is clear");
|
||||
expect(pixel(hidden, 3, 0)[3] > 0, "the others still drawn");
|
||||
faint := segment_overlay_pixels(*overlay, .[], 0.1);
|
||||
expect(pixel(faint, 0, 0)[3] < inner[3], "opacity sets the alpha");
|
||||
|
||||
// Read from a file as a viewer worker reads it.
|
||||
#if OS == .WINDOWS folder := tprint("%/mcap-tests", windows_expand_environment("%TEMP%"));
|
||||
else folder := "/tmp/mcap-tests";
|
||||
make_directory_if_it_does_not_exist(folder, recursive = true);
|
||||
path := tprint("%/overlay.mcap", folder);
|
||||
m: Mcap;
|
||||
m.text = "[[file \"metadata.json\"]]\nWindowName: \"photo.jpg\"\n";
|
||||
m.preview = .{width = 2, height = 1, png = cast([] u8) "preview"};
|
||||
m.original_name = "photo.jpg";
|
||||
m.original = cast([] u8) "the photo";
|
||||
mcap_write(path, mcap_serialize(m,, temp));
|
||||
s: Mcap_Segments;
|
||||
s.image = "original";
|
||||
s.image_width, s.image_height = 8, 6;
|
||||
s.segments.allocator = temp;
|
||||
array_add(*s.segments, .{id = 3, score = 0.9, area = 0.25, class = "dog"});
|
||||
array_add(*s.segments, .{id = 1, score = 0.8, area = 0.2, class = "person"});
|
||||
mcap_store(path, front = .[mcap_segments_chunk(SEGMENT_MODEL, s)], back = .[mcap_mask_chunk(SEGMENT_MODEL, 4, 3, map)]);
|
||||
read := segment_overlay_read(path);
|
||||
expect(!read.error && read.width == 4 && read.height == 3 && cast(string) read.map == cast(string) cast([] u8) map, "the mask read back: %", read.error);
|
||||
expect(read.segments.count == 2 && read.by_id[3] == 0 && read.by_id[1] == 1 && read.by_id[2] == -1, "segments found by id");
|
||||
segment_overlay_free(read);
|
||||
bare := tprint("%/bare.mcap", folder);
|
||||
mcap_write(bare, mcap_serialize(m,, temp));
|
||||
none := segment_overlay_read(bare);
|
||||
expect(none.error.count > 0, "a file not segmented says so: %", none.error);
|
||||
segment_overlay_free(none);
|
||||
file_delete(path);
|
||||
file_delete(bare);
|
||||
}
|
||||
|
||||
// What mcap_chunks_size counts, here where it cannot be seen.
|
||||
mcap_chunks_size_for_test :: (chunks: [] Mcap_Chunk) -> s64 {
|
||||
size: s64 = 4;
|
||||
@ -543,5 +622,6 @@ run_import_tests :: (builder: *String_Builder = null) -> bool {
|
||||
mcap_tests();
|
||||
segment_chunk_tests();
|
||||
library_folders_tests();
|
||||
segment_overlay_tests();
|
||||
return report();
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user