MusaMahmood/mcap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

[0.0.8] Import: any folder of screenshots and photos into the library, a folder for each source

Browse Source 
- Import window (and -import SOURCE DEST): every image under a folder (PNG, JPEG, HEIC, AVIF, WebP,
  GIF, BMP, TIFF, camera RAW) becomes an .mcap holding the file as it was, a preview turned by its
  EXIF orientation, and metadata.json and import.json Capture Text (date from EXIF, the file name or
  its time; file name as title; camera as app). .snagx files become the same .mcap SnagitConverter
  makes. Duplicates across the whole library are skipped by a hash index (.import-index); videos
  and sidecars are skipped; source files are only read.
- modules/Image_Decode: stb_image, then WIC for everything else; EXIF; orientation; previews.
  tools/image_probe.jai shows what it reads.
- modules/Snagx (and modules/jaison), copied from SnagitConverter.
- The library is a folder of folders: recursive scan, a folder filter, and a folder: search prefix.
- OCR, the viewer and ocr-bench decode any image format, turned the right way up.
- Rename D:\SnagitIndex to D:\McapIndex.
This commit is contained in:
Musa Mahmood 2026-10-08 22:56:54 +01:00
parent c333d95da7
commit 811c096dd0
31 changed files with 4689 additions and 73 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@ -1,13 +1,14 @@
# mcap # mcap
A browser for a library of screen captures: every `.mcap` file in a folder as a grid of thumbnails, A browser for a library of screenshots and photos: every `.mcap` file in a folder (and its folders)
searchable by the text in the images (OCR), by what they show (labels), by their window title, URL, as a grid of thumbnails, searchable by the text in the images (OCR), by what they show (labels), by
app and annotations, and by meaning ("a circuit board"). Written in [Jai](https://jai.community) their window title or file name, URL, app or camera and annotations, and by meaning ("a circuit board"). Written in [Jai](https://jai.community)
with [Dear ImGui](https://github.com/ocornut/imgui), for Windows (macOS and Linux build, but OCR with [Dear ImGui](https://github.com/ocornut/imgui), for Windows (macOS and Linux build, but OCR
and labels are tested on Windows with an NVIDIA GPU). and labels are tested on Windows with an NVIDIA GPU).
The `.mcap` files come from Snagit's `.snagx` captures, converted by `snagit-converter` The `.mcap` files come from the **Import** window, which takes any folder of images (PNG, JPEG, HEIC,
(`D:\SnagitIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format. AVIF, WebP, GIF, BMP, TIFF, camera RAW) and Snagit `.snagx` captures, or from `snagit-converter`
(`D:\McapIndex\SnagitConverter`); `modules/Capture_File/README.md` describes the format.
## Getting started ## Getting started
@ -27,7 +28,9 @@ The `.mcap` files come from Snagit's `.snagx` captures, converted by `snagit-con
3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime 3. **Build and run:** `jai first.jai - release`, then `bin\mcap.exe`. The build copies onnxruntime
and the models beside the program. and the models beside the program.
4. **The library** is `D:\SnagitIndex\mcap`; `-library FOLDER` shows another folder. 4. **The library** is `D:\McapIndex\mcap`, a folder for each place imported from
(`SnagitLibraryPersonal`, `SnagitLibraryWork`, ...); `-library FOLDER` shows another folder.
**Import** (button above the grid) brings in more; see [Importing](#importing).
5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**, 5. **Read the library:** open the **OCR** window (button above the grid), check **Labels too**,
and press **Scan the library**. Every capture is read and labelled once and the results are and press **Scan the library**. Every capture is read and labelled once and the results are
@ -108,6 +111,40 @@ model: captures with text, lines, words, characters, confidence, words per captu
a table by year, the captures with the most words (click to open), each capture's best label a table by year, the captures with the most words (click to open), each capture's best label
counted (click one to search for it), and the commonest words (click one to search for it). counted (click one to search for it), and the commonest words (click one to search for it).
## Importing
**Import** (above the grid) brings a folder of screenshots and photos into the library:
1. **Import from:** the folder, typed or chosen with **Browse...**. Everything under it is looked
at, its folders too (not hidden ones, such as `.thumbnails`).
2. **Into library folder:** the folder in the library it goes into, such as `SnagitLibraryWork` or
`Phone`. It starts as the name of the folder imported from, or the library folder that folder went
into last time, so importing the same place again adds to the same folder.
3. **Import.** The progress, the time left, and Pause and Cancel show while it runs; new captures
appear in the grid as they are written. With **Read text and labels** checked, the OCR scan then
reads and labels what came in.
What becomes of each file:
- **Images** (PNG, JPEG, HEIC and AVIF, WebP, GIF, BMP, PSD, TGA, TIFF, and camera RAW such as DNG,
CR2, CR3, NEF and ARW): an `.mcap` holding the file exactly as it was, a preview turned the right
way up (by its EXIF orientation), and where it came from: its file name (searched by `title:`), the
camera or phone (`app:`), and the date it was taken, from its EXIF, else a date in its name
(`IMG_20240512_101530`, `Screenshot 2024-05-12 at 10.15.30`), else the file's time. It is named
`2024-05-12_10-15-30_1a2b3c4d.mcap`, the date and the start of its hash. HEIC, AVIF, WebP and RAW
are read by Windows' own codecs: the HEIF, AV1, WebP and Raw Image Extensions from the Microsoft
Store.
- **Snagit `.snagx`:** the same `.mcap` as `snagit-converter mcap` makes, byte for byte, named as
the `.snagx` is.
- **Duplicates:** an image already in the library, in any of its folders, is not stored again; the
Import window lists each with where it is. Importing a folder a second time imports only what is
new.
- **Skipped:** videos (`.mp4`, `.mov`, ...), sidecar files (`.aae`, `.xmp`, `.json`) and anything
that is not an image, counted by kind.
- **Failed:** files that cannot be read or decoded, listed with why.
The files imported from are only read: nothing there is changed, moved or deleted.
## Searching ## Searching
Type in the search box above the grid; the grid shows the matches as you type, with "N of M Type in the search box above the grid; the grid shows the matches as you type, with "N of M
@ -131,12 +168,13 @@ A prefix limits a word, or a quoted phrase, to one field: `title:kicad`, `title:
| --- | --- | | --- | --- |
| *(none)* | everything below except `like:` | | *(none)* | everything below except `like:` |
| `ocr:` | the text read from the image | | `ocr:` | the text read from the image |
| `title:` | the window title when it was captured | | `title:` | the window title when it was captured; for an imported image, its file name |
| `url:` | the web page's address, for browser captures | | `url:` | the web page's address, for browser captures |
| `app:` | the app's name and version | | `app:` | the app's name and version; for a photo, the camera or phone |
| `note:` | the text of the capture's annotations (Snagit text boxes and callouts) | | `note:` | the text of the capture's annotations (Snagit text boxes and callouts) |
| `label:` | the capture's labels: its best label, and any scoring at least half as much | | `label:` | the capture's labels: its best label, and any scoring at least half as much |
| `like:` | the image's meaning (below) | | `like:` | the image's meaning (below) |
| `folder:` | the library folder it is in: `folder:work` |
A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work. A word with a colon that is not one of these stays an ordinary word, so `http://` and `C:\` work.
@ -171,7 +209,8 @@ then in that order instead of by date.
| Option | What it does | | Option | What it does |
| --- | --- | | --- | --- |
| `-library FOLDER` | Shows the `.mcap` files in FOLDER instead of `D:\SnagitIndex\mcap`. | | `-library FOLDER` | Shows the `.mcap` files in FOLDER (and its folders) instead of `D:\McapIndex\mcap`. |
| `-import SOURCE DEST` | Imports the folder SOURCE into the library folder DEST, as the Import window does, then exits: 0, or 1 if any file failed. |
| `-search QUERY` | Starts with QUERY in the search box. | | `-search QUERY` | Starts with QUERY in the search box. |
| `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. | | `-scan` | Scans the library (the chosen OCR model, and labels when the label model is up) once it has loaded, then exits: 0, or 1 if any capture failed. |
| `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. | | `-stats` | Logs the Stats window's numbers once the catalog has loaded, then exits; with `-search`, for the matches, and the first ten are logged. |
@ -181,7 +220,7 @@ then in that order instead of by date.
| `-headless` | No message boxes: a failure is logged and the app exits with 1. | | `-headless` | No message boxes: a failure is logged and the app exits with 1. |
| `-version` | Prints the version. | | `-version` | Prints the version. |
`-scan`, `-stats` and `-erase` still open the window, to show their progress; they are meant for `-import`, `-scan`, `-stats` and `-erase` still open the window, to show their progress; they are meant for
scripts and for trying things on a copy of the library: `bin\mcap.exe -library D:\copy -scan`. scripts and for trying things on a copy of the library: `bin\mcap.exe -library D:\copy -scan`.
## Where things are kept ## Where things are kept
@ -192,8 +231,11 @@ scripts and for trying things on a copy of the library: `bin\mcap.exe -library D
| The label list | `%APPDATA%\mcap\labels.config` | | The label list | `%APPDATA%\mcap\labels.config` |
| Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` | | Settings (theme, hotkeys, OCR model) | `%APPDATA%\mcap\settings.config` |
| Logs | `logs\<version>\` beside the program | | Logs | `logs\<version>\` beside the program |
| The hash of every capture's image, for finding duplicates | `.import-index` in the library folder |
| The folder each library folder was imported from | `.import-sources` in the library folder |
Nothing is written to `D:\Snagit`: the `.mcap` library is a converted copy. Nothing is written to the folders imported from (`D:\Snagit`, a phone's backup): the library is a
converted copy.
## How it works ## How it works
@ -212,6 +254,14 @@ Nothing is written to `D:\Snagit`: the `.mcap` library is a converted copy.
rewrite of its file; the erase; the windows. rewrite of its file; the erase; the windows.
- `modules/OCR`, `modules/Labels`: the models, run with onnxruntime (`modules/onnxruntime`, which - `modules/OCR`, `modules/Labels`: the models, run with onnxruntime (`modules/onnxruntime`, which
also holds the shared session and CUDA code); each has a README. also holds the shared session and CUDA code); each has a README.
- `src/app/import.jai`: the Import window and `-import`. A thread lists the files, brings the
library index up to date (hashing the image in any `.mcap` it has not seen), then imports on
several threads; captures written are handed to the grid each frame.
- `modules/Image_Decode`: any image file's bytes to pixels, the right way up: stb_image for PNG,
JPEG, GIF, BMP, PSD and TGA, Windows Imaging Component for the rest; EXIF (from JPEG, PNG, WebP,
HEIF and TIFF); previews. `bin\image-probe FILE...` (`jai first.jai - image_probe`) shows what it
makes of files.
- `modules/Snagx`: `.snagx` to `.mcap`, copied from SnagitConverter.
- `modules/Capture_File`: reading and writing `.mcap` files. - `modules/Capture_File`: reading and writing `.mcap` files.
# The application template # The application template

2
check_targets.jai
View File

@ -29,7 +29,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap";
GIT_BRANCH :: "check"; GIT_BRANCH :: "check";
GIT_HASH :: "-"; GIT_HASH :: "-";
VERSION :: "0.0.7"; VERSION :: "0.0.8";
BUILD_DATE :: "-"; BUILD_DATE :: "-";
DEBUG :: true; DEBUG :: true;
COMPILE_TESTS :: true; COMPILE_TESTS :: true;

6
first.jai
View File

@ -8,7 +8,7 @@ APP_NAME :: "mcap";
APP_FOLDER_NAME :: "mcap"; APP_FOLDER_NAME :: "mcap";
APP_BUNDLE_IDENTIFIER :: "com.example.mcap"; APP_BUNDLE_IDENTIFIER :: "com.example.mcap";
VERSION :: "0.0.7"; VERSION :: "0.0.8";
// Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated // Where everything goes: bin/, or bin-headless/ for builds made with no_crash_dialog, so automated
// builds never write over the ones you run by hand. // builds never write over the ones you run by hand.
build_directory := "bin"; build_directory := "bin";
@ -63,6 +63,8 @@ build_project :: () {
if build_release array_add(*pending, start_main_build(optimized = true)); if build_release array_add(*pending, start_main_build(optimized = true));
// jai first.jai - ocr_bench: also bin/ocr-bench, OCR timing and output over a folder of .mcap files. // jai first.jai - ocr_bench: also bin/ocr-bench, OCR timing and output over a folder of .mcap files.
if array_find(args, "ocr_bench") array_add(*pending, start_tool_build("ocr-bench", "tools/ocr_bench.jai", optimized = build_release)); if array_find(args, "ocr_bench") array_add(*pending, start_tool_build("ocr-bench", "tools/ocr_bench.jai", optimized = build_release));
// jai first.jai - image_probe: also bin/image-probe, what modules/Image_Decode reads from image files.
if array_find(args, "image_probe") array_add(*pending, start_tool_build("image-probe", "tools/image_probe.jai", optimized = build_release));
wait_for_builds(pending); wait_for_builds(pending);
@ -125,7 +127,7 @@ start_main_build :: (optimized: bool) -> Pending_Build {
file_delete(tprint("%/%.rdi", build_directory, options.output_executable_name)); file_delete(tprint("%/%.rdi", build_directory, options.output_executable_name));
} }
options.temporary_storage_size = 64 * 4096; // 256kB (could be more i guess?) options.temporary_storage_size = 64 * 4096;
options.output_path = build_directory; options.output_path = build_directory;
w := compiler_create_workspace(options.output_executable_name); w := compiler_create_workspace(options.output_executable_name);

43
modules/Capture_File/README.md
View File

@ -3,7 +3,10 @@
One capture in one file: a fixed magic, a file version stamp, the capture's data as One capture in one file: a fixed magic, a file version stamp, the capture's data as
[Capture Text](capture_text.md), what OCR read in it, what other models added (chunks), a small preview, [Capture Text](capture_text.md), what OCR read in it, what other models added (chunks), a small preview,
then the images, then any large chunks. There is no ZIP and no compression around it; the images keep then the images, then any large chunks. There is no ZIP and no compression around it; the images keep
their own PNG compression. their own compression.
A capture is a Snagit capture (from a `.snagx`) or an imported image: a photo or screenshot in any image
format (the mcap app's Import window).
The parts a library browser needs (header, text, OCR, front chunks and preview) come first, so it reads The parts a library browser needs (header, text, OCR, front chunks and preview) come first, so it reads
the start of each file and never touches the images; a search, which needs no preview, reads less still. the start of each file and never touches the images; a search, which needs no preview, reads less still.
@ -21,8 +24,8 @@ All integers are little-endian. Sizes count bytes.
| front chunks | chunk list | Small things models added, such as labels and an image vector (below). | | front chunks | chunk list | Small things models added, such as labels and an image vector (below). |
| preview width, height | u16, u16 | In pixels. | | preview width, height | u16, u16 | In pixels. |
| preview size, preview | u32, bytes | PNG. Size 0 when there is none. | | preview size, preview | u32, bytes | PNG. Size 0 when there is none. |
| original name size, name | u16, bytes | The original image's name in the .snagx, such as `{GUID}.png`; the page's `CaptureBackgroundImage` names it too. | | original name size, name | u16, bytes | The original image's name: in the .snagx, such as `{GUID}.png` (the page's `CaptureBackgroundImage` names it too), or an imported image's file name, such as `IMG_0001.HEIC`. |
| original size, original | u32, bytes | The capture as taken: PNG, exactly as it was in the .snagx. | | original size, original | u32, bytes | The capture as taken, exactly as it was: PNG from a .snagx; an imported image in its own format (JPEG, HEIC, WebP, RAW ...), told by its first bytes, not its name. |
| rendered kind | u8 | 0: none. 1: the same as the original (stored once). 2: its own, below. | | rendered kind | u8 | 0: none. 1: the same as the original (stored once). 2: its own, below. |
| rendered size, rendered | u32, bytes | Only for kind 2. The capture with its annotations drawn in (Snagit's `thumbnail.png`): PNG, exactly as it was. | | rendered size, rendered | u32, bytes | Only for kind 2. The capture with its annotations drawn in (Snagit's `thumbnail.png`): PNG, exactly as it was. |
| back chunks | chunk list | Large things models added, such as segmentation masks: after the images, so reading the front never reaches them. | | back chunks | chunk list | Large things models added, such as segmentation masks: after the images, so reading the front never reaches them. |
@ -43,7 +46,7 @@ the first time something is stored in it.
## The parts ## The parts
- **Text.** Everything the capture's JSON held: `index.json`, `metadata.json`, the page's `{GUID}.json`, - **Text.** Everything the capture's JSON held: `index.json`, `metadata.json`, the page's `{GUID}.json`,
and `{GUID}.backup.json` when there is one. and `{GUID}.backup.json` when there is one. An imported image has two files instead (below).
- **OCR.** One section per OCR model, so a capture read by several models keeps every reading; - **OCR.** One section per OCR model, so a capture read by several models keeps every reading;
reading it again with a model replaces only that model's section. See *OCR text* below. reading it again with a model replaces only that model's section. See *OCR text* below.
- **Chunks.** What other models add, each keyed by the model's name, so models do not overwrite each - **Chunks.** What other models add, each keyed by the model's name, so models do not overwrite each
@ -51,13 +54,41 @@ the first time something is stored in it.
know is kept as it is when the file is rewritten, so a later build can add kinds without a new file know is kept as it is when the file is rewritten, so a later build can add kinds without a new file
version. See *Label chunks* below. version. See *Label chunks* below.
- **Preview.** The rendered image (or the original, when there is no rendered one) scaled so its longest - **Preview.** The rendered image (or the original, when there is no rendered one) scaled so its longest
side is at most 320 pixels, as a PNG. An image already that small is used as it is, never scaled up. side is at most 320 pixels, as a PNG. A Snagit PNG already that small is used as it is, never scaled
up. An imported image's preview is always a PNG made from it, turned the right way up by its EXIF
orientation, which a reader applies to the original too.
- **Original and rendered.** A capture with no annotations has a rendered image identical to the original, - **Original and rendered.** A capture with no annotations has a rendered image identical to the original,
byte for byte; it is then stored once, as kind 1. byte for byte; it is then stored once, as kind 1.
- **One page.** An .mcap holds one page, as every capture so far does; a .snagx with more is not converted. - **One page.** An .mcap holds one page, as every capture so far does; a .snagx with more is not converted.
- **Left out:** `{GUID}.backup.png`, Snagit's image from before an edit. Its `{GUID}.backup.json` is kept, - **Left out:** `{GUID}.backup.png`, Snagit's image from before an edit. Its `{GUID}.backup.json` is kept,
as plenty of captures have one without the image. as plenty of captures have one without the image.
## Imported images
An image imported from a folder has no rendered image (kind 0), and its Capture Text holds two files:
```
[[file "metadata.json"]]
AppName: "Apple iPhone 15 Pro" the camera or phone, from EXIF; "" when unknown
CaptureDate: "2024-05-12 10:15:30" when it was taken, local time, as Snagit writes it
WindowName: "IMG_4021.HEIC" the file's name
[[file "import.json"]]
SourcePath: "E:/Phone/DCIM/IMG_4021.HEIC" where it was imported from
SourceSize: 2994394
SourceModified: "2024-05-12 10:15:31"
ContentHash: "d3fa07db58597cec" 64-bit FNV-1a of the original's bytes
Format: "HEIF"
Width: 3024 as shown: after the orientation is applied
Height: 4032
DateFrom: "EXIF" or "file name", or "modified time"
ImportDate: "2026-10-08 22:28:17"
Orientation: 6 and, when the EXIF has them: CameraMake, CameraModel,
Software, TimeZoneOffset, Latitude, Longitude, Altitude
```
So the parts a library reads from Snagit's `metadata.json` (date, title, app) work for photos too.
## Versions ## Versions
A reader checks the magic, then the file version. It refuses a version above its own, never writing A reader checks the magic, then the file version. It refuses a version above its own, never writing
@ -67,6 +98,8 @@ over a file from a later version, and reads every version up to its own. Version
`snagit-converter mcap` converts .snagx files (or extracted captures) to .mcap, and `snagit-converter `snagit-converter mcap` converts .snagx files (or extracted captures) to .mcap, and `snagit-converter
snagx` converts .mcap files back to a .snagx Snagit can open. See the README for what each checks. snagx` converts .mcap files back to a .snagx Snagit can open. See the README for what each checks.
The mcap app's Import makes the same .mcap from a .snagx (modules/Snagx). Only captures from a .snagx
convert back: an imported image has no Snagit JSON to make one from.
Back in a .snagx, the JSON is laid out as Snagit writes it, the images are the same bytes, and only the Back in a .snagx, the JSON is laid out as Snagit writes it, the images are the same bytes, and only the
backup image is missing. JSON numbers come back as values: `3.0` is written `3`. backup image is missing. JSON numbers come back as values: `3.0` is written `3`.

369
modules/Image_Decode/exif.jai Normal file
View File

@ -0,0 +1,369 @@
// EXIF: where a photo was taken, when, on what, and which way up. Found in a JPEG's APP1 segment, a
// PNG's eXIf chunk, a WebP's EXIF chunk, a HEIF or AVIF's Exif item, and as the whole of a TIFF or a
// TIFF-based RAW file; read as TIFF image file directories.
Exif :: struct {
found: bool;
orientation: s64 = 1; // 1-8; 1 is upright
date_original: string; // DateTimeOriginal, "2024:05:12 10:15:30"; else DateTimeDigitized, else DateTime
offset_original: string; // OffsetTimeOriginal, "+02:00"; often missing
make, model, software: string;
has_gps: bool;
latitude, longitude: float64; // degrees, south and west negative
altitude: float64; // metres; 0 when missing
}
// What bytes' EXIF says; the strings are views into bytes. kind may be passed when known.
exif_read :: (bytes: [] u8, kind := Image_Kind.UNKNOWN) -> Exif {
if kind == .UNKNOWN kind = image_kind(bytes);
return exif_parse_tiff(exif_find(bytes, kind));
}
// The TIFF block holding the EXIF, or empty.
exif_find :: (bytes: [] u8, kind: Image_Kind) -> [] u8 {
if kind == {
case .JPEG;
// Segments: FF, marker, a big-endian length counting itself; the image data starts at SOS.
at := 2;
while at + 4 <= bytes.count {
if bytes[at] != 0xFF return .[];
marker := bytes[at + 1];
if marker == 0xFF { at += 1; continue; } // fill byte
if marker == 0xDA || marker == 0xD9 return .[];
if marker == 0x01 || (marker >= 0xD0 && marker <= 0xD7) { at += 2; continue; } // no length
length := cast(s64) be16(bytes, at + 2);
if length < 2 return .[];
if marker == 0xE1 && length >= 8 && at + 10 <= bytes.count && memcmp(bytes.data + at + 4, "Exif\0\0".data, 6) == 0 return slice_bytes(bytes, at + 10, length - 8);
at += 2 + length;
}
case .PNG;
at := 8;
while at + 12 <= bytes.count {
length := cast(s64) be32(bytes, at);
type := cast(string) slice_bytes(bytes, at + 4, 4);
if type == "eXIf" return slice_bytes(bytes, at + 8, length);
if type == "IDAT" || type == "IEND" return .[]; // eXIf comes before the image data
at += 12 + length;
}
case .WEBP;
at := 12;
while at + 8 <= bytes.count {
type := cast(string) slice_bytes(bytes, at, 4);
length := cast(s64) le32(bytes, at + 4);
if type == "EXIF" {
data := slice_bytes(bytes, at + 8, length);
if data.count >= 6 && memcmp(data.data, "Exif\0\0".data, 6) == 0 data = slice_bytes(data, 6, data.count - 6); // some writers keep the JPEG prefix
return data;
}
at += 8 + length + (length & 1);
}
case .TIFF; #through;
case .RAW;
if bytes.count >= 8 && (bytes[0] == #char "I" || bytes[0] == #char "M") return bytes;
case .HEIF; #through;
case .AVIF;
return heif_info(bytes).exif;
}
return .[];
}
exif_parse_tiff :: (tiff: [] u8) -> Exif {
exif: Exif;
if tiff.count < 8 return exif;
little: bool;
if tiff[0] == #char "I" && tiff[1] == #char "I" little = true;
else if tiff[0] == #char "M" && tiff[1] == #char "M" little = false;
else return exif;
r := Tiff_Reader.{data = tiff, little = little};
if tiff_u16(*r, 2) != 42 return exif;
exif.found = true;
date_time, date_digitized: string;
exif_ifd, gps_ifd: s64;
for tiff_entries(*r, tiff_u32(*r, 4)) {
if it.tag == {
case 0x010F; exif.make = tiff_text(*r, it);
case 0x0110; exif.model = tiff_text(*r, it);
case 0x0112; exif.orientation = tiff_number(*r, it, 0);
case 0x0131; exif.software = tiff_text(*r, it);
case 0x0132; date_time = tiff_text(*r, it);
case 0x8769; exif_ifd = tiff_number(*r, it, 0);
case 0x8825; gps_ifd = tiff_number(*r, it, 0);
}
}
if exif_ifd {
for tiff_entries(*r, exif_ifd) {
if it.tag == {
case 0x9003; exif.date_original = tiff_text(*r, it);
case 0x9004; date_digitized = tiff_text(*r, it);
case 0x9011; exif.offset_original = tiff_text(*r, it);
}
}
}
if !exif.date_original exif.date_original = ifx date_digitized then date_digitized else date_time;
if exif.orientation < 1 || exif.orientation > 8 exif.orientation = 1;
if gps_ifd {
lat_ref, lon_ref: string;
lat, lon: [3] float64;
has_lat, has_lon: bool;
altitude_below := false;
for e: tiff_entries(*r, gps_ifd) {
if e.tag == {
case 1; lat_ref = tiff_text(*r, e);
case 2; if e.count >= 3 { for 0..2 lat[it] = tiff_rational(*r, e, it); has_lat = true; }
case 3; lon_ref = tiff_text(*r, e);
case 4; if e.count >= 3 { for 0..2 lon[it] = tiff_rational(*r, e, it); has_lon = true; }
case 5; altitude_below = tiff_number(*r, e, 0) == 1;
case 6; exif.altitude = tiff_rational(*r, e, 0);
}
}
if has_lat && has_lon {
exif.latitude = lat[0] + lat[1] / 60 + lat[2] / 3600;
exif.longitude = lon[0] + lon[1] / 60 + lon[2] / 3600;
if lat_ref == "S" exif.latitude = -exif.latitude;
if lon_ref == "W" exif.longitude = -exif.longitude;
if altitude_below exif.altitude = -exif.altitude;
// 0, 0 is what some phones write with no fix.
exif.has_gps = (exif.latitude != 0 || exif.longitude != 0) && abs(exif.latitude) <= 90 && abs(exif.longitude) <= 180;
}
}
return exif;
}
// HEIF and AVIF: the primary item's size, rotation and mirroring, and its Exif item.
Heif_Info :: struct {
width, height: s64; // ispe: the coded size, before any rotation
rotation: s64; // irot, degrees anticlockwise: 0, 90, 180 or 270
mirror: s64 = -1; // imir's axis, -1 for none
exif: [] u8; // the TIFF block, a view into the bytes
}
heif_info :: (bytes: [] u8) -> Heif_Info {
info: Heif_Info;
meta := box_find(bytes, "meta");
if meta.count < 4 return info;
meta = slice_bytes(meta, 4, meta.count - 4); // a full box: version and flags first
// The primary item.
primary: u32;
pitm := box_find(meta, "pitm");
if pitm.count >= 6 primary = ifx pitm[0] == 0 then be16(pitm, 4) else be32(pitm, 4);
// The Exif item's ID, from the item infos.
exif_id: u32;
iinf := box_find(meta, "iinf");
if iinf.count >= 6 {
at := ifx iinf[0] == 0 then 6 else 8;
while at + 8 <= iinf.count {
size := cast(s64) be32(iinf, at);
if size < 8 || at + size > iinf.count break;
if cast(string) slice_bytes(iinf, at + 4, 4) == "infe" {
infe := slice_bytes(iinf, at + 8, size - 8);
version := infe[0];
if version >= 2 {
id: u32;
type_at: s64;
if version == 2 { id = be16(infe, 4); type_at = 8; }
else { id = be32(infe, 4); type_at = 10; }
if cast(string) slice_bytes(infe, type_at, 4) == "Exif" exif_id = id;
}
}
at += size;
}
}
// Where the Exif item's bytes are, from the item locations.
iloc := box_find(meta, "iloc");
if exif_id && iloc.count >= 8 {
version := iloc[0];
offset_size := iloc[4] >> 4;
length_size := iloc[4] & 0xF;
base_offset_size := iloc[5] >> 4;
index_size := ifx version == 1 || version == 2 then iloc[5] & 0xF else 0;
at: s64 = 6;
count: s64;
if version < 2 { count = be16(iloc, at); at += 2; }
else { count = be32(iloc, at); at += 4; }
sized :: (b: [] u8, at: *s64, size: u8) -> u64 {
value: u64;
if size == 4 value = be32(b, at.*);
else if size == 8 value = be64(b, at.*);
else if size == 2 value = be16(b, at.*);
at.* += size;
return value;
}
for 1..count {
if at >= iloc.count break;
id: u32;
if version < 2 { id = be16(iloc, at); at += 2; }
else { id = be32(iloc, at); at += 4; }
method: u16;
if version == 1 || version == 2 { method = be16(iloc, at) & 0xF; at += 2; }
at += 2; // data reference index
base := sized(iloc, *at, base_offset_size);
extents := be16(iloc, at);
at += 2;
first_offset, first_length: u64;
for e: 0..cast(s64) extents - 1 {
if index_size sized(iloc, *at, index_size);
offset := sized(iloc, *at, offset_size);
length := sized(iloc, *at, length_size);
if e == 0 { first_offset = offset; first_length = length; }
}
if id != exif_id || method != 0 continue; // only items in the file itself, in one piece
data := slice_bytes(bytes, cast(s64) (base + first_offset), cast(s64) first_length);
// An Exif item starts with where the TIFF header is, past itself.
if data.count >= 4 {
skip := cast(s64) be32(data, 0);
info.exif = slice_bytes(data, 4 + skip, data.count - 4 - skip);
}
}
}
// The primary item's properties: ispe, irot and imir, through ipma's list of each item's.
iprp := box_find(meta, "iprp");
ipco := box_find(iprp, "ipco");
ipma := box_find(iprp, "ipma");
if primary && ipco.count && ipma.count >= 8 {
properties: [..] [] u8; // the boxes in ipco, each with its type at 4..8
properties.allocator = temp;
at := 0;
while at + 8 <= ipco.count {
size := cast(s64) be32(ipco, at);
if size < 8 || at + size > ipco.count break;
array_add(*properties, slice_bytes(ipco, at, size));
at += size;
}
version := ipma[0];
wide_index := (ipma[3] & 1) != 0;
entries := be32(ipma, 4);
at = 8;
for 1..entries {
if at >= ipma.count break;
id: u32;
if version < 1 { id = be16(ipma, at); at += 2; }
else { id = be32(ipma, at); at += 4; }
associations := ipma[at];
at += 1;
for 1..associations {
index: s64;
if wide_index { index = be16(ipma, at) & 0x7FFF; at += 2; }
else { index = ipma[at] & 0x7F; at += 1; }
if id != primary || index < 1 || index > properties.count continue;
box := properties[index - 1];
type := cast(string) slice_bytes(box, 4, 4);
if type == "ispe" && box.count >= 20 { info.width = be32(box, 12); info.height = be32(box, 16); }
if type == "irot" && box.count >= 9 info.rotation = (box[8] & 3) * 90;
if type == "imir" && box.count >= 9 info.mirror = box[8] & 1;
}
}
}
return info;
}
#scope_file
// The contents of the first box of type in data (a run of boxes), past its 8-byte header; empty when
// there is none.
box_find :: (data: [] u8, type: string) -> [] u8 {
at := 0;
while at + 8 <= data.count {
size := cast(s64) be32(data, at);
header := 8;
if size == 1 { size = cast(s64) be64(data, at + 8); header = 16; } // a 64-bit size
else if size == 0 size = data.count - at; // to the end
if size < header || at + size > data.count return .[];
if cast(string) slice_bytes(data, at + 4, 4) == type return slice_bytes(data, at + header, size - header);
at += size;
}
return .[];
}
Tiff_Reader :: struct {
data: [] u8;
little: bool;
}
Tiff_Entry :: struct {
tag, type: u16;
count: u32;
value_at: s64; // where its value is: inside the entry when it fits in 4 bytes, else where the entry points
}
tiff_u16 :: (r: *Tiff_Reader, at: s64) -> u16 {
if at < 0 || at + 2 > r.data.count return 0;
a, b := cast(u16) r.data[at], cast(u16) r.data[at + 1];
return ifx r.little then a | (b << 8) else (a << 8) | b;
}
tiff_u32 :: (r: *Tiff_Reader, at: s64) -> u32 {
if at < 0 || at + 4 > r.data.count return 0;
a, b := cast(u32) tiff_u16(r, at), cast(u32) tiff_u16(r, at + 2);
return ifx r.little then a | (b << 16) else (a << 16) | b;
}
TIFF_TYPE_SIZES :: s64.[0, 1, 1, 2, 4, 8, 1, 1, 2, 4, 8, 4, 8];
// The entries of the directory at offset; none when it is out of the data or unreasonable.
tiff_entries :: (r: *Tiff_Reader, offset: s64) -> [] Tiff_Entry {
entries: [..] Tiff_Entry;
entries.allocator = temp;
if offset < 8 || offset + 2 > r.data.count return entries;
count := cast(s64) tiff_u16(r, offset);
if count > 1000 return entries;
for 0..count - 1 {
at := offset + 2 + it * 12;
if at + 12 > r.data.count break;
e: Tiff_Entry;
e.tag = tiff_u16(r, at);
e.type = tiff_u16(r, at + 2);
e.count = tiff_u32(r, at + 4);
if e.type == 0 || e.type >= TIFF_TYPE_SIZES.count continue;
size := TIFF_TYPE_SIZES[e.type] * e.count;
e.value_at = ifx size <= 4 then at + 8 else cast(s64) tiff_u32(r, at + 8);
if e.value_at + size > r.data.count continue;
array_add(*entries, e);
}
return entries;
}
// An ASCII value, without its trailing nulls and spaces.
tiff_text :: (r: *Tiff_Reader, e: Tiff_Entry) -> string {
if e.type != 2 && e.type != 7 return "";
text := cast(string) slice_bytes(r.data, e.value_at, e.count);
while text.count && (text[text.count - 1] == 0 || text[text.count - 1] == #char " ") text.count -= 1;
zero := find_index_from_left(text, #char "\0");
if zero >= 0 text.count = zero;
return text;
}
// The index-th of a BYTE, SHORT or LONG value.
tiff_number :: (r: *Tiff_Reader, e: Tiff_Entry, index: s64) -> s64 {
if index >= e.count return 0;
if e.type == {
case 1; return r.data[e.value_at + index];
case 3; return tiff_u16(r, e.value_at + index * 2);
case 4; return tiff_u32(r, e.value_at + index * 4);
case 9; return cast,no_check(s32) tiff_u32(r, e.value_at + index * 4);
}
return 0;
}
// The index-th of a RATIONAL value, as a float.
tiff_rational :: (r: *Tiff_Reader, e: Tiff_Entry, index: s64) -> float64 {
if index >= e.count return 0;
if e.type == 5 || e.type == 10 {
numerator := tiff_u32(r, e.value_at + index * 8);
denominator := tiff_u32(r, e.value_at + index * 8 + 4);
if !denominator return 0;
if e.type == 10 return (cast(float64) cast,no_check(s32) numerator) / (cast(float64) cast,no_check(s32) denominator);
return (cast(float64) numerator) / denominator;
}
return cast(float64) tiff_number(r, e, index);
}

267
modules/Image_Decode/module.jai Normal file
View File

@ -0,0 +1,267 @@
// Image_Decode: any image file's bytes to RGBA pixels, the right way up, and the PNG previews .mcap
// files hold. stb_image reads PNG, JPEG, GIF, BMP, PSD and TGA; everything else (HEIC, AVIF, WebP,
// TIFF, camera RAW) goes to Windows' own codecs through WIC, which read what the installed Store
// extensions (HEIF, WebP, Raw Image) can. A photo's EXIF orientation is applied, so a phone's portrait
// photo comes out standing up.
//
// What the bytes are comes from their first bytes (image_kind), never from a file name, except for
// the camera RAW formats that are TIFF inside: image_kind_from_name tells those apart.
//
// Thread-safe: every call stands alone, and WIC is set up and released on the calling thread.
Image_Kind :: enum u8 {
UNKNOWN;
PNG; JPEG; GIF; BMP; PSD; TGA; // stb_image
WEBP; HEIF; AVIF; TIFF; RAW; // WIC
VIDEO; // an MP4 or QuickTime file: not an image
}
// Pixels from image_decode: RGBA, 4 bytes a pixel, rows packed. Free with image_free.
Image :: struct {
pixels: *u8;
width, height: s32;
from_stb: bool; // pixels are stb_image's, freed with stbi_image_free; otherwise the default allocator's
}
image_free :: (image: *Image) {
if image.pixels {
if image.from_stb stbi_image_free(image.pixels);
else free(image.pixels,, context.default_allocator);
}
image.* = .{};
}
// What the bytes hold, from their first bytes. A TIFF may be a camera's RAW file; see image_kind_from_name.
image_kind :: (bytes: [] u8) -> Image_Kind {
starts :: (bytes: [] u8, at: s64, magic: string) -> bool {
if bytes.count < at + magic.count return false;
return memcmp(bytes.data + at, magic.data, magic.count) == 0;
}
if starts(bytes, 0, "\x89PNG\r\n\x1A\n") return .PNG;
if starts(bytes, 0, "\xFF\xD8\xFF") return .JPEG;
if starts(bytes, 0, "GIF87a") || starts(bytes, 0, "GIF89a") return .GIF;
if starts(bytes, 0, "BM") && bytes.count > 26 return .BMP;
if starts(bytes, 0, "8BPS") return .PSD;
if starts(bytes, 0, "RIFF") && starts(bytes, 8, "WEBP") return .WEBP;
if starts(bytes, 0, "II*\0") || starts(bytes, 0, "MM\0*") return .TIFF;
if starts(bytes, 0, "IIRO") || starts(bytes, 0, "IIU\0") || starts(bytes, 0, "FUJIFILMCCD-RAW") return .RAW; // Olympus, Panasonic, Fuji
if starts(bytes, 4, "ftyp") {
// ISO base media: HEIF and AVIF images, Canon's CR3, or a video, by the major brand and the
// compatible ones. AVIF first: an AVIF often names the general mif1 brand too.
size := be32(bytes, 0);
end := min(cast(s64) size, bytes.count);
if end < 16 return .UNKNOWN;
found: [#run enum_highest_value(Image_Kind) + 1] bool;
found[ftyp_brand_kind(slice_bytes(bytes, 8, 4))] = true;
at := 16;
while at + 4 <= end {
found[ftyp_brand_kind(slice_bytes(bytes, at, 4))] = true;
at += 4;
}
if found[Image_Kind.AVIF] return .AVIF;
if found[Image_Kind.HEIF] return .HEIF;
if found[Image_Kind.RAW] return .RAW;
return .VIDEO; // an ftyp with no image brand: an MP4, MOV, 3GP and the like
}
if starts(bytes, 4, "moov") || starts(bytes, 4, "mdat") || starts(bytes, 4, "wide") return .VIDEO; // old QuickTime
return .UNKNOWN;
}
// The kind for bytes that image_kind cannot place, or places as TIFF, from the file's name: camera RAW
// formats built on TIFF, and TGA, which has no magic of its own. kind is image_kind's answer.
image_kind_from_name :: (kind: Image_Kind, name: string) -> Image_Kind {
extension := to_lower_copy(path_extension(name),, temp);
if kind == .TIFF {
for RAW_TIFF_EXTENSIONS if extension == it return .RAW;
return .TIFF;
}
if kind == .UNKNOWN && extension == "tga" return .TGA;
return kind;
}
RAW_TIFF_EXTENSIONS :: string.["dng", "cr2", "nef", "nrw", "arw", "srf", "sr2", "pef", "erf", "kdc", "dcr", "mos", "3fr", "iiq", "srw", "x3f", "mrw"];
image_kind_name :: (kind: Image_Kind) -> string {
if kind == {
case .PNG; return "PNG";
case .JPEG; return "JPEG";
case .GIF; return "GIF";
case .BMP; return "BMP";
case .PSD; return "PSD";
case .TGA; return "TGA";
case .WEBP; return "WebP";
case .HEIF; return "HEIF";
case .AVIF; return "AVIF";
case .TIFF; return "TIFF";
case .RAW; return "RAW";
case .VIDEO; return "video";
}
return "unknown";
}
// The image in bytes as RGBA, turned and flipped as its EXIF orientation (or, for HEIF and AVIF, its
// container) says, unless orient is false. kind may be passed when known; image_kind is used otherwise.
// On failure, why.
image_decode :: (bytes: [] u8, kind := Image_Kind.UNKNOWN, orient := true) -> Image, ok: bool, error: string {
image: Image;
if !bytes.count return image, false, "it is empty";
if kind == .UNKNOWN kind = image_kind(bytes);
if kind == .VIDEO return image, false, "it is a video";
stb_kinds :: Image_Kind.[.PNG, .JPEG, .GIF, .BMP, .PSD, .TGA, .UNKNOWN];
decoded := false;
error: string;
if array_find(stb_kinds, kind) && bytes.count < S32_MAX {
w, h, channels: s32;
pixels := stbi_load_from_memory(bytes.data, cast(s32) bytes.count, *w, *h, *channels, 4);
if pixels && w > 0 && h > 0 {
image = .{pixels = pixels, width = w, height = h, from_stb = true};
decoded = true;
} else {
if pixels stbi_image_free(pixels);
error = to_string(stbi_failure_reason());
}
}
if !decoded {
#if OS == .WINDOWS {
wic_ok: bool;
wic_error: string;
image, wic_ok, wic_error = wic_decode(bytes);
if !wic_ok {
if error return image, false, tprint("% (stb_image: %)", wic_error, error);
return image, false, wic_error;
}
} else {
return image, false, ifx error then tprint("stb_image: %", error) else tprint("% images are only read on Windows", image_kind_name(kind));
}
}
if orient {
orientation := 1;
if kind == .HEIF || kind == .AVIF {
// The container's irot and imir are the orientation; the EXIF tag only repeats it. WIC applies
// them (HEIF requires a reader to); a quarter turn it left out shows as the sides not swapped.
heif := heif_info(bytes);
if heif.rotation == 90 || heif.rotation == 270 {
if heif.width && image.width == heif.width && image.height == heif.height {
orientation = ifx heif.rotation == 90 then 8 else 6; // irot turns anticlockwise
}
}
} else {
exif := exif_read(bytes, kind);
orientation = exif.orientation;
}
if orientation > 1 && orientation <= 8 image_orient(*image, orientation);
}
return image, true, "";
}
// Turns and flips the pixels as EXIF orientation 1-8 says, into new pixels from the default allocator.
image_orient :: (image: *Image, orientation: s64) {
if orientation <= 1 || orientation > 8 || !image.pixels return;
w, h := image.width, image.height;
swaps := orientation >= 5;
out_w := ifx swaps then h else w;
out_h := ifx swaps then w else h;
out := cast(*u32) alloc(cast(s64) w * h * 4,, context.default_allocator);
source := cast(*u32) image.pixels;
for dy: 0..out_h - 1 {
row := out + cast(s64) dy * out_w;
for dx: 0..out_w - 1 {
sx, sy: s32;
if orientation == {
case 2; sx, sy = w - 1 - dx, dy; // mirrored left to right
case 3; sx, sy = w - 1 - dx, h - 1 - dy; // half a turn
case 4; sx, sy = dx, h - 1 - dy; // mirrored top to bottom
case 5; sx, sy = dy, dx; // transposed
case 6; sx, sy = dy, h - 1 - dx; // stored a quarter turn anticlockwise: turned clockwise
case 7; sx, sy = w - 1 - dy, h - 1 - dx; // transversed
case 8; sx, sy = w - 1 - dy, dx; // stored a quarter turn clockwise: turned anticlockwise
}
row[dx] = source[cast(s64) sy * w + sx];
}
}
image_free(image);
image.* = .{pixels = cast(*u8) out, width = out_w, height = out_h};
}
// The preview an .mcap holds: the pixels scaled so the longest side is at most max_side, as a PNG
// (temp, or stb's when reused is set: see below). The width and height are the preview's.
image_preview_png :: (image: Image, max_side: s64) -> png: [] u8, width: s64, height: s64, ok: bool, error: string {
w, h := image.width, image.height;
pixels := image.pixels;
longest := max(w, h);
if longest > max_side {
scale := cast(float64) max_side / longest;
out_w := max(cast(s32) 1, cast(s32) (w * scale + 0.5));
out_h := max(cast(s32) 1, cast(s32) (h * scale + 0.5));
scaled := NewArray(cast(s64) out_w * out_h * 4, u8, initialized = false,, temp);
if !stbir_resize_uint8_srgb(pixels, w, h, 0, scaled.data, out_w, out_h, 0, 4, 3, 0) return .[], 0, 0, false, "its image cannot be scaled";
pixels = scaled.data;
w, h = out_w, out_h;
}
length: s32;
encoded := stbi_write_png_to_mem(pixels, w * 4, w, h, 4, *length);
if !encoded return .[], 0, 0, false, "the preview cannot be encoded";
// stb_image_write allocates with the C runtime's malloc, and stb_image frees with its free: both
// libraries are linked against the same runtime.
defer stbi_image_free(encoded);
return cast([] u8) copy_string(string.{length, encoded},, temp), w, h, true, "";
}
// The width and height a PNG says it has, from its header, without decoding it.
png_size :: (png: [] u8) -> width: s64, height: s64, ok: bool {
if png.count < 24 return 0, 0, false;
for u8.[0x89, #char "P", #char "N", #char "G", 0x0D, 0x0A, 0x1A, 0x0A] if png[it_index] != it return 0, 0, false;
return be32(png, 16), be32(png, 20), true;
}
#load "exif.jai";
#if OS == .WINDOWS #load "wic.jai";
#scope_module
be16 :: (b: [] u8, at: s64) -> u16 {
if at < 0 || at + 2 > b.count return 0;
return ((cast(u16) b[at]) << 8) | (cast(u16) b[at + 1]);
}
be32 :: (b: [] u8, at: s64) -> u32 {
if at < 0 || at + 4 > b.count return 0;
return ((cast(u32) b[at]) << 24) | ((cast(u32) b[at + 1]) << 16) | ((cast(u32) b[at + 2]) << 8) | (cast(u32) b[at + 3]);
}
be64 :: (b: [] u8, at: s64) -> u64 {
return ((cast(u64) be32(b, at)) << 32) | (cast(u64) be32(b, at + 4));
}
le32 :: (b: [] u8, at: s64) -> u32 {
if at < 0 || at + 4 > b.count return 0;
return (cast(u32) b[at]) | ((cast(u32) b[at + 1]) << 8) | ((cast(u32) b[at + 2]) << 16) | ((cast(u32) b[at + 3]) << 24);
}
// count bytes at at, cut short at the end; empty when at is past it.
slice_bytes :: (b: [] u8, at: s64, count: s64) -> [] u8 {
if at < 0 || at >= b.count || count <= 0 return .[];
result: [] u8;
result.data = b.data + at;
result.count = min(count, b.count - at);
return result;
}
ftyp_brand_kind :: (brand: [] u8) -> Image_Kind {
if brand.count != 4 return .UNKNOWN;
text := cast(string) brand;
if text == "avif" || text == "avis" return .AVIF;
if text == "heic" || text == "heix" || text == "hevc" || text == "hevx" || text == "heim" || text == "heis" || text == "mif1" || text == "msf1" return .HEIF;
if text == "crx " return .RAW; // Canon CR3
return .UNKNOWN;
}
#import "Basic";
#import "String";
#import "Math";
#import "stb_image";
#import "stb_image_write";
#import "stb_image_resize";
#if OS == .WINDOWS #import "Windows";

151
modules/Image_Decode/wic.jai Normal file
View File

@ -0,0 +1,151 @@
// Windows Imaging Component: the codecs Windows has, and those the Store extensions add (HEIF and
// AVIF, WebP, Raw Image). Only the few COM methods used are named; the rest of each vtable is padding.
// The first frame of the image in bytes, as non-premultiplied RGBA (default allocator).
wic_decode :: (bytes: [] u8) -> Image, ok: bool, error: string {
image: Image;
if bytes.count > U32_MAX return image, false, "it is too large for WIC";
// Any apartment will do for WIC; a thread already in one keeps it.
initialized := CoInitializeEx(null, COINIT_MULTITHREADED);
defer if initialized == S_OK || initialized == S_FALSE CoUninitialize();
factory: *IWICImagingFactory;
hr := CoCreateInstance(*CLSID_WICImagingFactory, null, CLSCTX_INPROC_SERVER, *IID_IWICImagingFactory, xx *factory);
if failed(hr) return image, false, hr_text("WIC could not start", hr);
defer IUnknown_Release(factory);
stream: *IWICStream;
hr = factory.CreateStream(factory, *stream);
if failed(hr) return image, false, hr_text("WIC could not make a stream", hr);
defer IUnknown_Release(stream);
hr = stream.InitializeFromMemory(stream, bytes.data, cast(u32) bytes.count);
if failed(hr) return image, false, hr_text("WIC could not read the bytes", hr);
decoder: *IWICBitmapDecoder;
hr = factory.CreateDecoderFromStream(factory, stream, null, WICDecodeMetadataCacheOnDemand, *decoder);
if failed(hr) {
if hr == WINCODEC_ERR_COMPONENTNOTFOUND return image, false, "Windows has no codec for it (for HEIC, AVIF, WebP or RAW, install the HEIF, AV1, WebP or Raw Image Extension from the Microsoft Store)";
return image, false, hr_text("WIC cannot read it", hr);
}
defer IUnknown_Release(decoder);
frame: *IWICBitmapFrameDecode;
hr = decoder.GetFrame(decoder, 0, *frame);
if failed(hr) return image, false, hr_text("WIC cannot read its first frame", hr);
defer IUnknown_Release(frame);
converter: *IWICFormatConverter;
hr = factory.CreateFormatConverter(factory, *converter);
if failed(hr) return image, false, hr_text("WIC could not make a converter", hr);
defer IUnknown_Release(converter);
hr = converter.Initialize(converter, frame, *GUID_WICPixelFormat32bppRGBA, WICBitmapDitherTypeNone, null, 0, WICBitmapPaletteTypeCustom);
if failed(hr) return image, false, hr_text("WIC cannot convert it to RGBA", hr);
w, h: u32;
hr = converter.GetSize(converter, *w, *h);
if failed(hr) || !w || !h return image, false, hr_text("WIC gives it no size", hr);
if cast(u64) w * h * 4 > U32_MAX return image, false, tprint("it is % x %, too large to decode in one piece", w, h);
size := cast(s64) w * h * 4;
pixels := cast(*u8) alloc(size,, context.default_allocator);
hr = converter.CopyPixels(converter, null, w * 4, cast(u32) size, pixels);
if failed(hr) {
free(pixels,, context.default_allocator);
return image, false, hr_text("WIC could not decode it", hr);
}
image = .{pixels = pixels, width = cast(s32) w, height = cast(s32) h};
return image, true, "";
}
#scope_file
failed :: (hr: HRESULT) -> bool { return hr < 0; }
hr_text :: (what: string, hr: HRESULT) -> string {
return tprint("% (0x%)", what, formatInt(cast,no_check(u32) hr, base = 16));
}
Ole32 :: #library,system "ole32";
CoInitializeEx :: (pvReserved: *void, dwCoInit: u32) -> HRESULT #foreign Ole32;
CoUninitialize :: () #foreign Ole32;
CoCreateInstance :: (rclsid: *GUID, pUnkOuter: *IUnknown, dwClsContext: u32, riid: *GUID, ppv: **void) -> HRESULT #foreign Ole32;
COINIT_MULTITHREADED :: 0;
CLSCTX_INPROC_SERVER :: 1;
S_OK :HRESULT: 0;
S_FALSE :HRESULT: 1;
WINCODEC_ERR_COMPONENTNOTFOUND :HRESULT: cast,no_check(s32) 0x88982F50;
WICDecodeMetadataCacheOnDemand :: 0;
WICBitmapDitherTypeNone :: 0;
WICBitmapPaletteTypeCustom :: 0;
CLSID_WICImagingFactory := #run uid("CACAF262-9370-4615-A13B-9F5539DA4C0A");
IID_IWICImagingFactory := #run uid("EC5EC8A9-C395-4314-9C77-54D7A935FF70");
GUID_WICPixelFormat32bppRGBA := #run uid("F5C7AD2D-6A8D-43DD-A7A8-A29935261AE9");
IWICImagingFactory :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IWICImagingFactory_VTable;
}
IWICImagingFactory_VTable :: struct {
#as using base: IUnknown_VTable;
CreateDecoderFromFilename: *void;
CreateDecoderFromStream: (this: *IWICImagingFactory, stream: *IWICStream, vendor: *GUID, options: u32, decoder: **IWICBitmapDecoder) -> HRESULT #cpp_method;
CreateDecoderFromFileHandle: *void;
CreateComponentInfo: *void;
CreateDecoder: *void;
CreateEncoder: *void;
CreatePalette: *void;
CreateFormatConverter: (this: *IWICImagingFactory, converter: **IWICFormatConverter) -> HRESULT #cpp_method;
CreateBitmapScaler: *void;
CreateBitmapClipper: *void;
CreateBitmapFlipRotator: *void;
CreateStream: (this: *IWICImagingFactory, stream: **IWICStream) -> HRESULT #cpp_method;
}
IWICStream :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IWICStream_VTable;
}
IWICStream_VTable :: struct {
#as using base: IUnknown_VTable;
Read, Write: *void; // ISequentialStream
Seek, SetSize, CopyTo, Commit, Revert, LockRegion, UnlockRegion, Stat, Clone: *void; // IStream
InitializeFromIStream: *void;
InitializeFromFilename: *void;
InitializeFromMemory: (this: *IWICStream, buffer: *u8, size: u32) -> HRESULT #cpp_method;
}
IWICBitmapDecoder :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IWICBitmapDecoder_VTable;
}
IWICBitmapDecoder_VTable :: struct {
#as using base: IUnknown_VTable;
QueryCapability, Initialize, GetContainerFormat, GetDecoderInfo, CopyPalette, GetMetadataQueryReader, GetPreview, GetColorContexts, GetThumbnail, GetFrameCount: *void;
GetFrame: (this: *IWICBitmapDecoder, index: u32, frame: **IWICBitmapFrameDecode) -> HRESULT #cpp_method;
}
// Only ever passed on, to the converter.
IWICBitmapFrameDecode :: struct {
#as unknown: IUnknown;
}
IWICFormatConverter :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IWICFormatConverter_VTable;
}
IWICFormatConverter_VTable :: struct {
#as using base: IUnknown_VTable;
// IWICBitmapSource
GetSize: (this: *IWICFormatConverter, width: *u32, height: *u32) -> HRESULT #cpp_method;
GetPixelFormat: *void;
GetResolution: *void;
CopyPalette: *void;
CopyPixels: (this: *IWICFormatConverter, rect: *void, stride: u32, size: u32, buffer: *u8) -> HRESULT #cpp_method;
// IWICFormatConverter
Initialize: (this: *IWICFormatConverter, source: *IWICBitmapFrameDecode, format: *GUID, dither: u32, palette: *void, alpha_threshold: float64, palette_type: u32) -> HRESULT #cpp_method;
CanConvert: *void;
}

2
modules/Serialization/module.jai
View File

@ -1,4 +1,4 @@
// Capture_File (copied from D:\SnagitIndex\SnagitConverter) imports "Serialization" for Serializer // Capture_File (copied from D:\McapIndex\SnagitConverter) imports "Serialization" for Serializer
// and Deserializer. In this app they are part of Basic (modules/Basic/serialization.jai), so this // and Deserializer. In this app they are part of Basic (modules/Basic/serialization.jai), so this
// module only passes Basic on. // module only passes Basic on.

326
modules/Snagx/module.jai Normal file
View File

@ -0,0 +1,326 @@
// Snagx: Snagit's .snagx captures (ZIP archives of JSON and PNG) as .mcap files, as SnagitConverter's
// `snagit-converter mcap` makes them, byte for byte: the JSON as Capture Text, the page image as the
// original, thumbnail.png as the rendered image, and a preview. Copied from
// D:\McapIndex\SnagitConverter (src/snagx.jai, src/main.jai, src/json_convert.jai), reading only.
Archive_Entry :: struct {
name: string;
data: string; // inflated
}
SNAGX_RENDERED_NAME :: "thumbnail.png";
// A .snagx's bytes as an .mcap (its strings temp), checked by reading the serialized bytes back,
// which are returned too. On failure, why.
snagx_to_mcap :: (data: string) -> m: Mcap, bytes: [] u8, ok: bool, error: string {
entries, error := snagx_read_entries_from_memory(data);
if error return .{}, .[], false, error;
m, bytes, ok, mcap_error := snagx_mcap_from_entries(entries);
return m, bytes, ok, mcap_error;
}
// The entries of a .snagx, in archive order. Temp. On failure, the entries are empty and the error says why.
snagx_read_entries :: (path: string) -> [..] Archive_Entry, error: string {
data, ok := read_entire_file(path, log_errors = false,, temp);
if !ok {
entries: [..] Archive_Entry;
return entries, "cannot be read";
}
entries, error := snagx_read_entries_from_memory(data);
return entries, error;
}
snagx_read_entries_from_memory :: (data: string) -> [..] Archive_Entry, error: string {
entries: [..] Archive_Entry;
entries.allocator = temp;
if data.count < EOCD_SIZE return entries, "is too short to be a ZIP archive";
// The end of central directory record is the last thing in the file, followed only by a comment
// of up to 65535 bytes.
tail_start := max(0, data.count - EOCD_SIZE - 65535);
eocd := -1;
i := data.count - EOCD_SIZE;
while i >= tail_start {
if zip_u32(data, i) == EOCD_SIGNATURE { eocd = i; break; }
i -= 1;
}
if eocd < 0 return entries, "is not a ZIP archive (no end of central directory)";
entry_count := zip_u16(data, eocd + 10);
cd_size := cast(s64) zip_u32(data, eocd + 12);
cd_offset := cast(s64) zip_u32(data, eocd + 16);
if entry_count == 0xFFFF || cd_offset == 0xFFFFFFFF return entries, "is a ZIP64 archive, which is not supported";
if cd_offset + cd_size > data.count return entries, "has a central directory past its end";
cd := slice(data, cd_offset, cd_size);
at := 0;
for 1..entry_count {
if at + CD_ENTRY_SIZE > cd.count || zip_u32(cd, at) != CD_ENTRY_SIGNATURE return entries, "has a damaged central directory";
method := zip_u16(cd, at + 10);
compressed_size := cast(s64) zip_u32(cd, at + 20);
size := cast(s64) zip_u32(cd, at + 24);
name_length := zip_u16(cd, at + 28);
extra_length := zip_u16(cd, at + 30);
comment_length := zip_u16(cd, at + 32);
local_offset := cast(s64) zip_u32(cd, at + 42);
if at + CD_ENTRY_SIZE + name_length > cd.count return entries, "has a damaged central directory";
name := slice(cd, at + CD_ENTRY_SIZE, name_length);
at += CD_ENTRY_SIZE + name_length + extra_length + comment_length;
if size > MAX_ENTRY_SIZE return entries, tprint("has a % byte entry %, too large", size, name);
// The data starts after the local header, whose name and extra field lengths can differ from
// the central directory's.
if local_offset + LOCAL_HEADER_SIZE > data.count || zip_u32(data, local_offset) != LOCAL_HEADER_SIGNATURE return entries, tprint("has a damaged local header for %", name);
data_offset := local_offset + LOCAL_HEADER_SIZE + zip_u16(data, local_offset + 26) + zip_u16(data, local_offset + 28);
if data_offset + compressed_size > data.count return entries, tprint("cannot read entry %", name);
compressed := slice(data, data_offset, compressed_size);
text: string;
if method == 0 {
if compressed_size != size return entries, tprint("has a stored entry % whose sizes differ", name);
text = compressed;
} else if method == 8 {
text = talloc_string(size);
decoded := stbi_zlib_decode_noheader_buffer(text.data, cast(s32) size, compressed.data, cast(s32) compressed_size);
if decoded != cast(s32) size return entries, tprint("has an entry % that does not inflate to its % bytes", name, size);
} else {
return entries, tprint("has an entry % compressed with method %, which is not supported", name, method);
}
array_add(*entries, .{copy_string(name,, temp), text});
}
return entries, "";
}
// A capture's entries as an .mcap: its strings and bytes temp, checked by reading the bytes back.
snagx_mcap_from_entries :: (entries: [] Archive_Entry) -> m: Mcap, bytes: [] u8, ok: bool, error: string {
m: Mcap;
json_entries: [..] Archive_Entry;
json_entries.allocator = temp;
original: *Archive_Entry;
rendered: *Archive_Entry;
for * entries {
if ends_with_nocase(it.name, ".json") array_add(*json_entries, it.*);
else if it.name == SNAGX_RENDERED_NAME rendered = it;
else if ends_with_nocase(it.name, ".backup.png") {} // the image before an edit: left out
else if ends_with_nocase(it.name, ".png") {
if original return m, .[], false, tprint("has two page images, % and %; an .mcap holds one page", original.name, it.name);
original = it;
} else {
return m, .[], false, tprint("has an entry % that an .mcap has no place for", it.name);
}
}
if !original return m, .[], false, "has no page image";
text, text_ok, error := capture_text_from_json(json_entries);
if !text_ok return m, .[], false, error;
m.text = text;
m.original_name = original.name;
m.original = cast([] u8) original.data;
if !rendered m.rendered_kind = .NONE;
else if rendered.data == original.data m.rendered_kind = .SAME_AS_ORIGINAL;
else { m.rendered_kind = .OWN; m.rendered = cast([] u8) rendered.data; }
preview_ok: bool;
m.preview, preview_ok, error = snagx_preview(mcap_rendered(m));
if !preview_ok return m, .[], false, error;
push_allocator(temp);
serializer, serialized, serialize_error := mcap_serialize(m);
if !serialized return m, .[], false, serialize_error;
bytes := cast([] u8) string.{serializer.count, serializer.data};
// The check: the bytes read back as exactly what went in.
back, parsed, parse_error := mcap_parse(bytes);
if !parsed return m, .[], false, tprint("the .mcap it was written as does not read back: %", parse_error);
if back.text != m.text return m, .[], false, "the text came back different";
if back.original_name != original.name || cast(string) back.original != original.data return m, .[], false, "the original image came back different";
if back.rendered_kind != m.rendered_kind || cast(string) mcap_rendered(back) != cast(string) mcap_rendered(m) return m, .[], false, "the rendered image came back different";
width, height, size_ok := png_size(back.preview.png);
if back.preview.png.count && (!size_ok || width != back.preview.width || height != back.preview.height) return m, .[], false, "the preview's stated size is not its own";
return m, bytes, true, "";
}
// The preview, as SnagitConverter makes it: the PNG itself when it is small enough, else scaled down.
snagx_preview :: (png: [] u8) -> Mcap_Image, ok: bool, error: string {
preview: Mcap_Image;
if !png.count return preview, true, "";
image, decoded, error := image_decode(png, .PNG, orient = false);
if !decoded return preview, false, tprint("its image cannot be decoded: %", error);
defer image_free(*image);
if max(image.width, image.height) <= MCAP_PREVIEW_MAX_SIDE {
preview = .{width = cast(u16) image.width, height = cast(u16) image.height, png = png};
return preview, true, "";
}
scaled, w, h, scaled_ok, scale_error := image_preview_png(image, MCAP_PREVIEW_MAX_SIDE);
if !scaled_ok return preview, false, scale_error;
preview = .{width = cast(u16) w, height = cast(u16) h, png = scaled};
return preview, true, "";
}
// ---------------------------------------------------------------- JSON to Capture Text
// The JSON entries as Capture Text (temp), checked: the text reads back without a mistake, holds
// exactly the JSON, and writes out the same.
capture_text_from_json :: (json_entries: [] Archive_Entry) -> text: string, ok: bool, error: string {
// jaison allocates from the context; its values only need to last for this capture.
push_allocator(temp);
parsed: [..] Parsed_Entry;
for json_entries {
success, json := json_parse_string(it.data);
if !success return "", false, tprint("% is not valid JSON", it.name);
if json.type != .OBJECT return "", false, tprint("% holds a JSON %, not an object", it.name, json.type);
array_add(*parsed, .{name = it.name, json = json});
}
if !parsed.count return "", false, "holds no JSON entries";
order_entries(parsed);
doc: Capture_Document;
capture_document_init(*doc, temp);
for parsed {
root, ok, error := capture_from_json(*doc, it.json);
if !ok return "", false, tprint("%: %", it.name, error);
file := capture_add_file(*doc, it.name);
file.root = root;
}
text := capture_text_write(*doc);
// Comparing with the JSON cannot catch what the JSON reader itself got wrong, as with emoji
// before jaison joined surrogate pairs; text that is not UTF-8 is a sign of it.
if !is_valid_utf8(text) return "", false, "the text is not valid UTF-8 (a lone surrogate escape in the JSON?)";
back: Capture_Document;
capture_document_init(*back, temp);
if !capture_text_parse(*back, "the converted text", text) || back.mistakes return "", false, "the text it was written as does not read back cleanly (see above)";
if back.files.count != parsed.count return "", false, tprint("% JSON files went in, % came back", parsed.count, back.files.count);
for parsed {
file := capture_find_file(*back, it.name);
if !file return "", false, tprint("% did not come back", it.name);
if !capture_equals_json(file.root, it.json) return "", false, tprint("% came back with different data", it.name);
}
if capture_text_write(*back) != text return "", false, "the text read back writes out differently";
return text, true, "";
}
#scope_file
Parsed_Entry :: struct {
name: string;
json: JSON_Value;
rank: s64; // where it goes in the text
}
// index.json, metadata.json, the pages in the order index.json lists them, then any other files by name.
order_entries :: (parsed: [] Parsed_Entry) {
pages: [] JSON_Value;
for parsed {
if it.name != "index.json" continue;
found, value := table_find(it.json.object, "Pages");
if found && value.type == .ARRAY pages = value.array;
}
for * parsed {
if it.name == "index.json" { it.rank = 0; continue; }
if it.name == "metadata.json" { it.rank = 1; continue; }
it.rank = 2 + pages.count; // after every page
for page: pages {
if page.type == .STRING && page.str == it.name { it.rank = 2 + it_index; break; }
}
}
quick_sort(parsed, (a: Parsed_Entry, b: Parsed_Entry) -> s64 {
if a.rank != b.rank return ifx a.rank < b.rank then -1 else 1;
return compare(a.name, b.name);
});
}
// Object keys are sorted, as Snagit writes them; jaison keeps them in a hash table, so their order in
// the file is gone by now anyway.
capture_from_json :: (doc: *Capture_Document, json: JSON_Value) -> Capture_Value, ok: bool, error: string {
if #complete json.type == {
case .NULL; return capture_null(), true, "";
case .BOOLEAN; return capture_boolean(json.boolean), true, "";
case .NUMBER; return capture_number(json.number), true, "";
case .STRING; return capture_string(doc, json.str), true, "";
case .ARRAY;
result := capture_array(doc);
for json.array {
item, ok, error := capture_from_json(doc, it);
if !ok return result, false, error;
array_add(*result.items, item);
}
return result, true, "";
case .OBJECT;
result := capture_object(doc);
keys: [..] string;
keys.allocator = temp;
for value, key: json.object.* array_add(*keys, key);
intro_sort(keys, compare);
for keys {
if !capture_key_is_writable(it) return result, false, tprint("the key \"%\" holds a control character, which Capture Text cannot write", it);
_, json_value := table_find(json.object, it);
value, ok, error := capture_from_json(doc, json_value);
if !ok return result, false, error;
capture_object_add(doc, *result, it, value);
}
return result, true, "";
}
return .{}, false, "unknown JSON type";
}
// Whether value holds exactly what json does: the same types, keys, strings byte for byte, and
// numbers bit for bit.
capture_equals_json :: (value: Capture_Value, json: JSON_Value) -> bool {
if #complete json.type == {
case .NULL; return value.kind == .NULL;
case .BOOLEAN; return value.kind == .BOOLEAN && value.boolean == json.boolean;
case .NUMBER;
number := json.number;
return value.kind == .NUMBER && (cast(*u64) *value.number).* == (cast(*u64) *number).*;
case .STRING; return value.kind == .STRING && value.text == json.str;
case .ARRAY;
if value.kind != .ARRAY || value.items.count != json.array.count return false;
for json.array if !capture_equals_json(value.items[it_index], it) return false;
return true;
case .OBJECT;
if value.kind != .OBJECT || value.fields.count != json.object.count return false;
for json_value, key: json.object.* {
found := capture_object_find(*value, key);
if !found || !capture_equals_json(found.*, json_value) return false;
}
return true;
}
return false;
}
EOCD_SIGNATURE :: 0x06054b50;
CD_ENTRY_SIGNATURE :: 0x02014b50;
LOCAL_HEADER_SIGNATURE :: 0x04034b50;
EOCD_SIZE :: 22;
CD_ENTRY_SIZE :: 46;
LOCAL_HEADER_SIZE :: 30;
MAX_ENTRY_SIZE :: 256 * 1024 * 1024;
// Each byte's cast is parenthesized: a bare cast(u32) applies to the whole | expression, which is then
// worked out in u8 and keeps only the first byte.
zip_u16 :: (s: string, at: s64) -> u16 {
return (cast(u16) s[at]) | ((cast(u16) s[at + 1]) << 8);
}
zip_u32 :: (s: string, at: s64) -> u32 {
return (cast(u32) s[at]) | ((cast(u32) s[at + 1]) << 8) | ((cast(u32) s[at + 2]) << 16) | ((cast(u32) s[at + 3]) << 24);
}
#import "Basic";
#import "String";
#import "File";
#import "Sort";
#import "IntroSort";
#import "Hash_Table";
#import "stb_image";
#import "jaison";
#import,dir "../jaison/unicode_utils"; // is_valid_utf8
#import "Capture_Text";
#import "Capture_File";
#import "Image_Decode";

16
modules/jaison/.editorconfig Normal file
View File

@ -0,0 +1,16 @@
# EditorConfig helps developers define and maintain consistent
# coding styles between different editors and IDEs
# editorconfig.org
root = true
[*]
indent_style = space
indent_size = 4
# We recommend you to keep these unchanged
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true

21
modules/jaison/LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2021 Raphael Luba
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

471
modules/jaison/examples/example.jai Normal file
View File

@ -0,0 +1,471 @@
#import, file "../module.jai";
main :: () {
{
// Use the temporary allocator so we don't have to worry about individually freeing
// all the allocations.
push_allocator(temp);
typed_parsing();
typed_printing();
generic_parsing();
generic_printing();
rename_by_note();
custom_rename_procedure();
decode_into_struct_with_using();
enum_errors();
}
// Since the program ends here, this doesn't matter, but just setting an example.
reset_temporary_storage();
}
// Data structures for typed parsing/printing
LevelData :: struct {
kind: LevelKind;
flags: LevelFlags;
secret: bool;
player: Entity;
player2: *Entity;
union {
score: float;
score_v2: float;
}
entities: [..] Entity;
floats: [] float;
LevelKind :: enum {
EASY :: 0;
HARD :: 1;
LEGENDARY :: 2;
}
LevelFlags :: enum_flags {
FLAG_A :: 0x1;
FLAG_B :: 0x2;
FLAG_C :: 0x4;
}
}
Entity :: struct {
name: string;
x, y: int;
dirty: bool; @JsonIgnore
}
LEVEL_DATA_1_JSON := #string DONE
{"kind": ".HARD", "flags": "FLAG_A | LevelFlags.FLAG_C", "secret": false,"score":5.5,"player": {"name": "Pat","x": 10,"y": 10},"player2": {"name": "Chris"},"entities": [{"name": "fdsa","x": 0,"y": 0},{"name": "fdsa","x": 0,"y": 0}], "floats": [0.00, 1.11111111111111111, 2.0202, 3e-5, 4.444444, -5.0]}
DONE;
LEVEL_DATA_2_JSON := #string DONE
{"kind": 2, "score_v2": 25.1, "floats": [0.5, null]}
DONE;
Fixed_Size_Data :: struct {
data: [3] int;
}
DATA_ARRAY_1 := #string DONE
{"data": [1, 2, 3]}
DONE;
DATA_ARRAY_2 := #string DONE
{"data": [1, 2]}
DONE;
MessageEntry :: struct {
union {
player: int;
player_v2: int;
}
union {
tag: int;
tag_v2: int;
}
name: string;
strict: bool;
union {
messagemode: string;
message_mode: string;
}
roundtrip: int;
identifier: string;
message: string;
};
typed_parsing :: () {
{
success, level := json_parse_string(LEVEL_DATA_1_JSON, LevelData, ignore_unknown=false);
// success, level := json_parse_file("level.json", LevelData, ignore_unknown=false);
assert(success);
log("Typed parsing result 1:\n%\nscore: %\n\n", level, level.score);
assert(level.floats.count == 6);
assert(level.floats[1] > 1); // Regression test, we had a bug here…
}
{
// Test parsing integers into enum slots & alternative union fields
success, level := json_parse_string(LEVEL_DATA_2_JSON, LevelData, ignore_unknown=false);
assert(success);
log("Typed parsing result 2:\n%\nscore: %\n\n", level, level.score);
assert(level.kind == .LEGENDARY);
assert(level.score == 25.1);
assert(level.floats.count == 2);
assert(is_nan(level.floats[1]));
}
{
// Test parsing into fixed-sized arrays
success, result := json_parse_string(DATA_ARRAY_1, Fixed_Size_Data, ignore_unknown=false);
assert(success);
log("Typed parsing result 3:\n%\n\n", result);
assert(result.data[0] == 1);
assert(result.data[1] == 2);
assert(result.data[2] == 3);
}
{
// Test parsing into fixed-sized arrays (incorrect size)
success, result := json_parse_string(DATA_ARRAY_2, Fixed_Size_Data, ignore_unknown=false);
assert(!success);
}
{
success, result := json_parse_string("{}", MessageEntry, ignore_unknown=false);
assert(success);
log("Typed parsing result 4:\n%\n\n", result);
}
}
typed_printing :: () {
level := LevelData.{kind=.LEGENDARY, flags=LevelData.LevelFlags.FLAG_B|.FLAG_C, secret=true, score=500};
level.player = .{name="Pat", x=4, y=4, dirty=true};
array_add(*level.entities, .{name="Chris", x=6, y=6});
json_string := json_write_string(level);
log("Typed printing result:\n%\n\n", json_string);
// success := json_write_file("level.json", level, indent_char="");
// assert(success);
// Inf/Nan handlig:
floats := float.[
0.5,
0h7FFF_FFFF, // NaN
0hFF80_0001, // also NaN
0hFF80_0000, // -inf
0h7F80_0000, // +inf
];
json_string = json_write_string(floats, indent_char = "");
log("floats:\n%\n\n", json_string);
assert(json_string == "[0.5,null,null,null,null]");
json_string = json_write_string(floats, indent_char = "", float_handling = .SUPPORT_STRINGS);
log("floats:\n%\n\n", json_string);
assert(json_string == "[0.5,\"NaN\",\"NaN\",\"-Infinity\",\"Infinity\"]");
float64s := float64.[
0.5,
0h7FFF_FFFF_FFFF_FFFF, // NaN
0hFFF0_0000_0000_0001, // also NaN
0hFFF0_0000_0000_0000, // -inf
0h7FF0_0000_0000_0000, // +inf
];
json_string = json_write_string(float64s, indent_char = "");
log("float64s:\n%\n\n", json_string);
assert(json_string == "[0.5,null,null,null,null]");
json_string = json_write_string(float64s, indent_char = "", float_handling = .SUPPORT_STRINGS);
log("float64s:\n%\n\n", json_string);
assert(json_string == "[0.5,\"NaN\",\"NaN\",\"-Infinity\",\"Infinity\"]");
}
generic_parsing :: () {
// In this scenario, some parts of the structure are known, but other parts are not.
json := #string DONE
{
"version": 3,
"entities": [
{
"name": "Player",
"x": 2,
"y": 2,
"player_index": 0
},
{
"name": "Snake",
"x": 4,
"y": 4,
"snake_color": 1
}
],
"stuff": [null, true, false]
}
DONE
success, root := json_parse_string(json);
// success, root := json_parse_file("level.json");
assert(success);
log("Generic parsing result:");
// Print things out, for demonstration purposes
traverse_node :: (node: JSON_Value, depth: int) {
INDENTATION :: 4;
print("% ", node.type);
if node.type == {
case .NULL;
print("\n");
case .BOOLEAN;
print("%\n", node.boolean);
case .NUMBER;
print("%\n", node.number);
case .STRING;
print("%\n", node.str);
case .OBJECT;
print("{\n");
for node.object {
for 1..(depth+1)*INDENTATION print(" ");
print("%: ", it_index);
traverse_node(it, depth+1);
}
for 1..depth*INDENTATION print(" ");
print("}\n");
case .ARRAY;
print("[\n");
for node.array {
for 1..(depth+1)*INDENTATION print(" ");
traverse_node(it, depth + 1);
}
for 1..depth*INDENTATION print(" ");
print("]\n");
}
}
traverse_node(root, 0);
print("\n");
// Convenience function for grabbing object members
get :: (json_val: JSON_Value, key: string, expected_type: JSON_Type) -> JSON_Value {
assert(json_val.type == .OBJECT);
table := json_val.object;
success, val := Hash_Table.table_find(table, key);
assert(success);
assert(val.type == expected_type);
return val;
}
// Check for version number that may or may not exist
version: float64 = -1;
assert(root.type == .OBJECT);
success2, val := Hash_Table.table_find(root.object, "version");
if success2 {
if val.type == .NUMBER {
version = val.number;
}
}
log("version: %\n", version);
// Traverse a structure we are confident about
for get(root, "entities", .ARRAY).array {
entity_name := get(it, "name", .STRING).str;
x := get(it, "x", .NUMBER).number / 32;
y := get(it, "y", .NUMBER).number / 32;
if entity_name == {
case "Player";
player_index := cast(int) get(it, "player_index", .NUMBER).number;
log("Player with player_index=%\n", player_index);
case "Snake";
snake_color := cast(int) get(it, "snake_color", .NUMBER).number;
log("Snake with snake_color=%\n", snake_color);
case;
//...
}
}
log("\n");
}
generic_printing :: () {
// We want to write JSON with arbitrary structure.
// Create and initialize object
root_obj: JSON_Object;
root := json_value(*root_obj);
// Add music index to object, in certain cases
should_add_music := true;
if should_add_music {
json_set(*root_obj, "music_index", .{type=.NUMBER, number=3});
}
// Create an array of values
temp: [..] JSON_Value;
junk := JSON_Value.{type=.STRING, str="junk"};
array_add(*temp, junk);
array_add(*temp, junk);
// Create json array value
array := JSON_Value.{type=.ARRAY};
array.array = temp;
// Add array to object
json_set(*root_obj, "junk_array", array);
// Print result
json_string := json_write_string(root);
log("Generic_printing result:\n%\n\n", json_string);
//json_write_file("level.json", root);
// Inf/Nan handling
floats: [..] JSON_Value;
array_add(*floats, .{type=.NUMBER, number=0.5});
array_add(*floats, .{type=.NUMBER, number=0h7FFF_FFFF_FFFF_FFFF}); // NaN
array_add(*floats, .{type=.NUMBER, number=0hFFF0_0000_0000_0001}); // also NaN
array_add(*floats, .{type=.NUMBER, number=0hFFF0_0000_0000_0000}); // -inf
array_add(*floats, .{type=.NUMBER, number=0h7FF0_0000_0000_0000}); // +inf
array.array = floats;
json_string = json_write_string(array, indent_char = "");
log("numbers:\n%\n\n", json_string);
assert(json_string == "[0.5,null,null,null,null]");
}
rename_by_note :: () {
// Sometimes the JSON we are parsing contains members with names we cannot use or don´t wanna.
Message :: struct {
value: string;
_context: struct { // We cannot use "context" because it´s a reserved keyword in Jai.
channel: int;
parent: int;
} @JsonName(context) // This member in JSON is "context" but we need to have it as "_context". So we can use the JsonName note to encode and decode it as "context".
}
json := #string DONE
{
"value": "Hello!",
"context": {
"channel": 1,
"parent": 897820
}
}
DONE
success, message_decoded := json_parse_string(json, Message, ignore_unknown = false);
assert(success, "Could not decode message!");
log("Rename_by_note decode result:\n%\n\n", message_decoded);
assert(message_decoded._context.channel == 1);
message_encoded := json_write_string(message_decoded);
log("Rename_by_note encode result:\n%\n\n", message_encoded);
assert(message_encoded.count != 0);
assert(find_index_from_left(message_encoded, "_context") == -1);
}
custom_rename_procedure :: () {
// We can also pass a custom rename procedure for renaming certain members.
rename_to_upper :: (member: *Type_Info_Struct_Member) -> string {
return to_upper_copy(member.name,,temp);
}
player := Entity.{
name="Player",
x = 10,
y = 50
};
player_encoded := json_write_string(player, rename=rename_to_upper);
assert(player_encoded.count != 0);
log("encoded with rename_to_upper:\n%\n\n", player_encoded);
assert(find_index_from_left(player_encoded, "name") == -1);
assert(find_index_from_left(player_encoded, "NAME") != -1);
success, player_decoded := json_parse_string(player_encoded, Entity, ignore_unknown = false, rename = rename_to_upper);
assert(success);
log("decoded with rename_to_upper:\n%\n\n", player_decoded);
assert(player_decoded.name == "Player");
}
decode_into_struct_with_using :: () {
Coordinates :: struct {
x: int;
y: int;
}
Tile :: struct {
color: string;
using coordinates: Coordinates;
}
JSON :: #string END
{
"color": "green",
"x": 1,
"y": 3
}
END
success, tile := json_parse_string(JSON, Tile, ignore_unknown = false);
assert(success);
log("Decoded tile: %", tile);
assert(tile.color == "green");
assert(tile.x == 1);
assert(tile.y == 3);
}
enum_errors :: () {
// Testing error messages when parsing invalid enum values
{
LEVEL_DATA_BROKEN_JSON := #string DONE
{"kind": ".VERY_HARD"}
DONE;
success, level := json_parse_string(LEVEL_DATA_BROKEN_JSON, LevelData, ignore_unknown=false);
assert(!success);
}
{
LEVEL_DATA_BROKEN_JSON := #string DONE
{"flags": "FLAG_A | FLAG_D"}
DONE;
success, level := json_parse_string(LEVEL_DATA_BROKEN_JSON, LevelData, ignore_unknown=false);
assert(!success);
}
}
is_nan :: (value: float) -> bool {
v := value;
bits := (cast(*s32) *v).*;
is_special := (bits & 0x7F80_0000) == 0x7F80_0000;
mant := (bits & 0x007F_FFFF);
return (is_special && mant);
}
#import "Basic";
#import "Hash_Table"; // To be able to iterate over node.object
#import "String"; // For to_upper_copy
#import "Compiler"; // For Type_Info_Struct_Member
Hash_Table :: #import "Hash_Table";

405
modules/jaison/generic.jai Normal file
View File

@ -0,0 +1,405 @@
// Generic JSON parsing/writing functions. Result is always a JSON_Value,
// which is awful to read and even more awful to create for complex structures.
// But it’s useful for some cases where re-creating the whole JSON structure as
// custom Jai struct types is inconvenient or not possible.
// This generic interface was the very first part I wrote in Jai and hasn’t been thorougly tested.
// Tread with care. There may be dragons.
JSON_Type :: enum u8 {
NULL :: 0;
BOOLEAN :: 1;
NUMBER :: 3;
STRING :: 2;
ARRAY :: 5;
OBJECT :: 4;
}
JSON_Value :: struct {
type: JSON_Type;
union {
boolean: bool;
number: float64;
str: string;
array: [] JSON_Value;
object: *JSON_Object;
};
}
JSON_Object :: Table(string, JSON_Value);
json_free :: (using val: JSON_Value) {
if #complete type == {
case .NULL;
case .BOOLEAN;
case .NUMBER;
case .STRING;
free(str);
case .ARRAY;
for array {
json_free(it);
}
array_free(array);
case .OBJECT;
for object {
free(it_index);
json_free(it);
}
table_deinit(object);
free(object);
}
}
json_parse_file :: (filename: string) -> success: bool, JSON_Value {
file_data, success := read_entire_file(filename);
result: JSON_Value;
if !success {
log_error("Could not read file");
return false, result;
}
defer free(file_data);
if context.log_level >= .VERBOSE log("Read file: %", success);
success, result = json_parse_string(file_data);
return success, result;
}
json_parse_string :: (content: string) -> success: bool, JSON_Value {
if !content then return false, .{};
result, remainder, success := parse_value(content);
if !success return false, result;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if remainder.count {
log_error("Unexpected trailing characters: %", remainder);
return false, result;
}
return true, result;
}
// For debug purposes
print_val :: (using val: JSON_Value) {
if #complete type == {
case .NULL;
print("null");
case .BOOLEAN;
print("%", boolean);
case .NUMBER;
print("%", number);
case .STRING;
print("\"%\"", str);
case .ARRAY;
print("[");
for array print_val(it);
print("]");
case .OBJECT;
print("%", object.*);
}
}
json_value :: (str: string) -> JSON_Value {
val: JSON_Value;
val.type =.STRING;
val.str = str;
return val;
}
json_value :: (obj: *JSON_Object) -> JSON_Value {
val: JSON_Value;
val.type =.OBJECT;
val.object = obj;
return val;
}
json_set_null :: (val: *JSON_Value) {
val.type = .NULL;
}
json_set :: (val: *JSON_Value, value: bool) {
val.type = .BOOLEAN;
val.boolean = value;
}
json_set :: (val: *JSON_Value, value: int) {
val.type = .NUMBER;
val.number = cast(float64) value;
}
json_set :: (val: *JSON_Value, value: float64) {
val.type = .NUMBER;
val.number = value;
}
json_set :: (val: *JSON_Value, value: string) {
val.type = .STRING;
val.str = value;
}
json_set :: (val: *JSON_Value, value: [] JSON_Value) {
val.type = .ARRAY;
val.array = value;
}
json_set :: (val: *JSON_Value, value: *JSON_Object) {
val.type = .OBJECT;
val.object = value;
}
json_write_json_value :: (builder: *String_Builder, using val: JSON_Value, indent_char := "\t", level := 0, float_handling := Special_Float_Handling.SUPPORT_NULL) {
if #complete type == {
case JSON_Type.NULL;
append(builder, "null");
case JSON_Type.BOOLEAN;
append(builder, ifx boolean "true" else "false");
case JSON_Type.NUMBER;
is_nan, is_infinity, is_negative := get_nan_or_infinity((cast(*u64) *number).*);
if is_nan || is_infinity {
append_special_float(builder, float_handling, is_nan, is_negative);
} else {
print_item_to_builder(builder, number);
}
case JSON_Type.STRING;
json_append_escaped(builder, str);
case JSON_Type.ARRAY;
append(builder, "[");
for array {
if indent_char.count {
append(builder, "\n");
for 0..level append(builder, indent_char);
}
json_write_json_value(builder, it, indent_char, level + 1, float_handling);
if it_index != array.count - 1 append(builder, ",");
}
if indent_char.count {
append(builder, "\n");
for 0..level-1 append(builder, indent_char);
}
append(builder, "]");
case JSON_Type.OBJECT;
append(builder, "{");
obj := object;
keys: [..] string;
defer array_free(keys);
array_reserve(*keys, obj.count);
for v, k: obj.* {
array_add(*keys, k);
}
intro_sort(keys, compare);
for keys {
if indent_char.count {
append(builder, "\n");
for 0..level append(builder, indent_char);
}
json_append_escaped(builder, it);
append(builder, ": ");
found, v := table_find(obj, it);
assert(found, "Missing table value %", it);
json_write_json_value(builder, v, indent_char, level + 1, float_handling);
if it_index != obj.count - 1 append(builder, ",");
}
if indent_char.count {
append(builder, "\n");
for 0..level-1 append(builder, indent_char);
}
append(builder, "}");
}
}
json_set :: (obj: *JSON_Object, path: string, val: JSON_Value) -> bool {
dotpos := find_index_from_left(path, #char ".");
if dotpos == -1 {
table_set(obj, path, val);
return true;
}
next := slice(path, 0, dotpos);
remainder := advance(path, dotpos + 1);
if !next.count return false;
if !remainder.count return false;
success, next_value := table_find(obj, next);
next_obj: *JSON_Object;
if success {
if next_value.type != JSON_Type.OBJECT return false;
next_obj = xx next_value.object;
} else {
next_obj = cast(*JSON_Object) alloc(size_of(JSON_Object));
memset(next_obj, 0, size_of(JSON_Object));
next_value = json_value(next_obj);
table_add(obj, next, next_value);
}
return json_set(next_obj, remainder, val);
}
get_as :: (val: JSON_Value, $T: Type) -> T {
#insert #run () -> string {
if T == bool {
return #string END
assert(val.type == .BOOLEAN, "Expected a % but got %", T, val.type);
return val.boolean;
END;
} else if T == float || T == float64 {
return #string END
assert(val.type == .NUMBER, "Expected a % but got %", T, val.type);
return cast(T) val.number;
END;
} else if T == string {
return #string END
assert(val.type == .STRING, "Expected a % but got %", T, val.type);
return val.str;
END;
} else if T == [] JSON_Value {
return #string END
assert(val.type == .ARRAY, "Expected a % but got %", T, val.type);
return val.array;
END;
} else if T == JSON_Object {
return #string END
assert(val.type == .OBJECT, "Expected a % but got %", T, val.type);
return val.object.*;
END;
} else {
assert(false, "Unsupported type");
return "";
}
}();
}
#scope_module
// Deeper input fails rather than overflowing the stack. A debug build uses about 4 KB of stack per
// level and overflows a 1 MB thread stack near 200; web manifests nest about four deep.
MAX_NESTING :: 64;
parse_value :: (to_parse: string, depth := 0) -> JSON_Value, remainder: string, success: bool {
result: JSON_Value;
remainder := trim_left(to_parse, WHITESPACE_CHARS);
success := false;
if !remainder || depth > MAX_NESTING return result, remainder, false;
if remainder[0] == {
case #char "n";
remainder, success = expect_and_slice(remainder, "null");
if !success return result, remainder, false;
json_set_null(*result);
result.type = JSON_Type.NULL;
return result, remainder, true;
case #char "t";
remainder, success = expect_and_slice(remainder, "true");
if !success return result, remainder, false;
json_set(*result, true);
return result, remainder, true;
case #char "f";
remainder, success = expect_and_slice(remainder, "false");
if !success return result, remainder, false;
json_set(*result, false);
return result, remainder, true;
case #char "\"";
str: string;
str, remainder, success = parse_string(remainder);
json_set(*result, str);
case #char "[";
result.type = JSON_Type.ARRAY;
result.array, remainder, success = parse_array(remainder, depth + 1);
case #char "{";
obj := cast(*JSON_Object) alloc(size_of(JSON_Object));
obj.*, remainder, success = parse_object(remainder, depth + 1);
result = json_value(obj);
case;
result.type = JSON_Type.NUMBER;
result.number, success, remainder = string_to_float64(remainder);
}
return result, remainder, success;
}
parse_array:: (str: string, depth := 0) -> result: [] JSON_Value, remainder: string, success: bool {
assert(str[0] == #char "[", "Invalid object start %", str);
remainder := advance(str);
result: [..] JSON_Value;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder return result, remainder, false;
if remainder[0] == #char "]" {
remainder = advance(remainder);
return result, remainder, true;
}
while true {
value: JSON_Value;
success: bool;
value, remainder, success = parse_value(remainder, depth);
if !success return result, remainder, false;
array_add(*result, value);
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder || remainder[0] != #char "," break;
remainder = advance(remainder);
remainder = trim_left(remainder, WHITESPACE_CHARS);
}
if !remainder || remainder[0] != #char "]" return result, remainder, false;
remainder = advance(remainder);
return result, remainder, true;
}
parse_object :: (str: string, depth := 0) -> result: JSON_Object, remainder: string, success: bool {
assert(str[0] == #char "{", "Invalid object start %", str);
remainder := advance(str);
result: JSON_Object;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder return result, remainder, false;
if remainder[0] == #char "}" {
remainder = advance(remainder);
return result, remainder, true;
}
table_resize(*result, 32);
while true {
if !remainder || remainder[0] != #char "\"" return result, remainder, false;
key: string;
value: JSON_Value;
success: bool;
key, remainder, success = parse_string(remainder);
if !success return result, remainder, false;
existing := table_find_pointer(*result, key);
if existing {
free(key);
return result, remainder, false;
}
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder || remainder[0] != #char ":" {
free(key);
return result, remainder, false;
}
remainder = advance(remainder);
remainder = trim_left(remainder, WHITESPACE_CHARS);
value, remainder, success = parse_value(remainder, depth);
if !success {
free(key);
return result, remainder, false;
}
table_add(*result, key, value);
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder || remainder[0] != #char "," break;
remainder = advance(remainder);
remainder = trim_left(remainder, WHITESPACE_CHARS);
}
if !remainder || remainder[0] != #char "}" return result, remainder, false;
remainder = advance(remainder);
return result, remainder, true;
}

299
modules/jaison/module.jai Normal file
View File

@ -0,0 +1,299 @@
// This file contains just the JSON serialization functions. See generic.jai and typed.jai for parse fuctions.
// Generates a JSON string from either a JSON_Value or any custom type.
// "indent_char" does what it says on the tin.
// "ignore" is only used for custom types to determine which properties of your custom type should be serialized.
// The default ignore function ignores all struct members that have the note @JsonIgnore.
// "rename" is used for renaming certain members.
// It gets called with the Type_Info_Struct_Member and must return the new name of the field.
// The default procedure rename members by their @JsonName note. Eg: @JsonName(renamed_member).
json_write_string :: (value: $T, indent_char := "\t", ignore := ignore_by_note, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL) -> string {
builder: String_Builder;
defer free_buffers(*builder);
json_append_value(*builder, value, indent_char, ignore, rename, float_handling);
return builder_to_string(*builder);
}
json_write_file :: (filename: string, value: $T, indent_char := "\t", ignore := ignore_by_note, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL) -> bool {
builder: String_Builder;
defer free_buffers(*builder);
json_append_value(*builder, value, indent_char, ignore, rename, float_handling);
return write_entire_file(filename, *builder);
}
json_append_value :: (builder: *String_Builder, val: $T, indent_char := "\t", ignore := ignore_by_note, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL) {
#if T == JSON_Value {
json_write_json_value(builder, val, indent_char, float_handling=float_handling);
} else {
info := type_info(T);
json_write_native(builder, *val, info, indent_char, ignore, rename, float_handling=float_handling);
}
}
// This function is useful if you have a JSON template string and just want to
// safely insert a value without having to replicate the full json structure in Jai.
// The return value does NOT include quotes around the string.
//
// Example:
// JSON_TEMPLATE :: #string END
// {
// "complicated": {
// "json": {
// "structure": {
// "for_a_stupid_api": {
// "that_needs": [
// {"a_deeply_nested_value": "%1"}
// ]
// }
// }
// }
// }
// }
// END
// escaped_value := json_escape_string(my_unsafe_value);
// defer free(escaped_value);
// json_str := print(JSON_TEMPLATE, escaped_value);
json_escape_string :: (str: string) -> string {
builder: String_Builder;
defer free_buffers(*builder);
json_append_escaped(*builder, str);
escaped := builder_to_string(*builder);
return escaped;
}
// How shall NaN and Infinity be serialized / deserialized?
Special_Float_Handling :: enum_flags {
SUPPORT_NULL :: 0x01;
SUPPORT_STRINGS :: 0x02;
SUPPORT_ALL :: SUPPORT_NULL | SUPPORT_STRINGS;
}
Ignore_Proc :: #type (member: *Type_Info_Struct_Member) -> bool;
Rename_Proc :: #type (member: *Type_Info_Struct_Member) -> string;
ignore_by_note :: (member: *Type_Info_Struct_Member) -> bool {
for note: member.notes {
if note == "JsonIgnore" return true;
}
return false;
}
rename_by_note :: (member: *Type_Info_Struct_Member) -> string {
for note: member.notes {
if !begins_with(note, "JsonName(") continue;
if note.count <= 10 || note[note.count-1] != #char ")" {
log_error("Invalid JsonName note format. Expected a name in parenthesis, but the note was \"%\".", note);
continue;
}
return slice(note, 9, note.count-10);
}
return member.name;
}
#scope_module
WHITESPACE_CHARS :: " \t\n\r";
#load "generic.jai";
#load "typed.jai";
json_append_escaped :: (builder: *String_Builder, str: string) {
remaining := str;
next_pos := index_of_illegal_string_char(remaining);
append(builder, "\"");
while (next_pos >= 0) {
append(builder, slice(remaining, 0, next_pos));
if remaining[next_pos] == {
case #char "\\";
append(builder, "\\\\");
case #char "\"";
append(builder, "\\\"");
case #char "\n";
append(builder, "\\n");
case #char "\r";
append(builder, "\\r");
case #char "\t";
append(builder, "\\t");
case;
// ToDo: handle illegal multi-byte characters
// print("Escaping: %\n\n", slice(remaining, next_pos, remaining.count - next_pos));
print_to_builder(builder, "\\u%", formatInt(remaining[next_pos], base=16, minimum_digits=4));
}
remaining = advance(remaining, next_pos + 1);
next_pos = index_of_illegal_string_char(remaining);
}
append(builder, remaining);
append(builder, "\"");
}
index_of_illegal_string_char :: (str: string) -> s64 {
for 0..str.count - 1 {
if str[it] == #char "\\" || str[it] == #char "\"" || str[it] <= 0x1F {
return it;
}
}
return -1;
}
expect_and_slice :: (str: string, expected: string) -> remainder: string, success: bool {
if str.count < expected.count || !equal(slice(str, 0, expected.count), expected) {
log_error("Unexpected token. Expected \"%\" but got: %", expected, str);
return str, false;
}
remainder := advance(str, expected.count);
return remainder, true;
}
parse_string :: (str: string) -> result: string, remainder: string, success: bool {
assert(str[0] == #char "\"", "Invalid string start %", str);
inside := advance(str);
if !inside return "", "", false;
needsUnescape := false;
while true {
if !inside || (inside.count < 2 && inside[0] != #char "\"") return "", str, false;
if inside[0] == #char "\"" break;
if inside[0] == #char "\\" {
needsUnescape = true;
if inside.count < 2 return "", str, false;
advance(*inside);
}
advance(*inside);
}
length := inside.data - str.data - 1;
result := slice(str, 1, length);
if needsUnescape {
success: bool;
result, success = unescape(result);
if !success return "", str, false;
} else {
result = copy_string(result);
}
remainder := slice(str, length + 2, str.count - length - 2);
return result, remainder, true;
}
unescape :: (str: string) -> result: string, success: bool {
result := alloc_string(str.count);
rc := 0;
success := true;
for i: 0..str.count-1 {
if str[i] != #char "\\" {
// Check for invalid characters for JSON
if str[i] < 0x20 {
success = false;
break;
}
result[rc] = str[i];
rc += 1;
} else {
if i == str.count - 1 {
success = false;
break;
}
i += 1;
if str[i] == {
case #char "\""; #through;
case #char "/"; #through;
case #char "\\";
result[rc] = str[i];
rc += 1;
case #char "b";
result[rc] = 0x08;
rc += 1;
case #char "f";
result[rc] = 0x0c;
rc += 1;
case #char "n";
result[rc] = #char "\n";
rc += 1;
case #char "r";
result[rc] = #char "\r";
rc += 1;
case #char "t";
result[rc] = #char "\t";
rc += 1;
case #char "u";
if i + 4 >= str.count {
success = false;
break;
}
unicode_char, unicode_success := parse_unicode(slice(str, i + 1, 4));
if !unicode_success {
success = false;
break;
}
// A high surrogate followed by \u and a low one is a single character beyond
// U+FFFF, such as an emoji: four bytes of UTF-8, not two 3-byte halves (which are
// not valid UTF-8). A surrogate without its other half is encoded on its own, as before.
if unicode_char >= 0xD800 && unicode_char <= 0xDBFF && i + 10 < str.count && str[i + 5] == #char "\\" && str[i + 6] == #char "u" {
low, low_success := parse_unicode(slice(str, i + 7, 4));
if low_success && low >= 0xDC00 && low <= 0xDFFF {
code_point := 0x10000 + ((cast(u32) unicode_char - 0xD800) << 10) + (cast(u32) low - 0xDC00);
result[rc] = xx (0xF0 | (code_point >> 18));
result[rc + 1] = xx (0x80 | ((code_point >> 12) & 0x3F));
result[rc + 2] = xx (0x80 | ((code_point >> 6) & 0x3F));
result[rc + 3] = xx (0x80 | (code_point & 0x3F));
rc += 4;
i += 10;
continue;
}
}
utf8_len := encode_utf8(unicode_char, *(result.data[rc]));
rc += utf8_len;
i += 4;
case;
success = false;
break;
}
}
}
if !success {
free(result);
return "", false;
}
result.count = rc;
return result, true;
}
append_special_float :: (builder: *String_Builder, float_handling: Special_Float_Handling, is_nan: bool, is_negative: bool) {
if !(float_handling & .SUPPORT_STRINGS) {
append(builder, "null");
} else if is_nan {
append(builder, "\"NaN\"");
} else if is_negative {
append(builder, "\"-Infinity\"");
} else {
append(builder, "\"Infinity\"");
}
}
get_nan_or_infinity :: (value: u32) -> is_nan: bool, is_infinity: bool, is_negative: bool {
is_special := (value & 0x7F80_0000) == 0x7F80_0000;
mant := (value & 0x007F_FFFF);
return (is_special && mant), is_special && !mant, !!(value & 0x8000_0000);
}
get_nan_or_infinity :: (value: u64) -> is_nan: bool, is_infinity: bool, is_negative: bool {
is_special := (value & 0x7FF0_0000_0000_0000) == 0x7FF0_0000_0000_0000;
mant := (value & 0xF_FFFF_FFFF_FFF);
return (is_special && mant), is_special && !mant, !!(value & 0x8000_0000_0000_0000);
}
NAN :: 0h7FFF_FFFF_FFFF_FFFF;
INFINITY :: 0h7FF0_0000_0000_0000;
NEGATIVE_INFINITY :: 0h8FF0_0000_0000_0000;
#import "Basic";
#import "File";
#import "String";
#import "Hash_Table";
#import,dir "./unicode_utils";
#import "IntroSort";

60
modules/jaison/readme.md Normal file
View File

@ -0,0 +1,60 @@
# JSON serialization / deserialization module for Jai
This module offers two interfaces:
* one uses a "generic tree" built from `JSON_Value`
* the other is a typed version that serializes / deserializes your custom data structures.
The generic `JSON_Value` graphs are a pain to consume and even worse to produce by hand.
But they allow you to parse any JSON, even if you don’t know the structure (or can’t reproduce it in Jai because it varies).
The typed interface is what you want for most cases.
## Parsing / Deserialization
Parsing is as simple as:
```Jai
// Typed version:
success, result := json_parse_string(json_str, Your_Type_To_Parse_Into);
// … or if you want to get a generic structure back:
success, result := json_parse_string(json_str);
```
There are also a convenience functions for parsing if the JSON data is in a file:
```Jai
success, result := json_parse_file(json_filename, Your_Type_To_Parse_Into);
// … or
success, result := json_parse_file(json_filename);
```
See [`typed.jai`](./typed.jai) and [`generic.jai`](./generic.jai) for details and additional options.
### Mixed typed and generic data
If you don’t know the structure of some subfield of your `Your_Type_To_Parse_Into` structure, but still want to get these values from the JSON data,
you can declare these fields as the generic type `JSON_Value` or `*JSON_Value` and the generic parse function will take over at that point:
```
Your_Type_To_Parse_Into :: struct {
name: string;
age: int;
something_we_dont_know_much_about: *JSON_Value; // Whatever structure hides in the JSON, it will be parsed into JSON_Value.
}
```
## Printing / Serialization
Generating a string works the same for both interfaces:
```Jai
json_str := json_write_string(my_value);
```
where `my_value` is either a `JSON_Value` or any other data structure.
See [`module.jai`](./module.jai) for details and additional parameters.
## Dependencies
This module uses [the `unicode_utils` module](https://github.com/rluba/jai-unicode), which is included as a submodule.

698
modules/jaison/typed.jai Normal file
View File

@ -0,0 +1,698 @@
// Parse a JSON string into the given Type.
// All members of Type that are not present in the JSON are kept at their default values.
// All fields in the JSON that have no corresponding member in Type are ignored by default
// but you can pass ignore_unknown = false to fail instead.
json_parse_string :: (content: string, $T: Type, ignore_unknown := true, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL) -> success: bool, T {
result: T;
if !content then return false, result;
info := type_info(T);
remainder, success := parse_value(content, cast(*u8)*result, info, ignore_unknown, "", rename=rename, float_handling=float_handling);
if !success return false, result;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if remainder.count {
log_error("Unexpected trailing characters: %", remainder);
return false, result;
}
return true, result;
}
json_parse_file :: (filename: string, $T: Type, ignore_unknown := true, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL) -> success: bool, T {
file_data, success := read_entire_file(filename);
result: T;
if !success {
log_error("Could not read file: %", filename);
return false, result;
}
defer free(file_data);
if (context.log_level >= .VERBOSE) {
log("Read file: %", success);
}
success, result = json_parse_string(file_data, T, ignore_unknown, rename=rename, float_handling=float_handling);
return success, result;
}
json_write_native :: (builder: *String_Builder, data: *void, info: *Type_Info, indent_char := "\t", ignore := ignore_by_note, rename := rename_by_note, float_handling := Special_Float_Handling.SUPPORT_NULL, level := 0) {
if info.type == {
case .BOOL;
append(builder, ifx (.*)(cast(*bool) data) "true" else "false");
case .INTEGER;
any_val: Any;
any_val.type = info;
any_val.value_pointer = data;
print_item_to_builder(builder, any_val);
case .FLOAT;
is_nan, is_infinity, is_negative: bool;
if info.runtime_size == 4 {
is_nan, is_infinity, is_negative = get_nan_or_infinity((cast(*u32) data).*);
} else if info.runtime_size == 8 {
is_nan, is_infinity, is_negative = get_nan_or_infinity((cast(*u64) data).*);
}
if is_nan || is_infinity {
append_special_float(builder, float_handling, is_nan, is_negative);
} else {
any_val: Any;
any_val.type = info;
any_val.value_pointer = data;
print_item_to_builder(builder, any_val);
}
case .ENUM;
any_val: Any;
any_val.type = info;
any_val.value_pointer = data;
append(builder, #char "\"");
print_item_to_builder(builder, any_val);
append(builder, #char "\"");
case .STRING;
json_append_escaped(builder, (.*)(cast(*string) data));
case .ARRAY;
info_array := cast(*Type_Info_Array) info;
element_size := info_array.element_type.runtime_size;
assert(element_size != -1);
stride := element_size;
array_data := data;
array_count := info_array.array_count;
if info_array.array_count == -1 {
array_count = (.*) cast(*s64) data;
array_dest: **void = data + 8;
array_data = array_dest.*;
}
append(builder, "[");
if array_data {
for 0..array_count-1 {
if indent_char.count {
append(builder, "\n");
for 0..level append(builder, indent_char);
}
json_write_native(builder, array_data, info_array.element_type, indent_char, ignore, rename, float_handling=float_handling, level + 1);
if it != array_count - 1 append(builder, ",");
array_data += stride;
}
}
if indent_char.count {
append(builder, "\n");
for 0..level-1 append(builder, indent_char);
}
append(builder, "]");
case .STRUCT;
struct_info := cast(*Type_Info_Struct) info;
if is_generic_json_value(info) {
value := cast(*JSON_Value) data;
json_write_json_value(builder, value.*, indent_char, level);
} else {
append(builder, #char "{");
first := true;
json_write_native_members(builder, data, struct_info.members, indent_char, ignore, rename, float_handling, level, *first);
if indent_char.count {
append(builder, "\n");
for 0..level-1 append(builder, indent_char);
}
append(builder, "}");
}
case .POINTER;
ptr_info := cast(*Type_Info_Pointer) info;
ptr := (.*) cast(**void) data;
if ptr {
json_write_native(builder, ptr, ptr_info.pointer_to, indent_char, ignore, rename, float_handling, level);
} else {
append(builder, "null");
}
case;
assert(false, "Unsupported type: %", info.type);
}
}
#scope_file
json_write_native_members :: (builder: *String_Builder, data: *void, members: [] Type_Info_Struct_Member, indent_char := "\t", ignore := ignore_by_note, rename: Rename_Proc, float_handling: Special_Float_Handling, level := 0, first: *bool) {
for * member: members {
if member.flags & .CONSTANT continue;
if ignore(member) continue;
if (member.type.type == .STRUCT && member.flags & .USING) {
info := cast(*Type_Info_Struct) member.type;
json_write_native_members(builder, data + member.offset_in_bytes, info.members, indent_char, ignore, rename, float_handling, level, first);
} else {
if !first.* append(builder, ",");
first.* = false;
if indent_char.count {
append(builder, "\n");
for 0..level append(builder, indent_char);
}
renamed_name := rename(member);
name := ifx renamed_name.count > 0 renamed_name else member.name;
json_append_escaped(builder, name);
append(builder, ": ");
json_write_native(builder, data + member.offset_in_bytes, member.type, indent_char, ignore, rename, float_handling, level + 1);
}
}
}
is_generic_json_value_or_pointer_to_it :: (info: *Type_Info) -> bool {
value_info := info;
if info.type == .POINTER {
pointer_info := cast(*Type_Info_Pointer) info;
value_info = pointer_info.pointer_to;
}
return is_generic_json_value(info);
}
is_generic_json_value :: (info: *Type_Info) -> bool {
return info == type_info(JSON_Value);
}
parse_value :: (to_parse: string, slot: *u8, info: *Type_Info, ignore_unknown: bool, field_name: string, rename: Rename_Proc, float_handling: Special_Float_Handling) -> remainder: string, success: bool {
remainder := trim_left(to_parse, WHITESPACE_CHARS);
if !remainder return "", false;
success := true;
prepare_slot :: (expected_type: Type_Info_Tag, info: *Type_Info, slot: *u8, to_parse: string) -> *u8, success: bool, is_generic: bool, info: *Type_Info {
value_info := info;
if info.type == .POINTER {
pointer_info := cast(*Type_Info_Pointer) info;
value_info = pointer_info.pointer_to;
}
if info.type == .ENUM {
info_enum := cast(*Type_Info_Enum)info;
value_info = info_enum.internal_type;
}
is_generic := is_generic_json_value(value_info);
if !is_generic && value_info.type != expected_type {
teaser := to_parse;
if teaser.count > 50 teaser.count = 50;
builder: String_Builder;
print_type_to_builder(*builder, info);
type_name := builder_to_string(*builder,, temp);
log_error("Cannot parse % value into type \"%\". Remaining input is: %…", expected_type, type_name, teaser);
return null, false, false, value_info;
}
if info.type == .POINTER {
value_slot := alloc(value_info.runtime_size);
initializer: (*void) #no_context;
if value_info.type == .STRUCT {
struct_info := cast(*Type_Info_Struct) value_info;
initializer = struct_info.initializer;
}
if initializer {
initializer(value_slot);
} else {
memset(value_slot, 0, value_info.runtime_size);
}
(.*)cast(**u8)slot = value_slot;
return value_slot, true, is_generic, value_info;
} else {
return slot, true, is_generic, value_info;
}
}
store_float_value :: (info: *Type_Info, slot: *u8, to_parse: string, value: float64) -> bool {
value_slot, success, is_generic, value_info := prepare_slot(.FLOAT, info, slot, to_parse);
if success {
if is_generic {
json_set(cast(*JSON_Value)value_slot, value);
} else {
if value_info.runtime_size == 4 {
((.*) cast(*float) slot) = cast(float) value;
} else {
assert(value_info.runtime_size == 8);
((.*) cast(*float64) slot) = value;
}
}
}
return success;
}
is_generic: bool;
if remainder[0] == {
case #char "n";
remainder, success = expect_and_slice(remainder, "null");
if !success return remainder, false;
if slot {
if info.type == .POINTER {
(.*)cast(**void) slot = null;
} else if info.type == .FLOAT && float_handling & .SUPPORT_NULL {
success = store_float_value(info, slot, to_parse, NAN);
} else {
builder: String_Builder;
print_type_to_builder(*builder, info);
type_name := builder_to_string(*builder,, temp);
log_error("Found NULL value for non-pointer type \"%\" of field \"%\". Keeping default value instead.", type_name, field_name);
}
}
return remainder, true;
case #char "t";
remainder, success = expect_and_slice(remainder, "true");
if !success return remainder, false;
if slot {
value_slot: *u8;
value_slot, success, is_generic = prepare_slot(.BOOL, info, slot, to_parse);
if success {
if is_generic {
json_set(cast(*JSON_Value)value_slot, true);
} else {
(.*)cast(*bool)value_slot = true;
}
}
}
case #char "f";
remainder, success = expect_and_slice(remainder, "false");
if !success return remainder, false;
if slot {
value_slot: *u8;
value_slot, success, is_generic = prepare_slot(.BOOL, info, slot, to_parse);
if success {
if is_generic {
json_set(cast(*JSON_Value)value_slot, false);
} else {
(.*)cast(*bool)value_slot = false;
}
}
}
case #char "\"";
if slot && info && info.type == .ENUM {
info_enum := cast(*Type_Info_Enum)info;
value_slot: *u8;
value_slot, success, is_generic = prepare_slot(.INTEGER, info_enum.internal_type, slot, to_parse);
remainder, success = parse_enum_string(remainder, value_slot, info_enum);
} else {
value: string;
value, remainder, success = parse_string(remainder);
stored := false;
defer if !stored free(value);
if success && slot {
if info && info.type == .FLOAT && (float_handling & .SUPPORT_STRINGS) {
if value == {
case "NaN"; success = store_float_value(info, slot, to_parse, NAN);
case "Infinity"; success = store_float_value(info, slot, to_parse, INFINITY);
case "-Infinity"; success = store_float_value(info, slot, to_parse, NEGATIVE_INFINITY);
case;
builder: String_Builder;
print_type_to_builder(*builder, info);
type_name := builder_to_string(*builder,, temp);
log_error("Found unsupported string value \"%\" for field \"%\" of type %.", value, field_name, type_name);
}
} else {
value_slot: *u8;
value_slot, success, is_generic = prepare_slot(.STRING, info, slot, to_parse);
if success {
if is_generic {
json_set(cast(*JSON_Value)value_slot, value);
} else {
(.*)cast(*string)value_slot = value;
}
stored = true;
}
}
}
}
case #char "[";
value_slot: *u8;
value_info: *Type_Info;
if slot {
value_slot, success, is_generic, value_info = prepare_slot(.ARRAY, info, slot, to_parse);
}
if success {
if is_generic {
value: [] JSON_Value;
value, remainder, success = parse_array(remainder);
json_set(cast(*JSON_Value)value_slot, value);
} else {
remainder, success = parse_array(remainder, value_slot, cast(*Type_Info_Array) value_info, ignore_unknown, rename=rename, float_handling=float_handling);
}
}
case #char "{";
value_slot: *u8;
value_info: *Type_Info;
if slot {
value_slot, success, is_generic, value_info = prepare_slot(.STRUCT, info, slot, to_parse);
}
if success {
if is_generic {
value := New(JSON_Object);
value.*, remainder, success = parse_object(remainder);
json_set(cast(*JSON_Value)value_slot, value);
} else {
remainder, success = parse_object(remainder, value_slot, cast(*Type_Info_Struct) value_info, ignore_unknown, rename=rename, float_handling=float_handling);
}
}
case;
if slot == null || info.type == .FLOAT || is_generic_json_value_or_pointer_to_it(info) {
float_value: float64;
float_value, success, remainder = string_to_float64(remainder);
if success && slot {
success = store_float_value(info, slot, to_parse, float_value);
}
} else {
if slot {
value_slot: *u8;
value_info: *Type_Info;
value_slot, success, is_generic, value_info = prepare_slot(.INTEGER, info, slot, to_parse);
if success {
if is_generic {
int_value: s64;
int_value, success, remainder = string_to_int(remainder, T = s64);
if success {
json_set(cast(*JSON_Value)value_slot, int_value);
} else {
log_error("Could not parse \"%\" as an integer.", to_parse);
}
} else {
info_int := cast(*Type_Info_Integer) value_info;
success, remainder = parse_and_write_integer(info_int, value_slot, to_parse);
}
}
} else {
int_value: s64;
int_value, success, remainder = string_to_int(remainder, T = s64);
if !success {
log_error("Could not parse \"%\" as an integer.", to_parse);
}
}
}
}
return remainder, success;
}
parse_enum_string :: (str: string, slot: *u8, info_enum: *Type_Info_Enum) -> remainder: string, success: bool {
value, remainder, success := parse_string(str);
defer free(value);
if !success return remainder, false;
// Parse by members' names
normalize_enum_value :: inline (name: string) -> string #expand {
normalized := trim(name);
if normalized.count > info_enum.name.count && starts_with(normalized, info_enum.name) && normalized[info_enum.name.count] == #char "." {
normalized = slice(normalized, info_enum.name.count+1, normalized.count-info_enum.name.count-1);
} else if starts_with(normalized, ".") {
normalized = slice(normalized, 1, normalized.count-1);
}
return normalized;
}
int_info := info_enum.internal_type;
int_value: s64;
if info_enum.enum_type_flags & .FLAGS {
values := split(value, "|",, temp);
for v: values {
name := normalize_enum_value(v);
found_name := false;
for info_enum.names {
if name == it {
found_name = true;
int_value |= info_enum.values[it_index];
break;
}
}
if !found_name {
log_error("Enum \"%\" does not contain a member named \"%\".", info_enum.name, name);
success = false;
}
}
} else {
success = false;
name := normalize_enum_value(value);
for info_enum.names {
if name == it {
int_value = info_enum.values[it_index];
success = true;
break;
}
}
if !success {
log_error("Enum \"%\" does not contain a member named \"%\".", info_enum.name, name);
}
}
if success {
if int_info.signed {
valid, low, high := Reflection.range_check_and_store(int_value, int_info, slot);
if !valid {
log_error("The value '%' is out of range. (It must be between % and %.)", int_value, low, high);
return remainder, false;
}
} else {
valid, low, high := Reflection.range_check_and_store(cast(u64) int_value, int_info, slot);
if !valid {
log_error("The value '%' is out of range. (It must be between % and %.)", int_value, low, high);
return remainder, false;
}
}
}
return remainder, success;
}
parse_array :: (str: string, slot: *u8, info: *Type_Info_Array, ignore_unknown: bool, rename: Rename_Proc, float_handling: Special_Float_Handling) -> remainder: string, success: bool {
element_size: int;
if slot {
element_size = info.element_type.runtime_size;
assert(element_size != -1, "Unknown element size");
}
assert(str[0] == #char "[", "Invalid object start %", str);
remainder := advance(str);
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder return "", false;
success := true;
if slot {
array: Resizable_Array;
initializer: (*void) #no_context;
if info.element_type.type == .STRUCT {
struct_info := cast(*Type_Info_Struct) info.element_type;
initializer = struct_info.initializer;
}
while true {
if remainder[0] == #char "]" break;
if array.count {
if remainder[0] != #char "," {
success = false;
break;
}
remainder = advance(remainder);
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder {
success = false;
break;
}
if remainder[0] == #char "]" {
// Special case for trailing comma. This would also fail in parse_value below, but with a weird error.
success = false;
break;
}
}
if info.array_count != -1 && array.count >= info.array_count {
log_error("Expected array of size %, but found array of with at least % elements\n", info.array_count, info.array_count + 1);
success = false;
break;
}
maybe_grow(*array, element_size);
element_data := array.data + array.count * element_size;
array.count += 1;
if initializer {
initializer(element_data);
} else {
memset(element_data, 0, element_size);
}
remainder, success = parse_value(remainder, element_data, info.element_type, ignore_unknown, "", rename=rename, float_handling=float_handling);
if !success break;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder {
success = false;
break;
}
}
if info.array_type == .VIEW {
view := (cast(*Array_View_64) slot);
view.count = array.count;
view.data = array.data;
} else if info.array_count == -1 {
// Resizable array
(.*)(cast(*Resizable_Array) slot) = array;
} else {
// Fixed-size array
if info.array_count != array.count {
log_error("Expected array of size %, but found array of size %.\n", info.array_count, array.count);
success = false;
// Fill up the rest of the array so that we can safely copy the data. Maybe this is unnecessary, but it allows to print the successfully parsed values even in case of an error.
while array.count < info.array_count {
maybe_grow(*array, element_size);
array.count += 1;
element_data := array.data + array.count * element_size;
if initializer {
initializer(element_data);
} else {
memset(element_data, 0, element_size);
}
}
}
assert(info.array_count == array.count);
memcpy(slot, array.data, array.count * element_size);
free(array.data);
}
} else {
first := true;
while true {
if remainder[0] == #char "]" break;
if !first {
if remainder[0] != #char "," return remainder, false;
remainder = advance(remainder);
}
first = false;
remainder, success = parse_value(remainder, null, null, ignore_unknown, "", rename=rename, float_handling=float_handling);
if !success return remainder, false;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder return "", false;
}
}
if !remainder || remainder[0] != #char "]" return remainder, false;
remainder = advance(remainder);
return remainder, success;
}
Member_Offset :: struct {
member: *Type_Info_Struct_Member;
offset_in_bytes: s64;
}
// This procedure is somewhat copied from Basic.get_field.
fill_member_table :: (table: *Table(string, Member_Offset), info: *Type_Info_Struct, rename: Rename_Proc, base_offset := 0) {
for * member: info.members {
offset := base_offset + member.offset_in_bytes;
name := rename(member);
assert(!table_find_pointer(table, name), "Redeclaration of member \"%\": % vs. %", name, member.*, (.*)table_find_pointer(table, name));
table_set(table, name, .{member, offset});
if (member.flags & .USING) && (member.type.type == .STRUCT) {
fill_member_table(table, cast(*Type_Info_Struct)member.type, rename, offset);
}
}
}
parse_object :: (str: string, slot: *u8, info: *Type_Info_Struct, ignore_unknown: bool, rename: Rename_Proc, float_handling: Special_Float_Handling) -> remainder: string, success: bool {
assert(str[0] == #char "{", "Invalid object start %", str);
remainder := advance(str);
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder return "", false;
if remainder[0] == #char "}" {
remainder = advance(remainder);
return remainder, true;
}
// @Speed: Building this table every time is pretty silly.
// We should probably either not build it at all or cache it somewhere.
member_table: Table(string, Member_Offset);
defer table_deinit(*member_table);
if info fill_member_table(*member_table, info, rename);
while true {
if !remainder || remainder[0] != #char "\"" return remainder, false;
key: string;
success: bool;
key, remainder, success = parse_string(remainder);
if !success return remainder, false;
defer free(key);
member_found, member_offset := table_find(*member_table, key);
member_slot: *u8;
member_info: *Type_Info;
if member_found {
member_slot = slot + member_offset.offset_in_bytes;
member_info = member_offset.member.type;
} else if !ignore_unknown {
log_error("Missing member % in %", key, info.*);
return remainder, false;
}
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder || remainder[0] != #char ":" return remainder, false;
remainder = advance(remainder);
remainder, success = parse_value(remainder, member_slot, member_info, ignore_unknown, key, rename, float_handling=float_handling);
if !success return remainder, false;
remainder = trim_left(remainder, WHITESPACE_CHARS);
if !remainder || remainder[0] != #char "," break;
remainder = advance(remainder);
remainder = trim_left(remainder, WHITESPACE_CHARS);
}
if !remainder || remainder[0] != #char "}" return remainder, false;
remainder = advance(remainder);
return remainder, true;
}
parse_and_write_integer :: (info: *Type_Info_Integer, pointer: *void, string_value: string) -> bool, remainder: string {
if info.signed {
success, remainder := parse_and_write_integer(info, pointer, string_value, signed = true);
return success, remainder;
} else {
success, remainder := parse_and_write_integer(info, pointer, string_value, signed = false);
return success, remainder;
}
}
parse_and_write_integer :: (info: *Type_Info_Integer, pointer: *void, string_value: string, $signed: bool) -> bool, remainder: string {
#if signed {
Int_Type :: s64;
} else {
Int_Type :: u64;
}
int_value, int_success, remainder := string_to_int(string_value, T = Int_Type);
if !int_success {
#if signed {
log_error("Could not parse \"%\" as an integer.", string_value);
} else {
log_error("Could not parse \"%\" as an unsigned integer.", string_value);
}
return false, remainder;
}
valid, low, high := Reflection.range_check_and_store(int_value, info, pointer);
if !valid {
log_error("The value '%' is out of range. (It must be between % and %.)", int_value, low, high);
return false, remainder;
}
return true, remainder;
}
Reflection :: #import "Reflection";

59
modules/jaison/unicode_utils/module.jai Normal file
View File

@ -0,0 +1,59 @@
is_utf_cont :: inline (b: u8) -> bool {
return (b & 0xc0) == 0x80;
}
parse_unicode :: (str: string) -> result: u16, success: bool {
val, success, remainder := string_to_int(str, base = 16);
if !success || val > 0xFFFF || remainder.count return 0, false;
return xx val, true;
}
encode_utf8 :: (val: u16, result: *u8) -> len: u8 {
if val & 0xF800 {
result[0] = xx (0xE0 | ((val & 0xF000) >> 12));
result[1] = xx (0x80 | ((val & 0x0FC0) >> 6));
result[2] = xx (0x80 | (val & 0x003F));
return 3;
} else if val & 0x0F80 {
result[0] = xx (0xC0 | ((val & 0x0FC0) >> 6));
result[1] = xx (0x80 | (val & 0x003F));
return 2;
} else {
result[0] = xx (val & 0x7F);
return 1;
}
}
is_valid_utf8 :: (str: string) -> valid:bool {
for i: 0..str.count-1 {
cur := str[i];
if cur >= 0x80 {
// Must be between 0xc2 and 0xf4 inclusive to be valid
if (cur - 0xc2) > (0xf4 - 0xc2) return false;
if cur < 0xe0 { // 2-byte sequence
if i + 1 >= str.count || !is_utf_cont(str[i+1]) return false;
i += 1;
} else if cur < 0xf0 { // 3-byte sequence
if i + 2 >= str.count || !is_utf_cont(str[i+1]) || !is_utf_cont(str[i+2]) return false;
// Check for surrogate chars
if cur == 0xed && str[i+1] > 0x9f return false;
// ToDo: Check if total >= 0x800
// uc = ((uc & 0xf)<<12) | ((*str & 0x3f)<<6) | (str[1] & 0x3f);
i += 2;
} else { // 4-byte sequence
if i + 3 >= str.count || !is_utf_cont(str[i+1]) || !is_utf_cont(str[i+2]) || !is_utf_cont(str[i+3]) return false;
// Make sure its in valid range (0x10000 - 0x10ffff)
if cur == 0xf0 && str[i + 1] < 0x90 return false;
if cur == 0xf4 && str[i + 1] > 0x8f return false;
i += 3;
}
}
}
return true;
}
#scope_file
#import "Basic";

8
src/App.jai
View File

@ -12,6 +12,7 @@ USING_WINDOWS :: OS == .WINDOWS;
#load "test/catalog_tests.jai"; #load "test/catalog_tests.jai";
#load "test/ocr_tests.jai"; #load "test/ocr_tests.jai";
#load "test/labels_tests.jai"; #load "test/labels_tests.jai";
#load "test/import_tests.jai";
} }
#load "app/main.jai"; // entry point #load "app/main.jai"; // entry point
@ -22,6 +23,7 @@ USING_WINDOWS :: OS == .WINDOWS;
#load "app/stats_ui.jai"; #load "app/stats_ui.jai";
#load "app/background.jai"; #load "app/background.jai";
#load "app/labels_ui.jai"; #load "app/labels_ui.jai";
#load "app/import.jai";
#load "app/exit_shutdown.jai"; #load "app/exit_shutdown.jai";
#load "base/circle_buffer.jai"; #load "base/circle_buffer.jai";
@ -58,7 +60,10 @@ USING_WINDOWS :: OS == .WINDOWS;
Shared_Channel :: #import "Shared_Memory_Channel"; Shared_Channel :: #import "Shared_Memory_Channel";
#import "stb_image"; #import "stb_image";
#import "stb_image_resize"; #import "stb_image_resize";
#import "Capture_File"; // the .mcap format, copied from D:\SnagitIndex\SnagitConverter #import "Capture_File"; // the .mcap format, copied from D:\McapIndex\SnagitConverter
#import "Image_Decode"; // any image file to pixels: stb_image, and WIC on Windows
#import "Snagx"; // .snagx captures to .mcap, as SnagitConverter makes them
#import,dir "../modules/jaison/unicode_utils"; // is_valid_utf8
#import "OCR"; #import "OCR";
#import "Labels"; #import "Labels";
#import "Capture_Text"; #import "Capture_Text";
@ -185,6 +190,7 @@ App :: struct {
stats_and_exit: bool; // -stats: log the Statistics window's numbers once the catalog has loaded, then exit stats_and_exit: bool; // -stats: log the Statistics window's numbers once the catalog has loaded, then exit
start_search: string; // -search QUERY: the search box's text at startup start_search: string; // -search QUERY: the search box's text at startup
erase_and_exit: string; // -erase ocr|labels|all: erase that from the whole library once it has loaded, then exit erase_and_exit: string; // -erase ocr|labels|all: erase that from the whole library once it has loaded, then exit
import_source, import_dest: string; // -import SOURCE DEST: import SOURCE into the library folder DEST (app/import.jai), then exit
last_window_restore_time: Apollo_Time; last_window_restore_time: Apollo_Time;
imgui_context: *ImGui.ImGuiContext; imgui_context: *ImGui.ImGuiContext;

3
src/app/app_ui.jai
View File

@ -7,6 +7,7 @@ APP_MAIN_WINDOW_LABEL :: "Main";
// Once, after ImGui, the fonts, the theme and the settings are up and before the first frame. // Once, after ImGui, the fonts, the theme and the settings are up and before the first frame.
app_ui_startup :: () { app_ui_startup :: () {
viewer_startup(); viewer_startup();
import_startup();
ocr_ui_startup(); ocr_ui_startup();
labels_ui_startup(); labels_ui_startup();
} }
@ -18,6 +19,7 @@ app_ui_frame_start :: () {
// Once, on the shutdown thread after the window is asked to close (exit_shutdown.jai): save and // Once, on the shutdown thread after the window is asked to close (exit_shutdown.jai): save and
// stop background work here. Must not touch ImGui, since the main thread is still drawing frames. // stop background work here. Must not touch ImGui, since the main thread is still drawing frames.
app_ui_shutdown :: () { app_ui_shutdown :: () {
import_shutdown(); // before the viewer: it hands the viewer what it writes
viewer_shutdown(); viewer_shutdown();
ocr_ui_shutdown(); ocr_ui_shutdown();
labels_ui_shutdown(); // after the OCR thread and the background workers have stopped labels_ui_shutdown(); // after the OCR thread and the background workers have stopped
@ -30,4 +32,5 @@ app_main_window_ui :: () {
ocr_ui_frame(); ocr_ui_frame();
labels_ui_frame(); labels_ui_frame();
stats_ui_frame(); stats_ui_frame();
import_frame();
} }

5
src/app/background.jai
View File

@ -139,7 +139,10 @@ background_search_start :: () {
for viewer.search_terms request.terms[it_index] = .{it.field, copy_string(it.text)}; for viewer.search_terms request.terms[it_index] = .{it.field, copy_string(it.text)};
items: [..] Search_Item; items: [..] Search_Item;
array_reserve(*items, viewer_capture_count()); array_reserve(*items, viewer_capture_count());
for * viewer.captures if !it.removed array_add(*items, .{index = cast(s32) it_index, entry = it.catalog, name = it.name}); for * viewer.captures {
if it.removed || (viewer.folder_filter && it.folder != viewer.folder_filter) continue; // the folder chosen, or all
array_add(*items, .{index = cast(s32) it_index, entry = it.catalog, name = it.name});
}
request.items = items; request.items = items;
// like: phrases need their vectors first, from the label model's text encoder. // like: phrases need their vectors first, from the label model's text encoder.

22
src/app/catalog.jai
View File

@ -6,7 +6,8 @@
Catalog_Entry :: struct { Catalog_Entry :: struct {
date: string; // CaptureDate, "2026-10-07 08:11:15" (sorts as text); "" when unknown date: string; // CaptureDate, "2026-10-07 08:11:15" (sorts as text); "" when unknown
title, url, app: string; // WindowName, WebURL, AppName title, url, app: string; // WindowName, WebURL, AppName (an imported photo's: its file name, "", its camera)
folder: string; // the library folder it is in (app/viewer.jai); "" at the top
notes: string; // the annotations' text, one per line notes: string; // the annotations' text, one per line
ocr: [..] Mcap_Ocr_Section; // one per model that read it ocr: [..] Mcap_Ocr_Section; // one per model that read it
labels: [..] Mcap_Label; // the label model's (LABELS_MODEL) best labels, best first labels: [..] Mcap_Label; // the label model's (LABELS_MODEL) best labels, best first
@ -14,14 +15,14 @@ Catalog_Entry :: struct {
embedding: [] u16; // the label model's image vector, 16-bit floats; empty when not labelled embedding: [] u16; // the label model's image vector, 16-bit floats; empty when not labelled
// Lowercased copies, for matching without case. // Lowercased copies, for matching without case.
find_title, find_url, find_app, find_notes, find_ocr, find_labels: string; find_title, find_url, find_app, find_notes, find_ocr, find_labels, find_folder: string;
} }
// Everything the entry holds is from the default allocator. // Everything the entry holds is from the default allocator.
catalog_entry_free :: (entry: *Catalog_Entry) { catalog_entry_free :: (entry: *Catalog_Entry) {
if !entry return; if !entry return;
push_allocator(context.default_allocator); push_allocator(context.default_allocator);
free(entry.date); free(entry.title); free(entry.url); free(entry.app); free(entry.notes); free(entry.date); free(entry.title); free(entry.url); free(entry.app); free(entry.notes); free(entry.folder);
for entry.ocr { for entry.ocr {
free(it.model); free(it.engine); free(it.image); free(it.date); free(it.model); free(it.engine); free(it.image); free(it.date);
for line: it.lines free(line.text); for line: it.lines free(line.text);
@ -32,7 +33,7 @@ catalog_entry_free :: (entry: *Catalog_Entry) {
array_free(entry.labels); array_free(entry.labels);
free(entry.label_list); free(entry.label_list);
array_free(entry.embedding); array_free(entry.embedding);
free(entry.find_title); free(entry.find_url); free(entry.find_app); free(entry.find_notes); free(entry.find_ocr); free(entry.find_labels); free(entry.find_title); free(entry.find_url); free(entry.find_app); free(entry.find_notes); free(entry.find_ocr); free(entry.find_labels); free(entry.find_folder);
free(entry); free(entry);
} }
@ -100,6 +101,13 @@ catalog_load :: (path: string) -> *Catalog_Entry, error: string {
if half_ok entry.embedding = half; if half_ok entry.embedding = half;
} }
library := viewer_folder();
if path.count > library.count + 1 && begins_with_nocase(path, library) {
folder := path_strip_filename(advance(path, library.count + 1));
while folder.count && (folder[folder.count - 1] == #char "/" || folder[folder.count - 1] == #char "\\") folder.count -= 1;
entry.folder = copy_string(folder);
}
entry.find_folder = to_lower_copy(entry.folder);
entry.find_title = to_lower_copy(entry.title); entry.find_title = to_lower_copy(entry.title);
entry.find_url = to_lower_copy(entry.url); entry.find_url = to_lower_copy(entry.url);
entry.find_app = to_lower_copy(entry.app); entry.find_app = to_lower_copy(entry.app);
@ -130,14 +138,14 @@ catalog_section :: (entry: *Catalog_Entry, model: string) -> *Mcap_Ocr_Section {
// LIKE is by meaning: the captures whose image is nearest the phrase, ranked by the label model // LIKE is by meaning: the captures whose image is nearest the phrase, ranked by the label model
// (app/background.jai does the ranking); here it only asks for a labelled capture. // (app/background.jai does the ranking); here it only asks for a labelled capture.
Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; } Search_Field :: enum u8 { ANY; OCR; TITLE; URL; APP; NOTE; LABEL; LIKE; FOLDER; }
Search_Term :: struct { Search_Term :: struct {
field: Search_Field; field: Search_Field;
text: string; // lowercased text: string; // lowercased
} }
// Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like:. Temp. // Words, "quoted phrases", and either with a field prefix: ocr: title: url: app: note: label: like: folder:. Temp.
search_parse :: (query: string) -> [..] Search_Term { search_parse :: (query: string) -> [..] Search_Term {
terms: [..] Search_Term; terms: [..] Search_Term;
terms.allocator = temp; terms.allocator = temp;
@ -157,6 +165,7 @@ search_parse :: (query: string) -> [..] Search_Term {
case "note"; field = .NOTE; case "note"; field = .NOTE;
case "label"; field = .LABEL; case "label"; field = .LABEL;
case "like"; field = .LIKE; case "like"; field = .LIKE;
case "folder"; field = .FOLDER;
} }
if field != .ANY { term.field = field; rest = advance(rest, colon + 1); } // "http:..." and the like stay words if field != .ANY { term.field = field; rest = advance(rest, colon + 1); } // "http:..." and the like stay words
} }
@ -192,6 +201,7 @@ search_matches :: (entry: *Catalog_Entry, terms: [] Search_Term) -> bool {
case .NOTE; found = contains(entry.find_notes, it.text); case .NOTE; found = contains(entry.find_notes, it.text);
case .LABEL; found = contains(entry.find_labels, it.text); case .LABEL; found = contains(entry.find_labels, it.text);
case .LIKE; found = entry.embedding.count > 0; case .LIKE; found = entry.embedding.count > 0;
case .FOLDER; found = contains(entry.find_folder, it.text);
} }
if !found return false; if !found return false;
} }

918
src/app/import.jai Normal file
View File

@ -0,0 +1,918 @@
// Importing a folder of screenshots and photos into the library: every image under it (PNG, JPEG, HEIC,
// AVIF, WebP, GIF, BMP, TIFF, camera RAW) and every Snagit .snagx becomes an .mcap in a folder of its own
// in the library, one folder for each place imported from (D:\McapIndex\mcap\SnagitLibraryWork, ...).
// The files imported from are only read, never changed.
//
// An image's .mcap holds the file's bytes exactly as they were (its original), a preview turned the
// right way up, and Capture Text with metadata.json (CaptureDate, WindowName as the file's name,
// AppName as the camera or phone) and import.json (where it came from, its EXIF). A .snagx becomes the
// same .mcap SnagitConverter makes of it (modules/Snagx).
//
// Each image is stored once in the whole library: the library index (IMPORT_INDEX_NAME, in the library
// folder) holds a hash of every .mcap's original image, made once for the files already there and kept
// up to date after that; an image whose hash is in it is reported as a duplicate, with where it is.
//
// The import runs on a thread of its own, which lists the files, brings the index up to date, and then
// imports with IMPORT_WORKERS threads. New captures go into the grid as they are written; when it ends,
// the OCR scan reads and labels them, if asked to.
IMPORT_WINDOW :: "Import";
IMPORT_INDEX_NAME :: ".import-index"; // in the library folder: path, size, time and hash of each .mcap
IMPORT_SOURCES_NAME :: ".import-sources"; // in the library folder: the folder each library folder was last imported from
IMPORT_MAX_FILE :: 1024 * 1024 * 1024; // larger files are not read
IMPORT_NOTES_MAX :: 5000; // failures and duplicates listed; the rest are only counted
IMPORT_SAVE_EVERY :: 250; // captures imported between saves of the index
// Not images, by extension: skipped without being read. Anything else is read and looked at.
IMPORT_VIDEO_EXTENSIONS :: string.["mp4", "mov", "m4v", "avi", "mkv", "webm", "3gp", "3g2", "wmv", "mts", "m2ts", "mpg", "mpeg", "flv"];
IMPORT_OTHER_EXTENSIONS :: string.["aae", "xmp", "json", "txt", "ini", "db", "pdf", "zip", "lnk", "url", "html", "htm", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "mp3", "m4a", "wav", "aac", "exe", "dll", "mcap", "tmp", "thm", "lrv", "mpo"];
Import_Phase :: enum u8 { IDLE; LISTING; INDEXING; IMPORTING; DONE; }
Import_Outcome :: enum u8 { IMPORTED; DUPLICATE; SKIPPED; FAILED; }
// A file that failed, or was a duplicate, and why or of what. Heap.
Import_Note :: struct {
path: string;
text: string;
}
Index_Record :: struct {
size: s64;
modified: s64; // nanoseconds
hash: u64;
}
importer: struct {
window_open: bool;
source_buffer: [1024] u8;
dest_buffer: [256] u8;
dest_edited: bool; // typed in, so a new source does not replace it
read_after := true; // read and label what was imported, with the OCR scan
sources: Table(string, string); // library folder name -> the folder last imported into it; main thread
// Set by the main thread before the thread starts; read-only while it runs.
source: string;
dest_name: string;
dest_folder: string;
library: string;
thread: Thread;
thread_started: bool;
running: bool; // set by the main thread, cleared by the import thread when it ends
phase: Import_Phase;
paused, cancel: bool;
start: Apollo_Time;
paused_seconds: float64;
pause_start: Apollo_Time;
elapsed_seconds: float64; // once it has ended
files: [..] string; // what LISTING found; the import thread's
listed: s64;
index_total, index_done: s64;
next: s64; // the next file a worker takes
done, imported, duplicates, skipped, failed: s64;
error: string; // why the import could not run at all
// Under mutex.
mutex: Mutex;
failures: [..] Import_Note;
duplicate_notes: [..] Import_Note;
skipped_kinds: Table(string, s64); // "mp4", "video", "unknown" ... -> count
new_paths: [..] string; // written, for the main thread to put in the grid
by_hash: Table(u64, string); // every original in the library -> its path, relative to the library
records: Table(string, Index_Record);
reserved_names: Table(string, bool); // output paths being written
since_save: s64;
// Main thread.
new_indices: [..] s32; // the captures this import added, for the scan after it
scan_pending: bool;
flag_started: bool; // -import has started its import
}
// ---------------------------------------------------------------- startup and shutdown
import_startup :: () {
init(*importer.mutex, "import");
importer.files.allocator = context.default_allocator;
importer.failures.allocator = context.default_allocator;
importer.duplicate_notes.allocator = context.default_allocator;
importer.new_paths.allocator = context.default_allocator;
importer.new_indices.allocator = context.default_allocator;
import_sources_load();
}
import_shutdown :: () {
atomic_swap(*importer.cancel, true);
atomic_swap(*importer.paused, false);
if importer.thread_started thread_is_done(*importer.thread, -1);
}
// ---------------------------------------------------------------- starting
// Starts importing source into the library folder dest_name; false, with a toast, when it cannot.
import_start :: (source: string, dest_name: string) -> bool {
if importer.running return false;
source_path := trim(source);
while source_path.count > 3 && (source_path[source_path.count - 1] == #char "/" || source_path[source_path.count - 1] == #char "\\") source_path.count -= 1;
dest := trim(dest_name);
if !source_path.count || !dest.count { toast_show(.Error, "Choose a folder to import and a library folder to import it into"); return false; }
if !import_name_is_valid(dest) { toast_show(.Error, "\"%\" cannot be a folder's name", dest); return false; }
if !file_exists(source_path) { toast_show(.Error, "% does not exist", source_path); return false; }
push_allocator(context.default_allocator);
free(importer.source); free(importer.dest_name); free(importer.dest_folder); free(importer.library); free(importer.error);
importer.source = copy_string(source_path);
path_overwrite_separators(importer.source, #char "/");
importer.dest_name = copy_string(dest);
importer.library = copy_string(viewer_folder());
importer.dest_folder = sprint("%/%", importer.library, dest);
importer.error = "";
// The library is never imported into itself.
lower_source := to_lower_copy(importer.source,, temp);
lower_library := to_lower_copy(importer.library,, temp);
path_overwrite_separators(lower_library, #char "/");
if lower_source == lower_library || begins_with(lower_source, tprint("%/", lower_library)) || begins_with(lower_library, tprint("%/", lower_source)) {
toast_show(.Error, "The folder to import holds the library, or is inside it");
return false;
}
for importer.files free(it);
array_reset(*importer.files);
lock(*importer.mutex);
for importer.failures { free(it.path); free(it.text); }
for importer.duplicate_notes { free(it.path); free(it.text); }
array_reset(*importer.failures);
array_reset(*importer.duplicate_notes);
for _, key: importer.skipped_kinds free(key);
table_reset(*importer.skipped_kinds);
unlock(*importer.mutex);
array_reset(*importer.new_indices);
importer.listed = 0; importer.index_total = 0; importer.index_done = 0;
importer.next = 0; importer.done = 0; importer.imported = 0; importer.duplicates = 0; importer.skipped = 0; importer.failed = 0;
importer.paused = false; importer.cancel = false;
importer.paused_seconds = 0; importer.elapsed_seconds = 0;
importer.phase = .LISTING;
importer.start = current_time_monotonic();
importer.scan_pending = false;
table_set(*importer.sources, copy_string(dest), copy_string(importer.source));
import_sources_save();
importer.running = true;
if importer.thread_started thread_deinit(*importer.thread);
importer.thread_started = thread_init(*importer.thread, import_thread_proc, temporary_storage_size = 4 * 1024 * 1024, name = "import");
if !importer.thread_started { importer.running = false; importer.phase = .IDLE; toast_show(.Error, "Could not start the import"); return false; }
thread_start(*importer.thread);
log("[Import] Importing % into %.", importer.source, importer.dest_folder);
return true;
}
import_name_is_valid :: (name: string) -> bool {
if !name.count || name == "." || name == ".." || name[0] == #char "." return false;
for name if it < 32 || it == #char "/" || it == #char "\\" || it == #char ":" || it == #char "*" || it == #char "?" || it == #char "\"" || it == #char "<" || it == #char ">" || it == #char "|" return false;
return name[name.count - 1] != #char " " && name[name.count - 1] != #char ".";
}
// ---------------------------------------------------------------- the import thread
import_thread_proc :: (thread: *Thread) -> s64 {
context.allocator = context.default_allocator;
defer {
importer.elapsed_seconds = import_elapsed();
atomic_swap(*importer.phase, Import_Phase.DONE);
atomic_swap(*importer.running, false);
}
// The files to import, sorted so a second run goes through them in the same order.
Found :: struct { files: *[..] string; }
found := Found.{*importer.files};
visited := visit_files(importer.source, true, *found, (info: *File_Visit_Info, found: *Found) {
if info.is_directory {
name := info.short_name;
if name.count && (name[0] == #char "." || name[0] == #char "$" || name == "System Volume Information") info.descend_into_directory = false;
return;
}
array_add(found.files, copy_string(info.full_name));
atomic_add(*importer.listed, 1);
}, visit_files = true, visit_directories = true, follow_directory_symlinks = false);
if !visited || importer.cancel {
if !visited importer.error = sprint("% cannot be read", importer.source);
return 0;
}
quick_sort(importer.files, compare);
if !make_directory_if_it_does_not_exist(importer.dest_folder, recursive = true) {
importer.error = sprint("% cannot be made", importer.dest_folder);
return 0;
}
atomic_swap(*importer.phase, Import_Phase.INDEXING);
if !import_index_update() || importer.cancel return 0;
atomic_swap(*importer.phase, Import_Phase.IMPORTING);
import_run_workers(import_worker);
import_index_save();
log("[Import] % into %: % imported, % duplicates, % skipped, % failed, of % files, in %%.", importer.source, importer.dest_name,
importer.imported, importer.duplicates, importer.skipped, importer.failed, importer.files.count, ocr_duration_text(import_elapsed()),
ifx importer.cancel then " (cancelled)" else "");
return 0;
}
IMPORT_WORKERS :: 0; // 0: worked out from the processors
// Runs proc on several threads until it returns false on each, then waits for them.
import_run_workers :: (proc: (thread: *Thread) -> s64) {
count := clamp(get_number_of_processors(.ALL_LOGICAL) / 2, 2, 8);
if IMPORT_WORKERS count = IMPORT_WORKERS;
threads := NewArray(count, Thread,, context.default_allocator);
defer array_free(threads,, context.default_allocator);
started := 0;
for * threads {
// Reading an image and its .mcap takes about three times its size; temporary storage grows past this as needed.
if !thread_init(it, proc, temporary_storage_size = 16 * 1024 * 1024, name = tprint("import %", it_index)) break;
thread_start(it);
started += 1;
}
if !started proc(null); // on this thread, then
for 0..started - 1 {
thread_is_done(*threads[it], -1);
thread_deinit(*threads[it]);
}
}
import_worker :: (thread: *Thread) -> s64 {
context.allocator = context.default_allocator;
while true {
while importer.paused && !importer.cancel sleep_milliseconds(100);
if importer.cancel break;
n := atomic_add(*importer.next, 1);
if n >= importer.files.count break;
path := importer.files[n];
outcome, text, new_path := import_file(path);
lock(*importer.mutex);
if outcome == {
case .IMPORTED;
importer.imported += 1;
array_add(*importer.new_paths, copy_string(new_path));
importer.since_save += 1;
case .DUPLICATE;
importer.duplicates += 1;
if importer.duplicate_notes.count < IMPORT_NOTES_MAX array_add(*importer.duplicate_notes, .{copy_string(path), copy_string(text)});
case .SKIPPED;
importer.skipped += 1;
found, count := table_find(*importer.skipped_kinds, text);
if found table_set(*importer.skipped_kinds, text, count + 1);
else table_set(*importer.skipped_kinds, copy_string(text), 1);
case .FAILED;
importer.failed += 1;
if importer.failures.count < IMPORT_NOTES_MAX array_add(*importer.failures, .{copy_string(path), copy_string(text)});
}
save := importer.since_save >= IMPORT_SAVE_EVERY;
if save importer.since_save = 0;
unlock(*importer.mutex);
if outcome == .FAILED log_error("[Import] % %", path, text);
atomic_add(*importer.done, 1);
if save import_index_save();
reset_temporary_storage();
}
return 0;
}
// Imports one file. The text says why it failed or was skipped (what kind of file: "mp4", "video"),
// or, for a duplicate, where the image already is; new_path is where an import went. Temp.
import_file :: (path: string) -> Import_Outcome, text: string, new_path: string {
name := path_filename(path);
extension := to_lower_copy(path_extension(name),, temp);
if array_find(IMPORT_VIDEO_EXTENSIONS, extension) || array_find(IMPORT_OTHER_EXTENSIONS, extension) return .SKIPPED, extension, "";
modified, size, stat_ok := file_modtime_and_size(path);
if !stat_ok return .FAILED, "cannot be read", "";
if size == 0 return .SKIPPED, "empty", "";
if size > IMPORT_MAX_FILE return .FAILED, tprint("is % MB, larger than the % MB an import reads", size / (1024 * 1024), IMPORT_MAX_FILE / (1024 * 1024)), "";
// A look at the first bytes, before reading what may not be an image at all.
head := import_read_head(path, 64);
kind := image_kind_from_name(image_kind(head), name);
is_snagx := extension == "snagx" && head.count >= 4 && memcmp(head.data, "PK\x03\x04".data, 4) == 0;
if !is_snagx {
if kind == .VIDEO return .SKIPPED, "video", "";
if kind == .UNKNOWN return .SKIPPED, ifx extension then extension else "no extension", "";
}
data, read_ok := read_entire_file(path, log_errors = false,, temp);
if !read_ok return .FAILED, "cannot be read", "";
bytes := cast([] u8) data;
m: Mcap;
out_bytes: [] u8;
hash: u64;
out_name: string;
if is_snagx {
ok: bool;
error: string;
m, out_bytes, ok, error = snagx_to_mcap(data);
if !ok return .FAILED, error, "";
hash = import_hash(m.original);
out_name = tprint("%.%", path_strip_extension(name), MCAP_EXTENSION);
} else {
hash = import_hash(bytes);
}
// Once in the whole library: the hash is held for this file until it is written, or let go.
lock(*importer.mutex);
found, existing := table_find(*importer.by_hash, hash);
if found {
where := copy_string(existing,, temp);
unlock(*importer.mutex);
return .DUPLICATE, ifx where then where else "(another file in this import)", "";
}
table_set(*importer.by_hash, hash, ""); // "" while it is being written
unlock(*importer.mutex);
let_go := true;
defer if let_go {
lock(*importer.mutex);
table_remove(*importer.by_hash, hash);
unlock(*importer.mutex);
}
if !is_snagx {
ok: bool;
error: string;
m, out_name, ok, error = import_image_mcap(path, name, bytes, kind, hash, modified);
if !ok return .FAILED, error, "";
serializer, serialized, serialize_error := mcap_serialize(m,, temp);
if !serialized return .FAILED, serialize_error, "";
out_bytes = cast([] u8) string.{serializer.count, serializer.data};
}
// Its name in the library folder: another file there by that name (a different image) gets _2, _3 ...
lock(*importer.mutex);
base := path_strip_extension(out_name);
out_path := tprint("%/%", importer.dest_folder, out_name);
number := 2;
while file_exists(out_path) || table_contains(*importer.reserved_names, out_path) {
out_path = tprint("%/%_%.%", importer.dest_folder, base, number, MCAP_EXTENSION);
number += 1;
}
out_path = copy_string(out_path);
table_set(*importer.reserved_names, out_path, true);
unlock(*importer.mutex);
written, write_error := mcap_write(out_path, out_bytes);
out_modified, out_size, _ := file_modtime_and_size(out_path);
relative := import_relative(out_path);
lock(*importer.mutex);
table_remove(*importer.reserved_names, out_path);
if written {
table_set(*importer.by_hash, hash, copy_string(relative));
import_record_set(relative, .{size = out_size, modified = to_nanoseconds(out_modified), hash = hash});
}
unlock(*importer.mutex);
free(out_path);
if !written return .FAILED, write_error, "";
let_go = false;
return .IMPORTED, "", tprint("%/%", importer.library, relative);
}
// An image file as an .mcap (temp) and the name it gets: the date it was taken and its hash.
import_image_mcap :: (path: string, name: string, bytes: [] u8, kind: Image_Kind, hash: u64, modified: Apollo_Time) -> Mcap, name: string, ok: bool, error: string {
m: Mcap;
image, decoded, decode_error := image_decode(bytes, kind);
if !decoded return m, "", false, decode_error;
defer image_free(*image);
png, preview_w, preview_h, preview_ok, preview_error := image_preview_png(image, MCAP_PREVIEW_MAX_SIDE);
if !preview_ok return m, "", false, preview_error;
exif := exif_read(bytes, kind);
// When it was taken: the camera's clock, else a date in the file's name, else when the file was last changed.
date, date_from := import_exif_date(exif.date_original), "EXIF";
if !date { date = import_name_date(name); date_from = "file name"; }
if !date { date = import_calendar_text(to_calendar(modified, .LOCAL)); date_from = "modified time"; }
device := import_text(exif.model);
make := import_text(exif.make);
if make && !begins_with_nocase(device, make) device = tprint("% %", make, device);
device = trim(device);
doc: Capture_Document;
capture_document_init(*doc, temp);
doc.comment = tprint("Imported by % % from %.", APP_NAME, VERSION, import_text(path_filename(path)));
add :: (doc: *Capture_Document, file: *Capture_File, key: string, value: string) {
capture_object_add(doc, *file.root, key, capture_string(doc, value));
}
add_number :: (doc: *Capture_Document, file: *Capture_File, key: string, value: float64) {
capture_object_add(doc, *file.root, key, capture_number(value));
}
metadata := capture_add_file(*doc, "metadata.json");
add(*doc, metadata, "AppName", device);
add(*doc, metadata, "CaptureDate", date);
add(*doc, metadata, "WindowName", import_text(name));
info := capture_add_file(*doc, "import.json");
add(*doc, info, "SourcePath", import_text(path));
add_number(*doc, info, "SourceSize", xx bytes.count);
add(*doc, info, "SourceModified", import_calendar_text(to_calendar(modified, .LOCAL)));
add(*doc, info, "ContentHash", import_hash_text(hash));
add(*doc, info, "Format", image_kind_name(kind));
add_number(*doc, info, "Width", xx image.width);
add_number(*doc, info, "Height", xx image.height);
add(*doc, info, "DateFrom", date_from);
add(*doc, info, "ImportDate", import_calendar_text(to_calendar(current_time_consensus(), .LOCAL)));
if exif.found {
add_number(*doc, info, "Orientation", xx exif.orientation);
if make add(*doc, info, "CameraMake", make);
if exif.model add(*doc, info, "CameraModel", import_text(exif.model));
if exif.software add(*doc, info, "Software", import_text(exif.software));
if exif.offset_original add(*doc, info, "TimeZoneOffset", import_text(exif.offset_original));
if exif.has_gps {
add_number(*doc, info, "Latitude", exif.latitude);
add_number(*doc, info, "Longitude", exif.longitude);
if exif.altitude != 0 add_number(*doc, info, "Altitude", exif.altitude);
}
}
text := capture_text_write(*doc);
back: Capture_Document;
capture_document_init(*back, temp);
if !capture_text_parse(*back, path, text) || back.mistakes || back.files.count != 2 return m, "", false, "its Capture Text does not read back cleanly (see above)";
m.text = text;
m.preview = .{width = cast(u16) preview_w, height = cast(u16) preview_h, png = png};
m.original_name = name;
m.original = bytes;
m.rendered_kind = .NONE;
// 2024-05-12_10-15-30_1a2b3c4d.mcap: newest last, as Snagit names its captures.
stamp := replace(replace(date, " ", "_",, temp), ":", "-",, temp);
out_name := tprint("%_%.%", stamp, slice(import_hash_text(hash), 0, 8), MCAP_EXTENSION);
return m, out_name, true, "";
}
import_hash :: (bytes: [] u8) -> u64 {
return fnv1a_hash(bytes.data, bytes.count);
}
import_hash_text :: (hash: u64) -> string {
return tprint("%", formatInt(hash, base = 16, minimum_digits = 16));
}
// "2024-05-12 10:15:30", as Snagit writes CaptureDate. Temp.
import_calendar_text :: (c: Calendar_Time) -> string {
two :: (n: s64) -> string { return tprint("%", formatInt(n, minimum_digits = 2)); } // a string: a FormatInt would point at n
return tprint("%-%-% %:%:%", c.year, two(c.month_starting_at_0 + 1), two(c.day_of_month_starting_at_0 + 1), two(c.hour), two(c.minute), two(c.second));
}
// EXIF's "2024:05:12 10:15:30" as "2024-05-12 10:15:30"; "" when it is not a date (cameras with no
// clock set write zeros or blanks).
import_exif_date :: (text: string) -> string {
if text.count < 19 return "";
digits := "0000:00:00 00:00:00";
for 0..18 {
if digits[it] == #char "0" { if !is_digit(text[it]) return ""; }
else if text[it] != #char ":" && text[it] != #char "-" && text[it] != #char " " && text[it] != #char "T" return "";
}
year := string_to_int(slice(text, 0, 4));
month := string_to_int(slice(text, 5, 2));
day := string_to_int(slice(text, 8, 2));
if year < 1900 || month < 1 || month > 12 || day < 1 || day > 31 return "";
return tprint("%-%-% %", slice(text, 0, 4), slice(text, 5, 2), slice(text, 8, 2), slice(text, 11, 8));
}
// A date, and a time when there is one, in a file's name, as phones and screenshot tools write them:
// IMG_20240512_101530, PXL_20240512_101530123, Screenshot_20240512-101530, IMG-20240512-WA0001,
// Screenshot 2024-05-12 at 10.15.30, Screenshot_2024-05-12-10-15-30, 2024-05-12_10-15-30. "" for none.
import_name_date :: (name: string) -> string {
digits_at :: (s: string, at: s64, count: s64) -> s64 {
if at < 0 || at + count > s.count return -1;
for at..at + count - 1 if !is_digit(s[it]) return -1;
return string_to_int(slice(s, at, count));
}
separator :: (s: string, at: s64) -> bool {
return at < s.count && (s[at] == #char "-" || s[at] == #char "_" || s[at] == #char "." || s[at] == #char " " || s[at] == #char ":");
}
for start: 0..name.count - 8 {
if start > 0 && is_digit(name[start - 1]) continue; // the year starts a run of digits
year := digits_at(name, start, 4);
if year < 1990 || year > 2099 continue;
at := start + 4;
if separator(name, at) at += 1;
month := digits_at(name, at, 2);
at += 2;
if separator(name, at) at += 1;
day := digits_at(name, at, 2);
at += 2;
if month < 1 || month > 12 || day < 1 || day > 31 continue;
if at < name.count && is_digit(name[at]) && !(at + 6 <= name.count && digits_at(name, at, 6) >= 0) continue;
// The time: after a separator, or " at ", or nothing.
hour, minute, second := 0, 0, 0;
t := at;
if begins_with(advance(name, t), " at ") t += 4;
else if t < name.count && (separator(name, t) || name[t] == #char "T") t += 1;
h := digits_at(name, t, 2);
if h >= 0 && h < 24 {
t2 := t + 2;
if separator(name, t2) t2 += 1;
m := digits_at(name, t2, 2);
t3 := t2 + 2;
if separator(name, t3) t3 += 1;
s := digits_at(name, t3, 2);
if m >= 0 && m < 60 && s >= 0 && s < 60 { hour, minute, second = h, m, s; }
}
two :: (n: s64) -> string { return tprint("%", formatInt(n, minimum_digits = 2)); } // a string: a FormatInt would point at n
return tprint("%-%-% %:%:%", year, two(month), two(day), two(hour), two(minute), two(second));
}
return "";
}
// Text from a file (a name, an EXIF field) made safe for Capture Text: control characters as spaces,
// and bytes that are not UTF-8 (EXIF written in Latin-1) as ?. Temp.
import_text :: (text: string) -> string {
if !text return "";
out := copy_string(text,, temp);
for * out if it.* < 32 || it.* == 127 it.* = #char " ";
if !is_valid_utf8(out) for * out if it.* >= 128 it.* = #char "?";
return trim(out);
}
import_read_head :: (path: string, count: s64) -> [] u8 {
file, ok := file_open(path, log_errors = false);
if !ok return .[];
defer file_close(*file);
head := NewArray(count, u8,, temp);
read_ok, read := file_read(file, head.data, count);
if !read_ok return .[];
head.count = read;
return head;
}
// path inside the library, as the index keeps it: "SnagitLibraryWork/2024-05-12_10-15-30.mcap".
import_relative :: (path: string) -> string {
library := importer.library;
if path.count > library.count + 1 && begins_with_nocase(path, library) && (path[library.count] == #char "/" || path[library.count] == #char "\\") return advance(path, library.count + 1);
return path;
}
import_elapsed :: () -> float64 {
elapsed := to_float64_seconds(current_time_monotonic() - importer.start) - importer.paused_seconds;
if importer.paused elapsed -= to_float64_seconds(current_time_monotonic() - importer.pause_start);
return elapsed;
}
// ---------------------------------------------------------------- the library index
// Brings the index up to date with the library: files it does not know, or that changed, have their
// original hashed; files gone are dropped. False, with importer.error, when it cannot.
import_index_update :: () -> bool {
lock(*importer.mutex);
for _, key: importer.records free(key);
table_reset(*importer.records);
for value: importer.by_hash free(value);
table_reset(*importer.by_hash);
unlock(*importer.mutex);
// What the index says.
index_path := tprint("%/%", importer.library, IMPORT_INDEX_NAME);
data, read_ok := read_entire_file(index_path, log_errors = false,, temp);
if read_ok {
lines := split(data, "\n",, temp);
if lines.count && trim(lines[0]) == "@version 1" {
for 1..lines.count - 1 {
parts := split(trim_right(lines[it], "\r"), "\t",, temp);
if parts.count != 4 continue;
hash, hash_ok := string_to_int(parts[0], 16, u64);
size, size_ok := string_to_int(parts[1]);
modified, modified_ok := string_to_int(parts[2]);
if hash_ok && size_ok && modified_ok import_record_set(parts[3], .{size, modified, hash});
}
}
}
// What the library holds.
stale: [..] Stale_File;
stale.allocator = temp;
present: Table(string, bool);
present.allocator = temp;
for file_list(importer.library, recursive = true,, temp) {
if !ends_with_nocase(it, tprint(".%", MCAP_EXTENSION)) continue;
relative := import_relative(it);
table_set(*present, relative, true);
modified, size, ok := file_modtime_and_size(it);
if !ok continue;
nanoseconds := to_nanoseconds(modified);
found, record := table_find(*importer.records, relative);
if found && record.size == size && record.modified == nanoseconds continue;
array_add(*stale, .{copy_string(relative,, temp), size, nanoseconds});
}
gone: [..] string;
gone.allocator = temp;
for _, key: importer.records if !table_contains(*present, key) array_add(*gone, key);
for gone {
table_remove(*importer.records, it);
free(it,, context.default_allocator);
}
// The stale ones hashed, on the workers.
index_stale = stale;
importer.index_total = stale.count;
importer.index_done = 0;
index_next = 0;
if stale.count log("[Import] Indexing % of the library's .mcap files.", stale.count);
import_run_workers(import_index_worker);
if importer.cancel return false;
lock(*importer.mutex);
for record, relative: importer.records {
if !table_contains(*importer.by_hash, record.hash) table_set(*importer.by_hash, record.hash, copy_string(relative));
}
unlock(*importer.mutex);
import_index_save();
return true;
}
import_index_worker :: (thread: *Thread) -> s64 {
context.allocator = context.default_allocator;
while true {
while importer.paused && !importer.cancel sleep_milliseconds(100);
if importer.cancel break;
n := atomic_add(*index_next, 1);
if n >= index_stale.count break;
item := index_stale[n];
path := tprint("%/%", importer.library, item.relative);
data, ok := read_entire_file(path, log_errors = false,, temp);
if ok {
m, parsed, error := mcap_parse(cast([] u8) data);
if parsed {
hash := import_hash(m.original);
lock(*importer.mutex);
import_record_set(item.relative, .{item.size, item.modified, hash});
unlock(*importer.mutex);
} else {
log_error("[Import] % is not a readable .mcap (%); it is left out of the index.", path, error);
}
}
atomic_add(*importer.index_done, 1);
reset_temporary_storage();
}
return 0;
}
import_record_set :: (relative: string, record: Index_Record) {
pointer := table_find_pointer(*importer.records, relative);
if pointer { pointer.* = record; return; }
table_set(*importer.records, copy_string(relative,, context.default_allocator), record);
}
// Writes the index beside itself and swaps it in.
import_index_save :: () {
builder: String_Builder;
builder.allocator = temp;
append(*builder, "@version 1\n");
lock(*importer.mutex);
for record, relative: importer.records {
print_to_builder(*builder, "%\t%\t%\t%\n", import_hash_text(record.hash), record.size, record.modified, relative);
}
unlock(*importer.mutex);
path := tprint("%/%", importer.library, IMPORT_INDEX_NAME);
temp_path := tprint("%.tmp", path);
if !write_entire_file(temp_path, *builder) || !atomic_file_replace_or_just_rename_if_destination_file_does_not_exist(temp_path, path) {
log_error("[Import] Could not write %.", path);
}
}
// ---------------------------------------------------------------- where each library folder came from
import_sources_load :: () {
data, ok := read_entire_file(tprint("%/%", viewer_folder(), IMPORT_SOURCES_NAME), log_errors = false,, temp);
if !ok return;
for split(data, "\n",, temp) {
parts := split(trim_right(it, "\r"), "\t",, temp);
if parts.count != 2 || !parts[0].count continue;
table_set(*importer.sources, copy_string(parts[0],, context.default_allocator), copy_string(parts[1],, context.default_allocator));
}
}
import_sources_save :: () {
builder: String_Builder;
builder.allocator = temp;
for source, dest: importer.sources print_to_builder(*builder, "%\t%\n", dest, source);
write_entire_file(tprint("%/%", viewer_folder(), IMPORT_SOURCES_NAME), *builder);
}
// The library folder a source goes into: the one it went into last time, else its own folder's name.
import_dest_for :: (source: string) -> string {
normal := copy_string(trim(source),, temp);
path_overwrite_separators(normal, #char "/");
while normal.count > 3 && normal[normal.count - 1] == #char "/" normal.count -= 1;
for value, key: importer.sources if to_lower_copy(value,, temp) == to_lower_copy(normal,, temp) return key;
name := path_filename(normal);
return ifx import_name_is_valid(name) then name else "";
}
// ---------------------------------------------------------------- each frame
import_frame :: () { using ImGui;
import_take_new_paths();
if app.import_source import_flag_frame();
// Ended: the scan of what came in, when asked for.
if !importer.running && importer.thread_started && importer.phase == .DONE {
thread_is_done(*importer.thread, -1);
thread_deinit(*importer.thread);
importer.thread_started = false;
if importer.error toast_show(.Error, "Import: %", importer.error);
else toast_show(.Success, "Imported % into % (% duplicates, % skipped, % failed)", importer.imported, importer.dest_name, importer.duplicates, importer.skipped, importer.failed);
importer.scan_pending = importer.read_after && importer.new_indices.count > 0 && !app.import_source;
}
if importer.scan_pending && !ocr_ui.thread_busy && ocr_ui.engine && labels_ui_settled() {
importer.scan_pending = false;
ocr_scan_start(importer.new_indices);
}
if !importer.window_open return;
SetNextWindowSize(.{720, 520}, .FirstUseEver);
if Begin(IMPORT_WINDOW, *importer.window_open) {
import_window();
}
End();
}
// The captures written since the last frame, into the grid.
import_take_new_paths :: () {
lock(*importer.mutex);
paths := array_copy(importer.new_paths,, temp);
array_reset_keeping_memory(*importer.new_paths);
unlock(*importer.mutex);
for paths {
index := viewer_add_capture(it);
if index >= 0 array_add(*importer.new_indices, index);
free(it,, context.default_allocator);
}
}
// -import SOURCE DEST: imports once the viewer has started, then ends the app (1 if any file failed).
import_flag_frame :: () {
if !importer.flag_started {
importer.flag_started = true;
importer.read_after = false;
if !import_start(app.import_source, app.import_dest) {
log_error("[Import] -import % % could not start.", app.import_source, app.import_dest);
app.exit_code = 2; app.should_exit = true; app.import_source = "";
}
return;
}
if importer.running || importer.thread_started return;
log("[Import] -import is done: % imported, % duplicates, % skipped, % failed, in %.", importer.imported, importer.duplicates, importer.skipped, importer.failed, ocr_duration_text(importer.elapsed_seconds));
app.exit_code = ifx importer.failed || importer.error then cast(s32) 1 else 0;
app.should_exit = true;
app.import_source = "";
}
import_window :: () { using ImGui;
running := importer.running;
TextColored(app_theme.palette.dim, "Imports every image under a folder (PNG, JPEG, HEIC, AVIF, WebP, GIF, BMP, TIFF, camera RAW) and Snagit .snagx captures into a folder of its own in %. The files there are only read. Each image is stored once in the whole library; videos are skipped.", viewer_folder());
Spacing();
if running BeginDisabled();
SetNextItemWidth(-200);
if InputTextWithHint("##source", "The folder to import, e.g. D:\\Snagit or E:\\Phone backup\\DCIM", importer.source_buffer.data, importer.source_buffer.count) {
import_source_changed();
}
SameLine();
#if OS == .WINDOWS {
if Button("Browse...") {
chosen := windows_choose_folder("Choose a folder to import");
if chosen {
count := min(chosen.count, importer.source_buffer.count - 1);
memcpy(importer.source_buffer.data, chosen.data, count);
importer.source_buffer[count] = 0;
import_source_changed();
}
}
SameLine();
}
Text("Import from");
SetNextItemWidth(-200);
if InputTextWithHint("##dest", "A folder in the library, e.g. SnagitLibraryWork", importer.dest_buffer.data, importer.dest_buffer.count) {
importer.dest_edited = importer.dest_buffer[0] != 0;
}
SameLine();
Text("Into library folder");
source := to_string(importer.source_buffer.data);
dest := trim(to_string(importer.dest_buffer.data));
if dest {
if !import_name_is_valid(dest) TextColored(app_theme.palette.warning, "\"%\" cannot be a folder's name.", dest);
else TextColored(app_theme.palette.dim, "Into %/%", viewer_folder(), dest);
}
Checkbox("Read text and labels in what is imported, once it is in", *importer.read_after);
if running EndDisabled();
Spacing();
if !running {
can_start := trim(source).count > 0 && dest.count > 0 && import_name_is_valid(dest);
if !can_start BeginDisabled();
if Button("Import") import_start(source, dest);
if !can_start EndDisabled();
} else {
if importer.paused {
if Button("Resume") { importer.paused_seconds += to_float64_seconds(current_time_monotonic() - importer.pause_start); atomic_swap(*importer.paused, false); }
} else {
if Button("Pause") { importer.pause_start = current_time_monotonic(); atomic_swap(*importer.paused, true); }
}
SameLine();
if Button("Cancel") {
if importer.paused importer.paused_seconds += to_float64_seconds(current_time_monotonic() - importer.pause_start);
atomic_swap(*importer.cancel, true);
atomic_swap(*importer.paused, false);
}
}
if importer.phase == .IDLE return;
Separator();
if importer.error TextColored(app_theme.palette.warning, "%", importer.error);
elapsed := ifx running then import_elapsed() else importer.elapsed_seconds;
if importer.phase == {
case .LISTING;
Text("Looking through %: % files so far", importer.source, importer.listed);
case .INDEXING;
Text("Indexing the library (once; this finds duplicates):");
ProgressBar(cast(float) importer.index_done / max(importer.index_total, 1), .{-FLOAT32_MIN, 0}, temp_c_string(ocr_scan_progress_text(importer.index_done, importer.index_total)));
case;
total := importer.files.count;
done := importer.done;
rate := ifx elapsed > 0 then done / elapsed else 0;
left := ifx rate > 0 then (total - done) / rate else 0;
if running {
Text("Importing % into %:", importer.source, importer.dest_name);
ProgressBar(cast(float) done / max(total, 1), .{-FLOAT32_MIN, 0}, temp_c_string(ocr_scan_progress_text(done, total)));
Text("% elapsed, % left, % files/s%", ocr_duration_text(elapsed), ocr_duration_text(left), FF(rate, 1), ifx importer.paused then " (paused)" else "");
} else {
Text("Last import, % into %: % of % files in %%.", importer.source, importer.dest_name, done, total, ocr_duration_text(elapsed), ifx importer.cancel then " (cancelled)" else "");
}
}
Text("% imported, % duplicates, % skipped, % failed", importer.imported, importer.duplicates, importer.skipped, importer.failed);
lock(*importer.mutex);
defer unlock(*importer.mutex);
if importer.failures.count && CollapsingHeader(temp_c_string(tprint("Failed (%)###failed", importer.failed))) {
if SmallButton("Copy the list##failed") clipboard_set_text(import_notes_text(importer.failures));
BeginChild("failed", .{0, GetTextLineHeightWithSpacing() * min(cast(float) importer.failures.count + 1, 12)});
import_list(importer.failures, "% %");
EndChild();
}
if importer.duplicate_notes.count && CollapsingHeader(temp_c_string(tprint("Duplicates (%), already in the library###duplicates", importer.duplicates))) {
if SmallButton("Copy the list##duplicates") clipboard_set_text(import_notes_text(importer.duplicate_notes));
BeginChild("duplicates", .{0, GetTextLineHeightWithSpacing() * min(cast(float) importer.duplicate_notes.count + 1, 12)});
import_list(importer.duplicate_notes, "% is %");
EndChild();
}
if importer.skipped && CollapsingHeader(temp_c_string(tprint("Skipped (%), by kind###skipped", importer.skipped))) {
kinds: [..] string;
kinds.allocator = temp;
for _, key: importer.skipped_kinds array_add(*kinds, key);
quick_sort(kinds, compare);
for kinds {
_, count := table_find(*importer.skipped_kinds, it);
Text("%: %", it, count);
}
}
}
// The notes, a line each, only those in view drawn.
import_list :: (notes: [] Import_Note, format: string) { using ImGui;
clipper: ListClipper;
clipper.Constructor(*clipper);
defer clipper.Destructor(*clipper);
clipper.Begin(*clipper, xx notes.count);
while clipper.Step(*clipper) {
for clipper.DisplayStart..clipper.DisplayEnd - 1 TextUnformatted(tprint(format, notes[it].path, notes[it].text));
}
clipper.End(*clipper);
}
import_notes_text :: (notes: [] Import_Note) -> string {
builder: String_Builder;
builder.allocator = temp;
for notes print_to_builder(*builder, "%\t%\n", it.path, it.text);
return builder_to_string(*builder,, temp);
}
// A new source: the library folder it goes into, unless one was typed.
import_source_changed :: () {
if importer.dest_edited && importer.dest_buffer[0] return;
dest := import_dest_for(to_string(importer.source_buffer.data));
count := min(dest.count, importer.dest_buffer.count - 1);
memcpy(importer.dest_buffer.data, dest.data, count);
importer.dest_buffer[count] = 0;
importer.dest_edited = false;
}
#scope_file
Stale_File :: struct { relative: string; size: s64; modified: s64; }
index_stale: [] Stale_File;
index_next: s64;

8
src/app/main.jai
View File

@ -17,11 +17,17 @@ main :: () {
app.headless = array_find(app.args, "-headless") || NO_CRASH_DIALOG; app.headless = array_find(app.args, "-headless") || NO_CRASH_DIALOG;
runtime_support_headless = app.headless; runtime_support_headless = app.headless;
app.allow_multiple_instances = array_find(app.args, "-allow-multiple-instances"); app.allow_multiple_instances = array_find(app.args, "-allow-multiple-instances");
for app.args if it == "-library" && it_index + 1 < app.args.count app.library_folder = app.args[it_index + 1]; for app.args if it == "-library" && it_index + 1 < app.args.count {
// With / between its folders and none at the end, as the paths found in it are made.
app.library_folder = copy_string(app.args[it_index + 1]);
path_overwrite_separators(app.library_folder, #char "/");
while app.library_folder.count > 3 && app.library_folder[app.library_folder.count - 1] == #char "/" app.library_folder.count -= 1;
}
app.scan_and_exit = array_find(app.args, "-scan"); app.scan_and_exit = array_find(app.args, "-scan");
app.stats_and_exit = array_find(app.args, "-stats"); app.stats_and_exit = array_find(app.args, "-stats");
for app.args if it == "-search" && it_index + 1 < app.args.count app.start_search = app.args[it_index + 1]; for app.args if it == "-search" && it_index + 1 < app.args.count app.start_search = app.args[it_index + 1];
for app.args if it == "-erase" && it_index + 1 < app.args.count app.erase_and_exit = app.args[it_index + 1]; for app.args if it == "-erase" && it_index + 1 < app.args.count app.erase_and_exit = app.args[it_index + 1];
for app.args if it == "-import" && it_index + 2 < app.args.count { app.import_source = app.args[it_index + 1]; app.import_dest = app.args[it_index + 2]; }
defer report_memory_leaks(); defer report_memory_leaks();

22
src/app/ocr_ui.jai
View File

@ -282,7 +282,7 @@ ocr_read_and_store :: (items: [] Scan_Item, set: *Label_Set) -> [] string {
labels := labels_ui.engine; labels := labels_ui.engine;
// The image the Capture window shows: the rendered one, annotations and all, or the original. // The image the Capture window shows: the rendered one, annotations and all, or the original.
Prepared :: struct { pixels: *u8; width, height: s32; image: string; stored_vector: [] u16; } Prepared :: struct { decoded: Image; pixels: *u8; width, height: s32; image: string; stored_vector: [] u16; }
prepared := NewArray(items.count, Prepared,, temp); prepared := NewArray(items.count, Prepared,, temp);
ocr_images: [..] Ocr_Image; ocr_images: [..] Ocr_Image;
ocr_images.allocator = temp; ocr_images.allocator = temp;
@ -302,17 +302,18 @@ ocr_read_and_store :: (items: [] Scan_Item, set: *Label_Set) -> [] string {
if chunk prepared[n].stored_vector = mcap_embedding_parse(chunk.data,, temp); if chunk prepared[n].stored_vector = mcap_embedding_parse(chunk.data,, temp);
} }
if !item.ocr && !item.embed continue; if !item.ocr && !item.embed continue;
png := mcap_rendered(m); bytes := mcap_rendered(m);
kind := "rendered"; kind := "rendered";
if !png.count { png = m.original; kind = "original"; } if !bytes.count { bytes = m.original; kind = "original"; }
w, h, channels: s32; // Turned the right way up, as the Capture window shows it, so the boxes land on the text.
pixels := stbi_load_from_memory(png.data, cast(s32) png.count, *w, *h, *channels, 4); decoded, decoded_ok, decode_error := image_decode(bytes);
if !pixels { errors[n] = copy_string("its image cannot be decoded"); continue; } if !decoded_ok { errors[n] = sprint("its image cannot be decoded: %", decode_error); continue; }
prepared[n] = .{pixels, w, h, kind, prepared[n].stored_vector}; pixels, w, h := decoded.pixels, decoded.width, decoded.height;
prepared[n] = .{decoded, pixels, w, h, kind, prepared[n].stored_vector};
if item.ocr { array_add(*ocr_images, .{pixels, w, h, w * 4}); array_add(*ocr_of, n); } if item.ocr { array_add(*ocr_images, .{pixels, w, h, w * 4}); array_add(*ocr_of, n); }
if item.embed && labels && set { array_add(*label_images, .{pixels, w, h, w * 4}); array_add(*label_of, n); } if item.embed && labels && set { array_add(*label_images, .{pixels, w, h, w * 4}); array_add(*label_of, n); }
} }
defer for prepared if it.pixels stbi_image_free(it.pixels); defer for * prepared image_free(*it.decoded);
// Labels: vectors for the images to label, then each item's labels. // Labels: vectors for the images to label, then each item's labels.
Labelled :: struct { done: bool; labels: Mcap_Labels; vector: [] u16; } Labelled :: struct { done: bool; labels: Mcap_Labels; vector: [] u16; }
@ -508,7 +509,8 @@ ocr_ui_scan_and_exit :: (busy: bool) {
finish(ifx ocr_ui.scan.failed then cast(s32) 1 else 0, "[OCR scan] -scan is done: % read in %, % failed.", ocr_ui.scan.done, ocr_duration_text(ocr_ui.scan.elapsed_seconds), ocr_ui.scan.failed); finish(ifx ocr_ui.scan.failed then cast(s32) 1 else 0, "[OCR scan] -scan is done: % read in %, % failed.", ocr_ui.scan.done, ocr_duration_text(ocr_ui.scan.elapsed_seconds), ocr_ui.scan.failed);
} }
ocr_scan_start :: () { // Reads (and labels) every capture shown, or only those in only (an import's), that need it.
ocr_scan_start :: (only: [] s32 = .[]) {
if ocr_ui.thread_busy || !ocr_ui.engine return; if ocr_ui.thread_busy || !ocr_ui.engine return;
scan := *ocr_ui.scan; scan := *ocr_ui.scan;
for scan.items free(it.path); for scan.items free(it.path);
@ -519,7 +521,7 @@ ocr_scan_start :: () {
scan.model = copy_string(ocr_ui.engine.model.name,, context.default_allocator); scan.model = copy_string(ocr_ui.engine.model.name,, context.default_allocator);
if labels_ui.scan_labels && labels_ui.engine && labels_ui.set && labels_ui.set.hash == labels_ui.list_hash scan.label_set = labels_ui.set; if labels_ui.scan_labels && labels_ui.engine && labels_ui.set && labels_ui.set.hash == labels_ui.list_hash scan.label_set = labels_ui.set;
// Every capture this model has not read, or not labelled with this list, in the order the grid shows them. // Every capture this model has not read, or not labelled with this list, in the order the grid shows them.
for viewer.visible { for ifx only.count then only else viewer.visible {
capture := *viewer.captures[it]; capture := *viewer.captures[it];
if capture.removed continue; if capture.removed continue;
entry := capture.catalog; entry := capture.catalog;

140
src/app/viewer.jai
View File

@ -8,11 +8,14 @@
// newest first once the catalog is complete, and only those matching the search. Searching runs on // newest first once the catalog is complete, and only those matching the search. Searching runs on
// background workers (app/background.jai), so the UI thread never waits on it. // background workers (app/background.jai), so the UI thread never waits on it.
// //
// The library is a folder of folders, one for each place imported from (app/import.jai), and any .mcap
// files at its top. The grid shows them all, or one folder's.
//
// A capture keeps its index for as long as the app runs: the workers, the OCR scan and Stats refer // A capture keeps its index for as long as the app runs: the workers, the OCR scan and Stats refer
// to captures by index. Refresh (viewer_refresh) marks the files gone from the folder as removed and // to captures by index. Refresh (viewer_refresh) marks the files gone from the folder as removed and
// appends the new ones. // appends the new ones.
VIEWER_FOLDER :: "D:/SnagitIndex/mcap"; // hardcoded for now; -library FOLDER shows another VIEWER_FOLDER :: "D:/McapIndex/mcap"; // hardcoded for now; -library FOLDER shows another
viewer_folder :: () -> string { viewer_folder :: () -> string {
return ifx app.library_folder then app.library_folder else VIEWER_FOLDER; return ifx app.library_folder then app.library_folder else VIEWER_FOLDER;
@ -44,6 +47,7 @@ gpu_texture_ref :: (texture: Gpu_Texture) -> ImGui.ImTextureRef {
Viewer_Capture :: struct { Viewer_Capture :: struct {
name: string; // the file name without .mcap name: string; // the file name without .mcap
path: string; path: string;
folder: string; // the library folder it is in, "SnagitLibraryWork"; "" at the top
texture: Gpu_Texture; // its preview, once uploaded texture: Gpu_Texture; // its preview, once uploaded
last_drawn_frame: u64; last_drawn_frame: u64;
last_wanted_frame: u64; // last in the rows viewer_request_previews asked for last_wanted_frame: u64; // last in the rows viewer_request_previews asked for
@ -70,8 +74,8 @@ Viewer_Fit :: enum u8 {
// Pixels a worker decoded, waiting for the main thread to upload them. // Pixels a worker decoded, waiting for the main thread to upload them.
Decoded :: struct { Decoded :: struct {
index: s32; index: s32;
pixels: *u8; // RGBA; from stb_image, or from the default allocator when scaled down pixels: *u8; // RGBA; from stb_image, or from the default allocator when not
scaled: bool; // pixels came from the default allocator scaled: bool; // pixels came from the default allocator (scaled down, turned, or decoded by WIC)
width, height: s32; width, height: s32;
source_width, source_height: s32; // before any scaling down source_width, source_height: s32; // before any scaling down
failed: bool; failed: bool;
@ -116,6 +120,8 @@ viewer: struct {
sorted_by_date: bool; sorted_by_date: bool;
search_buffer: [256] u8; search_buffer: [256] u8;
search_terms: [..] Search_Term; search_terms: [..] Search_Term;
folders: [..] string; // the library's folders holding captures, sorted; "" for its top
folder_filter: string; // the folder the grid shows; "" for all (heap)
// The full view; main thread only. // The full view; main thread only.
full_open: bool; full_open: bool;
@ -143,15 +149,14 @@ viewer_startup :: () {
init(*viewer.mutex, "viewer"); init(*viewer.mutex, "viewer");
init(*viewer.wake); init(*viewer.wake);
viewer.folders.allocator = context.default_allocator;
start := current_time_monotonic(); start := current_time_monotonic();
files := file_list(viewer_folder(),, temp); for viewer_list_files() {
for files {
name := path_filename(it);
if !ends_with_nocase(name, tprint(".%", MCAP_EXTENSION)) continue;
capture := array_add(*viewer.captures); capture := array_add(*viewer.captures);
capture.path = copy_string(it); viewer_capture_init(capture, it);
capture.name = copy_string(path_strip_extension(name));
} }
viewer_update_folders();
// Newest first, roughly: dated names (2026-...) sort after the older {GUID} ones. // Newest first, roughly: dated names (2026-...) sort after the older {GUID} ones.
quick_sort(viewer.captures, (a: Viewer_Capture, b: Viewer_Capture) -> s64 { return compare(b.name, a.name); }); quick_sort(viewer.captures, (a: Viewer_Capture, b: Viewer_Capture) -> s64 { return compare(b.name, a.name); });
if !viewer.captures.count viewer.scan_error = sprint("No .% files in %.", MCAP_EXTENSION, viewer_folder()); if !viewer.captures.count viewer.scan_error = sprint("No .% files in %.", MCAP_EXTENSION, viewer_folder());
@ -253,7 +258,7 @@ viewer_decode_preview :: (path: string) -> Decoded {
m, ok, error := mcap_read_front(path); m, ok, error := mcap_read_front(path);
if !ok { log_error("[viewer] % %", path, error); return result; } if !ok { log_error("[viewer] % %", path, error); return result; }
if !m.preview.png.count { log_error("[viewer] % has no preview.", path); return result; } if !m.preview.png.count { log_error("[viewer] % has no preview.", path); return result; }
return viewer_decode_png(path, m.preview.png); return viewer_decode_image(path, m.preview.png, .PNG);
} }
// The rendered image, annotations and all, or the original when there is none. // The rendered image, annotations and all, or the original when there is none.
@ -267,22 +272,24 @@ viewer_decode_full :: (path: string) -> Decoded {
if !read_ok { log_error("[viewer] Cannot read %.", path); return result; } if !read_ok { log_error("[viewer] Cannot read %.", path); return result; }
m, ok, error := mcap_parse(cast([] u8) data); m, ok, error := mcap_parse(cast([] u8) data);
if !ok { log_error("[viewer] % %", path, error); return result; } if !ok { log_error("[viewer] % %", path, error); return result; }
png := mcap_rendered(m); image := mcap_rendered(m);
if !png.count png = m.original; if !image.count image = m.original;
return viewer_decode_png(path, png); return viewer_decode_image(path, image);
} }
viewer_decode_png :: (path: string, png: [] u8) -> Decoded { // Any image (modules/Image_Decode), turned the right way up.
viewer_decode_image :: (path: string, bytes: [] u8, kind := Image_Kind.UNKNOWN) -> Decoded {
result: Decoded; result: Decoded;
result.failed = true; result.failed = true;
width, height, channels: s32; image, ok, error := image_decode(bytes, kind);
pixels := stbi_load_from_memory(png.data, cast(s32) png.count, *width, *height, *channels, 4); if !ok {
if !pixels || width <= 0 || height <= 0 { log_error("[viewer] Cannot decode the image in %: %", path, error);
log_error("[viewer] Cannot decode the image in %.", path);
if pixels stbi_image_free(pixels);
return result; return result;
} }
pixels := image.pixels;
width, height := image.width, image.height;
result.pixels = pixels; result.pixels = pixels;
result.scaled = !image.from_stb;
result.width, result.height = width, height; result.width, result.height = width, height;
result.source_width, result.source_height = width, height; result.source_width, result.source_height = width, height;
result.failed = false; result.failed = false;
@ -295,13 +302,13 @@ viewer_decode_png :: (path: string, png: [] u8) -> Decoded {
out_height := max(cast(s32) 1, cast(s32) (height * scale)); out_height := max(cast(s32) 1, cast(s32) (height * scale));
scaled := cast(*u8) alloc(cast(s64) out_width * out_height * 4,, context.default_allocator); scaled := cast(*u8) alloc(cast(s64) out_width * out_height * 4,, context.default_allocator);
if stbir_resize_uint8_srgb(pixels, width, height, 0, scaled, out_width, out_height, 0, 4, 3, 0) { if stbir_resize_uint8_srgb(pixels, width, height, 0, scaled, out_width, out_height, 0, 4, 3, 0) {
stbi_image_free(pixels); image_free(*image);
result.pixels = scaled; result.pixels = scaled;
result.scaled = true; result.scaled = true;
result.width, result.height = out_width, out_height; result.width, result.height = out_width, out_height;
} else { } else {
free(scaled,, context.default_allocator); free(scaled,, context.default_allocator);
stbi_image_free(pixels); image_free(*image);
result.pixels = null; result.pixels = null;
result.failed = true; result.failed = true;
log_error("[viewer] Cannot scale down the % x % image in %.", width, height, path); log_error("[viewer] Cannot scale down the % x % image in %.", width, height, path);
@ -432,9 +439,7 @@ viewer_refresh :: () {
start := current_time_monotonic(); start := current_time_monotonic();
present: Table(string, bool); // the folder's .mcap files now present: Table(string, bool); // the folder's .mcap files now
present.allocator = temp; present.allocator = temp;
for file_list(viewer_folder(),, temp) { for viewer_list_files() table_set(*present, it, true);
if ends_with_nocase(path_filename(it), tprint(".%", MCAP_EXTENSION)) table_set(*present, it, true);
}
removed, added: s64; removed, added: s64;
known: Table(string, bool); // the captures still there known: Table(string, bool); // the captures still there
@ -451,12 +456,12 @@ viewer_refresh :: () {
if table_contains(*known, path) continue; if table_contains(*known, path) continue;
capture := array_add(*viewer.captures); capture := array_add(*viewer.captures);
capture.* = .{}; capture.* = .{};
capture.path = copy_string(path,, context.default_allocator); viewer_capture_init(capture, path);
capture.name = copy_string(path_strip_extension(path_filename(path)),, context.default_allocator);
added += 1; added += 1;
} }
unlock(*viewer.mutex); unlock(*viewer.mutex);
if added signal(*viewer.wake); // their catalog entries if added signal(*viewer.wake); // their catalog entries
viewer_update_folders();
free(viewer.scan_error); free(viewer.scan_error);
viewer.scan_error = ""; viewer.scan_error = "";
@ -469,6 +474,65 @@ viewer_refresh :: () {
toast_show(.Success, "% removed, % added", removed, added); toast_show(.Success, "% removed, % added", removed, added);
} }
// One capture written into the library since it was looked at (an import): added as a refresh would
// add it. Its index, or -1 when it is already there.
viewer_add_capture :: (path: string) -> s32 {
for viewer.captures if !it.removed && it.path == path return -1;
lock(*viewer.mutex);
capture := array_add(*viewer.captures);
capture.* = .{};
viewer_capture_init(capture, path);
index := cast(s32) (viewer.captures.count - 1);
unlock(*viewer.mutex);
signal(*viewer.wake); // its catalog entry
if viewer.scan_error { free(viewer.scan_error); viewer.scan_error = ""; }
if !array_find(viewer.folders, capture.folder) viewer_update_folders();
viewer.sorted_by_date = false; // until its entry is in
viewer.visible_dirty = true;
return index;
}
// Every .mcap in the library, in its folders too, but not in hidden ones. Temp.
viewer_list_files :: () -> [..] string {
files: [..] string;
files.allocator = temp;
extension := tprint(".%", MCAP_EXTENSION);
for file_list(viewer_folder(), recursive = true,, temp) {
if !ends_with_nocase(it, extension) continue;
relative := advance(it, viewer_folder().count + 1);
if begins_with(relative, ".") || contains(relative, "/.") continue;
array_add(*files, it);
}
return files;
}
viewer_capture_init :: (capture: *Viewer_Capture, path: string) {
push_allocator(context.default_allocator);
capture.path = copy_string(path);
capture.name = copy_string(path_strip_extension(path_filename(path)));
relative := path_strip_filename(advance(path, viewer_folder().count + 1));
while relative.count && (relative[relative.count - 1] == #char "/" || relative[relative.count - 1] == #char "\\") relative.count -= 1;
capture.folder = copy_string(relative);
}
// The folders that hold captures, for the folder filter.
viewer_update_folders :: () {
for viewer.folders free(it);
array_reset_keeping_memory(*viewer.folders);
for viewer.captures {
if it.removed || array_find(viewer.folders, it.folder) continue;
array_add(*viewer.folders, copy_string(it.folder,, context.default_allocator));
}
quick_sort(viewer.folders, compare);
}
viewer_set_folder_filter :: (folder: string) {
free(viewer.folder_filter);
viewer.folder_filter = copy_string(folder,, context.default_allocator);
viewer.visible_dirty = true;
viewer.visible_rebuilt_at = .{};
}
// Takes a capture out of everything shown, as though its file had never been there; its index stays. // Takes a capture out of everything shown, as though its file had never been there; its index stays.
viewer_mark_removed :: (index: s32) { viewer_mark_removed :: (index: s32) {
capture := *viewer.captures[index]; capture := *viewer.captures[index];
@ -748,15 +812,32 @@ viewer_ui :: () { using ImGui;
TextColored(app_theme.palette.warning, "%", viewer.scan_error); TextColored(app_theme.palette.warning, "%", viewer.scan_error);
SameLine(); SameLine();
if Button("Refresh") viewer_refresh(); if Button("Refresh") viewer_refresh();
SameLine();
if Button("Import") importer.window_open = !importer.window_open; // app/import.jai
return; return;
} }
SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520)); SetNextItemWidth(min(GetContentRegionAvail().x * 0.5, 520));
if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like:", viewer.search_buffer.data, viewer.search_buffer.count) { if InputTextWithHint("##search", "Search text, titles, URLs, apps, notes, labels. \"a phrase\" ocr: title: url: app: note: label: like: folder:", viewer.search_buffer.data, viewer.search_buffer.count) {
viewer_search_changed(); viewer_search_changed();
} }
SameLine(); SameLine();
if viewer.search_terms.count Text("% of % captures", viewer.visible.count, viewer_capture_count()); // Which library folder: all, or one.
if viewer.folders.count > 1 || (viewer.folders.count == 1 && viewer.folders[0] != "") {
SetNextItemWidth(200);
folder_label := ifx viewer.folder_filter then viewer.folder_filter else "All folders";
if BeginCombo("##folder", temp_c_string(folder_label)) {
if Selectable("All folders", !viewer.folder_filter) viewer_set_folder_filter("");
for viewer.folders {
label := ifx it then it else "(the library's top)";
if Selectable(temp_c_string(tprint("%##%", label, it_index)), it == viewer.folder_filter) viewer_set_folder_filter(it);
}
EndCombo();
}
if IsItemHovered() SetTooltip("The library folder shown: each holds what was imported from one place");
SameLine();
}
if viewer.search_terms.count || viewer.folder_filter Text("% of % captures", viewer.visible.count, viewer_capture_count());
else Text("% captures", viewer_capture_count()); else Text("% captures", viewer_capture_count());
if background.searching { SameLine(); TextColored(app_theme.palette.dim, "searching..."); } if background.searching { SameLine(); TextColored(app_theme.palette.dim, "searching..."); }
if viewer.catalog_loaded < viewer.captures.count { if viewer.catalog_loaded < viewer.captures.count {
@ -774,6 +855,9 @@ viewer_ui :: () { using ImGui;
SameLine(); SameLine();
if Button("Labels") labels_ui.window_open = !labels_ui.window_open; // app/labels_ui.jai if Button("Labels") labels_ui.window_open = !labels_ui.window_open; // app/labels_ui.jai
SameLine(); SameLine();
if Button("Import") importer.window_open = !importer.window_open; // app/import.jai
if IsItemHovered() SetTooltip("Bring in a folder of screenshots and photos");
SameLine();
busy := ocr_ui.thread_busy; // a scan or a reading refers to captures by index busy := ocr_ui.thread_busy; // a scan or a reading refers to captures by index
if busy BeginDisabled(); if busy BeginDisabled();
if Button("Refresh") viewer_refresh(); if Button("Refresh") viewer_refresh();

25
src/base/platform/platform_windows.jai
View File

@ -829,3 +829,28 @@ os_log_error :: () {
error_code := GetLastError(); error_code := GetLastError();
log_error(" > GetLastError code: %, %", error_code, get_error_string(error_code)); log_error(" > GetLastError code: %, %", error_code, get_error_string(error_code));
} }
// Asks for a folder with Explorer's own picker, over the app's window: its path (temp), or "" when
// none was chosen.
windows_choose_folder :: (title: string) -> string {
initialized := CoInitializeEx(null, COINIT_APARTMENTTHREADED);
defer if initialized >= 0 CoUninitialize();
dialog: *IFileOpenDialog;
if CoCreateInstance(*uid(CLSID_FileOpenDialog_UUID), null, CLSCTX_INPROC_SERVER, *uid(IFileOpenDialog_UUID), xx *dialog) < 0 return "";
defer IUnknown_Release(dialog);
options: u32;
dialog.GetOptions(dialog, *options);
dialog.SetOptions(dialog, options | FOS_PICKFOLDERS | FOS_FORCEFILESYSTEM);
wide_title := utf8_to_wide(title,, temp);
dialog.SetTitle(dialog, wide_title);
if dialog.Show(dialog, platform.hwnd) < 0 return ""; // cancelled
item: *IShellItem;
if dialog.GetResult(dialog, *item) < 0 return "";
defer IUnknown_Release(item);
name: *u16;
if item.GetDisplayName(item, SIGDN_FILESYSPATH, *name) < 0 return "";
defer CoTaskMemFree(name);
path, ok := wide_to_utf8(name,, temp);
return ifx ok then path else "";
}

38
src/base/platform/win32_bindings.jai
View File

@ -499,3 +499,41 @@ IMAGE_ICON :: 1;
IMAGE_CURSOR :: 2; IMAGE_CURSOR :: 2;
LR_LOADFROMFILE :: 0x0010; LR_LOADFROMFILE :: 0x0010;
// The Explorer folder picker (IFileOpenDialog with FOS_PICKFOLDERS), for windows_choose_folder.
CoTaskMemFree :: (pv: *void) #foreign Ole32;
RPC_E_CHANGED_MODE :HRESULT: cast,no_check(s32) 0x80010106;
FOS_PICKFOLDERS :: 0x20;
FOS_FORCEFILESYSTEM :: 0x40;
SIGDN_FILESYSPATH :: cast,no_check(s32) 0x80058000;
CLSID_FileOpenDialog_UUID :: "DC1C5A9C-E88A-4DDE-A5A1-60F82A20AEF7";
IFileOpenDialog_UUID :: "D57C7288-D4AD-4768-BE02-9D969532D960";
IFileOpenDialog :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IFileOpenDialog_VTable;
}
IFileOpenDialog_VTable :: struct {
#as using base: IUnknown_VTable;
Show: (this: *IFileOpenDialog, owner: HWND) -> HRESULT #cpp_method; // IModalWindow
SetFileTypes, SetFileTypeIndex, GetFileTypeIndex, Advise, Unadvise: *void;
SetOptions: (this: *IFileOpenDialog, options: u32) -> HRESULT #cpp_method;
GetOptions: (this: *IFileOpenDialog, options: *u32) -> HRESULT #cpp_method;
SetDefaultFolder, SetFolder, GetFolder, GetCurrentSelection, SetFileName, GetFileName: *void;
SetTitle: (this: *IFileOpenDialog, title: *u16) -> HRESULT #cpp_method;
SetOkButtonLabel, SetFileNameLabel: *void;
GetResult: (this: *IFileOpenDialog, item: **IShellItem) -> HRESULT #cpp_method;
AddPlace, SetDefaultExtension, Close, SetClientGuid, ClearClientData, SetFilter, GetResults, GetSelectedItems: *void;
}
IShellItem :: struct {
#as unknown: IUnknown;
#overlay (unknown.vtable) using vtable: *IShellItem_VTable;
}
IShellItem_VTable :: struct {
#as using base: IUnknown_VTable;
BindToHandler, GetParent: *void;
GetDisplayName: (this: *IShellItem, kind: s32, name: **u16) -> HRESULT #cpp_method;
GetAttributes, Compare: *void;
}

232
src/test/import_tests.jai Normal file
View File

@ -0,0 +1,232 @@
// The importer's pieces on their own (app/import.jai, modules/Image_Decode): what kind of file bytes
// are, EXIF, turning an image by its orientation, the dates found in EXIF and in file names, and the
// folder: search.
#scope_file
checks_run := 0;
checks_failed := 0;
MAX_GROUPS :: 8;
group_names : [MAX_GROUPS] string;
group_checks : [MAX_GROUPS] int;
group_fails : [MAX_GROUPS] int;
group_count := 0;
output_builder: *String_Builder;
emit :: (format_string: string, args: ..Any) {
if output_builder print_to_builder(output_builder, format_string, ..args);
else print(format_string, ..args);
} @PrintLike
group :: (name: string) {
assert(group_count < MAX_GROUPS, "raise MAX_GROUPS");
group_names[group_count] = name;
group_count += 1;
emit("\n-- % --\n", name);
}
expect :: (condition: bool, format: string, args: ..Any, loc := #caller_location) {
checks_run += 1;
if group_count group_checks[group_count-1] += 1;
if condition return;
checks_failed += 1;
if group_count group_fails[group_count-1] += 1;
emit(" FAIL [%:%] %\n", group_names[max(group_count-1, 0)], loc.line_number, tprint(format, ..args));
}
report :: () -> bool {
emit("\n");
for 0..group_count-1 {
if group_fails[it] emit(" % -- % checks, % FAILED\n", group_names[it], group_checks[it], group_fails[it]);
else emit(" % -- % checks\n", group_names[it], group_checks[it]);
}
if checks_failed emit("\n% OF % IMPORT CHECKS FAILED\n", checks_failed, checks_run);
else emit("\nALL IMPORT TESTS PASSED (% checks over % groups)\n", checks_run, group_count);
return checks_failed == 0;
}
kind_tests :: () {
group("kind");
kind :: (bytes: string) -> Image_Kind { return image_kind(cast([] u8) bytes); }
expect(kind("\x89PNG\r\n\x1A\n....") == .PNG, "PNG");
expect(kind("\xFF\xD8\xFF\xE1") == .JPEG, "JPEG");
expect(kind("GIF89a......") == .GIF, "GIF");
expect(kind("RIFF\0\0\0\0WEBPVP8 ") == .WEBP, "WebP");
expect(kind("II*\0\x08\0\0\0") == .TIFF, "TIFF, little-endian");
expect(kind("MM\0*\0\0\0\x08") == .TIFF, "TIFF, big-endian");
expect(kind("\0\0\0\x18ftypheic\0\0\0\0mif1heic") == .HEIF, "HEIC");
expect(kind("\0\0\0\x1Cftypmif1\0\0\0\0mif1avifmiaf") == .AVIF, "AVIF under the general mif1 brand");
expect(kind("\0\0\0\x18ftypisom\0\0\0\0isommp41") == .VIDEO, "MP4");
expect(kind("\0\0\0\x14ftypqt \0\0\0\0qt ") == .VIDEO, "QuickTime");
expect(kind("\0\0\0\x18ftypcrx \0\0\0\x01crx isom") == .RAW, "Canon CR3");
expect(kind("hello") == .UNKNOWN, "text");
expect(kind("") == .UNKNOWN, "nothing");
expect(image_kind_from_name(.TIFF, "IMG_0001.DNG") == .RAW, "a DNG is RAW");
expect(image_kind_from_name(.TIFF, "scan.tif") == .TIFF, "a .tif stays TIFF");
expect(image_kind_from_name(.UNKNOWN, "x.tga") == .TGA, "TGA by its name");
}
// A TIFF block with Orientation, Make, Model, an Exif directory with DateTimeOriginal, and GPS.
make_tiff :: (little: bool) -> [] u8 {
out: [..] u8;
out.allocator = temp;
u16_ :: (out: *[..] u8, v: u16, little: bool) {
if little { array_add(out, cast(u8) (v & 0xFF)); array_add(out, cast(u8) (v >> 8)); }
else { array_add(out, cast(u8) (v >> 8)); array_add(out, cast(u8) (v & 0xFF)); }
}
u32_ :: (out: *[..] u8, v: u32, little: bool) {
if little { u16_(out, cast(u16) (v & 0xFFFF), little); u16_(out, cast(u16) (v >> 16), little); }
else { u16_(out, cast(u16) (v >> 16), little); u16_(out, cast(u16) (v & 0xFFFF), little); }
}
entry :: (out: *[..] u8, tag: u16, type: u16, count: u32, value: u32, little: bool) {
u16_(out, tag, little); u16_(out, type, little); u32_(out, count, little);
if type == 3 && count == 1 { u16_(out, cast(u16) value, little); u16_(out, 0, little); }
else u32_(out, value, little);
}
// Layout: header 8; IFD0 at 8 (5 entries: 2 + 60 + 4 = 66) ends at 74; strings at 74 ("Pixel\0" 6, "Pixel 8 Pro\0" 12);
// Exif IFD at 92 (1 entry, 18) ends at 110; date at 110 (20); GPS IFD at 130 (4 entries, 54) ends at 184; rationals at 184 (48).
array_add(*out, ..cast([] u8) ifx little then "II" else "MM");
u16_(*out, 42, little);
u32_(*out, 8, little);
u16_(*out, 5, little);
entry(*out, 0x010F, 2, 6, 74, little);
entry(*out, 0x0110, 2, 12, 80, little);
entry(*out, 0x0112, 3, 1, 6, little);
entry(*out, 0x8769, 4, 1, 92, little);
entry(*out, 0x8825, 4, 1, 130, little);
u32_(*out, 0, little);
array_add(*out, ..cast([] u8) "Pixel\0");
array_add(*out, ..cast([] u8) "Pixel 8 Pro\0");
u16_(*out, 1, little);
entry(*out, 0x9003, 2, 20, 110, little);
u32_(*out, 0, little);
array_add(*out, ..cast([] u8) "2024:05:12 10:15:30\0");
u16_(*out, 4, little);
entry(*out, 1, 2, 2, ifx little then cast(u32) #char "S" else (cast(u32) #char "S") << 24, little);
entry(*out, 2, 5, 3, 184, little);
entry(*out, 3, 2, 2, ifx little then cast(u32) #char "W" else (cast(u32) #char "W") << 24, little);
entry(*out, 4, 5, 3, 208, little);
u32_(*out, 0, little);
for u32.[33, 1, 52, 1, 3000, 100, 151, 1, 12, 1, 0, 1] u32_(*out, it, little);
return out;
}
exif_tests :: () {
group("exif");
for little: bool.[true, false] {
tiff := make_tiff(little);
exif := exif_parse_tiff(tiff);
order := ifx little then "little-endian" else "big-endian";
expect(exif.found, "% TIFF found", order);
expect(exif.orientation == 6, "% orientation: %", order, exif.orientation);
expect(exif.make == "Pixel" && exif.model == "Pixel 8 Pro", "% make and model: \"%\" \"%\"", order, exif.make, exif.model);
expect(exif.date_original == "2024:05:12 10:15:30", "% DateTimeOriginal: \"%\"", order, exif.date_original);
expect(exif.has_gps && abs(exif.latitude - (-(33 + 52.0 / 60 + 30.0 / 3600))) < 0.000000001 && abs(exif.longitude - (-(151 + 12.0 / 60))) < 0.000000001,
"% GPS, south and west negative: %, %", order, exif.latitude, exif.longitude);
// The same block inside a JPEG's APP1 segment.
jpeg: [..] u8;
jpeg.allocator = temp;
array_add(*jpeg, 0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x04, 0x00, 0x00); // an APP0 first
length := tiff.count + 8;
array_add(*jpeg, 0xFF, 0xE1, cast(u8) (length >> 8), cast(u8) (length & 0xFF));
array_add(*jpeg, ..cast([] u8) "Exif\0\0");
array_add(*jpeg, ..tiff);
array_add(*jpeg, 0xFF, 0xDA);
from_jpeg := exif_read(jpeg);
expect(from_jpeg.orientation == 6 && from_jpeg.model == "Pixel 8 Pro", "% EXIF from a JPEG's APP1", order);
}
expect(!exif_parse_tiff(cast([] u8) "not a tiff").found, "garbage is no EXIF");
expect(exif_read(cast([] u8) "\xFF\xD8\xFF\xDA").orientation == 1, "a JPEG with none is upright");
}
orientation_tests :: () {
group("orientation");
// 3 x 2, pixels numbered by where they are: 0 1 2 / 3 4 5. Each orientation's result, row by row.
expected :: u32.[
0, 1, 2, 3, 4, 5, // 1
2, 1, 0, 5, 4, 3, // 2 mirrored
5, 4, 3, 2, 1, 0, // 3 half a turn
3, 4, 5, 0, 1, 2, // 4 flipped
0, 3, 1, 4, 2, 5, // 5 transposed (2 x 3)
3, 0, 4, 1, 5, 2, // 6 a quarter turn clockwise
5, 2, 4, 1, 3, 0, // 7 transversed
2, 5, 1, 4, 0, 3, // 8 a quarter turn anticlockwise
];
for orientation: 1..8 {
pixels := cast(*u32) alloc(6 * 4,, context.default_allocator);
for 0..5 pixels[it] = cast(u32) it;
image := Image.{pixels = cast(*u8) pixels, width = 3, height = 2};
image_orient(*image, orientation);
swapped := orientation >= 5;
expect(image.width == (ifx swapped then 2 else 3) && image.height == (ifx swapped then 3 else 2), "orientation %: % x %", orientation, image.width, image.height);
got: [6] u32;
for 0..5 got[it] = (cast(*u32) image.pixels)[it];
wanted := array_view(expected, (orientation - 1) * 6, 6);
same := true;
for 0..5 if got[it] != wanted[it] same = false;
expect(same, "orientation %: % (wanted %)", orientation, got, wanted);
image_free(*image);
}
}
date_tests :: () {
group("dates");
expect(import_exif_date("2024:05:12 10:15:30") == "2024-05-12 10:15:30", "EXIF's date");
expect(import_exif_date("0000:00:00 00:00:00") == "", "a camera with no clock set");
expect(import_exif_date(" : : : : ") == "", "blanks");
expect(import_exif_date("") == "", "nothing");
Name_Date :: struct { name, date: string; }
names :: Name_Date.[
.{"IMG_20240512_101530.jpg", "2024-05-12 10:15:30"},
.{"PXL_20240512_101530123.jpg", "2024-05-12 10:15:30"},
.{"Screenshot_20240512-101530.png", "2024-05-12 10:15:30"},
.{"IMG-20240512-WA0001.jpg", "2024-05-12 00:00:00"},
.{"Screenshot 2024-05-12 at 10.15.30.png", "2024-05-12 10:15:30"},
.{"Screenshot_2024-05-12-10-15-30.png", "2024-05-12 10:15:30"},
.{"2024-05-12_10-15-30.snagx", "2024-05-12 10:15:30"},
.{"Screenshot 2024-05-12 101530.png", "2024-05-12 10:15:30"},
.{"IMG_0001.JPG", ""},
.{"scan_12345678.png", ""},
.{"DSC_20241399.jpg", ""},
];
for names {
got := import_name_date(it.name);
expect(got == it.date, "% -> \"%\" (wanted \"%\")", it.name, got, it.date);
}
expect(import_name_is_valid("SnagitLibraryWork") && import_name_is_valid("Phone 2024"), "folder names");
expect(!import_name_is_valid("a/b") && !import_name_is_valid("..") && !import_name_is_valid(".hidden") && !import_name_is_valid("x:") && !import_name_is_valid("trailing."), "not folder names");
expect(import_text("Canon\0\0") == "Canon", "EXIF text loses its trailing nulls");
expect(import_text("caf\xE9") == "caf?", "Latin-1 bytes are not UTF-8");
}
folder_tests :: () {
group("folder search");
terms := search_parse("folder:SnagitLibraryWork kicad");
expect(terms.count == 2 && terms[0].field == .FOLDER && terms[0].text == "snagitlibrarywork", "folder: prefix: %", terms);
entry: Catalog_Entry;
entry.find_folder = "snagitlibrarywork";
entry.find_title = "kicad";
expect(search_matches(*entry, terms), "in the folder");
expect(search_matches(*entry, search_parse("folder:work")), "part of the folder's name");
expect(!search_matches(*entry, search_parse("folder:personal")), "another folder");
expect(!search_matches(*entry, search_parse("snagitlibrarywork")), "a plain word does not match the folder");
}
#scope_export
run_import_tests :: (builder: *String_Builder = null) -> bool {
output_builder = builder;
emit("=== import test suite ===\n");
push_allocator(context.default_allocator);
kind_tests();
exif_tests();
orientation_tests();
date_tests();
folder_tests();
return report();
}

4
src/test/testing.jai
View File

@ -41,6 +41,7 @@ Test_Type :: enum s32 {
catalog_suite; catalog_suite;
ocr_suite; ocr_suite;
labels_suite; labels_suite;
import_suite;
} }
testing_started :: () -> bool { testing_started :: () -> bool {
@ -84,6 +85,7 @@ run_all_tests :: () {
add_suite_test(.catalog_suite, "catalog"); add_suite_test(.catalog_suite, "catalog");
add_suite_test(.ocr_suite, "ocr"); add_suite_test(.ocr_suite, "ocr");
add_suite_test(.labels_suite, "labels"); add_suite_test(.labels_suite, "labels");
add_suite_test(.import_suite, "import");
} }
// One work item per suite. Each suite keeps its tallies in file-scope globals, so the whole // One work item per suite. Each suite keeps its tallies in file-scope globals, so the whole
@ -194,6 +196,8 @@ test_thread_group_proc :: (group: *Thread_Group, thread: *Thread, work: *void) -
task.pass = run_ocr_tests(task.output); task.pass = run_ocr_tests(task.output);
case .labels_suite; case .labels_suite;
task.pass = run_labels_tests(task.output); task.pass = run_labels_tests(task.output);
case .import_suite;
task.pass = run_import_tests(task.output);
} }
return .CONTINUE; return .CONTINUE;

58
tools/image_probe.jai Normal file
View File

@ -0,0 +1,58 @@
// image-probe: what modules/Image_Decode makes of image files. For each FILE, its kind, EXIF and the
// decoded size; with -out FOLDER, the decoded image (turned the right way up) and its .mcap preview
// written there as PNGs, to look at.
//
// Build: jai first.jai - image_probe Run: bin/image-probe FILE... [-out FOLDER]
main :: () {
args := get_command_line_arguments();
files: [..] string;
out_folder: string;
i := 1;
while i < args.count {
if args[i] == "-out" && i + 1 < args.count { out_folder = args[i + 1]; i += 2; continue; }
array_add(*files, args[i]);
i += 1;
}
if !files.count { print("Usage: image-probe FILE... [-out FOLDER]\n"); exit(2); }
if out_folder make_directory_if_it_does_not_exist(out_folder, recursive = true);
failures := 0;
for path: files {
data, ok := read_entire_file(path, log_errors = false);
if !ok { print("%: cannot be read\n", path); failures += 1; continue; }
bytes := cast([] u8) data;
kind := image_kind_from_name(image_kind(bytes), path);
exif := exif_read(bytes, kind);
print("%\n kind %, % bytes\n", path, image_kind_name(kind), bytes.count);
if exif.found {
print(" EXIF: orientation %, date \"%\" %, make \"%\", model \"%\", software \"%\"\n", exif.orientation, exif.date_original, exif.offset_original, exif.make, exif.model, exif.software);
if exif.has_gps print(" GPS: %, %, % m\n", exif.latitude, exif.longitude, exif.altitude);
}
if kind == .HEIF || kind == .AVIF {
heif := heif_info(bytes);
print(" container: % x %, irot %, imir %, Exif item % bytes\n", heif.width, heif.height, heif.rotation, heif.mirror, heif.exif.count);
}
start := current_time_monotonic();
image, decoded, error := image_decode(bytes, kind);
if !decoded { print(" cannot decode: %\n", error); failures += 1; continue; }
print(" decoded: % x % in % ms (%)\n", image.width, image.height, FF(to_float64_seconds(current_time_monotonic() - start) * 1000, 1), ifx image.from_stb then "stb_image" else "WIC or turned");
if out_folder {
name := path_strip_extension(path_filename(path));
full := stbi_write_png(temp_c_string(tprint("%/%.png", out_folder, name)), image.width, image.height, 4, image.pixels, image.width * 4);
png, w, h, preview_ok, preview_error := image_preview_png(image, 320);
if preview_ok write_entire_file(tprint("%/%.preview.png", out_folder, name), png.data, png.count);
print(" wrote %/%.png%\n", out_folder, name, ifx preview_ok then tprint(" and a % x % preview", w, h) else tprint("; no preview: %", preview_error));
}
image_free(*image);
reset_temporary_storage();
}
exit(xx ifx failures then 1 else 0);
}
#import "Basic";
#import "String";
#import "File";
#import "Image_Decode";
#import "stb_image_write";

10
tools/ocr_bench.jai
View File

@ -63,17 +63,16 @@ main :: () {
Capture :: struct { name: string; pixels: *u8; width, height: s32; } Capture :: struct { name: string; pixels: *u8; width, height: s32; }
captures: [..] Capture; captures: [..] Capture;
decode_start := current_time_monotonic(); decode_start := current_time_monotonic();
for file_list(folder) { for file_list(folder, recursive = true) {
if captures.count >= limit break; if captures.count >= limit break;
if !ends_with_nocase(it, ".mcap") continue; if !ends_with_nocase(it, ".mcap") continue;
data, ok := read_entire_file(it); data, ok := read_entire_file(it);
if !ok continue; if !ok continue;
m, parsed, error := mcap_parse(cast([] u8) data); m, parsed, error := mcap_parse(cast([] u8) data);
if !parsed { print("Skipping %: %\n", it, error); continue; } if !parsed { print("Skipping %: %\n", it, error); continue; }
w, h, channels: s32; image, decoded, decode_error := image_decode(m.original);
pixels := stbi_load_from_memory(m.original.data, cast(s32) m.original.count, *w, *h, *channels, 4); if !decoded { print("Skipping %: its image cannot be decoded (%).\n", it, decode_error); continue; }
if !pixels { print("Skipping %: its image cannot be decoded.\n", it); continue; } array_add(*captures, .{copy_string(path_strip_extension(path_filename(it))), image.pixels, image.width, image.height});
array_add(*captures, .{copy_string(path_strip_extension(path_filename(it))), pixels, w, h});
free(data); free(data);
} }
if !captures.count { print("No .mcap files in %.\n", folder); exit(1); } if !captures.count { print("No .mcap files in %.\n", folder); exit(1); }
@ -169,4 +168,5 @@ S64_MAX_VALUE :: 0x7FFF_FFFF_FFFF_FFFF;
#import "System"; #import "System";
#import "stb_image"; #import "stb_image";
#import "Capture_File"; #import "Capture_File";
#import "Image_Decode";
#import "OCR"; #import "OCR";