Musa Mahmood
72f2754226
- modules/Screen_Record/audio_source.jai: WASAPI, in loopback from the default output device (what the computer plays) and from the default input device (the microphone), asked for as 48 kHz stereo float, which Windows converts any device to; a device that will not convert is taken as it is when it gives 48 kHz float. Packets carry the time they were heard. A device lost (unplugged, default changed) is opened again from the new default once a second. - An audio thread mixes them onto the video's own timeline: the same time 0, less the same pauses (record_pause now keeps one account of time paused for both threads). A packet within 30 ms of where its source left off follows on exactly; a source that sends nothing is silence, passed 200 ms late. Each 1024 samples every source has covered become an AAC frame. - encoder.jai: an AAC stream (FFmpeg's aac, 160 kb/s) beside the video, written from the audio thread; both threads' packets meet in the file under a lock. The remux keeps it. - [video] system_audio (true) and microphone (false). capture.json: Audio, AudioSources. A toast when a source asked for could not be opened. - record-probe: -audio system|mic|both, the sound read back (length, level), and -sync-test, which flashes a white square and beeps together every 2 s, then finds both in the file. - Measured: 20 s, beeps 0-21 ms from their flashes, no drift; both sources mixed, 5-18 ms; a 6 s pause cuts the same 6 s from sound and video; 3 min, 85 of 90 beeps within 3-24 ms and -14 ms from first to last, with 5 outliers (53-147 ms) where two beeps went missing. Tests: an MP4 with sound keeps its AAC stream through the remux into the library (30 checks).
289 lines
16 KiB
Markdown
289 lines
16 KiB
Markdown
# The .mcap file
|
|
|
|
One capture in one file: a fixed magic, a file version stamp, the capture's data as
|
|
[Capture Text](capture_text.md), what OCR read in it, what other models added (chunks), a small preview,
|
|
then the original and rendered image, then any large chunks. There is no ZIP and no compression around
|
|
it; the images and videos keep their own compression.
|
|
|
|
A capture is a Snagit capture (from a `.snagx`), an imported image (a photo or screenshot in any image
|
|
format), an imported video (the mcap app's Import window), or a screen capture or recording (the mcap app's
|
|
PrintScreen and Ctrl+PrintScreen).
|
|
|
|
The parts a library browser needs (header, text, OCR, front chunks and preview) come first, so it reads
|
|
the start of each file and never touches the images; a search, which needs no preview, reads less still.
|
|
|
|
## Layout, version 4
|
|
|
|
All integers are little-endian. Sizes count bytes.
|
|
|
|
| Field | Type | |
|
|
| --- | --- | --- |
|
|
| magic | 8 bytes | `89 4D 43 41 50 0D 0A 1A`, which is `\x89MCAP\r\n\x1A`. Says this is an .mcap, in every version. |
|
|
| file version | u32 | `4`. Versions 1 to 3 are still read; see below. |
|
|
| text size, text | u32, bytes | Capture Text, UTF-8: every JSON file of the capture, as `[[file]]` sections. |
|
|
| OCR size, OCR | u32, bytes | OCR text, UTF-8 (below): what each OCR model read. Size 0 when none has. |
|
|
| front chunks | chunk list | Small things models added, such as labels and an image vector (below). |
|
|
| preview width, height | u16, u16 | In pixels. |
|
|
| preview size, preview | u32, bytes | PNG. Size 0 when there is none. |
|
|
| original name size, name | u16, bytes | The original's name: in the .snagx, such as `{GUID}.png` (the page's `CaptureBackgroundImage` names it too), or an imported file's name, such as `IMG_0001.HEIC` or `IMG_4022.MOV`. |
|
|
| original size, original | u64, bytes | The capture as taken, exactly as it was: PNG from a .snagx; an imported image or video in its own format (JPEG, HEIC, WebP, RAW, MP4, MOV, MKV ...), told by its first bytes, not its name. |
|
|
| rendered kind | u8 | 0: none. 1: the same as the original (stored once). 2: its own, below. |
|
|
| rendered size, rendered | u32, bytes | Only for kind 2. The capture with its annotations drawn in (Snagit's `thumbnail.png`): PNG, exactly as it was. A video's is a frame from it, as PNG (below). |
|
|
| back chunks | chunk list | Large things models added, such as segmentation masks: after the images, so reading the front never reaches them. |
|
|
|
|
The file ends there: anything after the last part is a mistake.
|
|
|
|
The magic borrows PNG's: the first byte has its high bit set, so a channel that strips it shows; `\r\n`
|
|
shows line endings being converted; and `\x1A` stops `type` on Windows.
|
|
|
|
A **chunk list** is a u32 count, then each chunk: a 4-byte tag of ASCII letters saying what it is
|
|
(`labl`, `embd`), a u32 size, and that many bytes: a u16 key size, the key (whose it is: a model's
|
|
name), and the chunk's own data. A file holds at most one chunk per tag and key.
|
|
|
|
**Version 3** is version 4 with the original's size a u32, so an original of at most 4 GB; **version 2**
|
|
is version 3 without the two chunk lists; **version 1** is version 2 without the OCR size and OCR. A
|
|
reader takes all four; anything written is version 4, so an older file becomes version 4 the first time
|
|
something is stored in it.
|
|
|
|
An original can be larger than memory, so a file is read and rewritten with its original left on disk:
|
|
`mcap_read_without_original` reads everything else and says where the original lies, and
|
|
`mcap_write_streamed` copies it across a few MB at a time. Storing what a scan adds works that way.
|
|
|
|
## The parts
|
|
|
|
- **Text.** Everything the capture's JSON held: `index.json`, `metadata.json`, the page's `{GUID}.json`,
|
|
and `{GUID}.backup.json` when there is one. An imported image has two files instead (below).
|
|
- **OCR.** One section per OCR model, so a capture read by several models keeps every reading;
|
|
reading it again with a model replaces only that model's section. See *OCR text* below.
|
|
- **Chunks.** What other models add, each keyed by the model's name, so models do not overwrite each
|
|
other; storing a chunk again replaces only the one with its tag and key. A chunk a reader does not
|
|
know is kept as it is when the file is rewritten, so a later build can add kinds without a new file
|
|
version. See *Label chunks* below.
|
|
- **Preview.** The rendered image (or the original, when there is no rendered one) scaled so its longest
|
|
side is at most 320 pixels, as a PNG. A Snagit PNG already that small is used as it is, never scaled
|
|
up. An imported image's preview is always a PNG made from it, turned the right way up by its EXIF
|
|
orientation, which a reader applies to the original too.
|
|
- **Original and rendered.** A capture with no annotations has a rendered image identical to the original,
|
|
byte for byte; it is then stored once, as kind 1.
|
|
- **One page.** An .mcap holds one page, as every capture so far does; a .snagx with more is not converted.
|
|
- **Left out:** `{GUID}.backup.png`, Snagit's image from before an edit. Its `{GUID}.backup.json` is kept,
|
|
as plenty of captures have one without the image.
|
|
|
|
## Imported images
|
|
|
|
An image imported from a folder has no rendered image (kind 0), and its Capture Text holds two files:
|
|
|
|
```
|
|
[[file "metadata.json"]]
|
|
AppName: "Apple iPhone 15 Pro" the camera or phone, from EXIF; "" when unknown
|
|
CaptureDate: "2024-05-12 10:15:30" when it was taken, local time, as Snagit writes it
|
|
WindowName: "IMG_4021.HEIC" the file's name
|
|
|
|
[[file "import.json"]]
|
|
SourcePath: "E:/Phone/DCIM/IMG_4021.HEIC" where it was imported from
|
|
SourceSize: 2994394
|
|
SourceModified: "2024-05-12 10:15:31"
|
|
ContentHash: "d3fa07db58597cec" 64-bit FNV-1a of the original's bytes
|
|
Format: "HEIF"
|
|
Width: 3024 as shown: after the orientation is applied
|
|
Height: 4032
|
|
DateFrom: "EXIF" or "file name", or "modified time"
|
|
ImportDate: "2026-10-08 22:28:17"
|
|
Orientation: 6 and, when the EXIF has them: CameraMake, CameraModel,
|
|
Software, TimeZoneOffset, Latitude, Longitude, Altitude
|
|
```
|
|
|
|
So the parts a library reads from Snagit's `metadata.json` (date, title, app) work for photos too.
|
|
|
|
## Screen captures
|
|
|
|
A region of the screen taken with the mcap app's PrintScreen is stored as a PNG original with no
|
|
rendered image (kind 0), and its Capture Text holds two files:
|
|
|
|
```
|
|
[[file "metadata.json"]]
|
|
AppName: "Google Chrome" the program of the window under the region's middle:
|
|
its description, else its file's name; "" when unknown
|
|
CaptureDate: "2026-10-10 14:05:09" when it was taken, local time, as Snagit writes it
|
|
WindowName: "Assign Footprints - KiCad" that window's title
|
|
|
|
[[file "capture.json"]]
|
|
Region: "selection" or "screen": a click took the whole screen
|
|
Left: -1920 on the virtual desktop, whose origin is the primary
|
|
Top: 40 screen's top left, so it can be negative
|
|
Width: 700
|
|
Height: 300
|
|
AppPath: "C:/Program Files/Google/Chrome/Application/chrome.exe"
|
|
ContentHash: "5c1e9a0d77b2f413" 64-bit FNV-1a of the PNG's bytes
|
|
Format: "PNG"
|
|
```
|
|
|
|
## Screen recordings
|
|
|
|
A recording made with the mcap app's Ctrl+PrintScreen is a video: its original is the H.264 MP4, and
|
|
its rendered image (kind 2) a frame from it as a PNG, as for an imported video. It is recorded as a
|
|
fragmented MP4, so a crash still leaves one that plays, and stored as a regular one, its frames copied
|
|
as they are: one header first, with its length and the index of every frame. Its
|
|
metadata.json is a still's. Its capture.json has a still's keys, with Format "MP4", and these:
|
|
|
|
```
|
|
Duration: 12.48 seconds
|
|
FrameRate: 30 as the container says; frames are written only when the screen changes
|
|
VideoCodec: "h264"
|
|
PosterTime: 1.25 seconds into it of the rendered frame
|
|
Encoder: "h264_nvenc" FFmpeg's encoder: h264_nvenc, h264_amf, h264_qsv, h264_mf or libx264
|
|
Capture: "DXGI DuplicateOutput1 on NVIDIA GeForce RTX 5080 Laptop GPU"
|
|
or "Windows.Graphics.Capture on ..., border off"
|
|
DroppedFrames: 0 frames the encoder fell too far behind for
|
|
Paused: 10.2 seconds paused, left out of the video; only when it was paused
|
|
Audio: true whether it has sound (AAC)
|
|
AudioSources: "SAMSUNG (NVIDIA High Definition Audio); Microphone Array (Realtek(R) Audio)"
|
|
the devices heard; only when it has sound
|
|
```
|
|
|
|
A recording a crash cut short, put in the library at the next start, has no window, Encoder or
|
|
Capture: its Width and Height are its frames', and its CaptureDate is from its file's name.
|
|
|
|
A reader tells a recording from a still by VideoCodec, as it tells an imported video by its import.json.
|
|
|
|
## Imported videos
|
|
|
|
A video imported from a folder (MP4, MOV, MKV, WebM, AVI, WMV, MPEG; read with FFmpeg, on Windows) is
|
|
stored as it was in the original. Its rendered image (kind 2) is a frame from it as a PNG, at its own
|
|
size and turned the right way up: a tenth of the way in, at most 3 seconds, past the black or blurred
|
|
frames phones often start with. The preview is made from that frame, and OCR and labels read it, as
|
|
they read a Snagit capture's rendered image; nothing reads the video itself but the import.
|
|
|
|
Its metadata.json is a photo's. Its import.json has the keys above (Format is the container: "MP4",
|
|
"QuickTime", "WebM" ...; Width and Height as shown) and these:
|
|
|
|
```
|
|
Duration: 83.4 seconds
|
|
FrameRate: 30
|
|
VideoCodec: "h264" FFmpeg's name for it: "hevc", "vp9", "av1" ...
|
|
Rotation: 90 degrees clockwise its frames are turned to be shown
|
|
PosterTime: 3 seconds into it of the rendered frame
|
|
Audio: true
|
|
DateFrom: "metadata" Apple's creationdate (local time), else the file name, else the
|
|
container's creation_time (UTC, as this computer's local time), else
|
|
the modified time
|
|
and, when the container has them: CameraMake, CameraModel, Software,
|
|
TimeZoneOffset, Latitude, Longitude, Altitude
|
|
```
|
|
|
|
An animated GIF is imported as an image; its import.json adds `Frames` and `Duration` (seconds).
|
|
|
|
## Versions
|
|
|
|
A reader checks the magic, then the file version. It refuses a version above its own, never writing
|
|
over a file from a later version, and reads every version up to its own. Version 0 does not exist.
|
|
|
|
## Converting
|
|
|
|
`snagit-converter mcap` converts .snagx files (or extracted captures) to .mcap, and `snagit-converter
|
|
snagx` converts .mcap files back to a .snagx Snagit can open. See the README for what each checks.
|
|
The mcap app's Import makes the same .mcap from a .snagx (modules/Snagx). Only captures from a .snagx
|
|
convert back: an imported image has no Snagit JSON to make one from.
|
|
|
|
Back in a .snagx, the JSON is laid out as Snagit writes it, the images are the same bytes, and only the
|
|
backup image is missing. JSON numbers come back as values: `3.0` is written `3`.
|
|
|
|
## OCR text
|
|
|
|
In the style of the Ex4 text formats:
|
|
|
|
```
|
|
@version 2
|
|
|
|
[ocr ppocr-v5-mobile]
|
|
engine: "onnxruntime 1.30.0, CUDA"
|
|
image: rendered 1920 1080
|
|
date: 2026-10-08T21:04:11Z
|
|
line: 412 96 640 18 412 96 1052 96 1052 114 412 114 0.993 "Assign Footprints - KiCad"
|
|
line: 20 140 220 14 20 140 240 140 240 154 20 154 0.981 "Symbol : Footprint Assignments"
|
|
line: 1700 300 20 160 1720 300 1720 460 1700 460 1700 300 0.952 "Read downwards"
|
|
```
|
|
|
|
| Key | Value |
|
|
| --- | --- |
|
|
| `[ocr NAME]` | A section: the model's name, as in its `models.config`. |
|
|
| `engine` | What ran it, quoted. |
|
|
| `image` | Which image was read (`rendered` or `original`) and its width and height; the lines' boxes are in its pixels. |
|
|
| `date` | When it was read, UTC. |
|
|
| `line` | A text line: x, y, width and height, then its four corners as x y pairs, its confidence (0 to 1), and its text, quoted with `""` for a quote. A control character in the text is written as a space. |
|
|
|
|
The corners go clockwise from the text's top left: top left, top right, bottom right, bottom left, as
|
|
the text reads. Level text has the box's corners; a turned line's say which way it reads (the third
|
|
line above reads downwards, so its top left is the box's top right). x, y, width and height bound them.
|
|
|
|
A line that cannot be read is skipped, as are keys this version does not know. **Version 1** OCR text
|
|
had no corners (`line: x y width height confidence "text"`); it is still read, with the box's corners.
|
|
The OCR text carries its own version, so this did not change the .mcap file version. `mcap_store_ocr`
|
|
(`modules/Capture_File/ocr_text.jai`) stores a section in a file.
|
|
|
|
## Label chunks
|
|
|
|
A label model (an image-text encoder such as SigLIP 2) adds two front chunks, keyed by its name.
|
|
|
|
**`labl`**: the capture's best labels from a label list, as text in the style above:
|
|
|
|
```
|
|
@version 1
|
|
engine: "onnxruntime 1.30.0, CUDA"
|
|
date: 2026-10-08T21:04:11Z
|
|
list: 9c4f0b2d81e3a675
|
|
label: 0.6720 "Settings"
|
|
label: 0.0951 "Installer"
|
|
```
|
|
|
|
| Key | Value |
|
|
| --- | --- |
|
|
| `engine`, `date` | As in OCR text. |
|
|
| `list` | Which label list they were scored against: a 64-bit FNV-1a hash of its names and prompts. Labels from another list are out of date. |
|
|
| `label` | A score, 0 to 1 (the model's own sigmoid; small, but their order is what counts), and the label's name, quoted with `""` for a quote. Best first. |
|
|
|
|
**`embd`**: the image's vector, binary: a u32 length, a u8 format (1: 16-bit floats), then the values,
|
|
little-endian. It is a unit vector, so any phrase's vector (from the same model's text encoder) scores
|
|
against it with a dot product: labels from a new list, or a search by meaning, need no new pass over
|
|
the images. 768 values, 1.5 KB, for SigLIP 2 base.
|
|
|
|
`mcap_store` (`modules/Capture_File/module.jai`) stores an OCR section and chunks in one rewrite;
|
|
`label_chunks.jai` reads and writes these two.
|
|
|
|
## Segmentation chunks
|
|
|
|
A segmentation model (Mask2Former, `modules/Segment`) adds two chunks, keyed by its name, in one
|
|
rewrite (`mcap_store` with `front` and `back`), so a file has both or neither. Images only: a video's
|
|
frame is not segmented.
|
|
|
|
**`segs`**, in front: what is in the image, as text in the style above. Search and Stats read it.
|
|
|
|
```
|
|
@version 1
|
|
engine: "onnxruntime 1.30.0, CUDA"
|
|
date: 2026-10-12T09:30:00Z
|
|
image: rendered 4032 3024
|
|
classes: 5c1e9a0d77b2f413
|
|
segment: 1 0.981 0.2140 1210 380 900 2210 "person"
|
|
segment: 4 0.993 0.3105 0 0 4032 1100 "sky"
|
|
```
|
|
|
|
| Key | Value |
|
|
| --- | --- |
|
|
| `engine`, `date` | As in OCR text. |
|
|
| `image` | Which image was read (`rendered` or `original`) and its size, as shown (turned by its EXIF orientation). The boxes are in its pixels. |
|
|
| `classes` | A 64-bit FNV-1a hash of the model's class list: segments from another list are out of date. |
|
|
| `segment` | Its id in the mask, its score (0 to 1), its area (a fraction of the image), its box (x y width height), and its class as people search it ("wall", not COCO's "wall-other-merged"), quoted with `""` for a quote. Largest first. |
|
|
|
|
**`mask`**, at the back: which segment each pixel is in, for drawing. Binary:
|
|
|
|
| Field | Type | |
|
|
| --- | --- | --- |
|
|
| format | u8 | `1`: run-length encoded. |
|
|
| width, height | u16, u16 | The map's: the image's shape, its longest side at most 1024. |
|
|
| id bits | u8 | `8`. |
|
|
| runs | | To the end of the chunk, row by row, wrapping from one row to the next: a u8 segment id (0: none) and its length, LEB128 (7 bits a byte, low bits first, the top bit set on every byte but the last). The runs cover the map exactly. |
|
|
|
|
On 27 photos and frames, a mask was 0.8 to 6.4 KB and the `segs` text 200 to 400 bytes.
|
|
`mcap_erase` with both tags takes them out together. `segment_chunks.jai` reads and writes them.
|