53 lines
2.8 KiB
Markdown
53 lines
2.8 KiB
Markdown
# Labels
|
|
|
|
Images and text in one space, from an image-text encoder: SigLIP 2 base at 384 px (Google, Apache
|
|
2.0), as converted to ONNX by onnx-community. An image can then be scored against any phrase: a label
|
|
list, or words typed into a search. It runs ONNX models through onnxruntime
|
|
([`modules/onnxruntime`](../onnxruntime)), sharing its session and CUDA code with
|
|
[`modules/OCR`](../OCR).
|
|
|
|
## Setting up
|
|
|
|
```
|
|
cd modules/onnxruntime && jai fetch.jai # if OCR has not fetched it already
|
|
cd modules/Labels && jai fetch_models.jai # the vision model (fp32, 373 MB), the text model (int8, 283 MB), tokenizer.json (34 MB)
|
|
```
|
|
|
|
Each file's SHA-256 is checked. Building the app copies the models into `bin/label-models`.
|
|
|
|
## Using it
|
|
|
|
```jai
|
|
#import "Labels";
|
|
|
|
engine, ok, error := labels_create(models_folder, "siglip2-base-patch16-384"); // prefer_gpu = true
|
|
images := labels_embed_images(engine, pictures); // [] Labels_Image (RGBA): one unit vector each, engine.dimensions long
|
|
phrases := labels_embed_texts(engine, .["a photo of a cat", "a screenshot of a terminal"]);
|
|
score := labels_score(engine, images.data, phrases.data); // 0 to 1: the model's sigmoid
|
|
labels_destroy(engine);
|
|
```
|
|
|
|
- **Images** are resized to 384 x 384 (not keeping their shape) with PIL's bilinear filter, which
|
|
averages over each output pixel's footprint when shrinking, then scaled to -1..1: as SigLIP's own
|
|
processor does. The vision model runs on CUDA when it can, else on the CPU.
|
|
- **Text** is lowercased (ASCII letters) and tokenized with Gemma's tokenizer (`tokenizer.jai`: BPE
|
|
with byte fallback, read from `tokenizer.json`), cut or padded to 64 tokens. The text model runs
|
|
on the CPU: short inputs, a few at a time.
|
|
- **Scores** are `sigmoid(e^logit_scale * cosine + logit_bias)`, with the trained model's
|
|
`logit_scale` and `logit_bias` (in `models.config`; the ONNX encoders do not have them). They are
|
|
small for most phrases; their order is what counts. Descriptive prompts ("a screenshot of a code
|
|
editor showing source code") score far better than single words.
|
|
- `labels_to_half` and `labels_from_half` turn a vector into 16-bit floats, as `.mcap` files store
|
|
them, and back.
|
|
|
|
## Checked against the reference
|
|
|
|
- **The tokenizer** gives the same ids as Hugging Face's `tokenizers` (0.23.2) on the same
|
|
`tokenizer.json`, for 336 strings: label phrases, punctuation, digits, accents, Chinese, Japanese,
|
|
emoji, tabs and newlines, and text past 64 tokens.
|
|
- **The vectors** match SigLIP's own preprocessing through the same ONNX files (Python, onnxruntime
|
|
and PIL): image vectors at a cosine of 0.9998 or more, text vectors exactly, scores within 0.001.
|
|
|
|
On the RTX 5080 laptop, an image takes about 18 ms on the GPU (preparing it included) and 190 ms on
|
|
the CPU; a phrase about 5 ms.
|