license	apache-2.0

FAB-G Salience Extension — Salient-Cue Prediction

Multi-label classifier that predicts which formal attributes of an artwork are salient emotion cues, from the image alone. Output is a set of 1–5 cues from {composition, color, line, brushstroke, light}.

The supervision label is the tags field of the parquet files (train 1250 / test 150; the rare junk tag abandon is dropped, and rows whose only tag was abandon carry no in-scope supervision).

Approach

Two fixed candidate vision backbones are fine-tuned as multi-label classifiers and the best by val Macro-F1 is recommended (both checkpoints kept):

backbone	HF id	size	resolution
DINOv3 ViT-L/16	camenduru/dinov3-vitl16-pretrain-lvd1689m	304M	224
SigLIP2 Giant (vision tower only)	google/siglip2-giant-opt-patch16-384	~1.16B	384

Key design points:

• Label-safe augmentation — color is a target so no colour ops; composition is global so crops stay near-identity (scale 0.85–1.0) and evaluation uses full-frame resize (no centre crop).
• Asymmetric Loss for the strong class imbalance (composition 93.6% … light 39%).
• Per-class threshold tuning on a multilabel-stratified val split (128 rows), with a top-1 floor and a cap of 5 cues per prediction.
• SigLIP2 full fine-tune on a 24GB GPU requires --optim-8bit --grad-ckpt --no-ema (fp32 AdamW states alone are ~9.3GB for the 1.16B tower).

Layout

src/data.py            parquet decode, 5-dim multihot, stratified split, label-safe augment
src/losses.py          Asymmetric Loss, weighted BCE, all metrics (micro/macro/per-sample F1, mAP, exact match)
src/models.py          backbone wrappers (pooling + per-backbone normalisation)
src/train_backbone.py  two-phase training: linear probe -> full fine-tune (EMA, cosine, early stop)
src/thresholds.py      per-class threshold tuning on val probabilities
src/eval.py            full metric report on test.parquet
src/infer.py           single-image inference -> predicted cue set
configs/backbones.yaml per-backbone hyperparameters
outputs/reports/       metric reports + smoke-test summary

The parquet data and model checkpoints are not in this repo (see .gitignore); outputs/manifests/facts.json records the label statistics.

Usage

conda activate salient            # python 3.12, torch 2.10 cu128
pip install -r requirements.txt   # note: transformers pinned >=4.56,<5
export CUDA_VISIBLE_DEVICES=<idle gpu ids>

python -m src.data --build-split                                   # manifests + label facts
python -m src.train_backbone --backbone dinov3_vitl16
python -m src.train_backbone --backbone siglip2_giant --optim-8bit --grad-ckpt --no-ema
python -m src.thresholds --backbone <key>                          # -> thresholds.json
python -m src.eval --backbone <key> --split test                   # full metric report
python -m src.infer --image <artwork.jpg> --backbone <key>         # single-image cues

See outputs/reports/smoke_summary.md for the end-to-end verification log and environment notes (transformers 4.x pin, SigLIP2 loading path, GPU memory).