Multi-task Emotion Recognition on EmoArt-5k

Joint prediction of three emotion dimensions for artworks — arousal (Low/High), valence (Negative/Positive) and dominant emotion (12 classes) — from a single shared vision backbone, on the EmoArt-5k dataset (5,532 annotated paintings across ~55 styles). Two interchangeable backbone tracks share the same data pipeline, split and evaluation protocol, so results are directly comparable:

Track	Backbone	Head(s)	Key imbalance tools
siglip2/	SigLIP2 giant-opt (1.4B, 384px, attention-pooled)	3 × linear	mild (1/freq)^0.5 class weights, emotion→arousal/valence consistency, weighted sampler, paged 8-bit AdamW for 24 GB GPUs
dinov3/	DINOv3 ViT-L/16 (LVD-1689M), CLS + mean-patch features	2 × linear + 1 MLP	effective-number focal loss (β=0.999), sqrt-freq weighted sampler, Kendall uncertainty weighting (optional)

Both use: stratified 80/10/10 split by dominant_emotion (seed 42, rare classes guaranteed in train), layer-wise LR decay, cosine schedule + warmup, bf16, EMA, label smoothing on the binary heads, gradient checkpointing, early stopping, and test-time evaluation with overall accuracy + macro-F1 + balanced accuracy + confusion matrices.

Repository layout

emoart-multitask/
├── data/README.md        # how to obtain/place the EmoArt-5k dataset (not redistributed)
├── siglip2/              # SigLIP2 track: prepare_data, train (accelerate/DDP), eval, infer
└── dinov3/               # DINOv3 track: prepare_data, train (torchrun/DDP), evaluate

Setup

python -m venv .venv && source .venv/bin/activate   # or conda
pip install -r requirements.txt
# fine-tuning siglip2-giant-opt on 24 GB GPUs additionally needs: pip install bitsandbytes

Tested with torch 2.10.0+cu128, torchvision 0.25.0, transformers 5.16.1. (transformers>=4.49 covers SigLIP2; >=4.56 is required for DINOv3.)

Data

The dataset is not included in this repository. Place it so that the repo root (or $EMOART_DATA_ROOT) contains:

annotation.json                     # 5,532 records with third_section labels
Images/Images/<Style>/*.jpg         # Images/<Style>/ also works

Then, in either track: python <prepare_data.py> — see data/README.md for details on path quirks the loader handles automatically.

Quickstart

# --- DINOv3 track ---
cd dinov3
python src/prepare_data.py
python src/train.py --smoke                                      # pipeline check
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 \
    src/train.py --config configs/dinov3_multitask.yaml          # real training
python src/evaluate.py --ckpt outputs/best.pt --tta              # test metrics

# --- SigLIP2 track ---
cd ../siglip2
python prepare_data.py
python train.py --smoke
CUDA_VISIBLE_DEVICES=0,1 accelerate launch --multi_gpu --num_processes 2 \
    --mixed_precision bf16 train.py                              # real training
python eval.py --ckpt outputs/best.pt --tta
python infer.py --ckpt outputs/best.pt --image some.jpg          # single-image inference

Pretrained backbone weights download from huggingface.co on first run (~1.2 GB for DINOv3 ViT-L, ~6 GB for SigLIP2 giant-opt).

License

Code is released under the MIT License. The EmoArt-5k dataset and the pretrained backbones (SigLIP2, DINOv3) are governed by their own respective terms.