RegCache

RegCache is a CLIP vision-encoder pipeline for register-token prefix caching and token deletion. The public code path focuses on ImageNet classification with three baselines:

• naive
• ptq4vit
• repqvit

Other internal baselines are intentionally not part of the public workflow.

What This Repository Provides

• CLIP register candidate extraction
• Top-100 prefix averaging
• Quantization-sensitive block search
• Target-block search over max(0, k-3)..k
• Prefix-count and token-delete-count search
• Final ImageNet classification evaluation
• Optional PTQ4ViT and RepQ-ViT calibration paths

Directory Overview

• src/open_clip: OpenCLIP model code used by RegCache
• src/open_clip_train/main_RegCache.py: main RegCache entrypoint
• models/clip_find_prefix_by_target_block.py: prefix candidate extraction modules
• models/clip_sweep_config_delete_directly.py: prefix injection, token deletion, and quantized CLIP modules
• models/custom_utils/sweep_avg.py: prefix cache application utilities
• scripts/run_clip_regcache_pipeline.sh: full end-to-end CLIP RegCache pipeline
• scripts/eval/classification/clip.sh: classification evaluation launcher
• scripts/extract_register_candidates/clip.sh: prefix extraction launcher
• scripts/search_regcache/clip.sh: prefix and delete-count search launcher

Installation

conda env create -f environment.yml
conda activate regcache

The launcher scripts default to the active environment's Python:

python

Override it when needed:

PYTHON=/path/to/python bash scripts/eval/classification/clip.sh

The scripts set PYTHONPATH internally. For manual python -m ... commands, run:

export PYTHONPATH=/path/to/RegCache/src

Data Layout

Expected datasets:

• ImageNet validation set
• ImageNet train-sample set for prefix extraction and search
• Optional calibration set for PTQ4ViT and RepQ-ViT

Set dataset paths with environment variables:

export DATA_ROOT=/path/to/imagenet
export IMAGENET_VAL=${DATA_ROOT}/val
export TRAIN_SAMPLE=${DATA_ROOT}/train_sample_50_per_class
export CALIB=${DATA_ROOT}/1024calib_train
export COCO_DATA_ROOT=/path/to/coco

Override any path per command when needed.

Quick Start

Run the full RegCache pipeline:

bash scripts/run_clip_regcache_pipeline.sh

Run a specific baseline:

BASELINE=naive bash scripts/run_clip_regcache_pipeline.sh
BASELINE=ptq4vit bash scripts/run_clip_regcache_pipeline.sh
BASELINE=repqvit bash scripts/run_clip_regcache_pipeline.sh

The pipeline performs:

1. Quantization-sensitive block search
2. Prefix-extraction block search
3. Prefix candidate extraction
4. Top-100 prefix averaging
5. Prefix-count and token-delete-count search
6. Final ImageNet classification evaluation

The final output prints:

• quantization-sensitive block k
• quantization-sensitive layer
• prefix-extraction block candidates
• selected prefix-extraction block
• selected prefix-add blocks
• selected prefix token count
• selected token-delete block
• selected token-delete count
• search accuracy
• final result CSV path

Manual Evaluation

Run classification evaluation directly:

BASELINE=naive bash scripts/eval/classification/clip.sh
BASELINE=ptq4vit bash scripts/eval/classification/clip.sh
BASELINE=repqvit bash scripts/eval/classification/clip.sh

Run MS-COCO retrieval evaluation:

BASELINE=naive bash scripts/eval/retrieval/clip.sh

Public Baseline Policy

Only these baselines are supported in the public entrypoint:

• naive
• ptq4vit
• repqvit

Passing any other baseline to main_RegCache.py raises an error.

Files That Should Not Be Committed

Generated outputs, checkpoints, cached prefixes, logs, and calibration tensors should stay out of Git. The .gitignore excludes:

• clip_global_prefix*
• result*
• checkpoint*
• logs
• *.pt
• *.pth
• *.log
• *.csv
• __pycache__

Citation

This repository builds on OpenCLIP, PTQ4ViT, and RepQ-ViT. Cite the original projects together with RegCache when using this code.

License

See LICENSE.