Graduate student based in Beijing, working at the intersection of vision foundation models and multimodal large language models. I build small, dependency-light open-source tooling for vision-language understanding, multimodal alignment, and the practical problem of connecting vision encoders to LLMs — with a bias toward reproducible, offline-by-default code.
- Vision foundation models — frozen encoders (CLIP-style ViTs) as reusable visual backbones.
- Multimodal LLMs — how visual features become language-model tokens, and what that costs in context budget.
- Multimodal alignment & retrieval — shared image-text embedding spaces for grounding and RAG.
- Evaluation & reproducibility — metrics implemented from scratch, runnable with no model downloads.
A small, connected stack — connect a vision encoder to an LLM, evaluate the result, and ground it with retrieval:
| Project | What it is |
|---|---|
| graft | Pluggable PyTorch connectors that map frozen vision-encoder features into an LLM's token embedding space for visual instruction tuning. |
| vlmscope | A dependency-light evaluation harness for vision-language models across VQA, captioning, and image-text retrieval. |
| tessera | A small, swappable multimodal RAG framework that indexes mixed image/text corpora and assembles grounded context for any LLM. |
🔗 graft
The connector (projector / resampler) is the smallest, most-swapped part of a
VLM, but it usually lives buried inside a modeling file. graft pulls it out
behind one tiny typed API and collects the common architectures —
linear (LLaVA v1), mlp (LLaVA v1.5), avgpool, pixel_shuffle,
perceiver (Flamingo), and qformer (BLIP-2) — so you can swap them with a
single string and treat the visual token budget as a first-class, comparable
number.
PyTorch · vision-language · visual-instruction-tuning
📏 vlmscope
A NumPy-only harness for the three benchmark families that keep coming up in practice: VQA (standard soft accuracy and exact match), captioning (BLEU, ROUGE-L, and CIDEr, all implemented from scratch in pure Python), and image-text retrieval (Recall@k plus median/mean rank). It runs on built-in toy datasets out of the box, so results are reproducible without downloading anything — and you can score your own predictions file directly.
evaluation · vqa · image-captioning · image-text-retrieval
🧩 tessera
Multimodal retrieval-augmented generation built from small, swappable pieces.
tessera embeds images and text into a single shared space — so a text query
can surface image chunks and vice versa — then retrieves source-attributed
context for a language model. The core depends only on NumPy and ships a
deterministic encoder that runs fully offline; drop in the optional CLIP encoder
or a FAISS store when you want real cross-modal semantics or scale.
rag · multimodal · clip · vector-search
Exploring how connector design trades visual token budget against vision-language quality, and keeping the evaluation tooling above honest and reproducible.
All three projects share the same design philosophy: a tiny dependency core, swappable components behind small interfaces, and something that runs end-to-end offline before you reach for a GPU.