Skip to content
sorryhyunPublic

About

LoRA Generation with Delta Supervision

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LLGBM: Learning LoRA Generator by Behavioral Matching

Train prompt-conditioned LoRA generators with behavioral supervision. Instead of only matching LoRA weights via MSE (as in DnD), the generator is trained to match delta activations: the difference in hidden states between base and adapted models.

L = λ_w * L_weight + λ_d * L_delta

Δ(M, B) = v(M) - v(B)
v(M)    = E[h^(L)_M(p, last_token)]  for p in P (probe texts)

The generator learns to produce LoRA weights that induce the same behavioral shift as the teacher adapters, not just numerically similar weights.

Delta-Guided Training

Computing Δ for a generated LoRA needs forward passes through the base model on every probe, which is expensive. Delta-guided training adds a cheap delta head that predicts Δ directly from the condition embeddings, and grounds it against the real Δ periodically:

Text Prompt ──► [Text Encoder] ──► condition
                                      │
                               [LoRA Generator]
                                 │          │
                           LoRA weights  [Delta Head]
                                 │          │
                          (every N steps)  δ_predicted
                                 ▼          │
                            δ_computed ◄────┘

L = λ_pred        * MSE(δ_pred,     δ_teacher)               every step
  + λ_computed    * MSE(δ_computed, δ_teacher)   ┐ every N steps
  + λ_consistency * MSE(δ_pred,     δ_computed)  ┘

This cuts base-model forward passes by roughly N× while keeping the generated weights grounded in real behavior.

Findings so far

  • DnD's tokenize→detokenize roundtrip keeps LoRA weight error < 1e-3.
  • Delta norms are ~0.1 and cluster by domain in t-SNE.
  • All LoRA tensors receive gradients through FunctionalLoRA hooks.
  • Delta-only and delta-guided training both converge.

Pipeline

Text Prompts
    ↓
[PretrainedTextEncoder] → condition embeddings (384-dim, MiniLM-L6-v2)
    ↓
[LoRAGenerator]         → LoRA weights (A, B matrices per layer)
    ↓
[FunctionalLoRA]        → apply LoRA via forward hooks (no weight cloning, gradient-enabled)
    ↓
[compute_delta_*]       → Δ_pred via probe texts
    ↓
[MultiTaskLoss / DeltaGuidedLoss]

Setup

uv venv && source .venv/bin/activate
uv pip install -r dnd_repo/requirements.txt
uv pip install peft trl datasets sentence-transformers pandas scikit-learn matplotlib seaborn

Usage

All scripts run from the repo root as modules and take --help.

# 1. Train teacher adapters (non-overlapping data batches) and compute their deltas
python -m scripts.train_lora_adapters --tasks arc_e boolq piqa

# 2. Train a generator
python -m scripts.train_generator --loss weight       # raw (A, B) MSE baseline
python -m scripts.train_generator --loss delta_w      # MSE on B@A*scaling (gauge-invariant)
python -m scripts.train_generator --loss multitask    # weights + probe deltas
python -m scripts.train_generator --loss delta_only   # probe deltas only
python -m scripts.train_delta_guided                  # dual-head generator, 3-part loss

# 3. Ablation: delta_guided vs multitask vs weight_only, 3 seeds each
python -m scripts.run_ablations --eval

Outputs go to outputs/<run>/ (config.json, results.json, training_curves.png, checkpoints). Task accuracy is evaluated when data/{task}_eval.json files exist.

Package Structure

llgbm/
├── probes.py        # create_generic_probes(), create_domain_probes()
├── delta.py         # compute_base_activation(), compute_adapter_delta(), DeltaCache
├── dataset.py       # RealAdapterDataset, DeltaAugmentedDataset
├── functional.py    # FunctionalLoRA, compute_delta_differentiable()
├── losses.py        # WeightLoss, DeltaWLoss, MultiTaskLoss, DeltaOnlyLoss, DeltaGuidedLoss
├── training.py      # TrainingConfig, train(), evaluate()
├── generator.py     # LoRAGenerator, LoRAGeneratorWithDeltaHead
├── text_encoder.py  # PretrainedTextEncoder, TrainableTextEncoder
├── evaluation.py    # compute_accuracy_with_lora_batched(), compute_base_accuracy()
└── ablations.py     # AblationConfig, run_ablations()
scripts/             # entry points (see Usage)
results/             # saved ablation results

Model Configuration

Config Qwen2.5-0.5B-Instruct (default) Qwen2.5-1.5B (--large_model)
Layers 24 28
Hidden 896 1536
Intermediate 4864 8960
LoRA rank / alpha 16 / 32 16 / 32

LoRA targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Data Layout

data/
├── ARC-e_train.json, BoolQ_train.json, ...   # {prompt, response[, system]} lists
└── {task}_eval.json                           # optional, for task accuracy

checkpoints/
├── manifest.json                              # adapter index
├── {task}/{task}_{idx:03d}/adapter/
│   ├── adapter_model.safetensors
│   ├── prompts.json
│   └── metrics.json
└── deltas/
    ├── delta_manifest.json
    ├── base_activation.npy
    └── {adapter}_delta.npy

References

About

LoRA Generation with Delta Supervision

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages