Skip to content

About

Experiments on how to combine denoising and transformer architectures

Resources

Stars

2 stars

Watchers

1 watching

Forks

Repository files navigation

Diffusion-LLM

A masked diffusion language model (LLaDA-style) for text-to-SQL generation, built on ModernBERT. Instead of writing SQL left to right, the model starts from a fully masked SQL window and fills it in over a few parallel denoising steps. This repo includes training, evaluation experiments, and a web playground.

Live demo: diffusion.lethonium.com

Diffusion Example

Results at a glance

Diffusion makes text-to-SQL faster, two ways

ModernBERT-base (150M), 10 epochs on gretelai/synthetic_text_to_sql, confidence decoding with adaptive early stop at 0.9. Evaluated on the first 256 filtered test rows:

Metric Value
Semantic accuracy (LLM judge) 0.50
Exact string match 0.32
Avg. denoising steps (forward passes) 11.8, vs ~30 tokens for autoregressive decoding
Largest failure mode invalid SQL (~14%)

Compared with an autoregressive (AR) model fine-tuned on the same data (Qwen experiment, same 256 rows, same LLM judge):

Model Params Semantic accuracy Forward passes
AR: Qwen2.5-Coder-0.5B SFT 500M 0.645 ~30 (one per token)
Diffusion: ModernBERT-base (this repo's main model) 150M 0.500 11.8
Diffusion: Qwen2.5-Coder-0.5B adapted to masked diffusion 500M 0.191 11.8

Findings so far:

  • Matched size, diffusion is 2.6× faster at tied accuracy. Ettin-150M encoder (diffusion) vs. decoder (AR), same data, 10 epochs, 1,000 test queries on one H100: 25 ms (CUDA graphs) vs 67 ms (vLLM), exec 0.564 vs 0.580 (−1.7 ± 3.5, a tie), exact match 0.277 vs 0.239 (+3.8 ± 2.3). Diffusion's gap is invalid SQL (−10 points), mostly closed by one verify pass (exec 0.591, 42 ms). See GPU session 2 and session 1.

  • A diffusion drafter makes the larger AR model 2.0× faster. Qwen2.5-0.5B (exec 0.735) on vLLM with a DFlash block-diffusion drafter for speculative decoding: 28 ms instead of 57 ms, at unchanged accuracy, and 1.3× faster than an EAGLE-3 drafter trained on the same data (37 ms). Above 150M, diffusion as the drafter beats diffusion as the model: Ettin-400M diffusion takes 32 ms at exec 0.606.

  • Adaptive early stop is the main speed lever. It cuts forward passes 1.8–3× with no quality loss (see Decoding efficiency).

  • More steps can hurt. Accuracy peaks around 11 steps (threshold 0.9) and falls ~10 points when decoding runs to a 0.99 threshold (~19 steps).

  • PAPL fine-tuning only saves steps. It gives ~7% fewer steps at equal accuracy (PAPL A/B).

  • On CPU (the hosted setup), AR is still the better trade. Cheap changes make diffusion 2.1× faster, but a 3.3× larger AR model matches its fp32 latency at higher accuracy (fair CPU benchmark, roadmap).

  • Dependency-ordered decoding (DOS) does not beat confidence ordering on this model (see below).

Repository layout

src/
  train.py              LLaDA-style training (ModernBERT, 1/t-weighted ELBO)
  denoising.py          confidence / dependency decoding shared by eval and UIs
  inference.py          Flask playground + API (what the live demo runs)
  gradio_inference.py   minimal local Gradio UI
  finetune_papl.py, bench_papl.py, dump_predictions.py, compare_strategies.py, sql_repair.py, augment.py
bench/                  fair diffusion-vs-AR benchmark (quality + latency, same engine)
docs/roadmap.md         research roadmap
scripts/                GPU training / A/B launch scripts, qualitative eval
experiments/            dated experiment write-ups with data, predictions and plots
docker/, Dockerfile, docker-compose.yml   CPU deployment (web + worker + redis)
checkpoints/            model weights (gitignored)

How It Works

Training (src/train.py)

  • Backbone: answerdotai/ModernBERT-base with an MLM head.
  • Inputs are built at the token level: [CLS] <PROMPT> ... </PROMPT> <CONTEXT> ... </CONTEXT> <SQL> sql + [PAD]s </SQL> [SEP] The SQL span is a fixed 128-token window; [PAD] tokens inside it are real prediction targets, so the model learns output length.
  • Forward (noising) process: per example a continuous mask ratio t ~ U(0, 1] is drawn and each SQL-span token is masked independently with probability t.
  • Loss: cross-entropy on masked positions weighted by 1/t (the discrete diffusion ELBO). This properly trains the fully-masked regime that generation starts from.
  • Eval: besides MLM loss, generation-based exact match is computed by actually denoising validation examples (eval/generation_exact_match in wandb).

Inference (src/denoising.py)

Confidence-based iterative unmasking (MaskGIT/LLaDA decoding):

  1. The SQL window starts fully masked.
  2. Each step predicts only the currently masked positions.
  3. The most confident predictions are committed permanently (cosine schedule: few commits early, more as context fills in).
  4. Committed tokens are never resampled, so generation converges instead of flickering. top_k > 1 adds Gumbel noise to the commit order for diversity.
  5. Adaptive early stop (on by default): on each step, any masked position whose top-token probability is already >= confidence_stop is also committed, so easy queries empty the window in far fewer forward passes. The step slider becomes a maximum; the actual number of steps adapts to difficulty.

Decoding efficiency (steps vs. autoregressive)

Each diffusion step is one forward pass over the whole sequence but commits many tokens at once, whereas autoregressive decoding needs roughly one forward pass per output token. Measured with scripts/inspect_eval.py on 58 held-out examples (8 sanity + 50 gretelai test), greedy decoding, 128-token SQL window:

Fixed steps (no early stop) — accuracy plateaus around 12–16 steps; beyond that adds cost without quality, and at 24 steps the model needs more passes than autoregressive would:

steps exact soft (Jaccard ≥ .9) speedup vs. autoregressive
4 0.50 0.50 4.7×
8 0.55 0.60 2.3×
12 0.65 0.65 1.6×
16 0.60 0.60 1.2×
24 0.65 0.65 0.8×

Adaptive early stop (cap = 16 steps) — fewer average passes and higher accuracy, because committing confident tokens sooner gives the model better context to resolve the rest:

confidence_stop exact soft avg steps used
off 0.328 0.345 16.0
0.95 0.362 0.379 10.0
0.90 (default) 0.397 0.414 8.8

Easy queries finish in ~4 passes, hard ones use up to the cap. Defaults: default_steps = 20 (the cap; early stop adapts down from there), CONFIDENCE_STOP = 0.9. Toggle per run in the playground's Advanced panel, or globally via --confidence-stop / CONFIDENCE_STOP (set 0 to disable).

Commit ordering: confidence vs. dependency (DOS)

denoise_steps supports an alternative strategy="dependency" that reorders commits by a DOS-style (arXiv:2603.15340) attention-groundedness score: for each masked position, the fraction of its attention mass (averaged over heads, read from a single tuned layer via dep_layer_index) that lands on already-revealed context. The paper's intuition is that committing the best-grounded tokens first respects inter-token dependencies. dep_alpha blends it with confidence (1.0 = standalone DOS).

We evaluated it faithfully with src/compare_strategies.py on the gretelai text-to-SQL test split (eager attention). It does not beat the confidence baseline on this model, across three tests:

1. Standalone DOS, single layer swept shallow→deep (fixed 24 steps, n=128):

layer read exact Δ vs baseline latency
confidence (default) 0.242 — —
DOS layer 4 (best) 0.250 +0.008 (1 example) +30%
DOS layers 0/2/6/8/10/21 0.195–0.234 −0.008 to −0.047 +22–30%

The paper tunes the attention layer (favoring shallow, syntax-bearing layers); sweeping layers 0–21 here, the best beats baseline by a single example (noise) and most layers are worse. No layer redeems it.

2. Under adaptive early-stop (confidence_stop=0.9, n=256): DOS reaches the threshold in fewer steps but commits the wrong tokens early —

config exact avg steps latency
confidence (default) 0.320 11.8 1017 ms/ex
DOS layer 4 (standalone) 0.102 5.4 472 ms/ex

It is ~2× faster only because it's a fast path to a −0.219 accuracy answer.

Why it doesn't transfer: DOS is training-free and can only exploit dependency structure the model already encodes in attention. The paper's gains are on 7–8B reasoning DLMs (LLaDA/Dream); this is a ~150M SQL-tuned ModernBERT encoder whose attention carries little exploitable dependency signal — consistent with standalone dependency being worse than confidence at every layer.

Decision: confidence remains the default; DOS is not wired into the inference frontend. The dependency path stays opt-in (strategy/dep_alpha/dep_layers/ dep_layer_index args to denoise_steps, exercised by compare_strategies.py) for future experimentation — e.g. against a larger or dependency-supervised model. Note it also silently no-ops to confidence on the ONNX backend and on any torch model not loaded with attn_implementation="eager".

Features

  • LLaDA-style masked diffusion training with 1/t-weighted ELBO loss.
  • Confidence-based parallel decoding shared by training eval and both UIs.
  • Adaptive confidence early-stopping: ~1.8–3× fewer forward passes with no quality loss (toggleable per run; see Decoding efficiency above).
  • Generation-based exact-match eval logged to wandb during training.
  • Public-ready Flask playground (src/inference.py) with SSE + polling fallback.
  • Redis-backed queue + worker for stable queue position/ETA and run TTL.
  • CPU safety controls for public deployment:
    • strict single-flight inference (default 1 active run)
    • bounded queue (default 3)
    • per-IP rate limits
    • request timeout and stale-run cleanup
    • GIF generation disabled by default

Local Quickstart

1. Install dependencies

pip install -r requirements.txt

2. Train a model

python src/train.py

Checkpoints and the final model are written to checkpoints/diffusion-sql-modernbert/. For a full GPU run see scripts/run_gh200.sh. Track eval/generation_exact_match in wandb for actual generation quality.

3. Run inference UI locally

python src/inference.py --model-dir checkpoints/diffusion-sql-modernbert      # Flask playground (as hosted)
python src/gradio_inference.py --model-dir checkpoints/diffusion-sql-modernbert  # minimal Gradio UI

Both old (roberta) and new (ModernBERT) checkpoints load via the Auto classes.

Docker Deployment (CPU-safe public playground)

Build image

docker build -t diffusion-llm:latest .

Run container (equivalent to local python src/inference.py ...)

docker run --rm -p 7860:7860 \
  -v /host/model:/models/model:ro \
  diffusion-llm:latest \
  --model-dir /models/model --host 0.0.0.0 --no-open

Run with compose

Set model location:

export MODEL_DIR=/host/model

Start web + worker + redis:

docker compose up -d --build

Web service listens on :7860 and is designed to be put behind Caddy/Nginx.

Production Runtime Defaults

  • Global request limit: 60/minute per IP.
  • /start limit: 5/minute and 30/hour per IP.
  • Max concurrent runs: 1.
  • Max queued runs: 3.
  • Timeout per run: 90s.
  • Max prompt chars: 1000.
  • Max context chars: 8000.
  • Max steps: 64.
  • Max max_len: 512.
  • Max sql_len: 256.
  • Adaptive early-stop threshold (CONFIDENCE_STOP): 0.9 (on; 0 disables).
  • Run TTL cleanup: 900s.
  • GIF generation: disabled unless --enable-gif is set.

Public API Behavior

  • POST /start
    • 200 with {run_id, state, queue_position, eta_seconds, queue_token} when accepted.
    • 400 for validation errors.
    • 429 when rate-limited.
    • 503 when queue is full.
  • GET /run/<run_id>?after=<n>
    • Incremental run state + snapshot deltas.
  • GET /queue/<run_id>
    • Queue position and ETA info.
  • GET /stream/<run_id>
    • Server-sent events: snapshot, status, done.
  • POST /stop/<run_id>
    • Requests cooperative cancellation.
  • GET /healthz
    • Liveness/readiness info.

Relevant CLI Options (src/inference.py)

  • --public (bind 0.0.0.0, disable browser auto-open)
  • --max-concurrent-runs
  • --max-queued-runs
  • --request-timeout-seconds
  • --max-prompt-chars
  • --max-context-chars
  • --max-steps
  • --max-max-len
  • --max-sql-len
  • --confidence-stop (adaptive early-stop threshold; default 0.9, 0 disables)
  • --enable-gif
  • --run-ttl-seconds
  • --worker
  • --redis-url

CPU Tuning Notes

The container sets:

  • TORCH_NUM_THREADS=1
  • OMP_NUM_THREADS=1
  • MKL_NUM_THREADS=1
  • PYTHONUNBUFFERED=1

This is intentional for predictable CPU load on a Linux CPU host.

For a faster profile (roughly ~2x on many CPU-only hosts), increase worker CPU/threads in .env:

TORCH_NUM_THREADS=4
OMP_NUM_THREADS=4
MKL_NUM_THREADS=4
WORKER_CPUS=3.0
WORKER_MEM_LIMIT=4g

Adaptive early-stopping already cuts average steps automatically; for further speedups lower the step cap in requests (for example steps=8) at a small quality cost.

License

MIT

About

Experiments on how to combine denoising and transformer architectures

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages