A masked diffusion language model (LLaDA-style) for text-to-SQL generation, built on ModernBERT. Instead of writing SQL left to right, the model starts from a fully masked SQL window and fills it in over a few parallel denoising steps. This repo includes training, evaluation experiments, and a web playground.
Live demo: diffusion.lethonium.com
ModernBERT-base (150M), 10 epochs on gretelai/synthetic_text_to_sql, confidence decoding with adaptive early stop at 0.9. Evaluated on the first 256 filtered test rows:
| Metric | Value |
|---|---|
| Semantic accuracy (LLM judge) | 0.50 |
| Exact string match | 0.32 |
| Avg. denoising steps (forward passes) | 11.8, vs ~30 tokens for autoregressive decoding |
| Largest failure mode | invalid SQL (~14%) |
Compared with an autoregressive (AR) model fine-tuned on the same data (Qwen experiment, same 256 rows, same LLM judge):
| Model | Params | Semantic accuracy | Forward passes |
|---|---|---|---|
| AR: Qwen2.5-Coder-0.5B SFT | 500M | 0.645 | ~30 (one per token) |
| Diffusion: ModernBERT-base (this repo's main model) | 150M | 0.500 | 11.8 |
| Diffusion: Qwen2.5-Coder-0.5B adapted to masked diffusion | 500M | 0.191 | 11.8 |
Findings so far:
-
Matched size, diffusion is 2.6× faster at tied accuracy. Ettin-150M encoder (diffusion) vs. decoder (AR), same data, 10 epochs, 1,000 test queries on one H100: 25 ms (CUDA graphs) vs 67 ms (vLLM), exec 0.564 vs 0.580 (−1.7 ± 3.5, a tie), exact match 0.277 vs 0.239 (+3.8 ± 2.3). Diffusion's gap is invalid SQL (−10 points), mostly closed by one verify pass (exec 0.591, 42 ms). See GPU session 2 and session 1.
-
A diffusion drafter makes the larger AR model 2.0× faster. Qwen2.5-0.5B (exec 0.735) on vLLM with a DFlash block-diffusion drafter for speculative decoding: 28 ms instead of 57 ms, at unchanged accuracy, and 1.3× faster than an EAGLE-3 drafter trained on the same data (37 ms). Above 150M, diffusion as the drafter beats diffusion as the model: Ettin-400M diffusion takes 32 ms at exec 0.606.
-
Adaptive early stop is the main speed lever. It cuts forward passes 1.8–3× with no quality loss (see Decoding efficiency).
-
More steps can hurt. Accuracy peaks around 11 steps (threshold 0.9) and falls ~10 points when decoding runs to a 0.99 threshold (~19 steps).
-
PAPL fine-tuning only saves steps. It gives ~7% fewer steps at equal accuracy (PAPL A/B).
-
On CPU (the hosted setup), AR is still the better trade. Cheap changes make diffusion 2.1× faster, but a 3.3× larger AR model matches its fp32 latency at higher accuracy (fair CPU benchmark, roadmap).
-
Dependency-ordered decoding (DOS) does not beat confidence ordering on this model (see below).
src/
train.py LLaDA-style training (ModernBERT, 1/t-weighted ELBO)
denoising.py confidence / dependency decoding shared by eval and UIs
inference.py Flask playground + API (what the live demo runs)
gradio_inference.py minimal local Gradio UI
finetune_papl.py, bench_papl.py, dump_predictions.py, compare_strategies.py, sql_repair.py, augment.py
bench/ fair diffusion-vs-AR benchmark (quality + latency, same engine)
docs/roadmap.md research roadmap
scripts/ GPU training / A/B launch scripts, qualitative eval
experiments/ dated experiment write-ups with data, predictions and plots
docker/, Dockerfile, docker-compose.yml CPU deployment (web + worker + redis)
checkpoints/ model weights (gitignored)
- Backbone:
answerdotai/ModernBERT-basewith an MLM head. - Inputs are built at the token level:
[CLS] <PROMPT> ... </PROMPT> <CONTEXT> ... </CONTEXT> <SQL> sql + [PAD]s </SQL> [SEP]The SQL span is a fixed 128-token window;[PAD]tokens inside it are real prediction targets, so the model learns output length. - Forward (noising) process: per example a continuous mask ratio
t ~ U(0, 1]is drawn and each SQL-span token is masked independently with probabilityt. - Loss: cross-entropy on masked positions weighted by
1/t(the discrete diffusion ELBO). This properly trains the fully-masked regime that generation starts from. - Eval: besides MLM loss, generation-based exact match is computed by actually
denoising validation examples (
eval/generation_exact_matchin wandb).
Confidence-based iterative unmasking (MaskGIT/LLaDA decoding):
- The SQL window starts fully masked.
- Each step predicts only the currently masked positions.
- The most confident predictions are committed permanently (cosine schedule: few commits early, more as context fills in).
- Committed tokens are never resampled, so generation converges instead of
flickering.
top_k > 1adds Gumbel noise to the commit order for diversity. - Adaptive early stop (on by default): on each step, any masked position
whose top-token probability is already
>= confidence_stopis also committed, so easy queries empty the window in far fewer forward passes. The step slider becomes a maximum; the actual number of steps adapts to difficulty.
Each diffusion step is one forward pass over the whole sequence but commits many
tokens at once, whereas autoregressive decoding needs roughly one forward pass
per output token. Measured with scripts/inspect_eval.py on 58 held-out examples
(8 sanity + 50 gretelai test), greedy decoding, 128-token SQL window:
Fixed steps (no early stop) — accuracy plateaus around 12–16 steps; beyond that adds cost without quality, and at 24 steps the model needs more passes than autoregressive would:
| steps | exact | soft (Jaccard ≥ .9) | speedup vs. autoregressive |
|---|---|---|---|
| 4 | 0.50 | 0.50 | 4.7× |
| 8 | 0.55 | 0.60 | 2.3× |
| 12 | 0.65 | 0.65 | 1.6× |
| 16 | 0.60 | 0.60 | 1.2× |
| 24 | 0.65 | 0.65 | 0.8× |
Adaptive early stop (cap = 16 steps) — fewer average passes and higher accuracy, because committing confident tokens sooner gives the model better context to resolve the rest:
confidence_stop |
exact | soft | avg steps used |
|---|---|---|---|
| off | 0.328 | 0.345 | 16.0 |
| 0.95 | 0.362 | 0.379 | 10.0 |
| 0.90 (default) | 0.397 | 0.414 | 8.8 |
Easy queries finish in ~4 passes, hard ones use up to the cap. Defaults:
default_steps = 20 (the cap; early stop adapts down from there),
CONFIDENCE_STOP = 0.9. Toggle per run in the playground's Advanced panel, or
globally via --confidence-stop / CONFIDENCE_STOP (set 0 to disable).
denoise_steps supports an alternative strategy="dependency" that reorders
commits by a DOS-style (arXiv:2603.15340)
attention-groundedness score: for each masked position, the fraction of its
attention mass (averaged over heads, read from a single tuned layer via
dep_layer_index) that lands on already-revealed context. The paper's intuition
is that committing the best-grounded tokens first respects inter-token
dependencies. dep_alpha blends it with confidence (1.0 = standalone DOS).
We evaluated it faithfully with src/compare_strategies.py on the gretelai
text-to-SQL test split (eager attention). It does not beat the confidence
baseline on this model, across three tests:
1. Standalone DOS, single layer swept shallow→deep (fixed 24 steps, n=128):
| layer read | exact | Δ vs baseline | latency |
|---|---|---|---|
| confidence (default) | 0.242 | — | — |
| DOS layer 4 (best) | 0.250 | +0.008 (1 example) | +30% |
| DOS layers 0/2/6/8/10/21 | 0.195–0.234 | −0.008 to −0.047 | +22–30% |
The paper tunes the attention layer (favoring shallow, syntax-bearing layers); sweeping layers 0–21 here, the best beats baseline by a single example (noise) and most layers are worse. No layer redeems it.
2. Under adaptive early-stop (confidence_stop=0.9, n=256): DOS reaches the
threshold in fewer steps but commits the wrong tokens early —
| config | exact | avg steps | latency |
|---|---|---|---|
| confidence (default) | 0.320 | 11.8 | 1017 ms/ex |
| DOS layer 4 (standalone) | 0.102 | 5.4 | 472 ms/ex |
It is ~2× faster only because it's a fast path to a −0.219 accuracy answer.
Why it doesn't transfer: DOS is training-free and can only exploit dependency structure the model already encodes in attention. The paper's gains are on 7–8B reasoning DLMs (LLaDA/Dream); this is a ~150M SQL-tuned ModernBERT encoder whose attention carries little exploitable dependency signal — consistent with standalone dependency being worse than confidence at every layer.
Decision: confidence remains the default; DOS is not wired into the inference
frontend. The dependency path stays opt-in (strategy/dep_alpha/dep_layers/
dep_layer_index args to denoise_steps, exercised by compare_strategies.py)
for future experimentation — e.g. against a larger or dependency-supervised
model. Note it also silently no-ops to confidence on the ONNX backend and on any
torch model not loaded with attn_implementation="eager".
- LLaDA-style masked diffusion training with
1/t-weighted ELBO loss. - Confidence-based parallel decoding shared by training eval and both UIs.
- Adaptive confidence early-stopping: ~1.8–3× fewer forward passes with no quality loss (toggleable per run; see Decoding efficiency above).
- Generation-based exact-match eval logged to wandb during training.
- Public-ready Flask playground (
src/inference.py) with SSE + polling fallback. - Redis-backed queue + worker for stable queue position/ETA and run TTL.
- CPU safety controls for public deployment:
- strict single-flight inference (default 1 active run)
- bounded queue (default 3)
- per-IP rate limits
- request timeout and stale-run cleanup
- GIF generation disabled by default
pip install -r requirements.txtpython src/train.pyCheckpoints and the final model are written to checkpoints/diffusion-sql-modernbert/.
For a full GPU run see scripts/run_gh200.sh.
Track eval/generation_exact_match in wandb for actual generation quality.
python src/inference.py --model-dir checkpoints/diffusion-sql-modernbert # Flask playground (as hosted)
python src/gradio_inference.py --model-dir checkpoints/diffusion-sql-modernbert # minimal Gradio UIBoth old (roberta) and new (ModernBERT) checkpoints load via the Auto classes.
docker build -t diffusion-llm:latest .docker run --rm -p 7860:7860 \
-v /host/model:/models/model:ro \
diffusion-llm:latest \
--model-dir /models/model --host 0.0.0.0 --no-openSet model location:
export MODEL_DIR=/host/modelStart web + worker + redis:
docker compose up -d --buildWeb service listens on :7860 and is designed to be put behind Caddy/Nginx.
- Global request limit:
60/minuteper IP. /startlimit:5/minuteand30/hourper IP.- Max concurrent runs:
1. - Max queued runs:
3. - Timeout per run:
90s. - Max prompt chars:
1000. - Max context chars:
8000. - Max steps:
64. - Max max_len:
512. - Max sql_len:
256. - Adaptive early-stop threshold (
CONFIDENCE_STOP):0.9(on;0disables). - Run TTL cleanup:
900s. - GIF generation: disabled unless
--enable-gifis set.
POST /start200with{run_id, state, queue_position, eta_seconds, queue_token}when accepted.400for validation errors.429when rate-limited.503when queue is full.
GET /run/<run_id>?after=<n>- Incremental run state + snapshot deltas.
GET /queue/<run_id>- Queue position and ETA info.
GET /stream/<run_id>- Server-sent events:
snapshot,status,done.
- Server-sent events:
POST /stop/<run_id>- Requests cooperative cancellation.
GET /healthz- Liveness/readiness info.
--public(bind0.0.0.0, disable browser auto-open)--max-concurrent-runs--max-queued-runs--request-timeout-seconds--max-prompt-chars--max-context-chars--max-steps--max-max-len--max-sql-len--confidence-stop(adaptive early-stop threshold; default0.9,0disables)--enable-gif--run-ttl-seconds--worker--redis-url
The container sets:
TORCH_NUM_THREADS=1OMP_NUM_THREADS=1MKL_NUM_THREADS=1PYTHONUNBUFFERED=1
This is intentional for predictable CPU load on a Linux CPU host.
For a faster profile (roughly ~2x on many CPU-only hosts), increase worker CPU/threads in .env:
TORCH_NUM_THREADS=4
OMP_NUM_THREADS=4
MKL_NUM_THREADS=4
WORKER_CPUS=3.0
WORKER_MEM_LIMIT=4gAdaptive early-stopping already cuts average steps automatically; for further
speedups lower the step cap in requests (for example steps=8) at a small
quality cost.

