Runnable code examples that accompany the Distributed ML Notes learning series.
Each subdirectory is a self-contained toy program with its own README.md and requirements.txt.
| Directory | What it covers | Related notes chapter |
|---|---|---|
ddp-testrun |
PyTorch DDP training with Ring AllReduce on MNIST | Chapter 3 — Worker-Only (AllReduce) |
vllm-test |
vLLM serving: prefill vs decode, static vs continuous batching, TTFT, KV cache / PagedAttention | Prefill vs Decode |
test-LoRA-SVP |
LoRA math (ΔW ≈ B@A) + PEFT on Qwen2.5-0.5B + SVD→LoRA intro |
— |
Each example is standalone. Navigate into the subdirectory and follow its README:
cd ddp-testrun
pip install -r requirements.txt
python ddp_train.pyallreduce-from-scratch— Ring AllReduce implemented manually without DDPdataset-loading— Distributed dataset sharding and DataLoader setup