4D Parallelism Qwen3.5 MoE - #20

Open

tomiock wants to merge 28 commits into

mainfrom

Commits on Apr 23, 2026

pipeline parallel init
tomiock
committed
680caf5
Enhance README with online datapacking and loading info
tomiock
committed
56c048d
Revise scalability metrics in SCALABILITY.md
tomiock
committed
ed1d147
jup config
tomiock
committed
ba3ed0e
jupiter data packing config
tomiock
committed
60f2680
9B PP=4 but error on 27B
tomiock
committed
fe08445

Commits on Apr 24, 2026

[feat] 27B model running w/ PP=6
tomiock
committed
92b5ccc

Commits on Apr 25, 2026

Merge branch 'pipeline' of github.com:VLR-CVC/vlm-training into pipeline
tomiock
committed
12f3bd2

Commits on Apr 28, 2026

broken loss, models gives `nan` on logits
tomiock
committed
d48c74f
[fix] corrent loss
tomiock
committed
2afd6c7
loss issues w/ pipeline
tomiock
committed
a4b5083
we need to load the weights in PP
tomiock
committed
1aff281
dev checkpoint
tomiock
committed
6a7de7c
[fix] PP loading weights
tomiock
committed
c89d450
[fix] layers well split in PP ranks
tomiock
committed
6fccd16
init moe
tomiock
committed
579efb3
[feat] moe implemented (bad performance)
tomiock
committed
e069634
Merge branch 'pipeline' into moe
tomiock
authored
4ef463d
merge `moe` into `pipeline`
tomiock
committed
6b22368
Merge branch 'moe' of github.com:VLR-CVC/vlm-training into moe
tomiock
committed
267c870
Merge pull request #17 from VLR-CVC/moe
tomiock
authored
7c40c3d

Commits on Apr 29, 2026

sync
tomiock
committed
7cf1de7
[feat] MoE TP & PP (not at the same time)
tomiock
committed
22241cc
test EP
tomiock
committed
1b0e6c4
[feat] EP + TP implemented
tomiock
committed
f036910
[feat] 4D parallelism
tomiock
committed
d61392e

Commits on May 2, 2026

[feat] properly behaving 4D moe set up
tomiock
committed
ae04d25
[fix] better PP stages
tomiock
committed
9e5ddfc