[AmitMY]

Language modeling on Nemotron-ClimbMix, WeLT against the subword baseline: five models, six latent transformer sizes (1.6M to 1.07B), and the encoder and decoder sizes at a fixed latent transformer.

Data

• data.dataset_tokenizer: datasets of token ids (a tokens column, e.g. ClimbMix's GPT-2 ids) are decoded back to text while streaming; each model then segments the text its own way. Test with a GPT-2-encoded English + Hebrew text.
• Long documents: chunks overlap by one word, and the last word of a continued chunk has no label, so every word is a label exactly once (as in the baseline's token chunks). Oracle test across chunks.

Configs

• climbmix-{baseline,welt-pythia,welt}.yaml: the same text per example (340 Pythia tokens = 256 words), no image encoder; validation = the first 32 documents (one micro batch, the same bytes for all).
• models/latent-{14m,31m,160m,410m,1b}.json: latent transformers with Pythia's dimensions.
• models/bytes-<size>.json: byte models as large as the embedding tables they replace (matched params); models/bytes-flops-<size>.json: sized so that FLOPs per step match the baseline's (within 5%); plus the byte sizes of the encoder/decoder study.
• modal_1b.py: the 1B runs on Modal H100s.

Baseline

• Vocabulary padded to a multiple of 128 (as Pythia's 50,304) and Transformer Engine's fused cross entropy: 2x faster steps (70m: 473 → 230 ms), the same losses.

Results (validation bits per byte after 10,000 steps; details in the README)

Latent	Baseline	WeLT, Pythia tokens	WeLT, words	WeLT, matched params	WeLT, matched FLOPs
1.6M	1.238	1.203	1.194	1.165	1.242
6.3M	1.145	1.146	1.145	1.097	1.156
25M	1.077	1.088	1.098	1.043	1.083
113M	1.038	1.057	1.075	1.020	1.040
403M	1.009	1.024	1.049	1.007	1.015
1.07B	1.024			1.038	1.036

At equal parameters WeLT is ahead up to 403M (by a shrinking margin), at equal FLOPs slightly behind everywhere; the decoder carries most of the byte models' effect; WeLT overtakes the baseline with 3x the training steps.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

---

Note

Medium Risk
Packing and label-mask changes affect all WeLT training data paths; incorrect continues handling would skew loss and metrics, though coverage is strong in tests.

Overview
Adds Nemotron-ClimbMix language-modeling benchmarks comparing WeLT to the Pythia subword baseline from ~1.6M to 1B parameters, with new climbmix-*.yaml configs, latent/bytes model JSONs (matched params and matched FLOPs), modal_1b.py for H100 runs, and scaling results in the README.

Long-document packing now matches the baseline’s overlapping token chunks: pack_words yields a continues flag per sequence, chunks overlap by one word, and process_single_example suppresses labels on the last word when the document continues in the next packed block so each word is scored exactly once. Tests and oracles are updated accordingly.

Data loading adds data.dataset_tokenizer to decode streamed tokens columns (e.g. GPT-2 ids) back to text before each model segments on its own.

Baseline training pads vocabulary to a multiple of 128 and enables Transformer Engine fused cross-entropy (~2× faster steps, same loss).

^{Reviewed by Cursor Bugbot for commit 485e2ac. Bugbot is set up for automated code reviews on this repo. Configure here.}