[AmitMY]

An audit of the code (three independent reviews of welt/, welt_training/ + benchmarks/, and the tests and repository), with the findings fixed.

Validation scored different texts per model (affects results)

• Evaluations saw only the validation micro batches that fit in validation.eval_iters. On ClimbMix (eval_iters 1), WeLT scored the first 64 of its 80 validation examples (80% of the text), the baseline the first 64 of its 72 (89%): not the same bytes, although the configs claimed so. Rescored on the whole set, WeLT's bits per byte rise by ~0.05 and the baseline's by ~0.02.
• Now every validation example is scored exactly once: copies with score 0 fill an evaluation's remaining micro batches (zeroing their loss and metrics), and an evaluation that cannot cover the examples fails instead of warning. ClimbMix configs: eval_iters: 2.
• All ClimbMix checkpoints are being rescored on the whole set with this branch (validation.skip_train=true), and the paper updated.

Correctness

• Shift blocks cut by pack_words: a sequence ending (beginning) within a block keeps it to its last (from its first) word, so WeLT scores the bytes the baseline scores (the parity test now packs at 4-8 words, cutting blocks); a ShiftOut within a block belongs to it, as the baseline's regex.
• A processor loaded with from_pretrained (inference) no longer splits long words differently from training (words-segmentation does not save max_bytes; also worth fixing upstream), and keeps trust_remote_code. Other pretokenizers' tokens longer than max_word_length are split, not truncated.
• Data: the stream is split across GPUs before skipping the held-out validation examples (they leaked into training with sharded datasets on several GPUs); dataloader workers decode only their share of the stream (each decoded every document).
• Baseline: model options set the transformer's (unknown ones raise, as WeLT's); data.trust_remote_code no longer clashes.
• Generation: each text samples with its own seed; generated words are counted (not re-segmented); only a too-long context (ContextTooLongError) answers 400, other errors are 500s; welt-generate --seed.
• flops.py reuses the provider's make_examples and build_config's batch defaults.

Tests: labels across documents split into short examples (also within shift blocks), bits-per-byte parity at short sequence lengths (now in CI: token chunking moved to the Megatron-free data_utils), dataloader workers' split, NUL and empty texts, the generation loop (stubbed engines), configs' model files, reloaded processors; the busy-server test can no longer hang; tests independent of the working directory. Full suite on a GB10: 136 passed, 1 skipped.

CI: Docker build test also on pyproject.toml changes; ruff pinned; one checkout version.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

---

Note

High Risk
Changes validation metrics semantics, multi-GPU train/val splits, and shift-block labeling—directly affecting reported bits-per-byte and training data exposure; inference/API behavior also shifts for context limits and seeds.

Overview
Fixes validation, data packing, and inference correctness after an audit, with broad test and CI updates.

Validation now scores every held-out example exactly once per eval: real micro-batches carry score 1, padding batches use score 0 so loss/metrics ignore them, and setup errors if validation.eval_iters is too small (replacing a warning that let WeLT vs baseline see different byte subsets on ClimbMix). ClimbMix configs bump eval_iters to 2.

Shift blocks when pack_words splits sequences: blocks can start/end mid-chunk (aligned with the baseline’s \x0E…\x0F regex), including nested ShiftOut inside a block; warnings for “unmatched” tokens are removed in favor of explicit block boundaries.

Data: GPU sharding happens before dropping validation holdout (fixes leak on multi-GPU); dataloader workers only format their shard. NUL bytes and empty strings are dropped. Baseline token chunking moves to Megatron-free data_utils; baseline accepts model.* transformer overrides like WeLT.

Processor / inference: BPE spans are byte-split to max_word_length (not truncated); from_pretrained restores max_bytes and trust_remote_code. Generation adds generate_words, ContextTooLongError for HTTP 400, per-text sampling seeds, and accurate generated_words counts; welt-generate gets --seed.

Tests & CI: parity tests for short packing, workers, inference stub, config paths from repo root; ruff pinned; docker workflow on pyproject.toml changes.

^{Reviewed by Cursor Bugbot for commit 9515114. Bugbot is set up for automated code reviews on this repo. Configure here.}