🌎 WeLT: Word Embedding Latent Transformer

See Motivational Examples

We present a language model that replaces conventional subword tokenization with a dual representation of text as both bytes and rendered images of "words", allowing the model to directly process the visual and symbolic structure of language. It segments text into words (with sign/words-segmentation by default, or any HuggingFace tokenizer), encodes each as a byte sequence and/or an image, and passes them through an encoder–decoder pipeline: a bytes encoder, an image encoder, a large latent transformer, and a bytes decoder. At inference, predicted bytes are rendered back into images (with an image encoder), closing the loop for the next prediction. This design could make learning and inference cheaper and faster for non-English languages, since the heavy latent transformer only predicts high-level token representations while the actual byte sequences are generated by a much smaller decoder.

Each word is embedded on its own, from its image and bytes, by small encoders: there is no subword tokenizer to train, no vocabulary to fall outside of, and no embedding table over tokens. Any word, in any script, misspelled or new, has an embedding, and the latent transformer sees one vector per word.

Model Setup

• Bytes Encoder - A transformer over BOS and the bytes of each word (not EOS, which the bidirectional encoder does not need), used bidirectionally. Its first (BOS) output is the word embedding.
• Image Encoder - A transformer over the 16x16 patches of each rendered word (NaViT-style: native size, with learned row and column embeddings, packed without padding; used bidirectionally). Its CLS output is the word embedding.
• Latent Transformer - A causal transformer over the word embeddings (recommended: large).
• Bytes Decoder - A causal transformer generating each next word's bytes from its latent vector, which takes the place of BOS: its input is the latent, then the word's bytes, and it predicts the bytes and EOS (recommended: small).

The bytes encoder, latent transformer, bytes decoder and patch image encoder are each a causal LM architecture supported by Megatron-Bridge (Llama, Qwen, Mistral, ...; RoPE, without sliding window attention or logit softcapping), given as a HuggingFace model id/path, or as a JSON HuggingFace config to train from scratch (see welt_training/experiments/models). With load_pretrained: true, transformers given by id/path start from their weights. Their token embeddings and LM heads are replaced: words are given as embeddings, and bytes have their own embeddings.

Each encoder is optional (omit it or set it to null), but at least one is required. See the experiments for configs.

Install

WeLT trains with Megatron-Bridge and generates with vLLM, both from the NeMo container (see the Dockerfile):

git clone https://github.com/sign/WeLT.git
cd WeLT
docker build -t welt .
docker run -it --rm --gpus all --ipc=host \
  -v "$(pwd)":/app \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e WANDB_API_KEY \
  welt bash

The repository is installed in editable mode at /app, so mounting your checkout there runs your code. All commands below run inside the container. Run the tests with pytest (GPU tests are skipped without a GPU).

Without the container, the CPU tests run in a Python 3.12 environment with Pango (Megatron and vLLM tests are skipped):

micromamba create -n welt -c conda-forge python=3.12 pango pycairo pygobject
micromamba activate welt
pip install ".[dev]"
pytest

Train

torchrun --nproc_per_node=1 -m welt_training.train welt_training/experiments/easy-tasks/string-repetition.yaml

See welt_training/README.md for config overrides, the config reference, data streaming, parallelism, optimizers and the run directory.

Training and validation log lm loss (per byte), bits per byte, byte accuracy and word accuracy. Bits per byte count every byte prediction, including the end of each word (like a space in a causal LM), per UTF-8 byte of text, except the end of a document and the text inside shift blocks (\x0E...\x0F, e.g. the source sentence), which neither WeLT nor the baseline predicts, so the two are comparable.

Export, Serve & Generate

Export the latest checkpoint (or a specific checkpoints/iter_* directory) to HuggingFace-format transformers, which vLLM serves, then start a generation server (one per GPU) in the container:

welt-export output/string-repetition-tiny/checkpoints --output output/string-repetition-tiny/export
welt-serve output/string-repetition-tiny/export --port 8080
# or: docker run --rm --gpus all --ipc=host -p 8080:8080 -v "$(pwd)":/app welt welt-serve /app/output/string-repetition-tiny/export

and generate through it:

welt-generate $'<text>\x0eHello world\x0f<repeat> ' --url http://localhost:8080
curl localhost:8080/generate -H "Content-Type: application/json" \
  -d '{"texts": ["<text>\u000eHello world\u000f<repeat> "], "max_generated_words": 50, "temperature": 0.0}'

welt-serve (welt/server.py) answers GET /health and POST /generate, whose JSON body has texts (a list of prompts), and optionally max_generated_words (default 50), temperature (default 0, greedy) and seed. Invalid requests, and prompts whose words plus max_generated_words exceed the latent transformer's context, get a 400. Decoding is greedy or sampled, valid UTF-8 only, with no beam search; a prompt ending mid-word continues that word. In Python, welt.server.generate(url, texts) is the client, and welt.inference.WeLTGenerator(export_dir).generate(texts) runs the model in-process (see welt/inference.py for how generation works).

Evaluate

With the model served:

welt-evaluate output/string-repetition-tiny/export/welt.yaml --url http://localhost:8080 --max_samples 256 --output results.json

Generates completions for the validation prefixes of the run's dataset (see Data), and reports exact match, chrF and generated words per second. benchmarks/run_task.sh trains, exports, serves and evaluates a config in one go.

Baseline

welt_training.baseline trains a causal LM over a standard tokenizer on the same data, reporting comparable bits per byte; see machine translation and language modeling.

Training Speed

A training step is 9.7x faster than with the HuggingFace Trainer implementation, kept at the huggingface-transformers git tag; see benchmarks.

Contributing

See open issues.

• sign/words-segmentation as a universal word level pretokenizer.
• sign/utf8-tokenizer as a robust byte-level tokenizer.
• sign/pixel-renderer as a reproducible text-to-image renderer.

Warning

Training runs are experimental until core issues are resolved.

Cite

If you use this code in your research, please consider citing the work:

@misc{moryossef2025welt,
  title={{WeLT}: Word Embedding Latent Transformer for Equitable Modeling of the Languages of the World},
  author={Moryossef, Amit},
  howpublished={\url{https://github.com/sign/WeLT}},
  year={2025}
}