A small GPT-style transformer trained from scratch on public-domain fairy tales (Grimm, Andersen, Aesop, English Fairy Tales). Character-level tokenizer, roughly 57M parameters, AdamW training, temperature + top-k generation.
config.py— hyperparameters: VOCAB_SIZE=93, BLOCK_SIZE=256, N_EMBD=256, N_HEAD=8, N_LAYER=72, DROPOUT=0.1, LEARNING_RATE=3e-4, MAX_STEPS=50000.tokenizer/— character-level tokenizer;train_tokenizer.pybuildsvocab.jsonfrom the training text.model/transformer.py—TaleWizard(aliasedGPT): token + positional embedding, 72MagicBlocklayers (causal attention + 4x-fan GELU MLP), final LayerNorm, linear output head.prepare_data.py— reads the four raw books, strips Gutenberg markers, writesdata/processed/combined.txt.split_data.py— 90/10 split ofcombined.txtintotrain.txtandval.txt.train.py— AdamW training, evaluates everyEVAL_INTERVALsteps, saves checkpoints, resumes fromcheckpoints/model.pt.generate.py— samples with temperature and top-k.
prepare_data.py->data/processed/combined.txtsplit_data.py->train.txt+val.txttokenizer/train_tokenizer.py->tokenizer/vocab.jsontrain.py->checkpoints/model.pt(50K steps)generate.py "prompt" 500 0.8