Hey there! This is my personal implementation of the GPT-2 architecture built from scratch using PyTorch. I built this while following along with Andrej Karpathy's excellent guide, and I've also documented the whole process and established this project in my YouTube video (in Ukrainian).
- Scale: This model hasn't been trained at full scale yet (the max run was around 100M tokens).
- Why? Honestly, I didn't want to leave my local GPU running for 2 days straight, and renting a proper GPU cloud server isn't in my budget right now.
- What's next: Right now, I'm actively working on cleaning up the codebase, optimizing it, and making it actually usable for real-world inference and experimenting.
Feel free to check out the code, and if you speak Ukrainian, definitely check out the video link below to see how it was put together!
I use NixOS BTW