My attempt to (re)create some transformer architectures. This is a learning project, with intent to build many NLP components, such as the tokenization, the model, datasets and training scripts more-or-less from scratch.

The majority of code is written without the help of AI coding agents. Gen AI is used to ask questions, or for some repetitive code (e.g. parsing through different dataset formats.)

There is currently a working implementation of the original *Attention is All You Need" architecture.

As of writing this, I have trained a first model on a combination of a few croatian-english sentence-pairs. Currently working on properly evaluating this model, and working on implementing a GPT-2-like architecture, to train a croatian-english bilingual model.