Repository navigation

OSB tokenizer - #1

Merged

whyscream merged 6 commits into

mainfrom

Jul 6, 2026

Merged

OSB tokenizer#1
whyscream merged 6 commits into
mainfrom
osb-tokenizer

Commits on Jul 3, 2026
- Refactor the version detection, no more workarounds needed
  whyscream
  committed
  0882926
Commits on Jul 6, 2026
- OSB tokenizer, content tokens only for now
  whyscream
  committed
  bbebb7d
- OSB tokenization of metadata
  whyscream
  committed
  69a2446
- Extract metadata unrolling into standalone method
  whyscream
  committed
  b44dd1f
- Extract word tokenization and delimiter setup into standalone methods
  whyscream
  committed
  61d1fe8
- Simplify OSB content tokenization code
  whyscream
  committed
  e3fff2a