An interactive CLI tool to explore how LLMs process text — from raw input to tokens, token IDs, and semantic embeddings.
- Tokenizes any input text using BERT tokenizer
- Shows token → token ID mapping
- Generates semantic embedding vectors
- Computes cosine similarity between two sentences
Every LLM pipeline starts here. Before RAG, before agents, before fine-tuning — text must become numbers. This tool makes that process visible and interactive.
pip install transformers torch sentence-transformers numpy
python explorer.py- HuggingFace Transformers
- Sentence Transformers (all-MiniLM-L6-v2)
- BERT tokenizer (bert-base-uncased)
- NumPy
