diff --git a/README.md b/README.md index ab1df7c..008ffeb 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,8 @@ Drop your own `.md` files in `notes/` and ask again. | `cli.py` | Question in → answer + citations out | | `notes/` | Sample teaching notes (git, Ollama, RAG, pytest) | +See [how scoring works](docs/how-scoring-works.md) for a hand-worked cosine example. + ## Change one thing 1. Add `notes/my-topic.md` and ask a question only that file can answer diff --git a/docs/how-scoring-works.md b/docs/how-scoring-works.md new file mode 100644 index 0000000..52081b1 --- /dev/null +++ b/docs/how-scoring-works.md @@ -0,0 +1,67 @@ +# How scoring works + +The scorer compares token frequencies, not meanings. Here is a tiny example you +can check by hand against [`retrieve.py`](../src/notes_rag/retrieve.py). + +## 1. Turn text into tokens + +Query: `Git remote?` Two Markdown notes: + +```markdown +# Git +git remote +``` + +```markdown +# Remote +remote remote +``` + +The title **and** body are scored. Text is lowercased, then `[a-z0-9]+` matches +runs of ASCII letters or digits. Punctuation separates tokens; it is not kept. +The query becomes `[git, remote]`, note A `[git, git, remote]`, and note B +`[remote, remote, remote]`. + +## 2. Count and normalize + +Term frequency is each token's count divided by the total number of tokens. +Using the coordinate order `(git, remote)`: + +| Text | Counts | Term-frequency vector | +| --- | --- | --- | +| Query | `(1, 1)` | `(1/2, 1/2)` | +| A: Git | `(2, 1)` | `(2/3, 1/3)` | +| B: Remote | `(0, 3)` | `(0, 1)` | + +Word order does not affect these vectors. Repeating only `remote` still gives +note B a frequency of 1 for that token. + +## 3. Compute cosine similarity + +Multiply matching coordinates and add them for the dot product. The norm is +the square root of the sum of squared coordinates. Then: + +```text +score = dot(query, note) / (norm(query) * norm(note)) + +norm(query) = sqrt(1/4 + 1/4) = sqrt(2)/2 + +A: dot = (1/2 * 2/3) + (1/2 * 1/3) = 1/2 + norm = sqrt(4/9 + 1/9) = sqrt(5)/3 + score = (1/2) / (sqrt(2)/2 * sqrt(5)/3) = 3/sqrt(10) = 0.948683... + +B: dot = (1/2 * 0) + (1/2 * 1) = 1/2 + norm = sqrt(0 + 1) = 1 + score = (1/2) / (sqrt(2)/2 * 1) = 1/sqrt(2) = 0.707107... +``` + +## 4. Rank and keep positive hits + +Scores are sorted highest first. With `k=2`, A comes before B; the CLI displays +their scores as `0.949` and `0.707`. With `k=1`, only A remains. +The scorer takes the first `k` entries, then keeps only scores greater than 0. +A query such as `pytest` shares no tokens with either note: both dot products +are 0, so neither note is returned, even with `k=2`. + +Raising a positive `--k` lets more lower-ranked, positive-score notes into the +answer's context; it does not change their scores or admit zero-score notes.