Repository navigation
feat(tools): the follow-up rate, measured on the user's own conversation - #882
Open
green-dalii wants to merge 2 commits into
Open
green-dalii wants to merge 2 commits into
green-dalii wants to merge 2 commits into
Conversation
green-dalii
force-pushed
the
feat/followup-rate-from-real-history
branch
from
October 24, 2026 13:49
80a01b3 to
6398244
Compare
The conversation harness measures a CONSTRUCTED follow-up: the question names nothing at all, so "no history answers it" is close to built in. That proves the history can carry a follow-up. It does not say how many real follow-ups there are, and the design decision that depends on that number — how much the retrieval should weight the history — was being made without it. The number is free data. The plugin already stores the user's own conversation in `data.json` under `queryHistory`, so this reads it, counts, and prints. No model call, no network, nothing leaves the machine. The classifier is the harness's own shape — the production `lexMatchByTitleAndAliases` plus `nameTierCoverage` over the top hit — so two tools cannot disagree about what "names nothing" means. **And it is imprecise in a way the output has to show.** On the production vault it classifies 「也就是说是不是用token换智力？」 as self-sufficient with coverage 1.000, because `token` appears in some page's title. That is the same defect a coverage gate would have had: the number measures whether the question carries an aspect word, not whether it names its own topic. So the rate is a lower bound, and the per-turn table is printed so a reader can correct it — 29 % by the statistic, 29-43 % by reading the same seven turns. First measurement, 7 user turns from the production vault: 2 follow-ups. n is small because the plugin keeps only the latest conversation. A real rate needs accumulation, and that would be a product change: a local counter that appends one line per user turn, so a user can hand over the numbers without sending anything anywhere.
Two corrections to the same tool, both the same defect: a statement that was
stronger than what the code does.
One, the header claimed the harness and this tool "cannot disagree about what
'names nothing' means". They agree on the predicate and disagree on the corpus.
The harness loads page bodies; this tool is given a `wiki` directory and reads
only `index.md`. `nameTierCoverage` skips a term whose document frequency is 0
from BOTH the numerator and the denominator, so a term that appears only in a
body is absent here and counted there. The comment now says that, with the
direction: coverage here is the higher of the two, so a question looks
self-sufficient, so this rate is the lower of the two. That is a second,
independent reason the number is a floor.
Two, a missing `queryHistory` field and an empty history both printed one line
and both exited 0. A caller cannot branch on a message, and "the argument is the
wrong file" is not the same answer as "the measurement ran and found nothing".
A missing field now exits 1 and names the field; an empty history still exits 0.
Verified by running it, not by reading it:
{"other":1} -> exit 1, "no queryHistory field in ... is this t..."
{"queryHistory":[]} -> exit 0, "no user turns in queryHistory"
Also rebased onto current main, because #890 changed the dependency set and this
branch predated it, which fails two third-party-notice tests locally.
green-dalii
force-pushed
the
feat/followup-rate-from-real-history
branch
from
October 25, 2026 02:51
6398244 to
078585d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The conversation harness measures a constructed follow-up: the question names nothing at all (
那它呢？), so "no history answers it" is close to built in. That proves the history can carry a follow-up. It does not say how many real follow-ups there are — and the design decision that depends on that number, how much the retrieval should weight the history, was being made without it.The number is free data. The plugin already stores the user's own conversation in
data.jsonunderqueryHistory. This tool reads it, counts, and prints. No model call, no network, nothing leaves the machine.The classifier reuses the harness's own shape — the production
lexMatchByTitleAndAliasesplusnameTierCoverageover the top hit — so two tools cannot disagree about what "names nothing" means.First measurement, 7 user turns from the production vault: 2 follow-ups (29 %).
The statistic is imprecise in a way the output has to show. Turn 2 is a colloquial follow-up (
也就是说= "so you mean"), and the classifier calls it self-sufficient with coverage 1.000 becausetokenappears in some page's title. That is the same defect a coverage gate would have had: the number measures whether the question carries an aspect word, not whether it names its own topic. So the printed rate is a lower bound — 29 % by the statistic, 29–43 % by reading the same seven turns — and the per-turn table is there so a reader can correct it.Why that matters beyond bookkeeping. The weighted history arm breaks even against dropping the history entirely at a 9.8 % follow-up share on the derived family and 33.9 % on the hand-written one. A real share of 29–43 % sits at or above that band, so the weighting's benefit is marginal if the rate is near the top of its plausible range. This tool is what turns that from an assumption into a number someone can check.
n = 7. The plugin keeps only the latest conversation (
maxConversationHistory). A real rate needs accumulation, and that would be a product change — a local counter appending one line per user turn — which this PR deliberately does not make.Gate 1: lint 0, tsc 0. No production code is touched.