Repository navigation
Cut the tokens in every prompt by half without losing answer quality #216
Description
Story
As an admin running famstack on a local model, I want every prompt famstack sends to be about half as long, so that filing, wiki rebuilds and the agent answer faster and hold the AI server for less time, with answers as good as today.
Context
Most prompts are written as prose: long field descriptions in the JSON schema, rules that repeat those descriptions, and examples for cases the model already gets right. On a local model every prompt token costs prefill time, and the server is shared by everything the household runs.
#215 tried a compact, pseudocode-like style on one rule in the document classifier. The rule went from prose to:
- action_items: manual family tasks only (pay by transfer, sign/return form, book, cancel by date). NOT: direct debit, SEPA mandate, card on file, standing order, info or confirmation only. [] if none.
Measured against the same model with four test letters, three runs each, it gave the same 12 of 12 answers as a prose version twice its length.
The prompts, by owner:
- Archivist,
stacklets/docs/bot/pipeline.py:_build_classify_prompt,_build_capture_prompt,_build_reformat_prompt,_build_synthesize_prompt - Wiki,
stacklets/memory/bot/:home_page.prompt,person_page.prompt,topic_page.prompt - Memory:
stacklets/memory/bot/cli/answer.py(build_prompt),stacklets/memory/lib.py(build_rewrite_prompt) - Diary,
stacklets/memory/bot/cli/diary.py:_READ_PROMPT,_SUMMARY_PROMPT,_EXTRACT_PROMPT,_CORRECT_PROMPT - AI client:
lib/stack/ai/transcripts.py(_CORRECT_PROMPT),lib/stack/ai/client.py(_CLEANUP_PROMPT) - Agent:
stacklets/agent/workspace/SOUL.md,AGENTS.md,skills/family-memory/SKILL.md, the tool descriptions instacklets/agent/runtime/*_tool.py,join_greeting.greeting_prompt
Proposal
For each prompt:
- Measure:
usage.prompt_tokensof one real call on a fixed input, and its output on a fixed set of inputs. That set is the eval cases where they exist (tests/e2e/eval/: classify, capture todos) and fabricated demo-family inputs where they do not. - Rewrite in the compact style:
field: rule,NOT:lists, one line per rule. Say each rule once, either in the schema or under Rules, never both. Keep an example only where removing it changes an answer. Keep what the output parser depends on. - Measure again with the same inputs and compare the answers side by side.
One PR per owner above, so each can be judged and reverted on its own.
Out of scope:
- Changing what a prompt asks for. This is the same behaviour in fewer tokens.
- The input data a prompt carries (OCR text, evidence, excerpts from the family memory). Trimming those is separate work.
skills/zz-cache-pad/SKILL.md, which is long on purpose.
Definition of done
- Every prompt listed above has before and after token counts, measured with
usage.prompt_tokenson the same input, in its PR - The fixed instructions total at least 40% fewer tokens than before, with a target of 50%
- The classify and capture-todos evals score the same as before, within the run-to-run spread of two baseline runs
- For the prompts without an eval, the PR shows before and after answers for the same inputs, and a reviewer finds no answer worse
- Unit tests stay green, and no test pins prompt wording
Activity