Skip to content

Cut the tokens in every prompt by half without losing answer quality #216

Open

Description

@arthware-dev

Story

As an admin running famstack on a local model, I want every prompt famstack sends to be about half as long, so that filing, wiki rebuilds and the agent answer faster and hold the AI server for less time, with answers as good as today.

Context

Most prompts are written as prose: long field descriptions in the JSON schema, rules that repeat those descriptions, and examples for cases the model already gets right. On a local model every prompt token costs prefill time, and the server is shared by everything the household runs.

#215 tried a compact, pseudocode-like style on one rule in the document classifier. The rule went from prose to:

- action_items: manual family tasks only (pay by transfer, sign/return form, book, cancel by date). NOT: direct debit, SEPA mandate, card on file, standing order, info or confirmation only. [] if none.

Measured against the same model with four test letters, three runs each, it gave the same 12 of 12 answers as a prose version twice its length.

The prompts, by owner:

  • Archivist, stacklets/docs/bot/pipeline.py: _build_classify_prompt, _build_capture_prompt, _build_reformat_prompt, _build_synthesize_prompt
  • Wiki, stacklets/memory/bot/: home_page.prompt, person_page.prompt, topic_page.prompt
  • Memory: stacklets/memory/bot/cli/answer.py (build_prompt), stacklets/memory/lib.py (build_rewrite_prompt)
  • Diary, stacklets/memory/bot/cli/diary.py: _READ_PROMPT, _SUMMARY_PROMPT, _EXTRACT_PROMPT, _CORRECT_PROMPT
  • AI client: lib/stack/ai/transcripts.py (_CORRECT_PROMPT), lib/stack/ai/client.py (_CLEANUP_PROMPT)
  • Agent: stacklets/agent/workspace/SOUL.md, AGENTS.md, skills/family-memory/SKILL.md, the tool descriptions in stacklets/agent/runtime/*_tool.py, join_greeting.greeting_prompt

Proposal

For each prompt:

  1. Measure: usage.prompt_tokens of one real call on a fixed input, and its output on a fixed set of inputs. That set is the eval cases where they exist (tests/e2e/eval/: classify, capture todos) and fabricated demo-family inputs where they do not.
  2. Rewrite in the compact style: field: rule, NOT: lists, one line per rule. Say each rule once, either in the schema or under Rules, never both. Keep an example only where removing it changes an answer. Keep what the output parser depends on.
  3. Measure again with the same inputs and compare the answers side by side.

One PR per owner above, so each can be judged and reverted on its own.

Out of scope:

  • Changing what a prompt asks for. This is the same behaviour in fewer tokens.
  • The input data a prompt carries (OCR text, evidence, excerpts from the family memory). Trimming those is separate work.
  • skills/zz-cache-pad/SKILL.md, which is long on purpose.

Definition of done

  • Every prompt listed above has before and after token counts, measured with usage.prompt_tokens on the same input, in its PR
  • The fixed instructions total at least 40% fewer tokens than before, with a target of 50%
  • The classify and capture-todos evals score the same as before, within the run-to-run spread of two baseline runs
  • For the prompts without an eval, the PR shows before and after answers for the same inputs, and a reviewer finds no answer worse
  • Unit tests stay green, and no test pins prompt wording

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions