Lossless Prompt Compressor

English | 中文 | 繁體中文 | 日本語 | Deutsch | Français | 한국어 | Español | Italiano | Português

Why this exists

Every token in your system prompt is a tax on every API call. A 35,000-word instruction file means ~48,000 tokens of context eaten before the model reads a single user message. Multiply by thousands of requests per day and you're burning money on formatting, redundancy, and prose that only a human would ever need.

I built this to fix that. It takes AI instruction documents -- system prompts, CLAUDE.md files, agent rules, product specs, master plans -- and compresses them hard. Not summarizes. Compresses. Every fact, field name, constraint, and cross-reference survives intact. What gets cut is the packaging: Markdown formatting the model never needed, duplicated statements, tutorial scaffolding, motivational filler, grammatical glue that costs tokens without carrying information.

On a typical scaffolding-heavy product spec you get 45-65% reduction across all four passes. Real money saved per API call, and real context window freed up for the conversation that actually matters.

This compressor also powers the compression pass inside vibecheck.

What "lossless" actually means

This is not summarization. A summary decides what matters and throws away the rest. That's dangerous for instruction files -- one missing constraint and the model hallucinates a feature you explicitly ruled out.

Lossless means: if the original says "cap free-tier subsidies at 3% of monthly revenue", the compressed version says exactly the same thing. No facts dropped, reworded into ambiguity, or silently merged. Section numbers stay pinned so cross-references ("see S8.7.3") never break. The AI produces identical code and architecture decisions from either version.

Enforced by two mechanisms: an approval gate -- every non-mechanical edit is proposed with a before/after diff and only applied after you sign off -- and a deterministic verifier that diffs every URL, file path, identifier, number, section reference, and negation between original and compressed. Anything missing must map to an edit you approved, or it gets restored.

The 35 techniques across 4 passes

Four progressive passes, each going deeper. You choose where to stop.

Pass 1 -- Mechanical cleanup (automatic, 3-10% reduction)

No judgment calls here. This pass strips formatting that LLMs parse identically whether it is there or not:

• Markdown stripping: ## headers, **bold**, - bullets, | table | pipes, > blockquotes, --- rules -- all removed. The text and structure stay; the decoration goes. Code fences are kept: the ``` boundary tells the model where prose ends and code begins, and carries the language tag -- worth far more than the ~4 tokens it costs.
• Format conversion: Tables become semicolon-delimited rows. Bullet lists become inline semicolon-separated items. Link syntax [text](url) collapses to the URL.
• JSON minification (Technique 24): Whitespace in JSON is non-semantic, so pretty-printed JSON blocks become single-line JSON -- byte-level lossless, the highest-confidence transform in the catalog.
• Emphasis-to-caps (Technique 25): **never** becomes NEVER instead of just losing its bold -- the emphasis signal on a constraint survives at zero token cost.
• Verification sweep: After stripping, a full scan catches stray Markdown survivors. Hex colors (#FF0000), regex globs (*.txt), shell pipes, and YAML frontmatter are recognized as content and left alone.

Techniques 1-2 and 24-25 from the catalog. Safe to run on anything. Typical savings: 3-10% token reduction, 20%+ on table- and JSON-heavy documents.

Pass 2 -- Fact-preserving compression (approval-gated, +5-15%)

Here it gets interesting. Each proposed edit is shown with estimated savings and only applied on approval:

• Cross-reference deduplication (Technique 3): Facts stated in three places get stated once, with the others replaced by a pointer like "per S6".
• Code block compression (Technique 4): Struct/class definitions become compact inline descriptions preserving all field names, types, and enums. Exception: Python/YAML/Makefile stay verbatim because whitespace is semantic.
• JSON config compression (Technique 5): Full JSON examples become schema descriptions.
• Version breakdown cuts (Technique 6): If a summary table and a per-version breakdown say the same thing, the breakdown goes.
• Motivational block compression (Technique 7): "Why this matters" sections get collapsed to the core technical insight.
• Design rationale compression (Technique 8): Four paragraphs of reasoning become one sentence with the top reasons.
• Use-case list compression (Technique 9): Verbose multi-paragraph use cases become one-liners each.
• ASCII mockup compression (Technique 10): Token-expensive ASCII art becomes a compact text description.
• Terminology normalization (Technique 28): "Free tier", "free-tier plan", and "complimentary tier" for the same thing collapse to one canonical name -- one of the few techniques that compresses AND improves fidelity, since a model may treat two names as two concepts. Every merge is listed in the proposal so a false synonym can be vetoed.

Every edit must pass validation: all facts survive, the AI would make identical decisions, nothing is ambiguous, every fact traces back to its canonical section. Dedup only collapses facts of the same kind -- a normative rule ("do not retry more than 5 times") is never merged into a config value (max_attempts: 5) that happens to state the same number, because a prohibition is not a tunable default. And code-to-inline descriptions keep every modifier (var/let, optionality, access levels, defaults) -- a description that drops var lets a code-generating model emit let and produce non-compiling code.

Pass 3 -- Human-only content removal (approval-gated, +15-25%)

The test for each candidate: "If I remove this, will the AI produce worse output?" If no, it goes.

• Beginner tutorials (Technique 11): "Install your IDE from the app store. This takes 30-60 minutes..." becomes a list of what to install, not how.
• Practice prompts and exercises (Technique 12): The AI does not need someone else's learning exercises.
• Day-by-day schedules (Technique 13): Wake times and coffee breaks get cut. Feature names and technical requirements stay.
• Coaching and motivational content (Technique 14): "Don't panic" and "Celebrate!" carry zero information for the model.
• Validation matrices (Technique 15): A 30-row table where every feature says "SHIP" becomes two lines.
• Competitive comparison tables (Technique 16): "We win on every dimension" motivates humans, not models.
• Step-by-step human workflow examples (Technique 17): "What your day looks like" walkthroughs.
• Changelogs and revision histories (Technique 26): The AI needs the current state, not the path there. Facts that only live in a changelog entry get migrated into the spec body first.
• Common-knowledge explanations (Technique 29): Paragraphs explaining what OAuth is or how REST works -- the model already knows. The project-specific decision and parameters stay; the education goes.
• No-op instructions (Technique 30): "Write clean code", "follow best practices" -- they direct the model to do what it does anyway. Any line that changes a decision stays.
• Redundant few-shot example pruning (Technique 31): One example per distinct behavior or edge case suffices; pure repeats are proposed for removal with per-example approval. Survivors are never edited.

Pass 3 preserves WHAT and WHY. It removes HOW-for-humans.

Pass 4 -- Telegram rewrite (explicit opt-in, +25-40%)

Full document rewrite in ultra-dense shorthand. Every word must carry information; if it only exists to make prose flow for a human reader, it goes. For documents where the AI is the sole consumer.

• Bridge phrase elimination (Technique 18): "This means that", "which allows", "in order to", "it's important to note that" -- all cut.
• Semicolon-delimited fragments (Technique 19): Full sentences become key-value fragments. Articles, "is/are", filler subjects dropped.
• Multi-sentence collapse (Technique 20): Five sentences explaining one concept become the essential fact.
• Implied subject dropping (Technique 21): When the section heading says "Search Feature", you do not need "The search feature" at the start of every sentence.
• Compact notation (Technique 22): Slashes for options (immediate/batched/silent), arrows for flow (offline -> local queue -> sync), colons for relationships (Auth: JWT + refresh).
• Repetitive pattern compression (Technique 23): Describe the pattern once, then use shorthand for each instance.
• Alias tables (Technique 27): "RPRS = Regional Provider Routing Subsystem" defined once at the top, then the short form everywhere. Only for names appearing 5+ times, and never for strings the model must reproduce exactly in its output.
• Heading compression and echo removal (Technique 32): Headings shrink to the anchor number plus the shortest unambiguous title, and a first sentence that restates its heading goes.
• Canonical abbreviations and units (Technique 33): e.g./vs/max/~, ms/s/min/h, 100,000 -> 100k -- exact values only, never on identifiers or strings the model must emit.
• Convergence loop (Technique 34): One rewrite is not the floor -- rewriting exposes redundancy invisible in the previous form. The full pass repeats until an iteration saves less than ~2%, with verification after every iteration.
• Tokenizer-in-the-loop scoring (Technique 35): Character savings and token savings diverge systematically ("databases" is 1 token; "DBs" can cost more). Every micro-edit class is scored against the real tokenizer, and only the classes that measurably pay are applied. This is also the stopping criterion: when every remaining edit scores ~zero, the document is at the token floor.

Modern LLMs parse telegram-style just fine. Semicolons, arrows, slash-separated options -- no model family has trouble with any of it. For production prompts, an A/B behavior-parity check (same tasks against original and compressed) is still recommended before deploying.

Modes

Three modes, each building on the last.

• Strict Lossless (default): Pass 1 + Pass 2. Every fact survives, document stays human-readable. Use for files that both you and the AI reference -- CLAUDE.md, shared agent rules, living specs.

• AI-Lossless: Adds Pass 3. Strips tutorials, coaching, day-by-day schedules, validation matrices -- anything only a human reader needs. Same signal to the AI. Use for system prompts and agent instructions you rarely re-read yourself.

• AI-Only: Adds Pass 4. Full telegram-style rewrite. Maximum density, not pleasant reading. For high-volume production prompts where every token saved multiplies across thousands of daily calls.

Saying "cut it to the bone", "maximum compression", or "as dry as possible" goes straight to AI-Only mode without per-pass re-asking. And blanket approval ("apply all, don't ask per item") skips per-item gating -- every edit still lands in the ledger, and the verifier report arrives at the end. In AI-Only mode the rewrite ends with a fact-inventory audit: every atomic fact in the pre-rewrite document is located in the final version, and anything unaccounted for is restored -- "probably nothing was lost" is not the standard.

Workflow

Pass	What it does	Approval	Typical reduction
1	Strips Markdown formatting, converts tables/lists, minifies JSON	Automatic	3-10%
2	Deduplicates facts, compresses code/JSON/rationales	Per-item	+5-15%
3	Removes tutorials, coaching, schedules, human scaffolding	Per-item	+15-25%
4	Full telegram-style rewrite, fragments over sentences	Explicit opt-in	+25-40%

Every pass ends with a verifier run and a real token measurement -- numbers come from scripts/measure.py, not word-count guesswork. All four passes on a typical scaffolding-heavy product spec: 45-65% reduction. If the document is a cached system prompt, savings apply at your provider's cache rates -- compression still pays through context-window headroom and reduced attention dilution, but the skill quotes the economics honestly instead of assuming full price per call.

Worked example

A complete run on a deliberately scaffolding-heavy 1,203-word product spec lives in examples/. Every number below is measured, not estimated -- reproduce them with the commands in that file.

Stage	Tokens	Reduction	Verifier
Original	1,742	—	--
Strict Lossless (Pass 1+2)	1,375	21.1%	clean, exit 0
AI-Only (Pass 1-4)	694	60.2%	7 findings, all mapped to ledger entries

The edit ledger traces every finding back to the approval that caused it -- including the negation audit proving no rule changed polarity.

Design principles

• Lossless by default: Strict Lossless is the default. You opt in to anything more aggressive.
• Verified, not promised: scripts/verify_fidelity.py diffs URLs, file paths, @imports, identifiers, numbers, versions, and section refs between original and compressed -- in both directions, so an invented path or version is caught as well as a missing one. Normative keywords are compared by count, not as a set, because dropping four of five MUSTs leaves the set unchanged. A modality guard catches the rewrites that keep polarity but lose force -- NEVER becoming "avoid", MUST becoming "should" -- which counting negations alone misses, since the replacement is itself a negative-sounding word. Every lost "not" still gets flagged: "do not delete" becoming "delete" is the one compression error that makes output actively harmful. Structural checks cover unbalanced code fences, dropped YAML frontmatter, and lost or duplicated section anchors.
• Regression-tested: scripts/test_verify.py runs 25 stdlib tests whose fixtures are real compression failures -- deleted MUSTs, an injected instruction, a broken fence. Each is a case that must never pass clean.
• Section numbers are sacred: Cross-reference anchors never get renumbered, even when surrounding sections are removed. "See S8.7.3" keeps working.
• Prompt anatomy is protected: XML tags (<instructions>, <example>) are semantic API, not formatting. Few-shot input/output pairs are behavior definitions. Output templates are verbatim. Tool and JSON Schema definitions keep every field. MUST never weakens to "should", and a rule's scope qualifier is part of the rule -- "no backend for sync" never widens to "no backend".
• Approval-gated with an audit trail: Non-mechanical edits get proposed with a diff and estimated savings, and every applied edit lands in a compression log -- so any verifier finding traces to an approval, and selective rollback doesn't need a full restore.
• 35 techniques with examples: Full before/after catalog in references/techniques.md.
• Edge cases handled: Short documents, code-heavy prompts, already-compressed input, non-English/CJK text, embedded URLs, mixed-language documents, documents about Markdown, XML-structured prompts, creative/legal prose, very large documents, and more.

Installation

Claude Code loads skills from ~/.claude/skills/ (personal) or .claude/skills/ (per project). Clone into either one:

git clone https://github.com/wallmage/lossless-prompt-compressor.git \
  ~/.claude/skills/lossless-prompt-compressor

Then just ask for what you want -- "compress my CLAUDE.md", "shrink this system prompt", "cut this to the bone". The skill triggers on the request; you never invoke it by name.

Optional but recommended -- install tiktoken so token counts come from a real tokenizer instead of the character heuristic:

pip install tiktoken

Verify the install (25 regression tests, stdlib only):

python3 ~/.claude/skills/lossless-prompt-compressor/scripts/test_verify.py

Structure

lossless-prompt-compressor/
├── SKILL.md                     — Operating manual
├── LICENSE                      — MIT license
├── references/
│   └── techniques.md            — Full technique catalog with before/after examples
├── examples/
│   ├── spec-original.md         — Worked example: 1,742-token input spec
│   ├── spec-strict-lossless.md  — After Pass 1+2 -- 21.1% smaller, verifies clean
│   ├── spec-ai-only.md          — After all four passes -- 60.2% smaller
│   └── spec.compression-log.md  — Edit ledger + verifier finding mapping
└── scripts/
    ├── measure.py               — Lines/words/chars/tokens, before-vs-after comparison
    ├── verify_fidelity.py       — Entity diff, modality guard, structural checks
    └── test_verify.py           — 25 regression tests

License

MIT -- see LICENSE.

Author

Wallny