Repository navigation
Characterize anchor compression and LLM token-cache accounting - #2
whoIsStella wants to merge 32 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: accd1606eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Seek to --offset and read only --length bytes. Reject negative bounds before opening the input, preserving existing empty-slice handling. Add focused regression coverage for bounded/default reads, EOF, empty slices, and invalid bounds. Validation: all 27 research unittest tests passed locally, including the independent oracle tests. The new bounded-I/O regression fails on the parent implementation. Exact oracle, R1 evidence, and prior-art notes are unchanged.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: abb15237f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| parser.add_argument("--anchor-length", type=int, default=4) | ||
| parser.add_argument("--min-match", type=int, default=4) |
There was a problem hiding this comment.
When a user passes --anchor-length 0 or a negative value, this CLI accepts it and later emits an uncaught ValueError; --min-match 0 is worse because the search silently considers one-byte matches and returns a result for a nonsensical configuration. Validate that both arguments are at least one before reading the input or invoking the search.
Useful? React with 👍 / 👎.
| "length": len(data), | ||
| "anchor_length": args.anchor_length, |
There was a problem hiding this comment.
When --output is used, the JSON records the anchor length but omits args.min_match, even though that option can materially change every emitted parse. Once the artifact is separated from its command line, there is no reliable way to determine which model configuration produced it, so include min_match alongside anchor_length.
Useful? React with 👍 / 👎.
Rename active research modules and documentation to anchor compression while preserving exact oracle logic, historical R1 artifacts, and prior-art caveats. Add the ABC1 codec, correctness tests, raw benchmark results, and a bounded source-cache research hypothesis that remains unimplemented and unvalidated. The measured simple anchor baseline loses to zstd on all six fixtures; this commit does not claim a compression or speed improvement. Apply the saved anchor-compression research patch and update the renamed module in the CI smoke command. Validation: all 38 research tests pass locally; the renamed exhaustive binary smoke sweep through length 7 passes. R1 artifact sweeps and native C++/Rust parity were not rerun.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e62b6b19c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| reports = [json.loads(p.read_text()) for p in sorted(directory.glob('n*.json'))] | ||
| if not reports: | ||
| parser.error(f'no completed lengths in {directory}') | ||
| assert [r['length'] for r in reports] == list(range(reports[0]['anchor_length'], reports[0]['max_length'] + 1)) |
There was a problem hiding this comment.
When this audit is run with python -O or PYTHONOPTIMIZE set, Python removes this and every other assert in the script, yet the script still writes a verification artifact reporting row and profile counts as checked. That can silently certify incomplete, corrupt, or mismatched archived results; use explicit conditions that raise regardless of optimization mode for all evidence-validation checks.
Useful? React with 👍 / 👎.
whoIsStella
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e2e352a04
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| PYTHONPATH=ppc/research python -m middle_out.characterize --max-length 16 --output-dir /tmp/r1-k2 | ||
| PYTHONPATH=ppc/research python -m middle_out.characterize --max-length 10 --anchor-length 1 --output-dir /tmp/r1-k1 | ||
| PYTHONPATH=ppc/research python -m middle_out.characterize --max-length 10 --anchor-length 3 --output-dir /tmp/r1-k3 | ||
| PYTHONPATH=ppc/research python -m middle_out.r1_followup --sweep-dir /tmp/r1-k2 --output /tmp/r1-followup.json.gz |
There was a problem hiding this comment.
In the current checkout, every command in this block fails immediately with No module named 'middle_out'. A repository-wide tree search shows that the implementation now exists only as ppc/research/anchor_compression, and the active-tool README explicitly instructs users to replace middle_out with anchor_compression; update these commands so the documented R1 sweeps and follow-up analysis are reproducible from this commit.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 066533ce52
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| result = subprocess.run(["git", "cat-file", "blob", f"{commit}:{name}"], cwd=root, | ||
| capture_output=True, check=False) | ||
| require(result.returncode == 0, f"recorded source absent from commit: {name}") |
There was a problem hiding this comment.
In a checkout retaining only reviewed commit 5a6cf713, this git cat-file check makes the documented resident-cache audit fail before replaying any artifact: the summaries reference 1b84f4b... and 4895400..., neither of which is an ancestor of the reviewed commit, so those objects disappear when the source-branch refs are absent or garbage-collected. I reproduced this with an isolated checkout, where verify_cache_results.py --complete exited with recorded source absent from commit; use a published/reachable commit mapping (as the R1 artifacts do) or verify the recorded hashes against files preserved in this commit.
Useful? React with 👍 / 👎.
650be40 to
d68b926
Compare
36e0156 to
2c78bb2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d68b9264e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| source[i : i + 2048] | ||
| + rng.randbytes(2048) | ||
| + records[i : i + 2048] | ||
| + repeated[i : i + 2048] |
There was a problem hiding this comment.
When --size is between 1024 and 2048 (which build_corpora explicitly accepts), the first source slice already fills or exceeds the requested corpus size, so the final repeat_to_size(mixed) truncates away the random, records, and repetition components. At 1024 bytes, the supposedly mixed corpus is byte-identical to source_text, causing the benchmark to report duplicate rather than mixed-corpus measurements; size each component so their concatenation, rather than each individual slice, fits the requested chunk.
Useful? React with 👍 / 👎.
| ) | ||
| args.output.parent.mkdir(parents=True, exist_ok=True) | ||
| args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") | ||
| args.output.with_suffix(".md").write_text(markdown_report(report), encoding="utf-8") |
There was a problem hiding this comment.
When the caller passes an output path whose suffix is already .md, with_suffix(".md") resolves to the same path, so this second write immediately replaces the JSON artifact written on the preceding line with Markdown. Either reject .md output names or derive a distinct report name so a successful run cannot silently discard its machine-readable results.
Useful? React with 👍 / 👎.
Continues the exact anchor-compression research, stacked on
research/codec-rehab. Historical R1 results and oracle semantics remain unchanged. The resident-cache experiment is a separately versioned static-byte model; the LLM token experiment is a separate accounting control.The resident-cache model now includes exact all-anchor parsing, equal-budget rolling and pinned directional controls, independent resident-only frame replay, and a written bounded-switch normal form for W < minimum match. Its frozen domain contains 8,176 binary strings and 51 diagnostic cases; all losses and ties are retained. An additional six-case exploratory grid includes a 52-byte mirrored input encoded in 51 bytes adaptively, versus 56 with the best fixed schedule and 60 with each directional control. These figures include metadata and raw fallback. Independent source-enumerating search checks all 147 anchor/schedule scores for that witness.
The new
llm_efficiency/control measures actual tokenizer counts over 24 frozen synthetic JSON snapshots and simulates an explicitly idealized full-block prefix cache. It checks equivalent JSON data under each representation. Mid-session minification can lower logical token totals while raising newly-prefilled token counts; consistent minification improves both on this workload. This reproduces established cache-invalidation behavior, not a novelty, model-quality, FLOPs, latency, or energy result. The research contract records primary-source collisions and open task-quality obligations.Local evidence: 61 compression research tests pass. The resident-cache artifact audit verifies all 8,233 cases and replays 90,563 selected frames, with Git source/hash provenance; it does not re-solve every stored optimum. The LLM baseline has separate tests and an independent artifact audit documented in its README. Historical R1 covers 131,068 binary inputs through length 16, its two sensitivity sweeps, and 102 finite family probes; those sweeps were not rerun for this change.
See
ppc/research/results/resident-cache-v1-2026-09-27/README.md,llm_efficiency/README.md, andllm_efficiency/RESEARCH_CONTRACT.mdfor reproduction and limits. Practical codec performance, held-out LLM task quality, broader prior art, and originality remain open. PR #1 remains separate and unmerged; Rust compilation and cross-language parity are not verified by this work. The preceding remote CI failed before any job steps because of account billing/spending limits, providing no test evidence. New commits use empty author and committer email fields; shared history is preserved.Prepared state/history experiment (not run): a frozen 24-case workload, seven query-agnostic controls, and a 168-request local-model runner are saved at
066533c. 21 local LLM tests pass, including independent truth/projection checks and a typed-store comparator. All 168 contexts were prepared without model calls. The scored batch has not started; seellm_efficiency/STATE_HISTORY_CHECKPOINT.mdfor the pause point and resume command. No LLM task-quality or novelty finding is claimed.