Add gold_fidelity vision scanner (v0.6.2) - #8
Open
MattFisher wants to merge 3 commits into
Open
MattFisher wants to merge 3 commits into
MattFisher wants to merge 3 commits into
Conversation
Extend the LLM scanner plumbing to image inputs and add the first vision scanner, which judges gold markdown against the rendered page image. - _llm.py: judge_batch_vision() sends the page image as an inspect_ai ContentImage block ahead of the prompt text; factor the shared YES/NO parse into _generate_judgment() so the text-only path is unchanged. - _artifacts.py: page_image() resolves <artifacts-dir>/page.<ext> from the --files-root cache, mirroring how the cross-artifact scanners opt in. - gold_fidelity scanner: flags material discrepancies (wrong numbers, dropped/invented rows, garbled text); skips records with no page image. Registered as an LLM scanner factory; credentials flow through the existing --model flag (e.g. anthropic/claude-opus-4-8). - Tests use a mocked model, so the suite needs no API key; one test asserts the image reaches the model as an image content block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Record where the vision-scanner work stands for the next session: image plumbing + gold_fidelity are done and mocked-tested; a live run is blocked on ANTHROPIC_API_KEY being added to the cloud environment's secrets (not settable from inside the container); gold_completeness and reading_order remain and ride the same judge_batch_vision path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The first vision LLM scanner, plus the image-input plumbing every future vision scanner builds on.
gold_fidelityshows the model the rendered page image alongside the gold markdown and flags material fidelity errors — wrong numbers, dropped/invented table rows, garbled or mis-attributed text — while ignoring benign formatting differences.Changes
_llm.py→judge_batch_vision()— sends the page image as an inspect_aiContentImageblock ahead of the prompt text. The shared YES/NO parse is factored into_generate_judgment(), so the existing text-onlyjudge_batch()path is unchanged._artifacts.py→page_image()— resolves<artifacts-dir>/page.<ext>from the--files-rootcache, the same opt-in mechanism the cross-artifact scanners use. No new extraction; it reuses thepage.pngthe pipeline already writes.gold_fidelityscanner — registered as an LLM factory, so--modelalone enables it. Skips any record without a page image.--modelhelp lists the scanner; PLAN.md marks the v0.6.2 plumbing +gold_fidelitydone.Verification
mypyclean,ruffclean.imagecontent block.Credentials
No new credential plumbing — the key flows through the existing
--modelflag (inspect_ai's Anthropic provider readsANTHROPIC_API_KEY). Recommended judge: Opus 4.8 (anthropic/claude-opus-4-8) for its high-res vision (2576px long edge) on dense tables/small text; Sonnet 5 for cheaper scale-out; not Haiku (no high-res vision).📌 Handoff note (where we're up to)
This is a cloud (Claude Code on the web) session — nobody has interactive shell access to the sandbox, so this PR covers everything that could be done without a live API call.
Done: image plumbing (
judge_batch_vision,page_image) +gold_fidelity, fully mocked-tested.Blocked on credentials, not code — a live end-to-end run:
ANTHROPIC_API_KEYmust be added as a secret / env var in the environment's configuration (web app → environment settings); it's injected at container start, so it only appears in a new session. Once a fresh session has it (verify withenv | grep -c ANTHROPIC_API_KEY, don't print the value), run the real pass against the benchmark's cached page images:Not started (next up):
gold_completeness(page regions absent from gold) andreading_order(gold block order vs visual reading order). Both ride the samejudge_batch_visionpath that's now in place — each is a new scanner module + prompt + mocked-model test, followinggold_fidelityas the template. Neither needs a key to build or unit-test.🤖 Generated with Claude Code
https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Generated by Claude Code