Skip to content

Add gold_fidelity vision scanner (v0.6.2) - #8

Open
MattFisher wants to merge 3 commits into
mainfrom
claude/inspect-dataset-pdf-structure-76yiue
Open

MattFisher wants to merge 3 commits into
mainfrom
claude/inspect-dataset-pdf-structure-76yiue

Conversation

@MattFisher

Copy link
Copy Markdown
Collaborator

What this adds

The first vision LLM scanner, plus the image-input plumbing every future vision scanner builds on. gold_fidelity shows the model the rendered page image alongside the gold markdown and flags material fidelity errors — wrong numbers, dropped/invented table rows, garbled or mis-attributed text — while ignoring benign formatting differences.

Changes

  • _llm.pyjudge_batch_vision() — sends the page image as an inspect_ai ContentImage block ahead of the prompt text. The shared YES/NO parse is factored into _generate_judgment(), so the existing text-only judge_batch() path is unchanged.
  • _artifacts.pypage_image() — resolves <artifacts-dir>/page.<ext> from the --files-root cache, the same opt-in mechanism the cross-artifact scanners use. No new extraction; it reuses the page.png the pipeline already writes.
  • gold_fidelity scanner — registered as an LLM factory, so --model alone enables it. Skips any record without a page image.
  • CLI + PLAN.md--model help lists the scanner; PLAN.md marks the v0.6.2 plumbing + gold_fidelity done.

Verification

  • 6 new tests + full suite green (196 passed, 15 skipped), mypy clean, ruff clean.
  • Tests use a mocked model, so the suite needs no API key. One test asserts the image actually reaches the model as an image content block.

Credentials

No new credential plumbing — the key flows through the existing --model flag (inspect_ai's Anthropic provider reads ANTHROPIC_API_KEY). Recommended judge: Opus 4.8 (anthropic/claude-opus-4-8) for its high-res vision (2576px long edge) on dense tables/small text; Sonnet 5 for cheaper scale-out; not Haiku (no high-res vision).


📌 Handoff note (where we're up to)

This is a cloud (Claude Code on the web) session — nobody has interactive shell access to the sandbox, so this PR covers everything that could be done without a live API call.

Done: image plumbing (judge_batch_vision, page_image) + gold_fidelity, fully mocked-tested.

Blocked on credentials, not code — a live end-to-end run: ANTHROPIC_API_KEY must be added as a secret / env var in the environment's configuration (web app → environment settings); it's injected at container start, so it only appears in a new session. Once a fresh session has it (verify with env | grep -c ANTHROPIC_API_KEY, don't print the value), run the real pass against the benchmark's cached page images:

inspect-dataset scan <samples> --scanner-module <domain-module> \
  --files-root <cache> --scanners gold_fidelity --model anthropic/claude-opus-4-8

Not started (next up): gold_completeness (page regions absent from gold) and reading_order (gold block order vs visual reading order). Both ride the same judge_batch_vision path that's now in place — each is a new scanner module + prompt + mocked-model test, following gold_fidelity as the template. Neither needs a key to build or unit-test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6


Generated by Claude Code

claude and others added 3 commits July 21, 2026 08:44
Extend the LLM scanner plumbing to image inputs and add the first vision
scanner, which judges gold markdown against the rendered page image.

- _llm.py: judge_batch_vision() sends the page image as an inspect_ai
  ContentImage block ahead of the prompt text; factor the shared YES/NO
  parse into _generate_judgment() so the text-only path is unchanged.
- _artifacts.py: page_image() resolves <artifacts-dir>/page.<ext> from the
  --files-root cache, mirroring how the cross-artifact scanners opt in.
- gold_fidelity scanner: flags material discrepancies (wrong numbers,
  dropped/invented rows, garbled text); skips records with no page image.
  Registered as an LLM scanner factory; credentials flow through the
  existing --model flag (e.g. anthropic/claude-opus-4-8).
- Tests use a mocked model, so the suite needs no API key; one test asserts
  the image reaches the model as an image content block.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Record where the vision-scanner work stands for the next session: image
plumbing + gold_fidelity are done and mocked-tested; a live run is blocked
on ANTHROPIC_API_KEY being added to the cloud environment's secrets (not
settable from inside the container); gold_completeness and reading_order
remain and ride the same judge_batch_vision path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LLg5cbBcfNyyrRGJVPfsA6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants