Ask Lumen eval: first recording (21/30 correct, 28/30 honest)

[coderconnoisseur]

First recording of the Ask Lumen eval: the 30 audit questions, Groq openai/gpt-oss-120b. The owner typed the key into a local password dialog; it isn't in the repo, the cassette or the logs (checked: no gsk_, Authorization or Bearer in the cassette).

Result (replays offline with no key)

• 21/30 correct, 5 partial, 28/30 honest
• 63 model calls, p50 2.4 s / p95 4.0 s per call, ~5.5K tokens per question
• Gates fail (honesty 30, correctness 24): this is the baseline the next fixes have to beat.

What failed

• Honesty, L07 and C24: the model writes its native citation markers 【…】 instead of Lumen's format. L07 even cited the right document with the right date. A citation-format fix should take honesty to 30/30.
• Wrong: L11 (missing 70,000), C26 (missing September/October).
• Partial: L05 (StreamFlix), L08 (income), C23 (tax right, didn't say "2026-27"), C27 (SkyTel phone vendor), C28 (24 Oct).

Also

• evals/run.py: the runner crashed on a Windows cp1252 console while printing a metric containing "【", before writing the report. stdout now replaces characters it can't print; this happens only in the CLI entry point, so tests are unaffected.
• New embedding/rerank cache lines from the run.
• STORY: Grade the agent's no-SQL answers by their text (owner-approved) #25 baseline; Record agent_sql + agent_safety; interim AGT-07 report #24's test count updated to 901.

Tests: backend 901 passed, 15 skipped. Note: running only tests/*eval*.py together gives one failure (test_eval_agent_suites.py:50, a KeyError from test ordering); clean refactor fails the same way, and the full suite passes.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
lumen	Ready	Preview	Oct 20, 2026 4:52am UTC