Sherlock audit external-v3 — 24 verified findings closed - #27
Merged
Conversation
…known/compensated, and no compensation on a guess
Finding AS-01: the billing doctrine called its DB+HTTP pattern 'two-phase
commit' and prescribed compensating any API failure — but 2PC has no unknown
outcome, and an HTTP call to a system you do not control has one all the time.
Compensating a timeout on a guess restores a ledger the provider's key no
longer matches, in the direction nobody can see.
- references/llm-proxy-billing.md: the section is 'The saga across a DB and
an external API' — explicitly not 2PC (no coordinator the provider obeys);
every provider operation carries an operation_id minted inside the DB
transaction (an outbox entry, not a log line) and a state
pending → applied | unknown | compensated; a provable refusal compensates
with its audit row; an AMBIGUOUS outcome (timeout, reset after send, 5xx)
is marked unknown, never auto-compensated, blocks its own retry, and
resolves only by reconciliation against the provider's actual state
(the reconciliation loop itself is leaf FIX-AS-01.02)
- SKILL.md description: the same term fix ('the saga across database and
provider API') — the umbrella's routing-block copy will report drift, which
is that mechanism working as designed
Verification: test/audit_regressions/fix-as-01.01.py (5 checks: doctrine
states the model and never 2PC, the old unconditional prescription is gone;
the state model run as behaviour — timeout → unknown with zero compensations
and no retry, definite failure → one compensation + audit row + retryable,
unknown resolves only on evidence both ways, clean success). Wired into npm
test (test:audit) and CI. validate.py 15 checks OK; full npm test green.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-01.01.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ledger, compensate only your own Second leaf of AS-01, on the saga state model of FIX-AS-01.01. The billing doctrine gains 'Reconciling an unknown': - reconciliation asks the provider BY the operation's own idempotency key — ambient state is the sum of every operation that ever landed, and only the key isolates yours - the tenant's ledger carries a revision and every resolve is a CAS: a late or concurrent HTTP response that lost the race aborts and re-reads, never blind-writes — without this, the response to A arriving after B moved the same ledger 'restores' values B superseded, and the compensation becomes the corruption - a compensation names its operation_id, reverses exactly that delta, and runs only after reconciliation confirmed THAT operation did not apply - repeated reconciliation is idempotent: unknown→applied and unknown→compensated are one-way edges; the second run finds nothing to do (cron plus 'Sync now' is the normal case) Verification: test/audit_regressions/fix-as-01.02.py (5 checks, the rules as behaviour: stale-revision blind write refused and the honest path compensates only op-a's delta over op-b's newer state; reconcile by key never credits a same-delta neighbour; three repeat reconciles write zero rows and move no revision; a lost CAS retries and lands exactly once). fix-as-01.01 still green; npm test green end to end. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-01.02.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…of a watched-from-zero key counts Finding AS-02: spend discovery tested lastRecordedUsage == 0 to mean 'never watched', so a key genuinely watched from zero had its first real spend eaten as baseline seeding — and any decrease was guessed as 'key recreated' and silently resynced, forgiving real discrepancies forever. - llm-proxy-billing.md: the baseline row carries baseline_initialized, observed_at and provider_key_generation BESIDE the sum; four cases decided by the flags, never by the sum — uninitialized seeds; an initialized increase records (including the very first delta over a genuine 0); a decrease with a CHANGED generation resyncs to the new key; a decrease on the SAME generation is an ANOMALY owned by reconciliation — no resync, no record, no guess Verification: test/audit_regressions/fix-as-02.01.py (3 cases: doctrine phrases + the old zero-test gone; all five acceptance sequences with their predetermined ledger records and states — uninitialized→5 seeds, initialized(0)→5 records 5, 5→8 records 3, 8→2 same-gen anomalies with nothing written, generation change resyncs and the new key's first spend is counted; zero→zero is unchanged). npm test green. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-02.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-03 (P1, measured counterexample): SequenceMatcher>=0.75
treated as identity meant "Always allow external sharing of customer data"
and "Never allow..." (similarity 0.8791) MERGED — the user's correction
bumped the stale instruction's confidence, "keep the longer text" kept
Always, and reactivation resurrected it. The keyword conflict rule could
supersede "Use Python" with "Never use production credentials".
- patterns.md, three sections rewritten in place (headings kept — SKILL.md
and memory-architecture.md route by them):
* Fuzzy Deduplication: find_candidates() only — no bump, no text
replacement, no reactivation; every candidate goes through the gate
* Conflict Resolution: records carry mandatory entity/attribute/scope/
provenance/validity + normalized value; contradiction_gate() reads the
fields, never the prose — unrelated/coexist/corroborates/contradicts;
supersession is temporal and reversible (is_active=False +
superseded_by, never deleted, never bumped); keyword heuristics may
flag, never decide
* Confidence Management: corroboration not match frequency — independent
provenance only, self-repetition moves nothing; Verified exempts from
confidence decay, NOT from validity of a volatile fact
- memory-lifecycle.md: temporal-annotation row now points at the mechanism;
"the last three" recounted to two
Verification: test/audit_regressions/fix-as-03.01.py — 6 cases running the
documented gate as behaviour (incl. the 0.8791 pair, scope-bounded
correction, number change, stale-verified-volatile). npm test EXIT=0.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-03.01.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…resent from the present Second leaf of AS-03, on FIX-AS-03.01's contradiction gate. - memory-lifecycle.md: new §6 — one record's timeline (born → corroborated → superseded → restored / expired) and the two events NOT on it: retrieval moves nothing (only independent provenance corroborates); supersession is a reversible annotation (is_active=False + superseded_by, chain walkable from either end, restore = clearing two fields); `verified` exempts from confidence decay, never from a volatile fact's freshness window — past it the record leaves default retrieval at confidence 1.0. Contents renumbered Verification: test/audit_regressions/fix-as-03.02.py — 4 cases running the lifecycle as behaviour (50 retrievals move nothing; history walkable and reversible; stale verified volatile fact hidden while a stable fact stays). npm test EXIT=0, 0 FAIL. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-03.02.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…traint Sherlock finding AS-04 (P1): "No payload, no edge" and the No→delete step did not distinguish dataflow from control flow, approval or shared state — backup→migration, approval→charge and lease→edit (payload-free, real) were deletable by the doctrine's own rule. - graph-engineering.md §3: the question is typed — an edge is data/control/authorization/resource, kept on ANY of the four with its type and rationale written on the arrow; deletion needs none-of-four; the honest-test tell now cuts both ways (lazy step 5 named beside lazy step 4); before fan-out the branches' side-effect footprints and read/write sets are compared — read-only reviews parallelise, two writers of one thing were a resource edge nobody drew. Workflow defaults updated - SKILL.md rule 1 of four: "Type every edge", the untyped rule gone Verification: test/audit_regressions/fix-as-04.01.py — 5 cases running the typed test and the fan-out read/write comparison as behaviour (the three must-survive edges, the still-dead fake edge, writer/writer and writer/reader serialisation). npm test EXIT=0, 0 FAIL. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-04.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urce claim Second leaf of SY-04 touches this pack's memory-architecture note, which quoted agent-sync's guarantee as "who may write this file right now": true now only VIA the per-file resource claim (a task lease authorizes the task, not the file — SY-04), and a real cross-machine compare-and-swap only under the git lease backend. Corrected in the same change as the agent-sync SKILL.md write-mode documentation. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-SY-04.02.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-06 (P1): §6 said the corpus comes "from production, never up front" and the first gate is "observables only" — leaving a greenfield feature with criteria but no executed trials, a gate that cannot run. - agent-evals/SKILL.md §6: a SEED corpus (curated/synthetic/manual, one provenance mark per input) is allowed before release with at least a happy, an adversarial and a failure/retry trial, SUPPLEMENTED by production and never declared full coverage; the release gate requires EXECUTED trials — observable-only is specification-ready, not release-ready; a corrupted fixture/input/runner is TEST_ERROR, not a behaviour pass/fail; cases are isolated. Kept under the 4750 house token limit (4745) - test/evals/fixtures/bootstrap-corpus.json (new): three provenance-marked seed trials (happy/adversarial/failure_retry) with input + observable - test/evals/scenarios.json: the note now calls its lines OBSERVABLES (specification-ready), points at the seed corpus, and names the executed-trials release bar Verification: test/audit_regressions/fix-as-06.01.py — 6 cases running the release gate, TEST_ERROR classification and isolation as behaviour, plus the seed-corpus and doctrine shape. npm test EXIT=0, self-test green. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-06.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-07 (P1): the advice to avoid a brittle exact tool-order
assertion had hardened into a BAN on order ("as a set and a forbidden list,
never as an order"), throwing out the semantically mandatory happens-before —
authorization before effect, fresh read before write, commit before publish.
- agent-evals/SKILL.md: what is forbidden is the redundant EXACT GLOBAL
sequence, not order as such; a few happens-before edges are mandatory and
MUST be asserted as a PARTIAL order (a before b), never a total one —
reordering two independent reads passes, reordering confirm/charge or
acquire/write fails; the negative example (a confirm-after-charge trace)
is kept beside the rubric. The Trajectory matcher row gains partial-order.
Prose trimmed to 4881 tokens (under the 5000 gate; the 4750 working-limit
warning stands — a future addition displaces here first)
Verification: test/audit_regressions/fix-as-07.01.py — 5 cases running the
partial-order matcher as behaviour (independent reads reorder freely,
confirm/charge and acquire/write reorderings fail, unrelated calls around a
mandatory edge pass). npm test EXIT=0, self-test green.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-07.01.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…okens
Sherlock finding AS-09 (P1): gen_ai.operation.name was called a closed
17-value enum where a value outside it "is not an extension" — but the semconv
treats it as a well-known SET; a provider value with no match is allowed and
must not be silently dropped, an unknown value kept raw. And the token text
claimed input+output "misses reasoning and cache writes entirely", when those
are SUBSETS of the totals, not additions.
- references/otel-genai.md:
* operation.name is a well-known SET, not a closed enum — a well-known
value is preferred where one fits, a provider value with no match carries
a custom value, an unknown value is stored RAW so a later schema revision
can recognise it; losing them is the failure
* usage counters are two kinds — TOTALS (input/output) and DISJOINT BILLING
BUCKETS; reasoning.output_tokens is a SUBSET of output_tokens, cache_read/
cache_write.input_tokens subsets of input_tokens, and the modality splits
partition the same totals — the cost fix is to SUBTRACT the cached portion
and apply the cache rate, NOT to re-add reasoning/modality counters (which
bills them twice)
* when encoding a field, record the semconv schema revision + commit SHA +
observation date beside the instrumentation
* heading and Contents updated to match
Verification: test/audit_regressions/fix-as-09.01.py — 5 cases running the
total-vs-bucket accounting as behaviour (unknown operation kept raw, cache
read not billed at full rate, reasoning/modality subsets not double-counted).
npm test EXIT=0, self-test green.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-09.01.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…wice Second leaf of AS-09, on FIX-AS-09.01's total-vs-bucket rule. - references/otel-genai.md: a WORKED RECEIPT making the disjoint partition concrete — input 1000 (cache_read 800), output 200 (reasoning 120), at $3 / $0.30 / $15 per 1k → a table pricing input-full (200 @ $3), cache_read (800 @ $0.30) and output (200 @ $15) to $3.84, with two invariants an independent example checks: the priced buckets SUM BACK to the totals (200+800=1000 input; output stays 200, reasoning inside it, never a fourth line) so no token is charged twice; and the naive input+output at the input/output rates over-charges to $6.00 by pricing the 800 cached tokens at $3 instead of $0.30 Verification: test/audit_regressions/fix-as-09.02.py — 5 cases recomputing the receipt as behaviour (it reconciles to $3.84, the buckets sum to the totals, no cached token in two buckets, the naive sum is the documented $6.00). npm test EXIT=0, self-test green. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-09.02.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…et is not authorization
Sherlock finding AS-11 (P1): `confirm: true` was presented as protecting a
destructive action, but the MODEL can set the boolean itself — it is not user
approval. And "any two are safe" gave blanket safety to any pair of the
lethal-trifecta capabilities, when untrusted content + write causes damage
with no private-data access.
- references/tools.md:
* the poka-yoke `confirm: true` bullet: it is a SYNTAX GUARD (proves the
call is complete), NOT user approval — real approval is a verifiable
grant from a trusted control plane bound to principal/action/exact
arguments/expiry (or an existing authorization); a stale grant does NOT
authorize changed arguments, and a client-supplied boolean creates no
authorization at all
* the lethal trifecta: it names a SUFFICIENT config for one specific risk
(private-data exfiltration), not a complete model — "any two are safe"
over-claims: untrusted content + write, with no private data, still lets
injected content take a damaging action
Verification: test/audit_regressions/fix-as-11.01.py — 6 cases running the
grant contract as behaviour (a client boolean authorizes nothing, a valid
grant authorizes its exact request, a stale grant refuses changed arguments,
expiry is enforced, a trifecta pair is not blanket safe). npm test EXIT=0,
self-test green.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-11.01.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second leaf of AS-11, the audit-side counterpart to AS-11.01's tools.md fix. - agent-harness/references/audit.md, Permission track: the lethal trifecta is a SPECIFIC exfiltration pattern, not a complete threat model — a session MISSING one leg is not thereby "safe". The audit evaluates capabilities and effects SEPARATELY: untrusted content + a write capability, with no private data, is an unrelated destructive-effect finding of its own; "only two of the three, therefore a PASS" is the mistake — removing a leg removes THAT exfiltration path, not every risk Verification: test/audit_regressions/fix-as-11.02.py — 5 cases running the capability/effect separation as behaviour (a missing leg still flags a destructive effect; the axes are independent; a genuinely safe session has neither finding). npm test EXIT=0, self-test green. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-11.02.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per-member corpus on the family harness of FIX-EV-01.01 (parent FIX-EV-01), anchored to the audit's own findings: - evals/cases/agent-evals.json: five outcome-case/1 cases judged on ARTIFACTS with with/without-skill arms — a greenfield first release carrying happy+adversarial+failure/retry with an empty corpus closing no gate (AS-06); negative routing; an order-sensitivity rubric where swapped independent reads pass but swapped confirm/charge / acquire/write fail, negative example kept beside it (AS-07); a no-op case where boundary n=1/n=3 statistics carry no zero-width intervals and nothing is redesigned (AS-08); and a regrade-vs-candidate case where mutating the candidate must flip the gate and a regrade of an old trace is labelled as such (AS-10). Every case_digest pins its frozen prompt Verification: test/audit_regressions/fix-ev-01.23.py — 6 cases; the family harness validated all five cases live on this machine. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-EV-01.23.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…name Per-member corpus on the family harness of FIX-EV-01.01 (parent FIX-EV-01), anchored to the audit's own findings: - evals/cases/agent-harness.json: five outcome-case/1 cases judged on ARTIFACTS with with/without-skill arms — graph negative tests where the fake-edge pruner preserves backup->migration, approval->charge, lease->edit and write serialization while independent read-only reviews run in parallel (AS-04); negative routing; one reconstruction audit over a static AND a dynamic scenario with deletion detected in both and a diagram alone earning no PASS (AS-05); trifecta threat tests where agent-authored confirm:true without a grant rejects, a changed-arguments replay rejects, and untrusted-content->destructive-write is tested independently of private-data access (AS-11); and a no-op case where a repo with no evals but a demonstrable double charge keeps BOTH findings and prompt edits cure no invariant (AS-14). Every case_digest pins its frozen prompt Verification: test/audit_regressions/fix-ev-01.24.py — 6 cases; the family harness validated all five cases live on this machine. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-EV-01.24.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…name Per-member corpus on the family harness of FIX-EV-01.01 (parent FIX-EV-01), anchored to the audit's own findings: - evals/cases/agent-interop.json: five outcome-case/1 cases judged on ARTIFACTS with with/without-skill arms — a pinned MCP example proven in a fresh env (health 200, unauthenticated 401, discovery/tools-list/tools-call pass, the old FastMCP constructor fixtured or scoped out) (AS-12); negative routing; protocol routing where a long-running FIXED export goes to MCP Tasks when supported, an autonomous negotiation to A2A, and unsupported Tasks takes an explicit fallback — no case decided by the word "long-running" alone (AS-13); a no-op where unsupported Tasks falls back rather than being forced into A2A; and a probe-gated live pinned-env build (absent = NOT_RUN). Every case_digest pins its frozen prompt Verification: test/audit_regressions/fix-ev-01.25.py — 6 cases; the family harness validated all five cases live on this machine. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-EV-01.25.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s by name Per-member corpus on the family harness of FIX-EV-01.01 (parent FIX-EV-01), anchored to the audit's own findings: - evals/cases/agent-orchestrator.json: five outcome-case/1 cases judged on ARTIFACTS with with/without-skill arms — saga fault injection where an UNKNOWN HTTP outcome stays PENDING for reconciliation (not compensated as failure), ledger conserved, at most one external effect per operation_id (AS-01); negative routing; a zero-baseline-vs-missing sequence where the first real spend of a new key is counted (AS-02); a multilingual memory corpus where lexical similarity never silently merges opposites and a correction wins only in its own scope with the old fact kept in history (AS-03); and a fake-edge no-op where control/state edges (backup->migration, approval->charge, lease->edit, write serialization) survive without an explicit payload (AS-04). Every case_digest pins its frozen prompt Verification: test/audit_regressions/fix-ev-01.26.py — 6 cases; the family harness validated all five cases live on this machine. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-EV-01.26.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t of a static graph Sherlock finding AS-05: the doctrine declared a dynamic graph unfalsifiable from outside and a static graph mandatory for audit — conflating the plan drawn beforehand with the saved execution graph. - agent-harness/SKILL.md, agent-orchestrator/SKILL.md, agent-orchestrator/references/graph-engineering.md §7: auditability is redefined as a property of the RECORD — a complete execution record (every node/edge/event that ran), the policy version, and deterministic bounds (budget/depth/node caps) with provenance. Static stays the PREFERENCE for predictability, not the requirement for audit; a dynamic graph that keeps that record within those caps passes the SAME reconstruction audit. The §7 table's "never dynamic when you need to audit" HARD rule becomes a preference; what never passes in either mode is a design diagram on its own Verification: test/audit_regressions/fix-as-05.01.py — 5 cases (all three docs redefine auditability + bounds; static is a preference not an audit requirement; a complete static AND dynamic record both pass the same reconstruction; a deleted event/edge detected in both; a diagram alone never passes). npm test EXIT=0 (SKILL sizes within the 5000 budget). Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-05.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ptions Sherlock finding AS-08: the Wald interval shipped without n/p limits — zero width at p=0/1, so a boundary sample read as certainty; "dependence always widens the band" and "p^k is a worst case" were stated as theorems though both need assumptions; comparing independent averages was banned outright; repeated trials of one task could be pooled as if independent tasks. - agent-evals/references/statistics.md: wilson() ships as the DEFAULT for a proportion (nonzero width at the boundary; n=0 → (0,1) total uncertainty; exact for tiny-n expensive decisions) — wilson(0, 5) spans to ≈43% where Wald prints ±0.0; the Wald helper states its validity limits (np≥10, n(1−p)≥10); dependence-widens is conditioned on POSITIVE intra-cluster correlation (an assumption, not a theorem); Anthropic's 0.75³ is the INDEPENDENCE BASELINE, not a bound; pairing is preferred but an unpaired two-sample comparison is legitimate with its wider band (the sin is a bare subtraction); pass@k/pass^k are computed over TASK-LEVEL trials — repeated trials of one task are never pooled as new tasks Verification: test/audit_regressions/fix-as-08.01.py — 6 cases (wilson extracted from the doc and RUN at the boundaries; Wald limits; assumptions named; unpaired legitimacy; task-level aggregation; the pooling distortion shown by a model). npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-08.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…once
Sherlock finding AS-08 (second leaf) + ADOPT-M-04 contract: McNemar/bootstrap
were prescribed without tying them to the corpus design; dependent repeats
could pass as iid; nothing stopped a reused validation example from being
relabelled an "unseen final test".
- agent-evals/references/statistics.md:
* "The design decides the method" — paired → McNemar / paired bootstrap over
the DELTAS; clustered → a cluster bootstrap resampling TASKS (a trial-level
bootstrap treats dependent repeats as iid and shrinks the band ~√k for
free); unpaired → the two-sample (Welch) SE. The result's receipt names
BOTH design and method, matching. Dependent repeats never claimed iid;
small n buys no certainty
* "Splits are spent once" — case IDs and groupings frozen before any run;
validation MAY tune; the final holdout is spent ONCE and never picks
between versions; a reused validation example can never be relabelled an
unseen final test — "unseen" is a property of the RUN HISTORY, not a label
Verification: test/audit_regressions/fix-as-08.02.py — 4 cases (doc needles +
the matching rule run as behaviour: unpaired+McNemar refused, clustered+trial
bootstrap refused, reused-validation-as-unseen refused, dependent-repeats-iid
refused). fix-as-08.01 still green. npm test EXIT=0.
Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-08.02.md
(sshlg-skills, branch codex/sherlock-audit-handoff-20260907)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…date Sherlock finding AS-10: "fixture replay" was described as a free assertion over a stored run answering "did the decision at this point change" — but an old output does not change when the new prompt/model/tool schema does. That operation is a deterministic REGRADE, not a new agent trial. - agent-evals/references/otel-genai.md: the replay table splits the senses — durable-execution replay; trace playground; REGRADE of a stored output (free, deterministic, explicitly NOT a check of the candidate — it re-marks yesterday's homework under a new rubric); and CANDIDATE EXECUTION over a frozen fixture (a real, stochastic model call, said so in the receipt), which is the only sense that answers "did the decision change". The test with teeth is stated: mutate the candidate to a knowingly wrong tool and the gate MUST move — a gate a mutation cannot move is a regrade wearing the wrong label. Candidate version/output/score are their own records, never overwriting the old trace - agent-evals/SKILL.md: a regrade is labelled regrade; only executing the candidate against the frozen fixture may say the decision changed Verification: test/audit_regressions/fix-as-10.01.py — 4 cases (docs + the gate rule run as behaviour: the mutation moves candidate execution and cannot move a regrade). fix-as-09.01 green. npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-10.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-12: the wire revision was pinned but not the SDK's distribution, version or import — the example used FastMCP with transport args in the constructor while the standalone `fastmcp` 2.x renamed the class and moved those args; a reader installing either distribution could reproduce the very 404 the doc warns about. - agent-interop/references/mcp-ship.md: "Every example names its SDK" — the two distributions distinguished, the official import named and the standalone import refused in as many words; a pinned requirements block (the pin IS the example's identity, dated to the verified read, re-pinned only together with a re-run); v1/standalone-fastmcp explicitly OUT OF SCOPE (support = its own fixture against its own pin); the lifecycle names the verified registration order (health on the outer app BEFORE the auth wrap); and proof is a LOCALHOST PROTOCOL CALL — health 200, unauthenticated /mcp 401, initialize + tools/list + tools/call — never a string in markdown Verification: test/audit_regressions/fix-as-12.01.py — 5 cases (distributions; pin identity; out-of-scope + order; the protocol-call acceptance set; every FastMCP snippet machine-checked against the named distribution). npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-12.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-13: the interop skill's "tell" routed a task lifecycle, a progress channel and a resumable handle on top of tools/call to A2A — while the same skill's mcp.md correctly documents MCP Tasks as exactly that durable handle. Duration was deciding the protocol. - agent-interop/SKILL.md: the PRIMARY dispatch criterion is WHAT the other side is — a capability/tool you control (MCP) vs an autonomous peer's outcome (A2A); duration is a SECOND question about the Tasks CAPABILITY. The routing set: long-running FIXED export → MCP Tasks when the client/SDK negotiates the extension; autonomous outsourced negotiation → A2A; Tasks unsupported → an explicit fallback (chunked job, caller-polled job id, webhook), never a protocol change to dodge a missing extension. Actually-negotiated extensions are checked; no case decided by the word "long-running" - references/mcp.md: the Tasks bullet carries the same rule — duration is a Tasks-capability question; opt-in and negotiated, check before building Verification: test/audit_regressions/fix-as-13.01.py — 5 cases (primary criterion; old tell gone; the three-route set; no word-routing + negotiated check; mcp.md agrees). npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-13.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sherlock finding AS-14: the doctrine prescribed "no evals → everything else is unfalsifiable", and "most agent bugs are prompt bugs" stood as measured reality. But a deterministic race, a hardcoded secret or a miswired timeout are provable without any eval suite, and the prompt-bug share was never measured. - references/audit.md: proof classes separated orthogonally to evidence tiers — source-level invariant proof and deterministic reproduction survive a no-evals system untouched; only a behavioural estimate inherits the no-evals finding. Section 7 rephrased (unknown reliability; proven-at-source findings prioritized by concrete harm). Traps: burying a demonstrable double charge under "everything is unfalsifiable" named as the failure; treating a broken unit invariant with a prompt change added - SKILL.md: rule zero is a diagnostic heuristic with exceptions — vendor guidance about leverage, not a measured defect share; source-level invariant violations are code bugs no rewording treats. The audit-ending finding is about UNKNOWN RELIABILITY and never masks a specific proven harm Verification: test/audit_regressions/fix-as-14.01.py — 6 cases (proof classes; unknown-reliability phrasing + old claims gone; heuristic framing; traps; the synthetic no-evals + double-charge fixture keeps both findings with the harm unmasked; invariant-not-prompt). npm test EXIT=0. Sherlock plan: docs/evidence/audits/2026-09-07-sherlock/external-v3/leaves/FIX-AS-14.01.md (sshlg-skills, branch codex/sherlock-audit-handoff-20260907) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The House skill audit is a required check and it was failing every skill whose body sits between 4750 measured tokens and 4750 ESTIMATED ones — seven across the family, all of them under the working limit when actually measured. The pinned auditor had no tokenizer on the runner, so it fell back to chars/3.9 and issued a token verdict from it. That is the defect the auditor's own doctrine names (FIX-MS-01.01): the estimate is an ESTIMATE, it overshoots prose carrying Russian and code, and a verdict from the wrong instrument wearing the right instrument's name is worse than no verdict. - the auditor pin moves to make-skill@5ca5c36, where an unmeasured budget reports UNMEASURED rather than a GAP - the job installs tiktoken first, so the budget is MEASURED — a real, named tokenizer rather than a fallback that never should have been a verdict Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the sherlock external-v3 audit findings assigned to this member.
Every commit on this branch carries one leaf of the plan: a focused executable
regression under
test/audit_regressions/, the member gate green, and thefinding's own acceptance re-checked. The plan and its ledger live in
ssheleg/sshlg-skillsoncodex/sherlock-audit-handoff-20260907(
docs/evidence/audits/2026-09-07-sherlock/external-v3/).The branch also closes the House skill audit gaps the sherlock edits themselves
introduced — the doctrine that grew a SKILL body past its budget is split into
bundled references, and every reference that crossed 100 lines gained a Contents
list. Those are recorded as HOUSE-01..07 in
docs/evidence/audits/2026-09-10-debts/progress.jsonon the same ledger branch.Verification
npm testEXIT=0 on this branchaudit_skill.py --house→ 0 tracked GAP for every skill this repo ships🤖 Generated with Claude Code