Explain unmatched identifiers, and make the disclosing tier worth choosing - #278
Merged
Merged
Conversation
…osing
Spec 011 Phase 4. The `identifiers` tier is built, so the endpoint no longer
refuses it.
Two things the real service settled that the spec had wrong.
**A proportion is not derivable at the aggregate tier.** The result carries
how many identifiers were *not* found and nothing about how many were
submitted; the denominator lives only behind `/token/{token}/found/all`, which
returns the reader's own identifiers and is the disclosing tier. US2 scenario
1 asks for a proportion, so the scenario is corrected rather than the number
invented -- the count is reported, the reader knows what they submitted, and
the model is told never to state a proportion or a percentage. Same class of
error as D9's "12 significant out of 1280", caught before it shipped this
time.
**The disclosing tier named nothing.** Run against a real result, the first
version fetched the reader's unmatched identifiers, sent them to the model,
and produced the aggregate summary -- because nothing in the prompt asked it
to mention them. Disclosure with no benefit, which is worse than not offering
the choice. There is now an instruction applied only when names are present,
and the served-path test asserts it is present for `identifiers` and absent
for `aggregate`.
The disclosure boundary is asserted in both directions by recording the
outbound call, not by reading the summary: the aggregate tier must never ask
for the identifiers, and the disclosing tier must. Verified by sabotage --
removing the tier check fails both, and the sabotage itself was asserted to
have applied before believing the result.
`fetch_not_found` is a separate function rather than a flag on `fetch_result`,
because a flag acquires a default and a default here is a disclosure nobody
chose. It is bounded at fifty names: a longer list would go to a model
provider and tell the reader nothing a sample does not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spec 011 Phase 4. The
identifierstier is built, so the endpoint no longer refuses it.Two things the real service settled that the spec had wrong
A proportion is not derivable at the aggregate tier. The result carries how many identifiers were not found and nothing about how many were submitted — the denominator lives only behind
/token/{token}/found/all, which returns the reader's own identifiers and is therefore the disclosing tier.US2 scenario 1 asks for a proportion. The scenario is corrected rather than the number invented: the count is reported, the reader knows what they submitted, and the model is explicitly told never to state a proportion or percentage. Same class of error as D9's "12 significant out of 1280" — caught before shipping this time rather than after.
The disclosing tier named nothing. Run against a real result, the first version fetched the reader's unmatched identifiers, sent them to a model provider, and produced the aggregate summary — because nothing in the prompt asked it to mention them.
That is disclosure with no benefit, which is worse than not offering the choice. There is now an instruction applied only when names are present, and the served-path test asserts it is present for
identifiersand absent foraggregate. Visible only by running it: every stubbed test passed.The disclosure boundary
Asserted in both directions by recording the outbound call, not by reading the summary and seeing nothing alarming — a model that simply did not mention the identifiers would pass the second check while they had already left the service.
Verified by sabotage: removing the tier check fails both. The sabotage was itself asserted to have applied before the result was believed — an earlier one in this session silently did not, and appeared to prove the opposite.
Design note
fetch_not_foundis a separate function rather than a flag onfetch_result, because a flag acquires a default and a default here is a disclosure nobody chose. Bounded at fifty names: a longer list would go to a model provider and tell the reader nothing a sample does not.600 passed, 1 skipped; mypy over 151 files, ruff clean.
🤖 Generated with Claude Code