Skip to content

tools: advisory semantic checks for invariant statements and sources - #23

Merged
enmand merged 4 commits into
mainfrom
invariant-semantic-checks
Sep 19, 2026
Merged

enmand merged 4 commits into
mainfrom
invariant-semantic-checks

Conversation

@enmand

@enmand enmand commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds two advisory CI checks over invariants/, plus the citation fixes the first calibration run surfaced.

Semantic checks (tools/check_invariant_semantics.py). Calls the TypeSafe System One API over the HTTP API, so there's no new dependency.

  • Statement changes. A changed statement is scored against its previous version as a rewording, a refinement, or a different rule. A refinement keeps its ID and records a revisions entry; a different rule needs a new ID. This enforces a policy maintenance/freshness.md already stated but nothing checked.
  • Source support. Each invariant is judged against the pages it cites: on a PR, for invariants that changed or that cite a page the PR edited; weekly, across the whole registry. The PR case catches a page edit that quietly invalidates an invariant, which nothing catches today.

The job never fails while the checks are calibrated, and skips when no API key is present, as on fork PRs. Findings go to the job summary. It needs a TYPESAFE_API_KEY repo secret.

Policy, kept in code rather than in the model. Reported: a source that contradicts a statement, and an invariant no source supports. Not reported: a context citation once another source states the rule, a dwn/ page contradicting an enbox-parity invariant, and a page that names a non-normative invariant's ID instead of restating it, which is the page deferring to the registry. A finding is asked a second time, because confidence drifts between runs, and a changed answer is reported as unstable rather than as a finding.

Metadata validation (tools/check_knowledge_metadata.py). Source paths must now exist, and the optional revisions and source_notes fields are checked for shape. Plain code, no model.

Findings from calibration, fixed here.

Invariant Problem Fix
DWN-AUTH-007, DWN-AUTH-008 cited dwn/authorization.md, which never states the descriptor/payload match rule cite dwn/permissions.md §"Invoking a Grant", which does; the old page moves to related
DWN-REC-008 cited dwn/records.md, which never mentions Count, Subscribe or snapshot populations cite dwn/queries-and-sync.md, which does
DID-DHT-003 the only did:dht invariant its page neither stated nor tagged state the 255-byte TXT segmentation rule in enbox/did-dht.md §"DNS mapping"

Calibration ran against the three historical statement edits (70ccea4, 7166e82, 3fb5c24). All three scored ~1.0, the refinement level, which matches how they were handled. A full --all run over 56 invariants produced 8 findings, of which 1 was real and 7 came from over-strict handling of secondary citations, now fixed.

Source / provenance checklist

  • I identified whether this change is normative, implementation, or decision knowledge — tooling and process, with enbox/did-dht.md as implementation knowledge.
  • Spec-derived claims checked; no spec-reviewed date changed.
  • Implementation-derived claims checked; enbox/did-dht.md states a rule already registered as DID-DHT-003 against the same upstream-baseline, so reviewed is unchanged.
  • TypeScript reference behavior and Rust behavior are distinguished where they differ — the TXT segmentation sentence states the rule; the TypeScript UTF-16 divergence stays under DID-DHT-009 in §"Known TypeScript divergences".
  • Known divergences linked rather than presented as resolved.
  • No review date or baseline updated without re-checking the source.

Behavior-change checklist

Not applicable. No engine behaviour changes; the invariant statements themselves are untouched.

Drift

  • The checks are advisory. Once a few PRs' worth of findings look right, a follow-up should make a confident contradiction, an unsupported invariant, and a level-2 statement change blocking. Uncertain and unstable verdicts should stay advisory, given the run-to-run drift seen in calibration.
  • revisions entries are not backfilled for the three historical edits (ENBOX-REC-001, DWN-REC-008, ENBOX-ENC-003). The field starts empty and applies going forward.
  • The invariants that were flagged and cleared have not been re-verified beyond their citations: the statements were read against the pages, not against TypeScript or Rust source.

dwn/authorization.md does not state the descriptor/payload grant
invocation match rule, and dwn/records.md does not describe snapshot
populations; both rules live in the pages now cited. The dropped pages
move to related, where they remain useful context.
DID-DHT-003 was the one did:dht invariant the page neither stated nor
tagged.
Source paths are now required to exist, and the optional revisions and
source_notes fields are checked for shape.
Two checks run against the TypeSafe System One API. A changed statement
is scored against its previous version as a rewording, a refinement, or a
different rule; a refinement keeps its ID and records a revisions entry,
while a different rule needs a new ID. Each invariant is also judged
against the pages it cites, on pull requests for invariants that changed
or that cite a changed page, and weekly across the registry.

Reported: a source that contradicts a statement, and an invariant no
source supports. Not reported: a context citation once another source
states the rule, a dwn/ page contradicting an enbox-parity invariant, and
a page that names a non-normative invariant's ID instead of restating it.
A finding is asked a second time, since confidence drifts between runs,
and a changed answer is reported as unstable rather than as a finding.

The job never fails while the checks are calibrated, and skips when no
API key is present, as on fork pull requests.
@enmand
enmand merged commit 8ee8c7f into main Sep 19, 2026
2 checks passed
@enmand
enmand deleted the invariant-semantic-checks branch September 19, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant