Skip to content

tools: rank the knowledge base against a change, and check comment hygiene - #25

Closed
enmand wants to merge 1 commit into
mainfrom
knowledge-ci-checks
Closed

enmand wants to merge 1 commit into
mainfrom
knowledge-ci-checks

Conversation

@enmand

@enmand enmand commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Two checks that run unattended, with no agent driving them.

Knowledge drift ranking (tools/check_knowledge_drift.py). Judges every invariant against a change and lists the pages documenting the affected ones, so the PR rule in maintenance/freshness.md — update affected knowledge, link a focused follow-up, or state why it is unaffected — arrives as a checklist. It reads a pull request (--pr), a local checkout (--repo), a diff on stdin, or an issue (--text). --pages also ranks the pages no invariant cites: 27 of 75 substantive pages are cited by an invariant, so builders/, learning/, examples/ and conformance/ are otherwise unreachable. --json and --top serve a calling agent.

Questions batch over one copy of the state, so 56 invariants ride in 3 requests rather than 56. The change dominates each request, which is where the saving comes from.

Comment hygiene (tools/check_comment_hygiene.py). AGENTS.md permits exactly one process reference in a code comment: an invariant ID. This finds the others in added comment lines — a URL, an issue reference, a severity label, a packet reference, a commit or milestone marker, an enumerated list marker. Every rule is exact, so it uses no model, needs no key, and needs no calibration. It gates this repository's CI with --strict.

It found two false positives in its own first run: --intent and --questions in a usage example read as SQL -- comments. Fixed by requiring the space, with tests both ways. An exact rule fails visibly; a threshold does not.

CI: the existing job now runs every tools/test_*.py rather than one, and adds the comment check on pull requests.

Source / provenance checklist

  • I identified whether this change is normative, implementation, or decision knowledge — tooling and process only.
  • Spec-derived claims checked; no spec-reviewed date changed.
  • Implementation-derived claims checked; no upstream-baseline or reviewed date changed.
  • TypeScript reference behavior and Rust behavior are distinguished where they differ — not applicable; no behavioural claim is added.
  • Known divergences linked rather than presented as resolved.
  • No review date or baseline updated without re-checking the source.

Behavior-change checklist

Not applicable. No engine behaviour changes, and no invariant statement is edited.

Drift

…giene

Two checks that run unattended. The ranking judges every invariant
against a pull request, a local checkout, a diff, or an issue, and lists
the pages documenting the affected ones, so the PR rule in the freshness
policy arrives as a checklist. --pages also ranks the pages no invariant
cites; --json and --top serve a calling agent.

Comment hygiene is exact — a URL, an issue reference, a severity label, a
packet reference, a commit marker, or an enumerated list marker in an
added comment — so it needs no model and gates this repository's own CI
with --strict.

Questions now batch over one copy of the state, which is where the saving
comes from when the state is a diff.
@enmand

enmand commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

Closing unmerged. The ranking half overlaps local semantic search, which already indexes the invariants and returns the same pages at section granularity for free; the discovery reading-order step is better served there. The comment-hygiene half is exact and cheap and may return on its own if the rule proves worth enforcing.

@enmand enmand closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant