Skip to content

Coverage drift check - #21

Closed
lucasbalieiro wants to merge 3 commits into
stratum-mining:mainfrom
lucasbalieiro:coverage-drift-check
Closed

lucasbalieiro wants to merge 3 commits into
stratum-mining:mainfrom
lucasbalieiro:coverage-drift-check

Conversation

@lucasbalieiro

Copy link
Copy Markdown
Collaborator

an idea to close #8

IT IS NOT PRETTY!

I'm starting to think it might be better to add a skill to this repo and let the AI handle this check instead of trying to do it programmatically.

And while we're at it, maybe we could add another skill to verify whether the Scenario Catalog is also up to date with the specs reference.

I remember seeing another conversation, I think on sv2-apps, about adding a .skills/ folder. Maybe we could use this project as a playground to experiment with that idea.

I relied heavily on AI for this one because most of the code is just about parsing the Markdown table.

I was reading COVERAGE.md again, and I have the feeling that we'll be changing the table structure quite often. Maintaining this code could quickly become a burden.

I also tried a simple Bash script using just grep and sed, but it wasn't reliable enough.


About what this PR is doing…

It prevents the Suite Coverage table in COVERAGE.md from drifting out of sync with the implemented scenario suite.

  • tests/coverage.rs - checks that:
    • Registered scenarios match the ✅ rows in Suite Coverage, in both directions.
    • Every registered / coverage ID has a corresponding Scenario Catalog row.
    • Every /// Covers: <ID> scenario function is registered, and every registry entry has a Covers: annotation.
    • Every Suite Coverage column maps to a suite registry.
  • .github/workflows/ci.yaml — runs cargo test --test coverage before the full test suite.
  • README.md/COVERAGE.md — document the required documentation updates and the check itself.

No runtime or CLI behavior changes, and no new dependencies.

I verified the failure output for the following cases:

  • Missing row.
  • ✅ flipped to ❌.
  • Stale ✅ without an implementation.
  • Documented but unregistered scenario function.

To run the check:

bash

cargo test --test coverage

Guard the Suite Coverage table against the scenario registry in both directions, require every scenario to have a Scenario Catalog row, and catch scenario fns that are documented but never registered.
Explain how to keep the Scenario Catalog and Suite Coverage table in sync with the scenario registry.
Fail pull requests that add or remove scenarios without updating the coverage documentation.
@lucasbalieiro

Copy link
Copy Markdown
Collaborator Author

closing in favor of an strategy using agents

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consider adding a CI check to verify COVERAGE.md drift

1 participant