Repository navigation
Coverage drift check - #21
Closed
lucasbalieiro wants to merge 3 commits into
Closed
lucasbalieiro wants to merge 3 commits into
lucasbalieiro wants to merge 3 commits into
Conversation
Guard the Suite Coverage table against the scenario registry in both directions, require every scenario to have a Scenario Catalog row, and catch scenario fns that are documented but never registered.
Explain how to keep the Scenario Catalog and Suite Coverage table in sync with the scenario registry.
Fail pull requests that add or remove scenarios without updating the coverage documentation.
Collaborator
Author
|
closing in favor of an strategy using agents |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
an idea to close #8
IT IS NOT PRETTY!
I'm starting to think it might be better to add a skill to this repo and let the AI handle this check instead of trying to do it programmatically.
And while we're at it, maybe we could add another skill to verify whether the
Scenario Catalogis also up to date with the specs reference.I remember seeing another conversation, I think on
sv2-apps, about adding a.skills/folder. Maybe we could use this project as a playground to experiment with that idea.I relied heavily on AI for this one because most of the code is just about parsing the Markdown table.
I was reading
COVERAGE.mdagain, and I have the feeling that we'll be changing the table structure quite often. Maintaining this code could quickly become a burden.I also tried a simple Bash script using just
grepandsed, but it wasn't reliable enough.About what this PR is doing…
It prevents the Suite Coverage table in
COVERAGE.mdfrom drifting out of sync with the implemented scenario suite.tests/coverage.rs- checks that:/// Covers: <ID>scenario function is registered, and every registry entry has aCovers:annotation..github/workflows/ci.yaml— runscargo test --test coveragebefore the full test suite.README.md/COVERAGE.md— document the required documentation updates and the check itself.No runtime or CLI behavior changes, and no new dependencies.
I verified the failure output for the following cases:
To run the check:
bash
cargo test --test coverage