test(pwsh): add opt-in contract tests against a disposable repository - #635
Open
DevSecNinja wants to merge 1 commit into
Open
test(pwsh): add opt-in contract tests against a disposable repository#635DevSecNinja wants to merge 1 commit into
DevSecNinja wants to merge 1 commit into
Conversation
Every other test mocks at the CLI wrapper boundary, which proves the
module's logic but says nothing about whether GitHub accepts the payloads
it builds or returns the shapes it parses. Two of the four bugs found in
review were exactly that gap: assumptions about API shapes that the mocks
confirmed because the fixtures were written from the same assumptions.
These drive the real audit/remediate cycle against a throwaway repository
and assert against the live API - reading with `gh api` directly rather
than through the module, so a symmetrical bug in audit and remediate
cannot hide:
- settings converge, and a second run applies nothing
- the ruleset is repository-owned, targets branch, covers the default
branch, and keeps the admin bypass
- require_code_owner_review and friends survive a merge-method
remediation (the regression the mocks could not catch)
- a same-named tag ruleset is left untouched
- the environment ends up pinned to the default branch and nothing
else, with the secret in the environment rather than repo-level
- a broader branch policy is removed rather than left beside the pin
Destructive, so off by default. Skipped unless DOTFILES_CONTRACT_TEST=1;
CI runs Pester with no tag filter, so a tag alone would not have been
enough. Verified that Pester does not execute the file's BeforeAll while
every test in it is skipped, which is what keeps a privileged token from
creating orphan repositories on an ordinary run.
The repository is named zz-delete-me-dotfiles-contract-<random> so an
orphan left by a hard kill is obviously safe to delete, and AfterAll
confirms the deletion rather than assuming it.
Not yet executed: the token in use cannot create repositories.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last outstanding item from the rubber-duck review.
Why
Every other test mocks at the CLI wrapper boundary. That proves the module's logic, but says nothing about whether GitHub accepts the payloads it builds or returns the shapes it parses.
That gap is not theoretical: two of the four bugs found in review were exactly this — assumptions about API shapes that 169 green tests confirmed, because the fixtures were written from the same assumptions. Making the fixtures realistic is what broke 7 of them.
What it verifies against the live API
Reading with
gh apidirectly rather than through the module, so a symmetrical bug in audit and remediate can't hide:branch, covers the default branch, admin bypass intactrequire_code_owner_reviewand friends survive a merge-method remediation — the regression the mocks could not catchSafety
Destructive, so off by default — skipped unless
DOTFILES_CONTRACT_TEST=1.A tag alone would not have been enough: CI runs
Invoke-PesterTests.ps1 -CIwith no tag filter, so aContract-tagged Describe would have run there. It's gated on the env var as well.I verified the setup block can't leak: Pester does not execute a file's
BeforeAllwhile every test in it is skipped — proven with a marker-file probe, not assumed. That's what stops a privileged token creating orphan repos on an ordinary run. Confirmed no repository was created during a default run.The repo is named
zz-delete-me-dotfiles-contract-<random>so an orphan left by a hard kill is obviously safe to delete, andAfterAllconfirms deletion rather than assuming it.Not yet executed
The token in this session cannot create repositories (
403 Resource not accessible by personal access token), so these have been written and safety-verified but never run green. See the PR discussion for the permissions needed.Testing
Docs: new "Contract tests" section in
docs/github-repo-config.md, including the token permission table.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com