Skip to content

test(pwsh): add opt-in contract tests against a disposable repository - #635

Open
DevSecNinja wants to merge 1 commit into
mainfrom
test/repo-config-contract
Open

test(pwsh): add opt-in contract tests against a disposable repository#635
DevSecNinja wants to merge 1 commit into
mainfrom
test/repo-config-contract

Conversation

@DevSecNinja

Copy link
Copy Markdown
Owner

The last outstanding item from the rubber-duck review.

Why

Every other test mocks at the CLI wrapper boundary. That proves the module's logic, but says nothing about whether GitHub accepts the payloads it builds or returns the shapes it parses.

That gap is not theoretical: two of the four bugs found in review were exactly this — assumptions about API shapes that 169 green tests confirmed, because the fixtures were written from the same assumptions. Making the fixtures realistic is what broke 7 of them.

What it verifies against the live API

Reading with gh api directly rather than through the module, so a symmetrical bug in audit and remediate can't hide:

  • settings converge, and a second run applies nothing (idempotency)
  • the ruleset is repository-owned, targets branch, covers the default branch, admin bypass intact
  • require_code_owner_review and friends survive a merge-method remediation — the regression the mocks could not catch
  • a same-named tag ruleset is left untouched
  • the environment ends up pinned to the default branch and nothing else, secret in the environment rather than repo-level
  • a broader branch policy is removed rather than left beside the pin

Safety

Destructive, so off by default — skipped unless DOTFILES_CONTRACT_TEST=1.

A tag alone would not have been enough: CI runs Invoke-PesterTests.ps1 -CI with no tag filter, so a Contract-tagged Describe would have run there. It's gated on the env var as well.

I verified the setup block can't leak: Pester does not execute a file's BeforeAll while every test in it is skipped — proven with a marker-file probe, not assumed. That's what stops a privileged token creating orphan repos on an ordinary run. Confirmed no repository was created during a default run.

The repo is named zz-delete-me-dotfiles-contract-<random> so an orphan left by a hard kill is obviously safe to delete, and AfterAll confirms deletion rather than assuming it.

Not yet executed

The token in this session cannot create repositories (403 Resource not accessible by personal access token), so these have been written and safety-verified but never run green. See the PR discussion for the permissions needed.

Testing

  • 650 Pester tests pass, 58 skipped (up from 44 — the 14 contract tests)
  • PSScriptAnalyzer clean, dprint clean
  • Default run confirmed to skip all 14 and create nothing

Docs: new "Contract tests" section in docs/github-repo-config.md, including the token permission table.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Every other test mocks at the CLI wrapper boundary, which proves the
module's logic but says nothing about whether GitHub accepts the payloads
it builds or returns the shapes it parses. Two of the four bugs found in
review were exactly that gap: assumptions about API shapes that the mocks
confirmed because the fixtures were written from the same assumptions.

These drive the real audit/remediate cycle against a throwaway repository
and assert against the live API - reading with `gh api` directly rather
than through the module, so a symmetrical bug in audit and remediate
cannot hide:

  - settings converge, and a second run applies nothing
  - the ruleset is repository-owned, targets branch, covers the default
    branch, and keeps the admin bypass
  - require_code_owner_review and friends survive a merge-method
    remediation (the regression the mocks could not catch)
  - a same-named tag ruleset is left untouched
  - the environment ends up pinned to the default branch and nothing
    else, with the secret in the environment rather than repo-level
  - a broader branch policy is removed rather than left beside the pin

Destructive, so off by default. Skipped unless DOTFILES_CONTRACT_TEST=1;
CI runs Pester with no tag filter, so a tag alone would not have been
enough. Verified that Pester does not execute the file's BeforeAll while
every test in it is skipped, which is what keeps a privileged token from
creating orphan repositories on an ordinary run.

The repository is named zz-delete-me-dotfiles-contract-<random> so an
orphan left by a hard kill is obviously safe to delete, and AfterAll
confirms the deletion rather than assuming it.

Not yet executed: the token in use cannot create repositories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant