Skip to content

test: use a secret-shaped fake credential in redaction tests (1/2) - #22736

Closed
bolichen97 wants to merge 1 commit into
mainfrom
test/redaction-secret-fixture-22552
Closed

bolichen97 wants to merge 1 commit into
mainfrom
test/redaction-secret-fixture-22552

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Redaction tests that only need "some credential" use a fake value the redactor treats as a secret, not an AWS access key ID.

About 430 test files use the AWS documentation key ID as their stand-in for a credential. The redaction design in #22552 moves key IDs to the Identifier tier, which is never redacted. Once that lands, every one of those tests would stop exercising redaction at all.

Why it matters

Without this swap, the key-ID change would have to rewrite these tests in the same PR as the security change, which buries the real behaviour change under thousands of mechanical lines. Done first, the swap is reviewable on its own, and the follow-up PR carries only the tests whose expected result really changes.

Not a goal

  • Changing what is redacted. No product code changes here.
  • Tests that check key-ID behaviour itself; those change with the key-ID PR.
  • The other half of the files, which is in the second PR of this pair.

What changed (motivation → approach → change)

Tests used a key ID to mean "a credential", so a tier change for key IDs would silently empty them. Each such test now uses a fake token that is a secret under every tier: a GitLab-shaped glpat- fake with the same 16-character body, a Slack-shaped xoxb- fake where the test depends on a 20-character length or on a body that stops at _, and a labelled fake secret key where the test reads a labelled value.

Where a test cut the fake at a fixed offset ([4:], a padding length, a split index), the offset moves with the longer prefix so the test still exercises the same boundary. The shared fixture in src/kiro_crew/testing/fake_computer_use.py is test-only (no runtime caller) and takes the same swap.

This is the first of two test-only PRs; together they cover every such test.

Backwards compatibility

Compatible: only tests and a test-only fixture module change; nothing a user runs is touched.

Removes nothing: every test keeps its assertion; only the fake input changes.

Tests

Every changed test passes on origin/main with the swap, run file by file (about 15,600 tests). The only reds in that run also fail on unmodified main on this host (test_design_tweak_backend.py, test_design_tweak_devproc.py: the host refuses the preview with 403; test_prompt_override_encoding.py: a host binary check), and CI runs them in its own environment.

No test is added, so no determinism proof applies.

Manual verification

N/A - test-only change; the suite is the verification.

Related Issues

Part of #22552

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 21, 2026 10:17
@bolichen97
bolichen97 requested a review from buluoray October 21, 2026 10:17
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7fb5855f14f7a48480b4f83da56bfeee7729f266 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7fb5855f14f7a48480b4f83da56bfeee7729f266 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only swap of fake credentials ahead of the key-ID tier change, and it leaves product code untouched.

[DESIGN-REVIEWED] 7fb5855

Description read

[DESCRIPTION-READ] 589f60dccaf3b374882f9d00a79b61f3a77f76644b121c9ec5f37bd78af9b14d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7fb5855f14f7a48480b4f83da56bfeee7729f266 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This is a test-only change of input values: it adds no product surface, defaults or riders, and the reason for it is recorded in #22552.

What this change ships

Inventory (3 items) — 3 justified

Intent: keep the redaction tests testing redaction after #22552 stops redacting AWS key IDs. This is a FIX done ahead of that change, in tests only.

  1. In the 190 files this pair covers, tests that need "a credential" now use a GitLab-shaped glpat- or Slack-shaped xoxb- fake instead of an AWS key ID — justified
  2. Fixed cut points ([4:] → [6:], padding, split indexes) move with the longer prefix, so each test still hits the same boundary — justified
  3. The test-only fixture FAKE_CREDENTIAL_FIXTURE in src/kiro_crew/testing/fake_computer_use.py gets the same swap; nothing in src/ imports it — justified

[FIRST-PRINCIPLES-REVIEWED] 7fb5855

Description read

[DESCRIPTION-READ] 275860c85c6b8104fe9696225446d052ea53b8d1df26cae20b856137b01a4c3e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7fb5855f14f7a48480b4f83da56bfeee7729f266 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7fb5855

Verdict parsed from the review's SHA-scoped output markers for commit 7fb5855f14f7a48480b4f83da56bfeee7729f266.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7fb5855f14f7a48480b4f83da56bfeee7729f266: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7fb5855f14f7a48480b4f83da56bfeee7729f266 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7fb5855

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7fb5855f14f7a48480b4f83da56bfeee7729f266: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 21, 2026 10:35
@bolichen97
bolichen97 force-pushed the test/redaction-secret-fixture-22552 branch from d958560 to b7eda21 Compare October 21, 2026 11:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@bolichen97
bolichen97 force-pushed the test/redaction-secret-fixture-22552 branch from b7eda21 to 8c3dcb5 Compare October 21, 2026 11:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@bolichen97
bolichen97 force-pushed the test/redaction-secret-fixture-22552 branch from 8c3dcb5 to 142abe5 Compare October 21, 2026 12:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
Tests that use an AWS access key ID as their stand-in for "a credential"
now use a fake token the redactor treats as a secret (a GitLab- or
Slack-shaped fake, or a labelled fake secret key). No product code changes;
every test passes on main before and after.
@bolichen97
bolichen97 force-pushed the test/redaction-secret-fixture-22552 branch from 142abe5 to 7fb5855 Compare October 21, 2026 13:17
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
@bolichen97
bolichen97 disabled auto-merge October 21, 2026 14:32
@bolichen97
bolichen97 marked this pull request as draft October 21, 2026 14:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 21, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Superseded by #22887, which carries this change together with the rest of #22552.

@bolichen97 bolichen97 closed this Oct 21, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant