Skip to content

ci: measure advisory redteam verdicts on fork commits - #12

Draft
tuannx wants to merge 2 commits into
mainfrom
codex/redteam-advisory
Draft

tuannx wants to merge 2 commits into
mainfrom
codex/redteam-advisory

Conversation

@tuannx

@tuannx tuannx commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Add fork-only advisory commit monitoring on main, codex/**, pull requests and manual comparisons. It records exact base/head SHA provenance, double-run schema/SHA/outcome/hash ratios, counters/applicability/routing, runtime-file/tool provenance and measured adapter wall times, with artifacts instead of comments. There is no aggregate quality score or merge authorization.

Private source access is a required owner decision

The first hosted push run37346856239 failed before checkout: Unable to resolve action tuannx/redteam-doctrine, not found. The doctrine repository is private and this fork is public; GitHub private-action sharing is restricted to private consumers. No hosted gate ran in that attempt.

This PR now prepares an explicit fallback: checkout tuannx/redteam-doctrine at 0ee5a2e48f16c5e3274fceb1f6c390a10aaaedb2 into a local Action path, verify that SHA, and invoke it with comment: false. Both checkout operations disable credential persistence. The private source directory is excluded from uploaded evidence.

The owner must provision a dedicated REDTEAM_DOCTRINE_TOKEN with Contents:read restricted to tuannx/redteam-doctrine and trust this exact workflow revision (55301b98f7537a55acd7d8db47bb1e39df397a91). Credential use requires the repository-owner actor, same-repository codex/ PR heads, or owner pushes/dispatches on main/codex/ branches. No personal CLI credential is reused, no secret is provisioned by this PR, and no source is made public. With a missing credential or an untrusted event, the check fails with an actionable message and uploads distribution.json with scores:null and all gates notRun.

The pinned unsigned staging source combines proposed doctrine fixes #28, #29, #30. It is not an accepted signed policy release. The workflow is advisory: BLOCK findings stay visible; malformed/stale evidence or nondeterminism fail verification. Architecture predicates, red-green truth, changed-line coverage, mutation sampling, signed policy and attestation remain notRun.

Both existing automatic PR comment writers are disabled only for tuannx/arcade-agent. Native checks and analysis artifacts continue.

Validation

Base: cb30434e64fde86419c75490cf3f9d766836bc3a; current head: 55301b98f7537a55acd7d8db47bb1e39df397a91.

  • Current literal private-access preflight controls: 3/3 expected outcomes. Missing credential and non-owner placeholder fail; owner placeholder only passes the presence check. Every case retains scores:null/gates:notRun/privateSourceFetched:false. This is not authenticated checkout or CI proof.
  • Earlier literal local workflow/installed Action gate replay at d551366...: schema 2/2, SHA binding 2/2, outcome agreement 2/2, normalized hashes 1/1. YAML-only scope is correctly unresolvedSymbols:notRun.
  • Disposable branch controls only, outside this production PR: clean Python PASS/F821=0/exit0; planted undefined Python name BLOCK/F821=1/exit2. Local false positives 0/1, detection 1/1; literal installed gate-shell runtimes 0.571s/0.478s. Hosted controls remain unverified pending private access.
  • Native focused self-analysis/baseline tests: 27 passed/12.82s; full unchanged-source suite: 526 passed, 3 skipped/8.73s. Ruff clean. Source and test bytes are unchanged.
  • Actionlint1.7.12 passes current new workflow; changed existing workflow expressions/syntax pass. Existing arch-drift shell SC2129 style advice is unchanged.
  • Exact base/current-head Arcade pkg self-analysis: 11 components, 143 entities, 145 edges, 1 smell; BalancedArchitectureScore0.6166 unchanged; A2A similarity1.0000. No metric improvement is claimed.

Owner credential/distribution decision, hosted gate artifacts, review/merge and signed enforcement remain pending.

Hosted fallback verification: push37348596018 and PR37348604002 reached the access preflight and uploaded artifacts successfully. Downloaded producer snapshots match their exact expected base/head pairs; distribution records show ownerControlledEvent:true, privateSourceFetched:false, scores:null, and every gate:notRun. Private checkout and all Action invocations were skipped. This is measured missing-access behavior, not hosted gate success. Current architecture drift CI 37348603959 succeeded; full native CI37348604521 is still pending.

Current disposable canary producer ranges (noPR): clean 4ef13a0463c52f53abb86f174095f9872026f18c..1798ec8f4618f4578cd420f3b063fea782f7dfcf, run37349063041; bad b779c18e539587fec3bc9427d5e9adeae92c9765..7ad15fcd9ab755af4eb7233f35a54b595d8fb622, run37349067783. A whitespace-only probe touch makes the exact diff include the existing clean/adverse fixture snapshot; literal installed gate shell replays report PASS/F8210/exit0 in0.576s and BLOCK/F8211/exit2 in0.457s. They are not red-green test truth. Hosted tool scores remain null until private access and real verdict artifacts exist; no further CI reruns while the credential is absent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant