Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/pr-risk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -367,7 +367,8 @@ on:
description: >-
Retain grade records as an Actions artifact without changing labels, posting comments,
or publishing the optional Check Run. The record is advisory and does not establish
that separate reads used one immutable snapshot.
that separate reads used one immutable snapshot. Overrides use the observed base SHA;
a changed or unreadable final snapshot discards the grade.
type: boolean
required: false
default: false
Expand Down
2 changes: 1 addition & 1 deletion docs/callers/pr-risk.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ fail the caller's next run at startup.
| `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect R2 floors from still-pending checks. |
| `repo_map_path` | `.github/risk.json` | Consumer risk-map override, read from the PR **base ref**. Repo-relative — a leading `/` or a `..` segment is refused, not resolved. |
| `repo_runbooks_path` | `.github/risk-runbooks.json` | Consumer runbook-registry override, read from the PR **base ref**. |
| `report_only` | `false` | Retain per-target records and results as a 30-day Actions artifact. Suppresses label, sticky comment and optional risk Check Run publication even when other inputs request them. GitHub Actions still creates run/check records, and the workflow retains its declared permissions. The record is advisory: separate reads may race with a base or head update. |
| `report_only` | `false` | Retain per-target records and results as a 30-day Actions artifact. Suppresses label, sticky comment and optional risk Check Run publication even when other inputs request them. GitHub Actions still creates run/check records, and the workflow retains its declared permissions. Policy overrides are read at the observed base SHA; a changed or unreadable final snapshot discards the grade. Separate reads can still race, including a revision changing and changing back, so the record is not readiness evidence. |
| `sticky_comment` | `false` | Publish ONE sticky PR comment — a single visible line (the tier, the axis that decided it and that axis's reason), with the formula, the per-axis reasons, the map versions and the caveats folded into a collapsed `<details>` and a "this grade is wrong" checkbox below it. Created once and updated in place, so N pushes leave one comment — with two bounds worth stating. The marker lookup scans only the newest 10 pages of 100 comments (`STICKY_MAX_PAGES`), so on a PR chatty enough to bury the sticky past ~1,000 comments it reads as absent: the next grade posts a DUPLICATE with `disputed=0`, which can also drop an already-applied `risk-grade-disputed` label while the reviewer's box is still ticked. And the upsert is list-then-create / read-then-PATCH with no atomicity — a `pr_numbers` batch and an event run cannot share a per-PR concurrency group, so two first grades can both see no marker and both post, and a tick landing between the read and the PATCH is overwritten. Ordinary single-run pushes behave exactly as described; the guarantee is weaker than it sounds at the edges. **Defaults to false on purpose:** a bot comment on every PR is a visible behaviour change, so it is a per-consumer decision, and a caller that does not opt in behaves byte-identically to before this input existed. Ticking the checkbox applies `risk-grade-disputed` on the next grade — a machine-maintained mirror, distinct from the human-owned `risk-dispute:*` overrides — and needs no permission beyond the `pull-requests: write` already granted. |
| `check_run` | `false` | Publish a Check Run on the PR's head commit carrying the tier and the reason: the immutable, timestamped, commit-attached record a mutable label cannot be. Its conclusion is hardcoded `neutral`, so it can never fail a PR even if a repo later marks it required. **Leaving it false does not excuse the `checks: write` grant** — see "Grant the whole union" above; the grant is validated at startup, the input only decides whether the `publish-check` job runs. **A Check Run also requires `enabled: true`.** `publish-check` is gated on `enabled` as well as on this input, while `grade` deliberately admits a `workflow_dispatch` regardless of it — so on exactly the staged-rollout path the `enabled` row advertises, a manual dispatch against a repo still switched off grades, labels and (with `sticky_comment`) comments, but publishes no Check Run. Evaluating the Check Run before switching the repo on therefore means making `enabled` true for that run — `vars.RISK_CONFIG` = `{"enabled": true}` (no PR) or `enabled: true` in the caller's `with:` — because grading alone will not publish one. |
| `check_name` | `PR risk (advisory)` | Name of the published Check Run. Only used when `check_run` is true. **Do not reuse the name of a required check.** Nothing validates this value, and the conclusion is hardcoded `neutral` — which branch protection treats as satisfied — so pointing it at a required check's name lets this advisory run stand in for the real merge gate. Keep it distinct and obviously advisory. |
Expand Down
7 changes: 5 additions & 2 deletions scripts/pr-risk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ Set `report_only: true` on the reusable workflow to retain grade records and per
results as a 30-day Actions artifact. It skips risk labels, sticky comments and the
optional risk Check Run, even if their options are enabled. The workflow itself still
creates GitHub Actions run/check records and retains its declared token permissions.
An ungraded result is unknown, never a low-risk verdict. Separate GitHub reads can
race with changes to the base or head; the artifact is not readiness evidence.
An ungraded result is unknown, never a low-risk verdict. The mode reads overrides
at the observed base SHA and discards a grade if the base or head differs on recheck,
the recheck fails, or the grade names another head. The results record includes the
base SHA. Reads can still race, including a revision changing and changing back;
this is not an atomic snapshot or readiness evidence.

## How a grade is computed

Expand Down
44 changes: 35 additions & 9 deletions scripts/pr-risk/grade-targets.sh
Original file line number Diff line number Diff line change
Expand Up @@ -248,25 +248,41 @@ settle_grade() { # <num> <record-file> <map-override|""> <rb-override|""> <settl
}

# ---- one target, end to end -------------------------------------------------------------------
record_result() { # <pr> <status> <tier> <label> <record|""> <waited> <base-ref> <note>
record_result() { # <pr> <status> <tier> <label> <record|""> <waited> <base-ref> <note> [base-sha]
jq -cn --argjson pr "$1" --arg status "$2" --arg tier "$3" --arg label "$4" \
--arg record "$5" --argjson waited "$6" --arg base "$7" --arg note "$8" \
--arg record "$5" --argjson waited "$6" --arg base "$7" --arg note "$8" --arg base_sha "${9:-}" \
'{pr:$pr, status:$status,
tier:(if $tier == "" then null else $tier end),
label:(if $label == "" then null else $label end),
record:(if $record == "" then null else $record end),
waited:$waited,
base_ref:(if $base == "" then null else $base end),
base_sha:(if $base_sha == "" then null else $base_sha end),
note:(if $note == "" then null else $note end)}' >> "$RESULTS"
}

process_target() { # <num> -> rc 0 = label in sync, rc 1 = failed
read_report_snapshot() { # <pr> -> base ref, base SHA, head SHA; no fallback to a branch name
init_scratch
local num="$1" base map rb record label deadline now label_rc
local num="$1" ref base_sha head_sha
retry_read "$OUTF" "repos/${REPO}/pulls/${num}" --jq '[.base.ref,.base.sha,.head.sha] | @tsv' || return 1
IFS=$'\t' read -r ref base_sha head_sha < "$OUTF"
[ -n "$ref" ] && [[ "$base_sha" =~ ^[0-9a-f]{40}$ ]] && [[ "$head_sha" =~ ^[0-9a-f]{40}$ ]] || return 1
printf '%s\t%s\t%s' "$ref" "$base_sha" "$head_sha"
}

process_target() { # <num> -> rc 0 = reported or label in sync, rc 1 = failed
init_scratch
local num="$1" base policy_ref base_sha="" head_sha="" snapshot end_snapshot grade_head map rb record label deadline now label_rc
record="record-${num}.json"
G_TIER=""; G_WAITED=0

if [ -n "$BASE_REF" ] && [ "$TARGET_COUNT" -eq 1 ]; then
if [ "$REPORT_ONLY" = 1 ]; then
snapshot="$(read_report_snapshot "$num")" || {
record_result "$num" failed "" "" "" 0 "" "the PR snapshot could not be read — NOT graded"
return 1
}
IFS=$'\t' read -r base base_sha head_sha <<< "$snapshot"
elif [ -n "$BASE_REF" ] && [ "$TARGET_COUNT" -eq 1 ]; then
# The event path: the payload already carries the base ref of the one PR being graded, so no
# API call is spent re-reading it. NOT honoured for a supplied number — the target may be a
# different PR than the event's (or there may be no event at all), and the event's base ref
Expand All @@ -279,11 +295,12 @@ process_target() { # <num> -> rc 0 = label in sync, rc 1 = failed
}
fi

map="$(fetch_override "$MAP_PATH" "repo-risk-map-${num}.json" "$base")" || {
policy_ref="${base_sha:-$base}"
map="$(fetch_override "$MAP_PATH" "repo-risk-map-${num}.json" "$policy_ref")" || {
record_result "$num" failed "" "" "" 0 "$base" "the risk-map override could not be read from ${base} — NOT graded"
return 1
}
rb="$(fetch_override "$RB_PATH" "repo-risk-runbooks-${num}.json" "$base")" || {
rb="$(fetch_override "$RB_PATH" "repo-risk-runbooks-${num}.json" "$policy_ref")" || {
record_result "$num" failed "" "" "" 0 "$base" "the runbook-registry override could not be read from ${base} — NOT graded"
return 1
}
Expand All @@ -304,11 +321,20 @@ process_target() { # <num> -> rc 0 = label in sync, rc 1 = failed
# Report-only runs must not call any publisher, even when conflicting options are set.
# A null label says no label was applied; unknown remains distinct from a low-risk grade.
if [ "$REPORT_ONLY" = 1 ]; then
if [ "$G_TIER" != unknown ] && [ -n "$G_TIER" ]; then
grade_head="$(jq -r '.head_sha // ""' "$record")"
end_snapshot="$(read_report_snapshot "$num")" || end_snapshot=""
if [ "$snapshot" != "$end_snapshot" ] || [ "$grade_head" != "$head_sha" ]; then
rm -f "$record"
record_result "$num" ungraded unknown "" "" "$G_WAITED" "$base" "snapshot changed or could not be rechecked; discarded grade" "$base_sha"
return 0
fi
fi
case "$G_TIER" in
""|unknown|null)
record_result "$num" ungraded unknown "" "$record" "$G_WAITED" "$base" "risk evidence unavailable; no label was written" ;;
record_result "$num" ungraded unknown "" "$record" "$G_WAITED" "$base" "risk evidence unavailable; no label was written" "$base_sha" ;;
*)
record_result "$num" graded "$G_TIER" "" "$record" "$G_WAITED" "$base" "report only; no label was written" ;;
record_result "$num" graded "$G_TIER" "" "$record" "$G_WAITED" "$base" "report only; no label was written; separate reads are not atomic" "$base_sha" ;;
esac
return 0
fi
Expand Down
56 changes: 55 additions & 1 deletion scripts/pr-risk/tests/test_grade_targets.sh
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,26 @@ case "$kind" in
echo "gh: Could not resolve to a PullRequest with the number of ${num} (HTTP 404)" >&2
exit 1
fi
cat "$STUB_DIR/base_ref"; echo ;;
if [[ "$filter" == *".base.sha"* ]]; then
count=0; [ ! -f "$STUB_DIR/snapshot_count" ] || count="$(cat "$STUB_DIR/snapshot_count")"
printf '%s' "$((count + 1))" > "$STUB_DIR/snapshot_count"
base_sha="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
head_sha="c0ffee1234567890abcdef1234567890abcdef12"
if [ "$count" -gt 0 ] && [ -f "$STUB_DIR/snapshot_changed" ]; then
head_sha="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
fi
if [ "$count" -gt 0 ] && [ -f "$STUB_DIR/snapshot_base_changed" ]; then
base_sha="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
fi
if [ "$count" -gt 0 ] && [ -f "$STUB_DIR/snapshot_fail_after" ]; then
echo 'gh: snapshot unavailable (HTTP 403)' >&2; exit 1
fi
[ ! -f "$STUB_DIR/snapshot_bad" ] || head_sha=""
printf '%s\t%s\t%s\n' "$(cat "$STUB_DIR/base_ref")" "$base_sha" "$head_sha"
else
cat "$STUB_DIR/base_ref"; echo
fi ;;

contents)
case "$(cat "$STUB_DIR/contents_mode" 2>/dev/null || echo 404)" in
404) echo 'gh: Not Found (HTTP 404)' >&2; exit 1 ;;
Expand All @@ -155,6 +174,7 @@ WORK=""
run_targets() {
WORK="$(mktemp -d "$SANDBOX/work.XXXXXX")"
: > "$STUB_LOG"
rm -f "$STUB_DIR/snapshot_count"
RC=0
OUT="$( cd "$WORK" && PATH="$SANDBOX/bin:$PATH" \
env REPO=test/repo TOOL_DIR="$TOOLDIR" DRY_RUN=1 \
Expand Down Expand Up @@ -484,13 +504,47 @@ eq "report-only grades successfully" 0 "$RC"
eq "report-only records the grade" graded "$(res '.status')"
eq "report-only does not claim a label was applied" null "$(res '.label')"
eq "report-only emits no check payload" 0 "$(jq -s length "$WORK/pr-risk-surfaces.jsonl")"
has_text "policy is read at the immutable base revision" "risk.json?ref=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" "$(calls)"
eq "stable evidence records the base revision" aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa "$(res '.base_sha')"
no_text "report-only makes no label or comment API call" "issues/42/labels" "$(calls)"
no_text "report-only makes no comment API call" "issues/42/comments" "$(calls)"
if [ -f "$WORK/record-42.json" ]; then
ok "report-only retains the grade record"
else
bad "grade record missing"
fi
touch "$STUB_DIR/snapshot_changed"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0
eq "a changed head makes the report ungraded" ungraded "$(res '.status')"
eq "a changed head cannot retain a confident tier" unknown "$(res '.tier')"
has_text "the report explains the changed revision" "snapshot changed" "$(results)"
rm -f "$STUB_DIR/snapshot_changed"
if [ ! -f "$WORK/record-42.json" ]; then
ok "stale grade record is discarded"
else
bad "stale grade retained"
fi
touch "$STUB_DIR/snapshot_base_changed"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0
eq "a changed base makes the report ungraded" ungraded "$(res '.status')"
eq "a changed base cannot retain a confident tier" unknown "$(res '.tier')"
rm -f "$STUB_DIR/snapshot_base_changed"
touch "$STUB_DIR/snapshot_fail_after"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0
eq "an unavailable final snapshot is ungraded" ungraded "$(res '.status')"
eq "an unavailable final snapshot has no usable record" null "$(res '.record')"
rm -f "$STUB_DIR/snapshot_fail_after"
cp "$STUB_DIR/fixture.json" "$STUB_DIR/fixture-snapshot.json"
jq '.data.repository.pullRequest.commits.nodes[0].commit.oid = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"' "$STUB_DIR/fixture-snapshot.json" > "$STUB_DIR/fixture.json"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0
eq "a grade for another head is ungraded" ungraded "$(res '.status')"
eq "a grade for another head is discarded" null "$(res '.record')"
cp "$STUB_DIR/fixture-snapshot.json" "$STUB_DIR/fixture.json"
touch "$STUB_DIR/snapshot_bad"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0
eq "a missing initial head fails the target" 1 "$RC"
eq "a missing initial head is not graded" failed "$(res '.status')"
rm -f "$STUB_DIR/snapshot_bad"
touch "$STUB_DIR/graphql_fail"
run_targets PR_NUMBERS=42 BASE_REF=main REPORT_ONLY=1 DRY_RUN=0 PUBLISH_CHECK=1 PUBLISH_COMMENT=1
eq "report-only preserves unavailable evidence" ungraded "$(res '.status')"
Expand Down
Loading