Skip to content

fix: PR quality coverage and risk scoring - #634

Merged
jeromevdl merged 4 commits into
mainfrom
fix/pr-quality-signals
Oct 28, 2026
Merged

jeromevdl merged 4 commits into
mainfrom
fix/pr-quality-signals

Conversation

@jeromevdl

@jeromevdl jeromevdl commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

PR quality comments could discard coverage because of Node minor-version differences, exaggerate small complexity increases, and fail on older PR branches that do not contain the reporting scripts.

  • Load reporting tools from the workflow's merge commit while testing and comparing the exact base and PR revisions.
  • Collect backend/frontend coverage independently, compare within the same Node major, and show frontend dependency counts as n/a.
  • Use complexity deltas with a +10 frontend / +5 backend floor, adjusted by coverage.
  • Keep the intro and purple → red → orange → green order. Show the first 10 functions (or all purple functions if more), fold the rest under “Show more”, and omit the analyzed-file count.
  • Publish no comment when zero source files are analyzed; remove the previous bot report if present.
  • Replace the ambiguous failure wording with “Quality report unavailable — see CI logs.”

This prepares useful reviewer signals ahead of automated PR reviews.

Validation: all 154 script tests passed, plus lint, formatting and secret checks. Regression tests cover older PR heads, revision-bound coverage, comment publication, sorting, and missing coverage. Full frontend suites on PR #575’s base (971 tests) and PR revision (998 tests) also passed.

@jeromevdl jeromevdl changed the title Fix PR quality coverage and risk scoring fix: PR quality coverage and risk scoring Oct 27, 2026
@POLAARK

POLAARK commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Just a heads-up: I received this comment on my PR, and I think it's related to the changes introduced in this PR.

From a contributor's perspective, the message is quite unclear. If I received it without any context, I wouldn't know what it means or whether I need to take action.

I think the message should explicitly explain:

  • That it's an automated comment.
  • Its purpose and what it's trying to achieve.
  • What actions, if any, the contributor should take depending on the outcome.

This would make the feedback much more actionable and easier to understand.

@aws-samples aws-samples deleted a comment from github-actions Bot Oct 27, 2026
@jeromevdl

Copy link
Copy Markdown
Collaborator Author

@POLAARK thanks for your feedback

image

Is it more clear like this ?

@eipasteur

Copy link
Copy Markdown
Contributor

On larger PRs, the report could still take up a lot of the conversation, even with the current 20-row cap. I’d keep it expanded by default, with:

  • The quality section inside <details open>, so readers can collapse it.
  • The first 10 function rows visible, and the remaining displayed rows under “Show N more functions”.
  • The short purpose, counts and coverage status outside the collapsible section, explicitly distinguishing missing or partial coverage from a comparable result.

This keeps the useful results immediately visible while preserving the measurements, ordering and complete-results artifact. Linking each function location to the analyzed commit would also make the report easier to act on.

I’m also testing a fixed repository map inspired by Isobar. It complements the function-level report by showing where changes sit in the codebase and whether they spread across components. That gives reviewers a quick overview before following the flagged functions into the diff. Red marks changed files; green marks static links, not proven runtime impact or a risk grade.

The map would have its own <details open> section in the same comment, with credits and one image updated in place as commits arrive. It would use deterministic computation, with no runtime inference.

I’d suggest a separate follow-up PR for the Isobar integration, reusing the existing reporting workflow and publisher. The table-folding change could fit here if you’re happy with that scope.

The screenshots below show the same merged-PR pilot in four states: both sections open, quality folded, map folded, and both folded. The measurements and ordering are unchanged.

Expanded and collapsed quality report and Isobar map

@POLAARK

POLAARK commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

It's seems alright yes !

@jeromevdl

Copy link
Copy Markdown
Collaborator Author

@eipasteur, I reduced to 10 items plus a show more/less (or all purple if more than 10)

@jeromevdl
jeromevdl merged commit fbeb357 into main Oct 28, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants