Skip to content

ci: attribute e2e Codecov uploads to the triggering run's head commit - #15339

Closed
mattmillerai wants to merge 2 commits into
mainfrom
matt/be-7699-e2e-codecov-attribution
Closed

mattmillerai wants to merge 2 commits into
mainfrom
matt/be-7699-e2e-codecov-attribution

Conversation

@mattmillerai

@mattmillerai mattmillerai commented Aug 16, 2026 •

Copy link
Copy Markdown
Member

Status: superseded by main

Main now sets override_branch, override_commit and override_pr on this same Codecov upload step: they were added in #15225, and #16404 made the fork-PR lookup secure through resolve-pr-from-workflow-run. Main's override_pr also works for fork PRs, which this PR's pull_requests[0] could not do. Merging the original diff into current main would have left duplicate keys in the step's with: map, and GitHub Actions rejects that as an invalid workflow. The branch has now been merged with main and keeps main's version, so this PR has no diff against main and can be closed. The original description is kept below for context.

ELI-5

E2E coverage is uploaded by a follow-on workflow that runs after the E2E tests finish. In that context GitHub tells the job it is on main, so every PR's E2E coverage has been getting filed under main's latest commit instead of the PR's. Codecov then sees a PR head with unit coverage only, compares it against a main baseline inflated by other branches' E2E data, and reports a coverage drop that never happened. This tells the upload step which commit, branch and PR it is actually reporting for.

Summary

Pass the triggering workflow run's head commit, branch and PR number to the Upload E2E coverage to Codecov step in ci-tests-e2e-coverage.yaml, so E2E coverage is attributed to the PR head instead of the default-branch tip.

Changes

  • What: three new inputs on the existing Codecov upload step — override_commit: ${{ github.event.workflow_run.head_sha }}, override_branch: ${{ github.event.workflow_run.head_branch }}, override_pr: ${{ github.event.workflow_run.pull_requests[0].number }}. Nothing else in the step changes (pinned action SHA, files, flags, token, fail_ci_if_error are all untouched), and codecov.yml is not touched.

Review Focus

The bug, measured today. ci-tests-e2e-coverage.yaml runs on workflow_run, where GITHUB_SHA/GITHUB_REF point at the default-branch tip rather than the code that was tested — so the upload, which passed no overrides, always landed on main. Every run of this workflow reports headBranch: main (gh run list --workflow ci-tests-e2e-coverage.yaml), with four runs stacked on the same main commit in one morning. On Codecov, main 9ecd0a5f carries 12 sessions / 125,112 lines while the two open PR heads I checked (aa265ec1, 12c38ff4) carry 2 sessions / ~115k lines each — the ~10k-line gap is the E2E flag's files, present on main and absent on every PR head. Because the project/default status is scoped to the unit + e2e flags with target: auto, that asymmetry is exactly the phantom codecov/project failure seen on PRs whose codecov/patch passes.

Why the three inputs are the right source. For a workflow_run event, github.event.workflow_run describes the run that did the testing. I confirmed against live runs of CI: Tests E2E that its head_sha is the PR head commit for pull_request runs (e.g. aa265ec133… = PR #14543's headRefOid) and the pushed commit for push runs on main. That is the same SHA the unit-coverage upload already reports under, which is what makes the two flags land on one commit.

The riskiest line is override_pr, and it is safe. workflow_run.pull_requests[] is empty for fork PRs (a documented GitHub limitation) and for pushes/merge-queue runs, so pull_requests[0].number evaluates to an empty string. I checked the action at the pinned SHA fb8b3582: override_pr flows to CC_PR, and the uploader's k_arg/v_arg helpers emit no argument at all for an empty variable — so Codecov simply associates the upload by the (now correct) head SHA via the GitHub app. override_commit/override_branch are correct on every path. All three inputs exist at the pinned SHA (action.yml lines 97–111).

Behaviour on main is unchanged — a push to main already had head_sha equal to that main commit, so the upload lands where it landed before (slightly more precisely, since it can no longer drift to a newer tip picked up mid-run).

Deliberately not changed. No ref: was added to the Checkout sources for genhtml step: the lcov is self-contained and that checkout only feeds the HTML report's rendering, so checking out PR code in this privileged, secrets-bearing workflow_run context would be a security-surface change this fix does not need. git_service: github was not added preemptively — add it only if the upload step starts logging a git-service detection error.

Known edge case, not addressed here: if one branch has multiple open PRs, pull_requests[0] picks whichever GitHub lists first. The head SHA is authoritative for Codecov regardless, so the worst case is a coverage comment attached to the sibling PR.

Verification

  • actionlint on the edited file reports exactly the same two pre-existing shellcheck notices (SC2086, SC2129, both in unrelated run: blocks) as it does on the unmodified file on main — zero new findings.
  • The file parses as YAML and the resulting step object carries the three inputs with the intended expressions.
  • oxfmt --check (the formatter lint-staged applies to .yaml) passes.
  • No source files are touched, so the unit and E2E suites are unaffected by this diff; there is no test that asserts on workflow contents.

What cannot be verified before merge, and is therefore still open: the workflow only runs from main, so confirming that a PR head actually receives the e2e flag requires this to be merged first. After merge, pick any open PR with a completed CI: Tests E2E run and check that (1) the triggered CI: E2E Coverage run's Codecov upload logs show the PR head SHA, (2) https://api.codecov.io/api/v2/github/Comfy-Org/repos/ComfyUI_frontend/commits/<pr-head-sha>/ reports totals.sessions >= 3, and (3) codecov/project on that head reports a delta consistent with codecov/patch. Also confirm a subsequent push to main still uploads E2E coverage to that main commit.

Provenance

  • Authored by: agent-work loop
  • Verified:
    • Branch head has an empty diff against origin/main (git diff origin/main HEAD is empty).
    • Merging the original diff into main with git produced duplicate override_* keys; a duplicate-key-strict YAML load rejected them.
    • No CI has run yet on the new head. The earlier playwright-tests (mobile-chrome) failure was a pan.spec.ts touch-pan screenshot mismatch on a base two months out of date; main has since updated that snapshot.
  • Deviations:
    • The intended change already landed on main through other PRs, so this PR is now a no-op and should be closed rather than merged.

@mattmillerai mattmillerai added the agent-coded Opened by the agent-work code loop label Aug 16, 2026
@mattmillerai
mattmillerai marked this pull request as ready for review August 16, 2026 09:59
@mattmillerai
mattmillerai requested a review from a team August 16, 2026 09:59
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Aug 16, 2026
@coderabbitai

coderabbitai Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 17 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 122 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5ce9c724-88b6-4bbf-9566-68666fdc1821

📥 Commits

Reviewing files that changed from the base of the PR and between b30ed2f and dc538b5.

📒 Files selected for processing (1)
  • .github/workflows/ci-tests-e2e-coverage.yaml

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Playwright: ❌ 1817 passed, 1 failed · 1 flaky

❌ Failed Tests

📊 Browser Reports
  • chromium: View Report (✅ 1798 / ❌ 0 / ⚠️ 1 / ⏭️ 5)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 16 / ❌ 1 / ⚠️ 0 / ⏭️ 0)

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 6b0f2bd013fa16c33932085ba519cf8967b03fa7

Full docs: ADR-DEVEX-LINT-0015

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

This candidate changes nothing the matrix measures, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

@github-actions github-actions Bot added the risk:R3 PR risk grade (advisory shadow check; grader-owned) label Aug 16, 2026
@mattmillerai mattmillerai added the cursor-review Trigger multi-model Cursor agent review on this PR label Aug 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @mattmillerai.

⚠️ Panel did not produce any findings.

Every reviewer in the matrix failed to contribute — see the panel summary for which cells errored, and the run logs for the underlying cause.

Panel: 0/8 reviewers contributed findings.

Reviewers that did not contribute: claude-opus-5-thinking-max:adversarial (error), gemini-3.1-pro:adversarial (error), gpt-5.6-sol-max:adversarial (error), kimi-k3-max:adversarial (error), claude-opus-5-thinking-max:edge-case (error), gemini-3.1-pro:edge-case (error), gpt-5.6-sol-max:edge-case (error), kimi-k3-max:edge-case (error)

@codecov

codecov Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@            Coverage Diff             @@
##             main   #15339      +/-   ##
==========================================
+ Coverage   78.90%   81.54%   +2.64%     
==========================================
  Files        2211     1884     -327     
  Lines      111989   106878    -5111     
  Branches    32583    31312    -1271     
==========================================
- Hits        88362    87154    -1208     
+ Misses      23172    19371    -3801     
+ Partials      455      353     -102     
Flag Coverage Δ
unit 72.89% <ø> (ø)
website-unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 344 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kevinmastascusa pushed a commit to kevinmastascusa/ComfyUI_frontend that referenced this pull request Aug 17, 2026
…omfy-Org#15356)

## Problem

`browser_tests/tests/vueNodes/interactions/canvas/pan.spec.ts:204` —
`@mobile Can pan with touch`, run by the `mobile-chrome` Playwright
project — fails intermittently across unrelated PRs with:

```
40 pixels (ratio 0.01 of all image pixels) are different.
```

Across 9 open PRs the `playwright-tests (mobile-chrome)` check failed on
3 and passed on 6. On one PR it failed, was re-run with no code change,
and passed. Two of those failures
([Comfy-Org#15352](Comfy-Org#15352),
[Comfy-Org#15339](Comfy-Org#15339)) are
this flake and produced **byte-identical** actual/diff PNGs on different
runners. The third (Comfy-Org#15337) is a genuine rendering change from that PR —
it moved 3146 pixels here and also broke 4 `mobileBaseline.spec.ts`
screenshots, so it is not an instance of this flake.

## Root cause

I pulled the `playwright-report-mobile-chrome` artifacts and located the
40 differing pixels in the diff image:

- Bounding box `x ∈ [202, 348]`, `y ∈ [612, 617]` — inside the glyphs of
the Load Checkpoint node's `ckpt_name` combo widget value,
`v1-5-pruned-emaonly-fp16.sa…`.
- **Every other pixel of the 393×727 image is byte-identical** between
the passing and failing renders — the node's `transform: translate(26px,
444px)`, its error ring, the `ckpt_name` label, the node title, the
toolbar and the canvas grid all match exactly.
- Glyph positions are identical (per-glyph ink-centroid drift < 0.09
px). The failing render simply has ~5% less ink — the same glyphs at the
same coordinates rasterized through a different text-antialiasing path.

So this is **not** an unsettled pan, a mid-frame screenshot, or
momentum: the pan transform is fully deterministic. It is also not
something a readiness signal can wait out — Playwright reports `captured
a stable screenshot` and then re-fails all 3 retries with the exact same
40 pixels.

## Fix

Two changes, neither of which touches the baseline:

1. **Assert the pan actually landed** before comparing pixels, matching
the sibling tests in this file. A pan that never applies now fails with
a readable offset assertion instead of surfacing as a pixel diff.
2. **Bound the comparison with `maxDiffPixels: 100`**, in line with
existing usage in `selectionToolbox.spec.ts`, `interaction.spec.ts`,
`widget.spec.ts` and `canvasSettings.spec.ts`.

### Why not the alternatives

- **Wait on a ready signal** — nothing is unsettled. The geometry is
byte-identical and Playwright already burns 5s of retries on a
screenshot it considers stable.
- **Disable animations** — already the default (`animations:
'disabled'`).
- **Mask the region** — would blank a genuine part of the assertion and
require regenerating the baseline.
- **Product bug in the pan transform** — ruled out; the transform is
identical in both renders.

## Before / after

Replaying the real CI bytes through Playwright's own image comparator
(`playwright-core` `getComparator('image/png')`) against the committed
baseline:

| Input | Before | After |
| --- | --- | --- |
| CI failing render (Comfy-Org#15352 / Comfy-Org#15339, byte-identical) | **FAIL** — 40 px
| **PASS** |
| CI passing render | PASS | PASS |
| Control: Comfy-Org#15337 real rendering change | FAIL — 3146 px | **FAIL** —
3146 px |
| Control: 1px pan error (dx=1) | FAIL — 3615 px | **FAIL** — 3615 px |
| Control: 1px pan error (dy=1) | FAIL — 5845 px | **FAIL** — 5845 px |

The tolerance flips exactly the flake and nothing else. The smallest
real regression measurable on this test is a 1-pixel pan error at 3615
px — 36× the bound and 90× the observed noise.

### Verification caveat

I did not run `--repeat-each` locally. The baseline is
`mobile-chrome-linux`, generated in the CI container, so a local run
would mismatch on unrelated font rendering and would give no signal on
this specific 40-pixel difference. The table above is a stronger check:
it is a deterministic replay of the exact pixels that turned CI red,
through the same comparator Playwright uses. Confirmation that CI is
green comes from this PR's own `mobile-chrome` run.
@huang47 huang47 added the risk:xhigh PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026 — with ChatGPT Codex Connector
@huang47 huang47 removed the risk:R3 PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026
DrJKL
DrJKL previously approved these changes Sep 3, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the workflow_run attribution change and pinned Codecov action inputs. The three overrides use the triggering run metadata without expanding the privileged checkout surface. The current mobile E2E failure is unrelated to this workflow-only diff.

Main's Codecov upload step already sets override_branch, override_commit
and override_pr (#15225, hardened for forks in #16404). Keeping this
branch's copies would produce duplicate keys in the step's `with:` map,
which GitHub Actions rejects as an invalid workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added risk:medium PR risk grade (advisory shadow check; grader-owned) and removed risk:xhigh PR risk grade (advisory shadow check; grader-owned) labels Oct 8, 2026
@mattmillerai

Copy link
Copy Markdown
Member Author

This PR's diff against main is empty — the three-dot main...HEAD comparison reports no changed files, so it proposes no change at all. Closing it as a no-op rather than shepherding it to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Opened by the agent-work code loop cursor-review Trigger multi-model Cursor agent review on this PR risk:medium PR risk grade (advisory shadow check; grader-owned) size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants