Repository navigation
ci: attribute e2e Codecov uploads to the triggering run's head commit - #15339
mattmillerai wants to merge 2 commits into
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 17 minutes Limit details: You’ve used all 1 included review currently available under your plan. You completed 122 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Comment |
🎭 Playwright: ❌ 1817 passed, 1 failed · 1 flaky❌ Failed Tests📊 Browser Reports
🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. No new findings in the changed files. Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 6b0f2bd013fa16c33932085ba519cf8967b03fa7
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measureExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight This candidate changes nothing the matrix measures, so every expensive stage was skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @mattmillerai.
Every reviewer in the matrix failed to contribute — see the panel summary for which cells errored, and the run logs for the underlying cause.
Panel: 0/8 reviewers contributed findings.
Reviewers that did not contribute: claude-opus-5-thinking-max:adversarial (error), gemini-3.1-pro:adversarial (error), gpt-5.6-sol-max:adversarial (error), kimi-k3-max:adversarial (error), claude-opus-5-thinking-max:edge-case (error), gemini-3.1-pro:edge-case (error), gpt-5.6-sol-max:edge-case (error), kimi-k3-max:edge-case (error)
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #15339 +/- ##
==========================================
+ Coverage 78.90% 81.54% +2.64%
==========================================
Files 2211 1884 -327
Lines 111989 106878 -5111
Branches 32583 31312 -1271
==========================================
- Hits 88362 87154 -1208
+ Misses 23172 19371 -3801
+ Partials 455 353 -102
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
…omfy-Org#15356) ## Problem `browser_tests/tests/vueNodes/interactions/canvas/pan.spec.ts:204` — `@mobile Can pan with touch`, run by the `mobile-chrome` Playwright project — fails intermittently across unrelated PRs with: ``` 40 pixels (ratio 0.01 of all image pixels) are different. ``` Across 9 open PRs the `playwright-tests (mobile-chrome)` check failed on 3 and passed on 6. On one PR it failed, was re-run with no code change, and passed. Two of those failures ([Comfy-Org#15352](Comfy-Org#15352), [Comfy-Org#15339](Comfy-Org#15339)) are this flake and produced **byte-identical** actual/diff PNGs on different runners. The third (Comfy-Org#15337) is a genuine rendering change from that PR — it moved 3146 pixels here and also broke 4 `mobileBaseline.spec.ts` screenshots, so it is not an instance of this flake. ## Root cause I pulled the `playwright-report-mobile-chrome` artifacts and located the 40 differing pixels in the diff image: - Bounding box `x ∈ [202, 348]`, `y ∈ [612, 617]` — inside the glyphs of the Load Checkpoint node's `ckpt_name` combo widget value, `v1-5-pruned-emaonly-fp16.sa…`. - **Every other pixel of the 393×727 image is byte-identical** between the passing and failing renders — the node's `transform: translate(26px, 444px)`, its error ring, the `ckpt_name` label, the node title, the toolbar and the canvas grid all match exactly. - Glyph positions are identical (per-glyph ink-centroid drift < 0.09 px). The failing render simply has ~5% less ink — the same glyphs at the same coordinates rasterized through a different text-antialiasing path. So this is **not** an unsettled pan, a mid-frame screenshot, or momentum: the pan transform is fully deterministic. It is also not something a readiness signal can wait out — Playwright reports `captured a stable screenshot` and then re-fails all 3 retries with the exact same 40 pixels. ## Fix Two changes, neither of which touches the baseline: 1. **Assert the pan actually landed** before comparing pixels, matching the sibling tests in this file. A pan that never applies now fails with a readable offset assertion instead of surfacing as a pixel diff. 2. **Bound the comparison with `maxDiffPixels: 100`**, in line with existing usage in `selectionToolbox.spec.ts`, `interaction.spec.ts`, `widget.spec.ts` and `canvasSettings.spec.ts`. ### Why not the alternatives - **Wait on a ready signal** — nothing is unsettled. The geometry is byte-identical and Playwright already burns 5s of retries on a screenshot it considers stable. - **Disable animations** — already the default (`animations: 'disabled'`). - **Mask the region** — would blank a genuine part of the assertion and require regenerating the baseline. - **Product bug in the pan transform** — ruled out; the transform is identical in both renders. ## Before / after Replaying the real CI bytes through Playwright's own image comparator (`playwright-core` `getComparator('image/png')`) against the committed baseline: | Input | Before | After | | --- | --- | --- | | CI failing render (Comfy-Org#15352 / Comfy-Org#15339, byte-identical) | **FAIL** — 40 px | **PASS** | | CI passing render | PASS | PASS | | Control: Comfy-Org#15337 real rendering change | FAIL — 3146 px | **FAIL** — 3146 px | | Control: 1px pan error (dx=1) | FAIL — 3615 px | **FAIL** — 3615 px | | Control: 1px pan error (dy=1) | FAIL — 5845 px | **FAIL** — 5845 px | The tolerance flips exactly the flake and nothing else. The smallest real regression measurable on this test is a 1-pixel pan error at 3615 px — 36× the bound and 90× the observed noise. ### Verification caveat I did not run `--repeat-each` locally. The baseline is `mobile-chrome-linux`, generated in the CI container, so a local run would mismatch on unrelated font rendering and would give no signal on this specific 40-pixel difference. The table above is a stronger check: it is a deterministic replay of the exact pixels that turned CI red, through the same comparator Playwright uses. Confirmation that CI is green comes from this PR's own `mobile-chrome` run.
DrJKL
left a comment
There was a problem hiding this comment.
Reviewed the workflow_run attribution change and pinned Codecov action inputs. The three overrides use the triggering run metadata without expanding the privileged checkout surface. The current mobile E2E failure is unrelated to this workflow-only diff.
Main's Codecov upload step already sets override_branch, override_commit and override_pr (#15225, hardened for forks in #16404). Keeping this branch's copies would produce duplicate keys in the step's `with:` map, which GitHub Actions rejects as an invalid workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This PR's diff against |
Status: superseded by main
Main now sets
override_branch,override_commitandoverride_pron this same Codecov upload step: they were added in #15225, and #16404 made the fork-PR lookup secure throughresolve-pr-from-workflow-run. Main'soverride_pralso works for fork PRs, which this PR'spull_requests[0]could not do. Merging the original diff into current main would have left duplicate keys in the step'swith:map, and GitHub Actions rejects that as an invalid workflow. The branch has now been merged with main and keeps main's version, so this PR has no diff against main and can be closed. The original description is kept below for context.ELI-5
E2E coverage is uploaded by a follow-on workflow that runs after the E2E tests finish. In that context GitHub tells the job it is on
main, so every PR's E2E coverage has been getting filed under main's latest commit instead of the PR's. Codecov then sees a PR head with unit coverage only, compares it against a main baseline inflated by other branches' E2E data, and reports a coverage drop that never happened. This tells the upload step which commit, branch and PR it is actually reporting for.Summary
Pass the triggering workflow run's head commit, branch and PR number to the
Upload E2E coverage to Codecovstep inci-tests-e2e-coverage.yaml, so E2E coverage is attributed to the PR head instead of the default-branch tip.Changes
override_commit: ${{ github.event.workflow_run.head_sha }},override_branch: ${{ github.event.workflow_run.head_branch }},override_pr: ${{ github.event.workflow_run.pull_requests[0].number }}. Nothing else in the step changes (pinned action SHA,files,flags,token,fail_ci_if_errorare all untouched), andcodecov.ymlis not touched.Review Focus
The bug, measured today.
ci-tests-e2e-coverage.yamlruns onworkflow_run, whereGITHUB_SHA/GITHUB_REFpoint at the default-branch tip rather than the code that was tested — so the upload, which passed no overrides, always landed on main. Every run of this workflow reportsheadBranch: main(gh run list --workflow ci-tests-e2e-coverage.yaml), with four runs stacked on the same main commit in one morning. On Codecov, main9ecd0a5fcarries 12 sessions / 125,112 lines while the two open PR heads I checked (aa265ec1,12c38ff4) carry 2 sessions / ~115k lines each — the ~10k-line gap is the E2E flag's files, present on main and absent on every PR head. Because theproject/defaultstatus is scoped to theunit+e2eflags withtarget: auto, that asymmetry is exactly the phantomcodecov/projectfailure seen on PRs whosecodecov/patchpasses.Why the three inputs are the right source. For a
workflow_runevent,github.event.workflow_rundescribes the run that did the testing. I confirmed against live runs ofCI: Tests E2Ethat itshead_shais the PR head commit forpull_requestruns (e.g.aa265ec133…= PR #14543'sheadRefOid) and the pushed commit forpushruns on main. That is the same SHA the unit-coverage upload already reports under, which is what makes the two flags land on one commit.The riskiest line is
override_pr, and it is safe.workflow_run.pull_requests[]is empty for fork PRs (a documented GitHub limitation) and for pushes/merge-queue runs, sopull_requests[0].numberevaluates to an empty string. I checked the action at the pinned SHAfb8b3582:override_prflows toCC_PR, and the uploader'sk_arg/v_arghelpers emit no argument at all for an empty variable — so Codecov simply associates the upload by the (now correct) head SHA via the GitHub app.override_commit/override_branchare correct on every path. All three inputs exist at the pinned SHA (action.ymllines 97–111).Behaviour on main is unchanged — a push to main already had
head_shaequal to that main commit, so the upload lands where it landed before (slightly more precisely, since it can no longer drift to a newer tip picked up mid-run).Deliberately not changed. No
ref:was added to theCheckout sources for genhtmlstep: the lcov is self-contained and that checkout only feeds the HTML report's rendering, so checking out PR code in this privileged, secrets-bearingworkflow_runcontext would be a security-surface change this fix does not need.git_service: githubwas not added preemptively — add it only if the upload step starts logging a git-service detection error.Known edge case, not addressed here: if one branch has multiple open PRs,
pull_requests[0]picks whichever GitHub lists first. The head SHA is authoritative for Codecov regardless, so the worst case is a coverage comment attached to the sibling PR.Verification
actionlinton the edited file reports exactly the same two pre-existing shellcheck notices (SC2086, SC2129, both in unrelatedrun:blocks) as it does on the unmodified file onmain— zero new findings.oxfmt --check(the formatter lint-staged applies to.yaml) passes.What cannot be verified before merge, and is therefore still open: the workflow only runs from
main, so confirming that a PR head actually receives thee2eflag requires this to be merged first. After merge, pick any open PR with a completedCI: Tests E2Erun and check that (1) the triggeredCI: E2E Coveragerun's Codecov upload logs show the PR head SHA, (2)https://api.codecov.io/api/v2/github/Comfy-Org/repos/ComfyUI_frontend/commits/<pr-head-sha>/reportstotals.sessions >= 3, and (3)codecov/projecton that head reports a delta consistent withcodecov/patch. Also confirm a subsequent push to main still uploads E2E coverage to that main commit.Provenance
origin/main(git diff origin/main HEADis empty).override_*keys; a duplicate-key-strict YAML load rejected them.playwright-tests (mobile-chrome)failure was apan.spec.tstouch-pan screenshot mismatch on a base two months out of date; main has since updated that snapshot.