Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6a25054
ci: give every unified PR report section one status line
Glary-Bot Sep 21, 2026
2bf6883
ci: tighten the normalized PR report status lines
Glary-Bot Sep 21, 2026
4dcbd37
ci: drop the report boxes that reveal nothing
Glary-Bot Sep 22, 2026
b11189a
ci: fix the report content that broke on being nested
Glary-Bot Sep 22, 2026
4c8f868
ci: flag the website E2E couplings that outlive this change
Glary-Bot Sep 22, 2026
8f733ff
ci: lead the perf heading with the failing verdict
Glary-Bot Sep 22, 2026
e80abe0
Merge remote-tracking branch 'origin/main' into glary/unified-pr-comm…
christian-byrne Oct 28, 2026
259eeb1
Merge remote-tracking branch 'origin/main' into glary/unified-pr-comm…
christian-byrne Oct 28, 2026
1d79859
ci: address review on the one-line PR report sections
christian-byrne Oct 28, 2026
a72772a
ci: keep unsampled perf runs out of the FPS verdict
christian-byrne Oct 28, 2026
0014503
refactor(perf-report): split the report into small helpers
christian-byrne Oct 28, 2026
b995459
ci: keep unsampled runs out of the perf tables; drop added comments
christian-byrne Oct 28, 2026
e79c446
Merge remote-tracking branch 'origin/main' into glary/unified-pr-comm…
christian-byrne Oct 28, 2026
96b906c
test: drop duplicate section renderer cases and assert exact padded o…
christian-byrne Oct 29, 2026
e8f41fa
fix: leave unsampled runs out of historical frame time stats
christian-byrne Oct 29, 2026
6e6b6e4
test: cover both failed-test outcomes of the Playwright report section
christian-byrne Oct 29, 2026
486f96c
test: cover the no-line-coverage heading at the coverage producer
christian-byrne Oct 29, 2026
3d7217b
test: reject invalid explain-check slugs before any report file is wr…
christian-byrne Oct 29, 2026
08b1da6
Merge remote-tracking branch 'origin/main' into glary/unified-pr-comm…
christian-byrne Oct 29, 2026
ed286a8
fix: stop history table headings overstating frame sample counts
christian-byrne Oct 29, 2026
b2e0b9b
test: accept a hyphen-and-digit explain-check slug and write its section
christian-byrne Oct 29, 2026
a94d877
Merge remote-tracking branch 'origin/main' into ci/unified-report-status
christian-byrne Oct 31, 2026
746a362
ci: address final report review findings
christian-byrne Oct 31, 2026
3654782
ci: format report comment test
christian-byrne Oct 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .claude/skills/perf-fix-with-proof/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,16 @@ Implement the fix. The `@perf` test from PR1 is now on main and will run automat
The `ci-perf-report.yaml` posts a comment like:

```markdown
## ⚡ Performance Report
## ⚡ Performance: ✅ No regressions detected

<details>
<summary>Details</summary>

| Metric | Baseline | PR (n=3) | Δ | Sig |
| --------------------- | -------- | -------- | ---- | --- |
| <name>: style recalcs | 450 | 12 | -97% | 🟢 |

</details>
```

If Δ is negative for the target metric, the fix is proven.
Expand Down
11 changes: 11 additions & 0 deletions .github/actions/comment-release-links/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ runs:
fi

SECTION_NAME='release'
SECTION_TITLE='Release'
MESSAGE='Publish jobs finished successfully:'
LINKS_VALUE=''

Expand All @@ -64,18 +65,22 @@ runs:
;;
packages/design-system/package.json)
SECTION_NAME='design-system-release'
SECTION_TITLE='Design System Release'
LINKS_VALUE='npm design-system|https://www.npmjs.com/package/@comfyorg/design-system/v/{{version}}'
;;
packages/account-core/package.json)
SECTION_NAME='account-core-release'
SECTION_TITLE='Account Core Release'
LINKS_VALUE='npm account-core|https://www.npmjs.com/package/@comfyorg/account-core/v/{{version}}'
;;
packages/billing-contract/package.json)
SECTION_NAME='billing-contract-release'
SECTION_TITLE='Billing Contract Release'
LINKS_VALUE='npm billing-contract|https://www.npmjs.com/package/@comfyorg/billing-contract/v/{{version}}'
;;
packages/ingest-types/package.json)
SECTION_NAME='ingest-types-release'
SECTION_TITLE='Ingest Types Release'
LINKS_VALUE='npm ingest-types|https://www.npmjs.com/package/@comfyorg/ingest-types/v/{{version}}'
;;
esac
Expand All @@ -86,6 +91,11 @@ runs:
COMMENT_FILE=$(mktemp)

{
echo "## 🚀 $SECTION_TITLE: ✅ Publish jobs finished for \`$NEW_VERSION\`"
echo ""
echo "<details>"
echo "<summary>Details</summary>"
echo ""
echo "$MESSAGE"
echo ""
echo "- $DIFF_LABEL: [\`$DIFF_PREFIX$PREV_VERSION...$DIFF_PREFIX$NEW_VERSION\`]($DIFF_URL)"
Expand All @@ -105,6 +115,7 @@ runs:
done <<< "$LINKS_VALUE"

echo ""
echo "</details>"
} > "$COMMENT_FILE"

{
Expand Down
5 changes: 5 additions & 0 deletions .github/actions/explain-check/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@ runs:
;;
esac

if ! [[ "$INPUT_SLUG" =~ ^[a-z0-9-]+$ ]]; then
echo "::error::explain-check: slug '$INPUT_SLUG' must match ^[a-z0-9-]+$"
exit 1
fi
Comment thread
christian-byrne marked this conversation as resolved.
Resolved

# Must live inside the workspace: read-and-upsert-comment-section guards
# on hashFiles(), which only resolves paths under GITHUB_WORKSPACE and
# would silently no-op on a RUNNER_TEMP path.
Expand Down
10 changes: 8 additions & 2 deletions .github/workflows/ci-tests-storybook.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -236,8 +236,14 @@ jobs:
pr-number: ${{ github.event.pull_request.number }}
section-name: chromatic
section-content: |
### 🎨 Chromatic Visual Tests
- 📊 [View Chromatic Build](${{ needs.chromatic-deployment.outputs.chromatic-build-url }})
## 🎨 Chromatic: 📤 Uploaded — [View Build](${{ needs.chromatic-deployment.outputs.chromatic-build-url }})

<details>
<summary>Details</summary>

**Links**
- 📚 [View Chromatic Storybook](${{ needs.chromatic-deployment.outputs.chromatic-storybook-url }})

</details>
comment-marker: '<!-- COMFYUI_FRONTEND_PR_REPORT -->'
token: ${{ github.token }}
10 changes: 5 additions & 5 deletions .github/workflows/ci-vercel-billing-web-preview.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -127,13 +127,13 @@ jobs:
section-name: billing-web-preview
token: ${{ secrets.GITHUB_TOKEN }}
section-content: |-
## 💳 Billing Web Preview

**Billing Web Preview:** ${{ steps.urls.outputs.stable-url }}
## 💳 Billing Web Preview: ✅ Deployed — [View Preview](${{ steps.urls.outputs.stable-url }})

<details>
<summary>Deployment details</summary>
<summary>Details</summary>

<sub>This commit: ${{ steps.deploy.outputs.url }}</sub>
**Links**
- 🔗 Preview: ${{ steps.urls.outputs.stable-url }}
- 📌 This commit: ${{ steps.deploy.outputs.url }}

</details>
9 changes: 9 additions & 0 deletions .github/workflows/publish-package-on-merge.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,15 @@ jobs:
run: |
set -euo pipefail
mkdir -p publish-report
PACKAGE_COUNT=$(jq -r '.include | length' <<<"$MATRIX")
{
printf '## 🚀 Release: ✅ Publish jobs finished for %s package' "$PACKAGE_COUNT"
[ "$PACKAGE_COUNT" = 1 ] || printf 's'
echo ''
echo ''
echo '<details>'
echo '<summary>Details</summary>'
echo ''
echo 'Publish jobs finished successfully:'
echo ''
jq -r '
Expand All @@ -133,6 +141,7 @@ jobs:
' <<<"$MATRIX"
echo "- Published from [\`${MERGE_SHA:0:10}\`](https://github.com/${REPO}/commit/${MERGE_SHA})"
echo ''
echo '</details>'
} > publish-report/report.md
printf '%s' "$PR_NUMBER" > publish-report/pr-number.txt

Expand Down
29 changes: 19 additions & 10 deletions scripts/cicd/pr-playwright-deploy-and-comment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -285,14 +285,9 @@ else
fi

if [ $total_failed -gt 0 ]; then
comment="$comment

> ❌ $total_failed tests failed. Expand the failed tests below, review their reports and fix the failures before merging.

<details>
<summary>Failed tests</summary>

"
failed_test_lines=""
failed_tests_label="$total_failed tests"
[ "$total_failed" -eq 1 ] && failed_tests_label="1 test"

for counts_json in "${counts_array[@]}"; do
[ -z "$counts_json" ] || [ "$counts_json" = "{}" ] && continue
Expand All @@ -312,15 +307,29 @@ else
test_line="$test_line: [View test report]($report_url)"
fi

comment="$comment
failed_test_lines="$failed_test_lines
$test_line"
done <<< "$failures"
fi
fi
done
comment="$comment

if [ -n "$failed_test_lines" ]; then
Comment thread
christian-byrne marked this conversation as resolved.
Resolved
comment="$comment

> ❌ $failed_tests_label failed. Expand the failed tests below, review their reports and fix the failures before merging.

<details>
<summary>Failed tests</summary>

${failed_test_lines#$'\n'}

</details>"
else
comment="$comment

> ❌ $failed_tests_label failed. Open the browser reports below, review the failed tests and fix the failures before merging."
fi
fi

screenshot_diffs=$(count_screenshot_diffs) || screenshot_diffs=0
Expand Down
96 changes: 85 additions & 11 deletions scripts/cicd/pr-playwright-deploy-and-comment.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ import { describe, expect, it, onTestFinished } from 'vitest'
import { parse } from 'yaml'
import { z } from 'zod'

import type { extractTestCounts } from './extract-playwright-counts'

type TestCounts = ReturnType<typeof extractTestCounts>

const script = join(import.meta.dirname, 'pr-playwright-deploy-and-comment.sh')

const e2eWorkflow = z
Expand Down Expand Up @@ -42,9 +46,18 @@ const uploadedProjects = [
)
]

const passingCounts = '{"passed":2,"failed":0,"flaky":0,"skipped":0,"total":2}'
const passingCounts: TestCounts = {
passed: 2,
failed: 0,
flaky: 0,
skipped: 0,
total: 2
}

function runWithReports(projects: string[], counts = passingCounts) {
function runWithReports(
projects: string[],
countsByProject: Record<string, TestCounts> = {}
) {
const root = mkdtempSync(join(tmpdir(), 'playwright-pr-report-'))
const bin = join(root, 'bin')
const wranglerLog = join(root, 'wrangler.log')
Expand All @@ -64,15 +77,19 @@ echo "https://abc.$project.pages.dev"
`#!/bin/bash
echo "$*" >> "${extractorLog}"
echo 'Could not list failed tests: diagnostic' >&2
echo '${counts}'
cat "$2/counts.json"
Comment thread
christian-byrne marked this conversation as resolved.
Resolved
`
)
chmodSync(join(bin, 'wrangler'), 0o755)
chmodSync(join(bin, 'tsx'), 0o755)
for (const project of projects)
mkdirSync(join(root, 'reports', `playwright-report-${project}`), {
recursive: true
})
for (const project of projects) {
const reportDir = join(root, 'reports', `playwright-report-${project}`)
mkdirSync(reportDir, { recursive: true })
writeFileSync(
join(reportDir, 'counts.json'),
JSON.stringify(countsByProject[project] ?? passingCounts)
)
}

const result = spawnSync('bash', [script, '1', 'feature', 'completed'], {
cwd: root,
Expand Down Expand Up @@ -115,10 +132,22 @@ describe('PR Playwright report section', () => {
})

it('links a failed test at its path in the repository and in the report', () => {
const { summary, extractorCalls } = runWithReports(
['chromium'],
'{"passed":0,"failed":1,"flaky":0,"skipped":0,"total":1,"failures":[{"name":"saves a workflow","file":"browser_tests/tests/save.spec.ts","reportUrl":"https://abc.chromium.pages.dev/#?testId=fail-1"}]}'
)
const { summary, extractorCalls } = runWithReports(['chromium'], {
chromium: {
passed: 0,
failed: 1,
flaky: 0,
skipped: 0,
total: 1,
failures: [
{
name: 'saves a workflow',
file: 'browser_tests/tests/save.spec.ts',
reportUrl: 'https://abc.chromium.pages.dev/#?testId=fail-1'
}
]
}
})

expect(extractorCalls).toContain('browser_tests')
expect(summary).toContain(
Expand All @@ -135,4 +164,49 @@ describe('PR Playwright report section', () => {
expect(summary).toMatch(/^## 🎭 Playwright: ❌ .*, 1 missing/)
expect(summary).toContain('- **cloud**: ❌ Report missing')
})

it('lists each named failed test with its report in a collapsed box', () => {
const { status, summary } = runWithReports(uploadedProjects, {
chromium: {
passed: 1,
failed: 1,
flaky: 0,
skipped: 0,
total: 2,
failures: [
{
name: 'saves a workflow',
file: 'browser_tests/tests/save.spec.ts',
reportUrl: 'https://abc.chromium.pages.dev/#?testId=fail-1'
}
]
}
})

expect(status).toBe(0)
expect(summary).toMatch(/^## 🎭 Playwright: ❌ .*, 1 failed/)
expect(summary).toContain(
`> ❌ 1 test failed. Expand the failed tests below, review their reports and fix the failures before merging.

<details>
<summary>Failed tests</summary>

- [saves a workflow](https://github.com/Comfy-Org/ComfyUI_frontend/blob/sha/browser_tests/tests/save.spec.ts): [View test report](https://abc.chromium.pages.dev/#?testId=fail-1)

</details>`
)
})

it('points to the browser reports when failed tests have no extracted names', () => {
const { status, summary } = runWithReports(uploadedProjects, {
chromium: { passed: 1, failed: 1, flaky: 0, skipped: 0, total: 2 }
})

expect(status).toBe(0)
expect(summary).toMatch(/^## 🎭 Playwright: ❌ .*, 1 failed/)
expect(summary).toContain(
'> ❌ 1 test failed. Open the browser reports below, review the failed tests and fix the failures before merging.'
)
expect(summary).not.toContain('Failed tests')
})
})
Loading
Loading