Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
74415ce
test(website): enforce router drift checks in CI
christian-byrne Oct 1, 2026
e3586c1
test(website): rely on shared Vitest cleanup
christian-byrne Oct 1, 2026
baeb945
ci(website): rely on runner CI environment
christian-byrne Oct 1, 2026
0c09930
test(website): parse the CI flag explicitly
christian-byrne Oct 1, 2026
f90412f
test(website): normalize source fetch failures
christian-byrne Oct 1, 2026
6338c20
test(website): retry published router sources
christian-byrne Oct 1, 2026
3b2c055
test(website): budget time for source retries
christian-byrne Oct 1, 2026
d45169b
test(website): remove unreachable source skips
christian-byrne Oct 1, 2026
4e516b3
test(website): cover deterministic provider drift
christian-byrne Oct 1, 2026
1e7c48d
test(website): accept readonly provider fixtures
christian-byrne Oct 1, 2026
c1cffc3
test(website): discriminate router drift failure
christian-byrne Oct 1, 2026
e8c12d6
test(website): cover explicit CI flag values
christian-byrne Oct 1, 2026
4051e8d
test(website): advance source retry timers
christian-byrne Oct 1, 2026
65f0b46
test(website): use the retry constant directly
christian-byrne Oct 1, 2026
70e718e
test(website): derive the source-test budget
christian-byrne Oct 1, 2026
6fb4b34
test(website): name the provider mismatch case precisely
christian-byrne Oct 1, 2026
9393a7c
test(website): satisfy parameterized async lint
christian-byrne Oct 1, 2026
9d96e1e
test(website): handle retry rejection before timers
christian-byrne Oct 2, 2026
348f312
ci(website): schedule router provider drift checks
christian-byrne Oct 2, 2026
29fe947
fix(ci): keep drift script within website package
christian-byrne Oct 2, 2026
833d709
fix(website): satisfy router drift lint checks
christian-byrne Oct 3, 2026
93441fa
test(website): close router drift review gaps
christian-byrne Oct 3, 2026
eb7bc54
chore(website): use kebab-case for drift scripts
christian-byrne Oct 3, 2026
f29e4cb
chore(ci): name router drift workflow consistently
christian-byrne Oct 3, 2026
457212e
docs(website): document router drift command
christian-byrne Oct 3, 2026
9428b4e
test(website): prove router source comparisons
christian-byrne Oct 3, 2026
3b1a86c
fix(ci): distinguish drift from workflow failures
christian-byrne Oct 3, 2026
d8e51dd
docs(website): describe router drift alert
christian-byrne Oct 3, 2026
4292ec2
fix(website): remove stale drift test constant
christian-byrne Oct 3, 2026
20043b5
[automated] Apply ESLint and Oxfmt fixes
actions-user Oct 3, 2026
0d64678
test(website): cover remaining router drift paths
christian-byrne Oct 3, 2026
a34f386
docs(website): clarify router drift issue behavior
christian-byrne Oct 3, 2026
e4bd988
test(website): pin Comfy coverage drift error
christian-byrne Oct 3, 2026
1cbab6f
test(website): isolate router drift cases
christian-byrne Oct 3, 2026
62865c2
test(website): match drift errors as text
christian-byrne Oct 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions .github/workflows/ci-router-provider-drift.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Published Router schemas and docs are external state, so this alarm runs on a
# schedule instead of gating unrelated pull requests.
name: 'Router Provider Drift'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): .github/workflows/README.md names workflows <prefix>-<descriptive-name>.yaml, with ci- for validation, as in ci-crdt-skew-alarm.yaml. ci-router-provider-drift.yaml would match.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in f29e4cb5b2: the workflow is now ci-router-provider-drift.yaml.


on:
schedule:
- cron: '37 13 * * *'
workflow_dispatch:

permissions:
contents: read
issues: write

env:
ISSUE_LABEL: 'area:testing'
ISSUE_MARKER: '<!-- router-provider-drift -->'
ISSUE_TITLE: 'Router provider coverage has drifted from published sources'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The title is the issue's only identity. If someone edits the open issue's title, the next failure opens a second issue, and a passing run leaves the edited one open. That breaks the PR's "creates or updates one tracking issue" claim. A hidden marker in the body, such as <!-- router-provider-drift -->, searched in addition to the label, would survive title edits.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 3b1a86c0f8: the workflow identifies its issue with a hidden body marker, so title edits do not create duplicates or prevent recovery closure.


concurrency:
group: ${{ github.workflow }}
cancel-in-progress: true

jobs:
drift-check:
if: github.repository == 'Comfy-Org/ComfyUI_frontend'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: ./.github/actions/setup-frontend
- name: Compare checked-in coverage with published sources
id: drift
run: pnpm --filter @comfyorg/website check:router-provider-drift
- name: File or update the drift issue
if: failure() && steps.drift.outcome == 'failure'
uses: actions/github-script@v9

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These two github-script steps repeat the tracking-issue lifecycle from multiplayer-pin-currency.yaml (create or comment on failure, then comment and close on success) with different strings. The pull-request filter and title identity issues above exist in both copies. A small local composite action that takes the title, label, failure body and recovery body would let both workflows share one implementation. That could be a follow-up rather than part of this PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that the shared issue lifecycle is a useful follow-up. I kept this PR scoped to the Router drift alarm; the marker and pull-request filtering are applied consistently to both steps here.

with:
github-token: ${{ github.token }}
script: |
const title = process.env.ISSUE_TITLE
const marker = process.env.ISSUE_MARKER
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`
const body = `${marker}\nThe scheduled [Router Provider Drift](${runUrl}) check failed. The checked-in Router provider coverage may no longer match the published documentation or schemas, or a published source may be unavailable. See the run for the exact failure. This issue is maintained by \`.github/workflows/ci-router-provider-drift.yaml\` and closes itself after a passing run.`
const openIssues = await github.paginate(github.rest.issues.listForRepo, {
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
labels: process.env.ISSUE_LABEL,
per_page: 100
})
const existing = openIssues.find((issue) =>
!issue.pull_request && (issue.body ?? '').includes(marker)
)
if (existing) {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body
})
} else {
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title,
body,
labels: [process.env.ISSUE_LABEL]
})
}
- name: Close the drift issue after recovery
if: success()
uses: actions/github-script@v9
with:
github-token: ${{ github.token }}
script: |
const marker = process.env.ISSUE_MARKER
const openIssues = await github.paginate(github.rest.issues.listForRepo, {
owner: context.repo.owner,
repo: context.repo.repo,
state: 'open',
labels: process.env.ISSUE_LABEL,
per_page: 100
})
const existing = openIssues.find((issue) =>
!issue.pull_request && (issue.body ?? '').includes(marker)
)
if (!existing) return
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body: 'Router provider coverage matches the published sources again. Closing.'
})
await github.rest.issues.update({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
state: 'closed'
})
1 change: 1 addition & 0 deletions apps/website/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -508,6 +508,7 @@ the hosted script once, and renders the documented embed container.
- `pnpm build:e2e` — indexable build to `dist/`, the one e2e and screenshots run against
- `pnpm typecheck` — `astro check`
- `pnpm test:unit` — Vitest unit tests
- `pnpm check:router-provider-drift` — compare Router coverage with published sources (requires network access)
- `pnpm test:e2e` — Playwright E2E tests (requires `pnpm build:e2e` first)
- `pnpm ashby:refresh-snapshot` — refresh the committed careers snapshot
- `pnpm cloud-nodes:refresh-snapshot` — refresh the committed cloud nodes snapshot
1 change: 1 addition & 0 deletions apps/website/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"typecheck": "astro check && tsx ../../scripts/vue-tsc-native.ts --noEmit -p tsconfig.json",
"test:unit": "vitest run",
"test:router-models": "tsx ./scripts/test-router-models.ts",
"check:router-provider-drift": "tsx ./scripts/router-provider-drift.ts",
"test:router-validation": "vitest run scripts/router-model-validation.test.ts src/config/workshop-input-validation.test.ts src/config/workshop-media-validation.test.ts src/config/workshop-media-metadata.test.ts",
"test:workflow-pages": "tsx ./scripts/test-workflow-pages.ts",
"test:workshop-upload": "tsx ./scripts/test-workshop-upload.ts",
Expand Down
10 changes: 10 additions & 0 deletions apps/website/scripts/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Website Scripts

## `router-provider-drift.ts`

Compares the checked-in Router provider coverage with the published schemas and
documentation. The command requires network access and is also run by the
scheduled Router Provider Drift workflow:

```sh
pnpm --filter @comfyorg/website check:router-provider-drift
```

## `generate-workshop-display.ts`

Imports the content team's `workshop-display.json`, keyed by catalogue model ID,
Expand Down
222 changes: 222 additions & 0 deletions apps/website/scripts/router-provider-drift.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
import { describe, expect, it, vi } from 'vitest'

import {
ROUTER_COMFY_ONLY_PREVIEW,
ROUTER_PROVIDER_COVERAGE,
ROUTER_SERVING_PROVIDERS
} from '../src/config/router-providers'
import {
alternateProviders,
checkRouterProviderDrift,
fetchText,
parseCoverageTable
} from './router-provider-drift'

describe('fetchText', () => {
it('recovers when a source fails once and then responds', async () => {
const fetchImpl = vi
.fn<typeof fetch>()
.mockRejectedValueOnce(new Error('offline'))
.mockResolvedValueOnce(new Response('providers'))
const sleep = vi.fn(async () => {})

await expect(
fetchText('https://example.com', { fetchImpl, sleep })
).resolves.toBe('providers')
expect(fetchImpl).toHaveBeenCalledTimes(2)
expect(sleep).toHaveBeenCalledOnce()
})

it('reports an unavailable source after every attempt', async () => {
const fetchImpl = vi
.fn<typeof fetch>()
.mockRejectedValue(new Error('offline'))

await expect(
fetchText('https://example.com', {
fetchImpl,
sleep: async () => {}
})
).rejects.toThrow('Could not fetch https://example.com')
expect(fetchImpl).toHaveBeenCalledTimes(3)
})
Comment on lines +30 to +42

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Removing if (!response.ok) from fetchText leaves all four tests passing. Both fetch tests use either a rejected promise or a 200 Response. One case where fetchImpl returns new Response('maintenance', { status: 503 }) and the test asserts Could not fetch https://example.com would pin that branch. Without it, a 503 or 404 body would be parsed as Markdown or JSON and fail as drift instead of as an unavailable source.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 93441faf4a: the tests now cover an HTTP 503 response and require it to surface as an unavailable source.


it('reports an HTTP error as an unavailable source', async () => {
const fetchImpl = vi
.fn<typeof fetch>()
.mockResolvedValue(new Response('maintenance', { status: 503 }))

await expect(
fetchText('https://example.com', {
fetchImpl,
sleep: async () => {}
})
).rejects.toThrow('Could not fetch https://example.com')
})
})

it('reads alternate providers from a published model schema', () => {
expect(
alternateProviders(
JSON.stringify({
'x-comfy-router-alt-providers': [
{ provider: 'fal' },
{ provider: 'runware' }
]
})
)
).toEqual(['fal', 'runware'])
})

it('reads the provider coverage table from Markdown', () => {
const markdown = `
## Provider coverage
| **Model / provider** | **Comfy (default)** | **fal** |
| --- | --- | --- |
| [Model](/models/model/code) | ✓ | ✓ |
`

expect(parseCoverageTable(markdown)).toEqual({
providers: ['fal'],
rows: [
{
name: 'Model',
docsUrl: 'https://docs.comfy.org/models/model/code',
comfy: '✓'
}
]
})
})

describe('checkRouterProviderDrift', () => {
const coverageRows = ROUTER_PROVIDER_COVERAGE.map(
({ docsName, docsUrl, name }) =>
`| [${docsName ?? name}](${new URL(docsUrl).pathname}) | ✓ |`
).join('\n')
Comment on lines +92 to +95

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): The matching fixture rebuilds the published Markdown and schemas from the same three arrays that serve as the expected values. That's fine for the "everything matches" case, but it means a reader has to decode a small Markdown generator to see what the test controls. docsUrl.slice(DOCS_ORIGIN.length) also repeats the docs-origin knowledge that the script holds. new URL(docsUrl).pathname gives the path without copying the origin, here and in the script's two catalog checks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9428b4e21e: fixture and production path handling now use new URL(docsUrl).pathname.

const coverage = `
## Provider coverage
| **Model / provider** | **Comfy (default)** | ${ROUTER_SERVING_PROVIDERS.map(({ name }) => `**${name}**`).join(' | ')} |
| --- | --- | --- |
${coverageRows}
`
const catalog = ROUTER_COMFY_ONLY_PREVIEW.map(
({ docsUrl, name }) => `[${name}](${new URL(docsUrl).pathname})`
).join('\n')
const [driftedRow] = ROUTER_PROVIDER_COVERAGE
const [driftedProvider] = ROUTER_SERVING_PROVIDERS
const [previewRow] = ROUTER_COMFY_ONLY_PREVIEW

function createFetch({
coverageMarkdown = coverage,
catalogMarkdown = catalog,
providerOverrides = {}
}: {
coverageMarkdown?: string
catalogMarkdown?: string
providerOverrides?: Readonly<Record<string, readonly string[]>>
} = {}) {
return vi.fn<typeof fetch>(async (input) => {
const url = input instanceof Request ? input.url : String(input)
if (url.endsWith('/providers.md')) {
return new Response(coverageMarkdown)
}
if (url.endsWith('/models.md')) return new Response(catalogMarkdown)
const row = ROUTER_PROVIDER_COVERAGE.find(({ modelId }) =>
url.endsWith(`/${modelId}.json`)
)
const providers = row
? (providerOverrides[row.modelId] ?? row.providers)
: []
return new Response(
JSON.stringify({
'x-comfy-router-alt-providers': providers.map((provider) => ({
provider
}))
})
)
})
}

it('accepts matching published sources', async () => {
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch(),
sleep: async () => {}
})
).resolves.toBeUndefined()
})

it('detects serving-provider header drift', async () => {
const coverageMarkdown = coverage.replace(
`**${driftedProvider.name}**`,
`**${driftedProvider.name}-drift**`
)
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({ coverageMarkdown }),
sleep: async () => {}
})
).rejects.toThrow(`${driftedProvider.name}-drift`)
})

it('detects alternate-provider drift', async () => {
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({
providerOverrides: {
[driftedRow.modelId]: [...driftedRow.providers, 'drift-provider']
}
}),
sleep: async () => {}
})
).rejects.toThrow(/drift-provider/)
})

it('detects provider documentation row drift', async () => {
const coverageMarkdown = coverage.replace(
new URL(driftedRow.docsUrl).pathname,
`${new URL(driftedRow.docsUrl).pathname}-drift`
)
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({ coverageMarkdown }),
sleep: async () => {}
})
).rejects.toThrow(/-drift/)
})

it('detects a provider row not served by Comfy', async () => {
const coverageMarkdown = coverage.replace(
`| [${driftedRow.docsName ?? driftedRow.name}](${new URL(driftedRow.docsUrl).pathname}) | ✓ |`,
`| [${driftedRow.docsName ?? driftedRow.name}](${new URL(driftedRow.docsUrl).pathname}) | - |`
)
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({ coverageMarkdown }),
sleep: async () => {}
})
).rejects.toThrow('every provider coverage row must be served by Comfy')
})

it('detects a missing catalog link', async () => {
const catalogMarkdown = catalog.replace(`[${previewRow.name}]`, '[Drift]')
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({ catalogMarkdown }),
sleep: async () => {}
})
).rejects.toThrow(previewRow.name)
})

it('detects a preview path in provider coverage', async () => {
const previewPath = new URL(previewRow.docsUrl).pathname
await expect(
checkRouterProviderDrift({
fetchImpl: createFetch({
coverageMarkdown: `${previewPath}\n${coverage}`
}),
sleep: async () => {}
})
).rejects.toThrow(previewPath)
})
})
Loading
Loading