Repository navigation
test(website): schedule router provider drift checks #19713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
74415ce
e3586c1
baeb945
0c09930
f90412f
6338c20
3b2c055
d45169b
4e516b3
1e7c48d
c1cffc3
e8c12d6
4051e8d
65f0b46
70e718e
6fb4b34
9393a7c
9d96e1e
348f312
29fe947
833d709
93441fa
eb7bc54
f29e4cb
457212e
9428b4e
3b1a86c
d8e51dd
4292ec2
20043b5
0d64678
a34f386
e4bd988
1cbab6f
62865c2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Published Router schemas and docs are external state, so this alarm runs on a | ||
| # schedule instead of gating unrelated pull requests. | ||
| name: 'Router Provider Drift' | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '37 13 * * *' | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| issues: write | ||
|
|
||
| env: | ||
| ISSUE_LABEL: 'area:testing' | ||
| ISSUE_MARKER: '<!-- router-provider-drift -->' | ||
| ISSUE_TITLE: 'Router provider coverage has drifted from published sources' | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: The title is the issue's only identity. If someone edits the open issue's title, the next failure opens a second issue, and a passing run leaves the edited one open. That breaks the PR's "creates or updates one tracking issue" claim. A hidden marker in the body, such as
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed in |
||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| drift-check: | ||
| if: github.repository == 'Comfy-Org/ComfyUI_frontend' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: ./.github/actions/setup-frontend | ||
| - name: Compare checked-in coverage with published sources | ||
| id: drift | ||
| run: pnpm --filter @comfyorg/website check:router-provider-drift | ||
| - name: File or update the drift issue | ||
| if: failure() && steps.drift.outcome == 'failure' | ||
| uses: actions/github-script@v9 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: These two
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that the shared issue lifecycle is a useful follow-up. I kept this PR scoped to the Router drift alarm; the marker and pull-request filtering are applied consistently to both steps here. |
||
| with: | ||
| github-token: ${{ github.token }} | ||
| script: | | ||
| const title = process.env.ISSUE_TITLE | ||
| const marker = process.env.ISSUE_MARKER | ||
| const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}` | ||
| const body = `${marker}\nThe scheduled [Router Provider Drift](${runUrl}) check failed. The checked-in Router provider coverage may no longer match the published documentation or schemas, or a published source may be unavailable. See the run for the exact failure. This issue is maintained by \`.github/workflows/ci-router-provider-drift.yaml\` and closes itself after a passing run.` | ||
| const openIssues = await github.paginate(github.rest.issues.listForRepo, { | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| state: 'open', | ||
| labels: process.env.ISSUE_LABEL, | ||
| per_page: 100 | ||
| }) | ||
| const existing = openIssues.find((issue) => | ||
| !issue.pull_request && (issue.body ?? '').includes(marker) | ||
| ) | ||
| if (existing) { | ||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: existing.number, | ||
| body | ||
| }) | ||
| } else { | ||
| await github.rest.issues.create({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| title, | ||
| body, | ||
| labels: [process.env.ISSUE_LABEL] | ||
| }) | ||
| } | ||
| - name: Close the drift issue after recovery | ||
| if: success() | ||
| uses: actions/github-script@v9 | ||
| with: | ||
| github-token: ${{ github.token }} | ||
| script: | | ||
| const marker = process.env.ISSUE_MARKER | ||
| const openIssues = await github.paginate(github.rest.issues.listForRepo, { | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| state: 'open', | ||
| labels: process.env.ISSUE_LABEL, | ||
| per_page: 100 | ||
| }) | ||
| const existing = openIssues.find((issue) => | ||
| !issue.pull_request && (issue.body ?? '').includes(marker) | ||
| ) | ||
| if (!existing) return | ||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: existing.number, | ||
| body: 'Router provider coverage matches the published sources again. Closing.' | ||
| }) | ||
| await github.rest.issues.update({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: existing.number, | ||
| state: 'closed' | ||
| }) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,222 @@ | ||
| import { describe, expect, it, vi } from 'vitest' | ||
|
|
||
| import { | ||
| ROUTER_COMFY_ONLY_PREVIEW, | ||
| ROUTER_PROVIDER_COVERAGE, | ||
| ROUTER_SERVING_PROVIDERS | ||
| } from '../src/config/router-providers' | ||
| import { | ||
| alternateProviders, | ||
| checkRouterProviderDrift, | ||
| fetchText, | ||
| parseCoverageTable | ||
| } from './router-provider-drift' | ||
|
|
||
| describe('fetchText', () => { | ||
| it('recovers when a source fails once and then responds', async () => { | ||
| const fetchImpl = vi | ||
| .fn<typeof fetch>() | ||
| .mockRejectedValueOnce(new Error('offline')) | ||
| .mockResolvedValueOnce(new Response('providers')) | ||
| const sleep = vi.fn(async () => {}) | ||
|
|
||
| await expect( | ||
| fetchText('https://example.com', { fetchImpl, sleep }) | ||
| ).resolves.toBe('providers') | ||
| expect(fetchImpl).toHaveBeenCalledTimes(2) | ||
| expect(sleep).toHaveBeenCalledOnce() | ||
| }) | ||
|
|
||
| it('reports an unavailable source after every attempt', async () => { | ||
| const fetchImpl = vi | ||
| .fn<typeof fetch>() | ||
| .mockRejectedValue(new Error('offline')) | ||
|
|
||
| await expect( | ||
| fetchText('https://example.com', { | ||
| fetchImpl, | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow('Could not fetch https://example.com') | ||
| expect(fetchImpl).toHaveBeenCalledTimes(3) | ||
| }) | ||
|
Comment on lines
+30
to
+42
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3: Removing
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed in |
||
|
|
||
| it('reports an HTTP error as an unavailable source', async () => { | ||
| const fetchImpl = vi | ||
| .fn<typeof fetch>() | ||
| .mockResolvedValue(new Response('maintenance', { status: 503 })) | ||
|
|
||
| await expect( | ||
| fetchText('https://example.com', { | ||
| fetchImpl, | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow('Could not fetch https://example.com') | ||
| }) | ||
| }) | ||
|
|
||
| it('reads alternate providers from a published model schema', () => { | ||
| expect( | ||
| alternateProviders( | ||
| JSON.stringify({ | ||
| 'x-comfy-router-alt-providers': [ | ||
| { provider: 'fal' }, | ||
| { provider: 'runware' } | ||
| ] | ||
| }) | ||
| ) | ||
| ).toEqual(['fal', 'runware']) | ||
| }) | ||
|
|
||
| it('reads the provider coverage table from Markdown', () => { | ||
| const markdown = ` | ||
| ## Provider coverage | ||
| | **Model / provider** | **Comfy (default)** | **fal** | | ||
| | --- | --- | --- | | ||
| | [Model](/models/model/code) | ✓ | ✓ | | ||
| ` | ||
|
|
||
| expect(parseCoverageTable(markdown)).toEqual({ | ||
| providers: ['fal'], | ||
| rows: [ | ||
| { | ||
| name: 'Model', | ||
| docsUrl: 'https://docs.comfy.org/models/model/code', | ||
| comfy: '✓' | ||
| } | ||
| ] | ||
| }) | ||
| }) | ||
|
|
||
| describe('checkRouterProviderDrift', () => { | ||
| const coverageRows = ROUTER_PROVIDER_COVERAGE.map( | ||
| ({ docsName, docsUrl, name }) => | ||
| `| [${docsName ?? name}](${new URL(docsUrl).pathname}) | ✓ |` | ||
| ).join('\n') | ||
|
Comment on lines
+92
to
+95
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit (P3): The matching fixture rebuilds the published Markdown and schemas from the same three arrays that serve as the expected values. That's fine for the "everything matches" case, but it means a reader has to decode a small Markdown generator to see what the test controls.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed in |
||
| const coverage = ` | ||
| ## Provider coverage | ||
| | **Model / provider** | **Comfy (default)** | ${ROUTER_SERVING_PROVIDERS.map(({ name }) => `**${name}**`).join(' | ')} | | ||
| | --- | --- | --- | | ||
| ${coverageRows} | ||
| ` | ||
| const catalog = ROUTER_COMFY_ONLY_PREVIEW.map( | ||
| ({ docsUrl, name }) => `[${name}](${new URL(docsUrl).pathname})` | ||
| ).join('\n') | ||
| const [driftedRow] = ROUTER_PROVIDER_COVERAGE | ||
| const [driftedProvider] = ROUTER_SERVING_PROVIDERS | ||
| const [previewRow] = ROUTER_COMFY_ONLY_PREVIEW | ||
|
|
||
| function createFetch({ | ||
| coverageMarkdown = coverage, | ||
| catalogMarkdown = catalog, | ||
| providerOverrides = {} | ||
| }: { | ||
| coverageMarkdown?: string | ||
| catalogMarkdown?: string | ||
| providerOverrides?: Readonly<Record<string, readonly string[]>> | ||
| } = {}) { | ||
| return vi.fn<typeof fetch>(async (input) => { | ||
| const url = input instanceof Request ? input.url : String(input) | ||
| if (url.endsWith('/providers.md')) { | ||
| return new Response(coverageMarkdown) | ||
| } | ||
| if (url.endsWith('/models.md')) return new Response(catalogMarkdown) | ||
| const row = ROUTER_PROVIDER_COVERAGE.find(({ modelId }) => | ||
| url.endsWith(`/${modelId}.json`) | ||
| ) | ||
| const providers = row | ||
| ? (providerOverrides[row.modelId] ?? row.providers) | ||
| : [] | ||
| return new Response( | ||
| JSON.stringify({ | ||
| 'x-comfy-router-alt-providers': providers.map((provider) => ({ | ||
| provider | ||
| })) | ||
| }) | ||
| ) | ||
| }) | ||
| } | ||
|
|
||
| it('accepts matching published sources', async () => { | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch(), | ||
| sleep: async () => {} | ||
| }) | ||
| ).resolves.toBeUndefined() | ||
| }) | ||
|
|
||
| it('detects serving-provider header drift', async () => { | ||
| const coverageMarkdown = coverage.replace( | ||
| `**${driftedProvider.name}**`, | ||
| `**${driftedProvider.name}-drift**` | ||
| ) | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ coverageMarkdown }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow(`${driftedProvider.name}-drift`) | ||
| }) | ||
|
|
||
| it('detects alternate-provider drift', async () => { | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ | ||
| providerOverrides: { | ||
| [driftedRow.modelId]: [...driftedRow.providers, 'drift-provider'] | ||
| } | ||
| }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow(/drift-provider/) | ||
| }) | ||
|
|
||
| it('detects provider documentation row drift', async () => { | ||
| const coverageMarkdown = coverage.replace( | ||
| new URL(driftedRow.docsUrl).pathname, | ||
| `${new URL(driftedRow.docsUrl).pathname}-drift` | ||
| ) | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ coverageMarkdown }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow(/-drift/) | ||
| }) | ||
|
|
||
| it('detects a provider row not served by Comfy', async () => { | ||
| const coverageMarkdown = coverage.replace( | ||
| `| [${driftedRow.docsName ?? driftedRow.name}](${new URL(driftedRow.docsUrl).pathname}) | ✓ |`, | ||
| `| [${driftedRow.docsName ?? driftedRow.name}](${new URL(driftedRow.docsUrl).pathname}) | - |` | ||
| ) | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ coverageMarkdown }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow('every provider coverage row must be served by Comfy') | ||
| }) | ||
|
|
||
| it('detects a missing catalog link', async () => { | ||
| const catalogMarkdown = catalog.replace(`[${previewRow.name}]`, '[Drift]') | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ catalogMarkdown }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow(previewRow.name) | ||
| }) | ||
|
|
||
| it('detects a preview path in provider coverage', async () => { | ||
| const previewPath = new URL(previewRow.docsUrl).pathname | ||
| await expect( | ||
| checkRouterProviderDrift({ | ||
| fetchImpl: createFetch({ | ||
| coverageMarkdown: `${previewPath}\n${coverage}` | ||
| }), | ||
| sleep: async () => {} | ||
| }) | ||
| ).rejects.toThrow(previewPath) | ||
| }) | ||
| }) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit (P3):
.github/workflows/README.mdnames workflows<prefix>-<descriptive-name>.yaml, withci-for validation, as inci-crdt-skew-alarm.yaml.ci-router-provider-drift.yamlwould match.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in
f29e4cb5b2: the workflow is nowci-router-provider-drift.yaml.