Skip to content

[backport core/1.59] feat(telemetry): record requests the shared status retry retried and how they ended (#22204) - #22323

Merged
comfy-pr-bot merged 2 commits into
core/1.59from
backport-22204-to-core-1.59
Oct 30, 2026
Merged

comfy-pr-bot merged 2 commits into
core/1.59from
backport-22204-to-core-1.59

Conversation

@christian-byrne

Copy link
Copy Markdown
Contributor

Manual backport of #22204 (8cee1d7a92) to core/1.59. The auto-backport failed on a conflict in src/scripts/api.ts.

What

Each request that the shared 429/502/503/504 retry touched records one app:fetch_status_retry Datadog action and one Sentry breadcrumb: route group, attempts, last status and outcome (recovered, exhausted, Retry-After too long, timed out, failed). Capped at 20 per route and outcome per minute. Route groups and hosts are allowlisted, so nothing user-chosen is recorded. Shows whether retries are hiding a real backend problem.

Conflict resolution

  • api.ts: import-only conflict. Kept this branch's NodeExecutionId import path and added the getFetchRouteTemplate import; the moved route-group helper applied cleanly.
  • This branch has no @comfyorg/telemetry package, so the new and changed src/utils files import @/platform/telemetry and @/platform/telemetry/types. The telemetry type and provider changes landed in src/platform/telemetry/ by rename detection.

Verification

  • src/utils, src/scripts and the cloud telemetry provider tests pass locally.
  • pnpm typecheck:app, oxfmt and oxlint pass.

Linear: https://linear.app/comfyorg/issue/FRONT-3683

…etried and how they ended (#22204)

nothing records that a retry happened, so a backend that fails often but
recovers on retry looks healthy. This records one
`app:fetch_status_retry` event per request that hit a retryable status:
a Datadog RUM action and a Sentry breadcrumb with transport, method,
host, route group, attempts, last status and outcome (`recovered`,
`exhausted`, `retry_after_too_long`, `timeout_budget`, `failed`). Retry
behaviour is unchanged.

Design calls worth a look:
- One pure classifier (`classifyStatusRetry`) decides the outcome from
how each transport's loop ended. A retry that ends on a 500/404 or with
no response is `failed`, not `recovered`. The request timeout
(`TimeoutError` abort on fetch, `ECONNABORTED`/`ETIMEDOUT` on axios) is
told apart from a caller cancel, which records nothing.
- Routes reuse the allowlisted route groups of `app:fetch_timeout`
(`getFetchRouteTemplate`, moved from `api.ts` to
`src/utils/fetchRouteTemplate.ts`, now also `billing`, `customers`,
`workspaces`), so user-chosen names never leave the client and the two
events join on `route`. Side effect: `app:fetch_timeout` for those three
groups now reports the group instead of `/other`.
- At most 20 events per host/route/outcome per minute, so an outage
cannot flood RUM while a long-lived tab keeps reporting. Hosts outside
`comfy.org` and the page's own host report as `other`.
- Recording can never throw into a request.
- `retryDelayMs` is replaced by `nextRetry`, which also says why it
stopped.

Linear: https://linear.app/comfyorg/issue/FRONT-3683

- New cases red against the first cut of this change, green now:
per-outcome tables for both transports, timeout vs cancel, route
redaction, the per-key cap, telemetry throwing, opt-out reads; plus a
classifier table and provider tests (Datadog action, Sentry breadcrumb
level).
- 2,200+ tests in `src/utils`, `src/scripts`, auth, workspace API,
settings and telemetry providers pass. `pnpm typecheck`, knip, oxlint,
oxfmt and the pre-push fallow audit pass.
- An independent adversarial review (Skeptic, Architect, Minimalist;
fresh Claude reviewers, since the cross-model CLI is unavailable on this
host) ran on the first cut; its findings (mislabelled `recovered`,
timeout classification, duplicated mapping, weak route scrubbing,
unbounded volume, telemetry inside the request `try`) are fixed here.
Panel, CodeRabbit and review findings on 0531273 are fixed in 37b5a87
and the per-minute cap commit.
- After release: in Datadog RUM, `@action.name:app:fetch_status_retry`
grouped by `@context.route` and `@context.outcome`.
@christian-byrne christian-byrne added the backport Backporting a PR onto a release candidate label Oct 30, 2026
@christian-byrne
christian-byrne requested a review from a team as a code owner October 30, 2026 00:27
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords: 2
  • [release]
  • [backport

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle Size

⏳ Size data collection in progress…

⚡ Performance

⏳ Performance tests in progress…

🔬 E2E Coverage

⚠️ Coverage collection failed. Check the CI workflow logs.

🎭 Playwright: ✅ 3123 passed, 0 failed · 3 flaky

📊 Browser Reports
  • chromium: View Report (✅ 2323 / ❌ 0 / ⚠️ 2 / ⏭️ 0)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • cloud: View Report (✅ 769 / ❌ 0 / ⚠️ 1 / ⏭️ 0)
  • desktop: View Report (✅ 6 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 19 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-safari: View Report (✅ 3 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 10/30/2026, 12:39:25 AM UTC

Links

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

No verdict is available. Open the workflow run to confirm the check was intentionally skipped, or fix the collection failure and re-run it.

Findings and check details

The matrix runs on same-repository version-bump PRs with relevant changes, main-branch pushes, and manual dispatches. This candidate does not qualify, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

Findings and check details

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since c0a3ce9490451caf8ce39b132cf75b4bcdbf15cc

Full docs: ADR-DEVEX-LINT-0015

🏛️ Architecture: ✅ Passed

Findings and check details

Warnings (222)

  • workspace path matches no source files: src/platform/workspace/fixtures/**. Remove or update it in docs/architecture/domains/records/workspace.domain.json.
  • src/components/graph/selectionToolbox/LinkSelectionActions.vue is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/searchbox/v2/filterChip.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/Popover.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/card/Card.stories.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/card/Card.vue is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/card/CardFooter.vue is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/card/CardHeader.vue is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/ui/card/card.variants.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/widgetHistory/WidgetHistoryButtons.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/components/widgetHistory/WidgetHistoryButtons.vue is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/graph/menuOption.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/useLiveGraphNode.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/useLiveGraphNode.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/useWidgetValueHistory.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/useWidgetValueHistory.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/widgetHistoryScope.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/composables/widgetHistoryScope.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/graph/litegraphServiceAccess.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/graph/nodeDef/ComfyNodeDefImpl.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/graph/widgets/dynamicGroupDiagnostics.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/graph/widgets/dynamicGroupDiagnostics.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/graph/widgets/dynamicGroupEnforcementBoundary.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/legacyUI/attachLegacyUI.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/core/selection/selectionIntent.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/agentPanel.historyIdentity.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/cloudBadges.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/dynamicPrompts.serializeValue.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/electronAdapter.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/index.cloud.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/index.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/cameraInfo.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/findNumericWidget.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dHost.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dHost.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dInstances.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dLazy.retry.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dSettings.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dWidgets.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/load3dWidgets.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/reportLoadModelOutcome.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/load3d/reportLoadModelOutcome.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/slotDefaults.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/touchCanvasAccess.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/touchCanvasAccess.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/extensions/core/touchPointerGuards.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/i18n.literalMessages.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/lib/litegraph/src/ContextMenu.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/lib/litegraph/src/LGraphCanvas.startRendering.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • src/lib/litegraph/src/LiteGraphGlobal.cloneObject.test.ts is not claimed by any domain. Add it to a module in docs/architecture/domains/records.
  • …and 172 more
Reproduce locally
pnpm architecture:check

Full docs: Frontend domain catalog

@github-actions
github-actions Bot requested a review from thedatalife October 30, 2026 00:28
@datadog-official

This comment has been minimized.

…ispatcher type for fallow

useTelemetry() returns the registry typed as TelemetryDispatcher, so fallow
cannot see the call in statusRetryTelemetry.ts. Same suppression the other
dispatcher-only members in this file carry.
@github-actions github-actions Bot added the risk:xhigh PR risk grade (advisory shadow check; grader-owned) label Oct 30, 2026

@dante01yoon dante01yoon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6ad65b1 against merged #22204 and the complete parent discussions. The import conflict preserves this branch's NodeExecutionId path, and retry logic matches the merged implementation after the telemetry package-path adaptation. Outcome classification, caller cancellation, route/host redaction, per-minute budget and both provider sinks retain their intended behavior. The added metadata is client telemetry, not a server API contract. Own isolated frozen-install validation: 176 tests across six suites, app typecheck and scoped lint/format PASS. Independently accepting redirects as recovery, removing host redaction, disabling the minute reset or allowing response-body cleanup failure to escape each fails its retained regression; restored source passes. No remaining code blocker.

@dante01yoon dante01yoon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 2791edb: the sole change is a one-line Fallow machine directive on the dispatcher implementation. I ran the owning audit without it: the only introduced finding is trackFetchStatusRetry as an unused class member, with incomplete-import-graph caveat. A private test using the real TelemetryRegistry, setTelemetryRegistry and trackStatusRetry reaches the registered provider; 64 registry/classifier/probe tests PASS. Restoring the directive makes the new-only audit PASS with zero introduced findings; app typecheck and scoped Oxlint/Oxfmt also PASS. The actual runtime caller and authoritative TelemetryDispatcher type confirm this is a verified analyzer limitation, not dead code. The repository explicitly permits scoped tool directives; behavior and previously reviewed retry coverage are unchanged. One nonblocking documentation note is inline. CI completion is not claimed.

this.dispatch((provider) => provider.trackFetchTimeout?.(metadata))
}

// fallow-ignore-next-line unused-class-member

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit-pick (non-blocking): Please copy the external-tool constraint from commit 2791edb into the PR description: Fallow cannot credit this concrete method through useTelemetry()’s TelemetryDispatcher return type, while the real registry/provider path is reachable. AGENTS.md asks that a justified rare suppression’s constraint be recorded there. Keep the machine directive itself free of prose as it is now.

@codecov

codecov Bot commented Oct 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.70968% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/platform/telemetry/TelemetryRegistry.ts 0.00% 2 Missing ⚠️
@@              Coverage Diff              @@
##           core/1.59   #22323      +/-   ##
=============================================
- Coverage      87.11%   86.98%   -0.14%     
=============================================
  Files           3027     3030       +3     
  Lines         158494   158949     +455     
  Branches       42267    42444     +177     
=============================================
+ Hits          138078   138261     +183     
- Misses         19740    20012     +272     
  Partials         676      676              
Flag Coverage Δ
unit 86.42% <98.70%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...try/providers/cloud/DatadogRumTelemetryProvider.ts 94.80% <100.00%> (+0.13%) ⬆️
...lemetry/providers/cloud/SentryTelemetryProvider.ts 100.00% <100.00%> (ø)
src/platform/telemetry/types.ts 100.00% <ø> (ø)
src/scripts/api.ts 82.16% <ø> (-0.19%) ⬇️
src/utils/axiosRetryableStatus.ts 97.19% <100.00%> (+4.33%) ⬆️
src/utils/fetchRouteTemplate.ts 100.00% <100.00%> (ø)
src/utils/fetchWithRetry.ts 95.55% <100.00%> (+1.80%) ⬆️
src/utils/retryPolicy.ts 97.95% <100.00%> (+2.04%) ⬆️
src/utils/statusRetryTelemetry.ts 100.00% <100.00%> (ø)
src/platform/telemetry/TelemetryRegistry.ts 55.70% <0.00%> (-0.50%) ⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@comfy-pr-bot
comfy-pr-bot merged commit 117b1d2 into core/1.59 Oct 30, 2026
106 checks passed
@comfy-pr-bot
comfy-pr-bot deleted the backport-22204-to-core-1.59 branch October 30, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Backporting a PR onto a release candidate risk:xhigh PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants