From f18e2fa3221afc97d142ada9ae6fa0a66a6672a7 Mon Sep 17 00:00:00 2001 From: bymyself Date: Thu, 3 Sep 2026 16:07:10 +0000 Subject: [PATCH 1/9] docs: add agent telemetry taxonomy ADR --- docs/adr/README.md | 1 + ...S-0030-agent-consumable-error-telemetry.md | 105 ++++++++++++++++++ src/AGENTS.md | 3 + 3 files changed, 109 insertions(+) create mode 100644 docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md diff --git a/docs/adr/README.md b/docs/adr/README.md index 1448f3230b5..89b706f6787 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -47,6 +47,7 @@ An Architecture Decision Record captures an important architectural decision mad | [SUBGRAPH-PROMOTION-0009](SUBGRAPH-PROMOTION-0009-represent-promoted-widgets-as-linked-inputs.md) | Represent Promoted Widgets as Linked Inputs | Proposed | 2026-05-05 | | [SUBGRAPH-PROMOTION-0027](SUBGRAPH-PROMOTION-0027-defer-promoted-widget-registration-to-onadded.md) | Defer Promoted-Widget Registration to `onAdded()` | Accepted | 2026-09-01 | | [TELEMETRY-DIAGNOSTICS-0019](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md) | Recoverable Event Diagnostics | Proposed | 2026-08-25 | +| [TELEMETRY-ERRORS-0030](TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) | Agent-Consumable Error Telemetry | Accepted | 2026-09-03 | | [TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) | Telemetry Routing Across Consumers | Accepted | 2026-07-28 | | [TELEMETRY-STARTUP-0030](TELEMETRY-STARTUP-0030-production-startup-telemetry.md) | Production Startup Telemetry | Proposed | 2026-09-08 | | [TESTING-BILLING-0031](TESTING-BILLING-0031-real-cloud-checkout-tests.md) | Real Cloud Checkout Tests | Proposed | 2026-09-11 | diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md new file mode 100644 index 00000000000..a4e09816ad6 --- /dev/null +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -0,0 +1,105 @@ +# ADR-TELEMETRY-ERRORS-0030: Agent-Consumable Error Telemetry + +Date: 2026-09-03 + +## Status + +Accepted + +## Context + +Real-user state exposes failures that tests and pre-deploy checks cannot see: +invariants, bad user states, expected-but-missing events, unexpected catches, +and degraded paths. These signals need intentional common tags so agents can +query them by family and Sentry can route useful alerts without creating one +rule per error slug. + +The frontend already provides `reportError` and mandates it in +`src/AGENTS.md`; 33 snake_case `errorType` slugs exist across 29 files. +`src/base/assert.ts` is used much less widely, while hundreds of catch blocks +remain. The gap is coverage and discipline, not another reporting API. + +## Decision + +1. **Keep `reportError` and `assert` as the only emitters.** Do not add another + abstraction. Widen coverage by converting swallowed or unexpected catches + to `reportError`, adding soft assertions at invariant boundaries, and + emitting `missing_event` observations where expected completion has a + bounded window. +2. **Use a fixed snake_case tag taxonomy:** + - `error_type`: existing stable slug shaped as `__` + - `failure_kind`: `invariant`, `bad_state`, `missing_event`, + `caught_unexpected`, or `degraded` + - `feature_area`: `workflow`, `queue`, `canvas`, `nodes`, `auth`, `cloud`, + `agent`, `crdt`, `billing`, `extensions`, `settings`, or `assets` + - `operation`: `load`, `save`, `execute`, `sync`, `import`, `export`, + `render`, `navigate`, or `auth` + - `outcome`: `failed`, `recovered`, `aborted`, `timed_out`, or `missing` + - `assert_mode`: `soft`, `hard`, or `sampled` for assertions + + New enum values require an entry under Registry Amendments. Dynamic values + such as ids, names, and paths are forbidden in tags; permitted diagnostic + values belong in `context`. + +3. **Choose levels by impact.** Use `error` for `invariant`, `bad_state`, + `caught_unexpected`, and `outcome:failed`. Use `warning` for `degraded`, + `outcome:recovered`, and `missing_event` unless the user lost work. +4. **Retain default grouping.** The default fingerprint plus the stable slug + subdivides issues without merging distinct causes. Never use a bare + `[errorType]` fingerprint. Configure a fingerprint at the emitter when one + is needed, not globally in `beforeSend`, so it remains reviewable. +5. **Use soft assertions by default in the browser.** Hard assertions are only + for cases where continuing corrupts document or CRDT state, authentication, + or billing. Sampled assertions are only for render and pointer hot paths and + never for data-loss invariants. Assertion messages are static strings. +6. **Classify every touched catch.** A catch is expected-handled, + degraded-recovered, unexpected, or rethrown. Only degraded-recovered and + unexpected catches report. Never report and rethrow the same error. +7. **Do not include PII.** Prompts, workflow names or paths, file names, node + titles, emails, and tokens are forbidden in tags and context. Limit context + to ids, counts, enums, and booleans. +8. **Route alerts by family.** Use one Sentry rule per family rather than per + slug: new invariant, hard assertion, broken user operation, unexpected + catch, missing completion, failure-rate deviation, dynamic anomaly by + feature area, and a recovered-degradation digest. +9. **Add denominators in a later phase.** Paired success counters should use + Sentry Application Metrics with the same attribute names as these tags so + deviation alerts have a denominator. +10. **Keep this decision frontend-scoped.** Backend telemetry transports need + their own amendment before adopting this taxonomy. + +## Consequences + +- Sentry issues become queryable by agents using stable tag conjunctions such + as `failure_kind:invariant feature_area:workflow release:X`. +- Event volume rises, so alert families and existing filtering must keep noisy + deployment and chunk-load failures out of these signals. +- `feature_area` becomes shared vocabulary for error tags, future metric + attributes, and operational follow-up. +- Instrumentation changes stay small and reviewable: one feature area or catch + cluster per pull request, with no behavior change to recovery paths. + +## Alternatives Considered + +- A new `reportInvariant()` or `Telemetry` wrapper was rejected because it + duplicates `reportError`, which already fans out to every configured sink. +- Kebab-case tags were rejected because existing slugs use snake_case. +- One Slack alert per `error_type` was rejected because it creates an + unbounded rule count and alert fatigue. +- A bare `[errorType]` fingerprint was rejected because it merges distinct root + causes. +- Global sampling was rejected because it can discard the rare invariants this + decision is intended to expose. +- Datadog-first instrumentation was rejected because Sentry already receives + frontend errors and supports the intended query and alert path. + +## Registry Amendments + +Append `YYYY-MM-DD tag=value — reason — PR` lines here. + +## References + +- [`reportError`](../../src/platform/telemetry/reportError.ts) +- [`assert`](../../src/base/assert.ts) +- [Source decision and research](https://github.com/christian-byrne/in-app-agent-program/blob/main/decisions/ADR-029-agent-telemetry-tag-taxonomy.md) +- [ADR-TELEMETRY-ROUTING-0013: Telemetry Routing Across Consumers](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) diff --git a/src/AGENTS.md b/src/AGENTS.md index ca6c38d8b8d..55926d3e2dd 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -40,6 +40,9 @@ a migration of affected queries and alerts that accounts for old and new releases reporting different names. + Follow [ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) for + agent-consumable tags, assertion modes, catch classification, and PII rules. + ## Security - Sanitize HTML with DOMPurify From 11f822ddd70ee1c696baa5e210f1d1809ff0d12f Mon Sep 17 00:00:00 2001 From: Christian Byrne Date: Fri, 4 Sep 2026 12:30:02 -0700 Subject: [PATCH 2/9] docs(adr): mark the telemetry taxonomy Proposed until its emitter APIs land Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#pullrequestreview-5107191660 Verified the report: assert() is (condition, message) with no mode parameter, and ReportErrorOptions is { errorType, tags, context, level } with no fingerprint field, so rule 5's soft/hard/sampled selection and rule 4's emitter-level fingerprint are both inexpressible. The only fingerprint seam that exists is datadogRumBeforeSend, which is the global beforeSend path rule 4 explicitly rules out. Takes the first of the two remedies offered in review: Status drops to Proposed with the blocking APIs named, the index row follows, and src/AGENTS.md now points at it as direction rather than mandating it, so nothing in the accepted policy binds before the emitters support it. --- docs/adr/README.md | 2 +- ...TRY-ERRORS-0030-agent-consumable-error-telemetry.md | 10 +++++++++- src/AGENTS.md | 7 +++++-- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/adr/README.md b/docs/adr/README.md index 89b706f6787..cfe386175e3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -47,7 +47,7 @@ An Architecture Decision Record captures an important architectural decision mad | [SUBGRAPH-PROMOTION-0009](SUBGRAPH-PROMOTION-0009-represent-promoted-widgets-as-linked-inputs.md) | Represent Promoted Widgets as Linked Inputs | Proposed | 2026-05-05 | | [SUBGRAPH-PROMOTION-0027](SUBGRAPH-PROMOTION-0027-defer-promoted-widget-registration-to-onadded.md) | Defer Promoted-Widget Registration to `onAdded()` | Accepted | 2026-09-01 | | [TELEMETRY-DIAGNOSTICS-0019](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md) | Recoverable Event Diagnostics | Proposed | 2026-08-25 | -| [TELEMETRY-ERRORS-0030](TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) | Agent-Consumable Error Telemetry | Accepted | 2026-09-03 | +| [TELEMETRY-ERRORS-0030](TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) | Agent-Consumable Error Telemetry | Proposed | 2026-09-03 | | [TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) | Telemetry Routing Across Consumers | Accepted | 2026-07-28 | | [TELEMETRY-STARTUP-0030](TELEMETRY-STARTUP-0030-production-startup-telemetry.md) | Production Startup Telemetry | Proposed | 2026-09-08 | | [TESTING-BILLING-0031](TESTING-BILLING-0031-real-cloud-checkout-tests.md) | Real Cloud Checkout Tests | Proposed | 2026-09-11 | diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index a4e09816ad6..7edd45babd8 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -4,7 +4,15 @@ Date: 2026-09-03 ## Status -Accepted +Proposed + +Two rules below are not expressible with the APIs as they stand: rule 5 +selects soft, hard, and sampled assertion modes, but `assert()` takes only a +condition and a message; rule 4 configures a fingerprint at the emitter, but +`ReportErrorOptions` has no fingerprint field and the only fingerprint seam +today is the global `beforeSend` that rule 4 rules out. This stays Proposed +until those two emitter APIs land, so it reads as direction rather than an +immediately-binding gate. ## Context diff --git a/src/AGENTS.md b/src/AGENTS.md index 55926d3e2dd..a5261328f63 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -40,8 +40,11 @@ a migration of affected queries and alerts that accounts for old and new releases reporting different names. - Follow [ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) for - agent-consumable tags, assertion modes, catch classification, and PII rules. + [ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) + proposes agent-consumable tags, assertion modes, catch classification, and + PII rules. It is Proposed, not Accepted, until `assert()` takes an assertion + mode and `ReportErrorOptions` takes a fingerprint, so treat it as direction + rather than a gate. ## Security From f66f41b7ca756ff4eb7ed1ccb5d2440098d3b1b8 Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 20 Sep 2026 03:27:21 +0000 Subject: [PATCH 3/9] docs(adr): align error type naming contract Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r3974382605 --- .../TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index 7edd45babd8..6f418e6e7c5 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -35,7 +35,9 @@ remain. The gap is coverage and discipline, not another reporting API. emitting `missing_event` observations where expected completion has a bounded window. 2. **Use a fixed snake_case tag taxonomy:** - - `error_type`: existing stable slug shaped as `__` + - `error_type`: existing stable slug shaped as + `__[_detail]`, with the operation written as + a present participle - `failure_kind`: `invariant`, `bad_state`, `missing_event`, `caught_unexpected`, or `degraded` - `feature_area`: `workflow`, `queue`, `canvas`, `nodes`, `auth`, `cloud`, From 73492bdf53709b7025e5fe30749fb2edf8cd2bdf Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 20 Sep 2026 03:27:33 +0000 Subject: [PATCH 4/9] docs(adr): correct Sentry grouping semantics Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r3974382599 --- ...TRY-ERRORS-0030-agent-consumable-error-telemetry.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index 6f418e6e7c5..2b593a8365f 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -54,10 +54,12 @@ remain. The gap is coverage and discipline, not another reporting API. 3. **Choose levels by impact.** Use `error` for `invariant`, `bad_state`, `caught_unexpected`, and `outcome:failed`. Use `warning` for `degraded`, `outcome:recovered`, and `missing_event` unless the user lost work. -4. **Retain default grouping.** The default fingerprint plus the stable slug - subdivides issues without merging distinct causes. Never use a bare - `[errorType]` fingerprint. Configure a fingerprint at the emitter when one - is needed, not globally in `beforeSend`, so it remains reviewable. +4. **Retain default grouping unless a slug must subdivide an issue.** Sentry's + default grouping ignores tags, including `error_type`. When separate issues + are needed per slug, configure an emitter fingerprint containing both + `{{ default }}` and `errorType`. Never use a bare `[errorType]` fingerprint, + and do not set fingerprints globally in `beforeSend`, so each override stays + reviewable at its call site. 5. **Use soft assertions by default in the browser.** Hard assertions are only for cases where continuing corrupts document or CRDT state, authentication, or billing. Sampled assertions are only for render and pointer hot paths and From 5ee65a913ff8302023bed4c6737d70d5fd69520a Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 20 Sep 2026 03:27:48 +0000 Subject: [PATCH 5/9] docs(adr): align operational alerts with Datadog Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r3974382598 --- ...S-0030-agent-consumable-error-telemetry.md | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index 2b593a8365f..9985b7c747a 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -19,8 +19,8 @@ immediately-binding gate. Real-user state exposes failures that tests and pre-deploy checks cannot see: invariants, bad user states, expected-but-missing events, unexpected catches, and degraded paths. These signals need intentional common tags so agents can -query them by family and Sentry can route useful alerts without creating one -rule per error slug. +query them by family and Datadog RUM can route useful alerts without creating +one monitor per error slug. The frontend already provides `reportError` and mandates it in `src/AGENTS.md`; 33 snake_case `errorType` slugs exist across 29 files. @@ -70,13 +70,16 @@ remain. The gap is coverage and discipline, not another reporting API. 7. **Do not include PII.** Prompts, workflow names or paths, file names, node titles, emails, and tokens are forbidden in tags and context. Limit context to ids, counts, enums, and booleans. -8. **Route alerts by family.** Use one Sentry rule per family rather than per - slug: new invariant, hard assertion, broken user operation, unexpected - catch, missing completion, failure-rate deviation, dynamic anomaly by - feature area, and a recovered-degradation digest. -9. **Add denominators in a later phase.** Paired success counters should use - Sentry Application Metrics with the same attribute names as these tags so - deviation alerts have a denominator. +8. **Route operational alerts through Datadog by family.** Following + [ADR-TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md), + use one RUM monitor per family rather than per slug: new invariant, hard + assertion, broken user operation, unexpected catch, missing completion, + failure-rate deviation, dynamic anomaly by feature area, and a + recovered-degradation digest. Sentry remains an error exploration and + grouping sink, not the operational alert route. +9. **Add denominators in a later phase.** Paired Datadog RUM success actions + should use the same bounded attributes as these error tags so Datadog + calculated metrics and monitors can measure failure rates. 10. **Keep this decision frontend-scoped.** Backend telemetry transports need their own amendment before adopting this taxonomy. @@ -96,14 +99,15 @@ remain. The gap is coverage and discipline, not another reporting API. - A new `reportInvariant()` or `Telemetry` wrapper was rejected because it duplicates `reportError`, which already fans out to every configured sink. - Kebab-case tags were rejected because existing slugs use snake_case. -- One Slack alert per `error_type` was rejected because it creates an +- One Datadog monitor per `error_type` was rejected because it creates an unbounded rule count and alert fatigue. - A bare `[errorType]` fingerprint was rejected because it merges distinct root causes. - Global sampling was rejected because it can discard the rare invariants this decision is intended to expose. -- Datadog-first instrumentation was rejected because Sentry already receives - frontend errors and supports the intended query and alert path. +- Sentry-only alerting was rejected because accepted + [ADR-TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) + assigns low-latency incident response to Datadog RUM and monitors. ## Registry Amendments From dbcf6e7d06a228a1f5a728cc3bf06d5dabc97b9e Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 20 Sep 2026 03:46:32 +0000 Subject: [PATCH 6/9] docs(adr): define telemetry tag requiredness Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r4055973253 --- ...EMETRY-ERRORS-0030-agent-consumable-error-telemetry.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index 9985b7c747a..b2320d65060 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -47,6 +47,14 @@ remain. The gap is coverage and discipline, not another reporting API. - `outcome`: `failed`, `recovered`, `aborted`, `timed_out`, or `missing` - `assert_mode`: `soft`, `hard`, or `sampled` for assertions + A conforming event always supplies `failure_kind`, `feature_area`, and + `outcome`. It supplies `operation` when the failure occurs during one of the + enumerated operations; invariants and bad states with no discrete operation + omit it. Assertion events also supply `assert_mode`; non-assertion events + omit it. `ReportErrorOptions.tags` remains optional for existing callers, + but an event does not conform to this taxonomy unless its tags satisfy this + record shape. + New enum values require an entry under Registry Amendments. Dynamic values such as ids, names, and paths are forbidden in tags; permitted diagnostic values belong in `context`. From 9408f7543b5fe1f1087941ecdbe0dd66bbb3437a Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 20 Sep 2026 03:46:53 +0000 Subject: [PATCH 7/9] docs(adr): define telemetry level precedence Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r4055973257 --- .../TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index b2320d65060..c9ac058750a 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -61,7 +61,8 @@ remain. The gap is coverage and discipline, not another reporting API. 3. **Choose levels by impact.** Use `error` for `invariant`, `bad_state`, `caught_unexpected`, and `outcome:failed`. Use `warning` for `degraded`, - `outcome:recovered`, and `missing_event` unless the user lost work. + `outcome:recovered`, and `missing_event` unless the user lost work. When + multiple rules match, `error` takes precedence over `warning`. 4. **Retain default grouping unless a slug must subdivide an issue.** Sentry's default grouping ignores tags, including `error_type`. When separate issues are needed per slug, configure an emitter fingerprint containing both From 5c0940d16dbd4fc59cf449f7f95ca681992769d8 Mon Sep 17 00:00:00 2001 From: bymyself Date: Mon, 21 Sep 2026 01:22:41 -0700 Subject: [PATCH 8/9] docs(adr): reconcile telemetry taxonomy provenance Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r4056154013 Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r4056154015 Addresses https://github.com/Comfy-Org/ComfyUI_frontend/pull/16784#discussion_r4056154016 --- ...S-0030-agent-consumable-error-telemetry.md | 46 +++++++++++++++---- 1 file changed, 36 insertions(+), 10 deletions(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index c9ac058750a..e0f7d3b417c 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -22,10 +22,20 @@ and degraded paths. These signals need intentional common tags so agents can query them by family and Datadog RUM can route useful alerts without creating one monitor per error slug. -The frontend already provides `reportError` and mandates it in -`src/AGENTS.md`; 33 snake_case `errorType` slugs exist across 29 files. -`src/base/assert.ts` is used much less widely, while hundreds of catch blocks -remain. The gap is coverage and discipline, not another reporting API. +The frontend already provides `reportError`, mandates it in `src/AGENTS.md`, +and has production callers using snake_case `errorType` slugs. +`src/base/assert.ts` is used much less widely, while many catch blocks remain. +The gap is coverage and discipline, not another reporting API. Counts are +deliberately omitted because they change independently of this decision. + +This decision originated in a 2026-09-03 product directive to expose +invariants, bad states, missing bounded completions, unexpected catches, and +degraded paths through one queryable taxonomy. The supporting research checked +the existing frontend emitters and compared bounded indexed tags with diagnostic +context, default grouping with explicit fingerprints, family-level alerting with +per-slug alerts, and failure counts with rates that have denominators. Those +tradeoffs lead to the fixed low-cardinality tags and alternatives recorded +below; the decision does not depend on access to an external design document. ## Decision @@ -43,17 +53,21 @@ remain. The gap is coverage and discipline, not another reporting API. - `feature_area`: `workflow`, `queue`, `canvas`, `nodes`, `auth`, `cloud`, `agent`, `crdt`, `billing`, `extensions`, `settings`, or `assets` - `operation`: `load`, `save`, `execute`, `sync`, `import`, `export`, - `render`, `navigate`, or `auth` - - `outcome`: `failed`, `recovered`, `aborted`, `timed_out`, or `missing` + `render`, `navigate`, `auth`, or `insert` + - `outcome`: `failed`, `recovered`, `aborted`, `timed_out`, `missing`, or + `degraded` - `assert_mode`: `soft`, `hard`, or `sampled` for assertions A conforming event always supplies `failure_kind`, `feature_area`, and `outcome`. It supplies `operation` when the failure occurs during one of the enumerated operations; invariants and bad states with no discrete operation omit it. Assertion events also supply `assert_mode`; non-assertion events - omit it. `ReportErrorOptions.tags` remains optional for existing callers, - but an event does not conform to this taxonomy unless its tags satisfy this - record shape. + omit it. An assertion event is any invariant or bad-state check reported with + assertion semantics, including a semantic soft assertion emitted through + `reportError` instead of `assert()`. Such an event supplies + `assert_mode:soft`. `ReportErrorOptions.tags` remains optional for existing + callers, but an event does not conform to this taxonomy unless its tags + satisfy this record shape. New enum values require an entry under Registry Amendments. Dynamic values such as ids, names, and paths are forbidden in tags; permitted diagnostic @@ -122,9 +136,21 @@ remain. The gap is coverage and discipline, not another reporting API. Append `YYYY-MM-DD tag=value — reason — PR` lines here. +- 2026-09-13 `error_type=workflow_insert_aborted_canvas_changed` — workflow + insertion was dropped after the canvas or graph changed while loading — + [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) +- 2026-09-13 `operation=insert` — workflow insertion into an existing graph — + [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) +- 2026-09-13 `outcome=degraded` — operation was dropped safely without + user-visible corruption — + [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) + ## References - [`reportError`](../../src/platform/telemetry/reportError.ts) - [`assert`](../../src/base/assert.ts) -- [Source decision and research](https://github.com/christian-byrne/in-app-agent-program/blob/main/decisions/ADR-029-agent-telemetry-tag-taxonomy.md) +- [Sentry tags](https://docs.sentry.io/platforms/javascript/enriching-events/tags/) +- [Sentry event grouping](https://docs.sentry.io/concepts/data-management/event-grouping/) +- [Google SRE: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) +- [TigerBeetle assertion guidance](https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md) - [ADR-TELEMETRY-ROUTING-0013: Telemetry Routing Across Consumers](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) From 67aed68459740e3dff3132fc90a15614735f6231 Mon Sep 17 00:00:00 2001 From: Alexander Brown Date: Mon, 12 Oct 2026 08:36:31 +0000 Subject: [PATCH 9/9] docs(adr): reconcile telemetry taxonomy contract Amp-Thread-ID: https://ampcode.com/threads/T-01a088bd-b179-74af-a7bd-06866bee405e --- ...S-0030-agent-consumable-error-telemetry.md | 155 +++++++++++------- src/AGENTS.md | 8 +- 2 files changed, 97 insertions(+), 66 deletions(-) diff --git a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md index e0f7d3b417c..599ccd3fded 100644 --- a/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md +++ b/docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md @@ -6,36 +6,40 @@ Date: 2026-09-03 Proposed -Two rules below are not expressible with the APIs as they stand: rule 5 -selects soft, hard, and sampled assertion modes, but `assert()` takes only a -condition and a message; rule 4 configures a fingerprint at the emitter, but -`ReportErrorOptions` has no fingerprint field and the only fingerprint seam -today is the global `beforeSend` that rule 4 rules out. This stays Proposed -until those two emitter APIs land, so it reads as direction rather than an -immediately-binding gate. +Three parts of this decision are not expressible with the current emitters. +Rule 5 selects soft, hard, and sampled assertion modes, but +`assert(condition, message, context)` has no mode or taxonomy-tag input. Its +desktop reporter calls Sentry `captureMessage` directly, while the cloud +reporter emits `invariant_assert` without taxonomy tags and caps repeated and +distinct messages. Rule 4 configures a fingerprint at the emitter, but +`ReportErrorOptions` has no fingerprint field. This stays Proposed until +assertion mode, assertion taxonomy tags, and emitter fingerprints are supported. ## Context Real-user state exposes failures that tests and pre-deploy checks cannot see: invariants, bad user states, expected-but-missing events, unexpected catches, -and degraded paths. These signals need intentional common tags so agents can -query them by family and Datadog RUM can route useful alerts without creating -one monitor per error slug. +and degraded paths. These signals need common tags so agents can query them by +alert family and Datadog RUM can route alerts without one monitor per error +slug. The frontend already provides `reportError`, mandates it in `src/AGENTS.md`, -and has production callers using snake_case `errorType` slugs. -`src/base/assert.ts` is used much less widely, while many catch blocks remain. -The gap is coverage and discipline, not another reporting API. Counts are -deliberately omitted because they change independently of this decision. - -This decision originated in a 2026-09-03 product directive to expose -invariants, bad states, missing bounded completions, unexpected catches, and -degraded paths through one queryable taxonomy. The supporting research checked -the existing frontend emitters and compared bounded indexed tags with diagnostic -context, default grouping with explicit fingerprints, family-level alerting with -per-slug alerts, and failure counts with rates that have denominators. Those -tradeoffs lead to the fixed low-cardinality tags and alternatives recorded -below; the decision does not depend on access to an external design document. +and has production callers using snake_case `errorType` slugs and product +`surface` values. `src/base/assert.ts` is used much less widely, while many +catch blocks remain. The gap is coverage and discipline, not another reporting +API. Counts are deliberately omitted because they change independently of this +decision. + +The tradeoffs considered were bounded indexed tags versus diagnostic context, +default grouping versus explicit fingerprints, family-level alerting versus +per-slug alerts, and failure counts versus rates with denominators. Those +tradeoffs lead to the low-cardinality tags and alternatives recorded below. + +[ADR-TELEMETRY-DIAGNOSTICS-0019](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md) +proposes one diagnostics system and defers its concrete API while requiring +existing `reportError` infrastructure to be reused or evolved. This ADR narrows +that direction for error telemetry by defining tags for `reportError` and +`assert`; it does not add a second reporting path. ## Decision @@ -45,9 +49,11 @@ below; the decision does not depend on access to an external design document. emitting `missing_event` observations where expected completion has a bounded window. 2. **Use a fixed snake_case tag taxonomy:** - - `error_type`: existing stable slug shaped as - `__[_detail]`, with the operation written as - a present participle + - `error_type`: written by `reportError` from `errorType`; a stable slug + shaped as `__[_detail]`, with the operation + written as a present participle + - `surface`: written by `reportError` from the required product owner in + `ReportErrorOptions` - `failure_kind`: `invariant`, `bad_state`, `missing_event`, `caught_unexpected`, or `degraded` - `feature_area`: `workflow`, `queue`, `canvas`, `nodes`, `auth`, `cloud`, @@ -58,48 +64,75 @@ below; the decision does not depend on access to an external design document. `degraded` - `assert_mode`: `soft`, `hard`, or `sampled` for assertions + `surface` identifies who owns the response; `feature_area` identifies the + failing product area used by alert families. The outcomes mean: + - `failed`: the requested work did not complete and no fallback ran. + - `recovered`: the requested work completed after recovery. + - `aborted`: the requested work stopped safely without applying its result. + - `timed_out`: a bounded wait expired. + - `missing`: an expected event did not arrive. + - `degraded`: work continued with reduced data or capability. + A conforming event always supplies `failure_kind`, `feature_area`, and `outcome`. It supplies `operation` when the failure occurs during one of the enumerated operations; invariants and bad states with no discrete operation - omit it. Assertion events also supply `assert_mode`; non-assertion events - omit it. An assertion event is any invariant or bad-state check reported with - assertion semantics, including a semantic soft assertion emitted through - `reportError` instead of `assert()`. Such an event supplies - `assert_mode:soft`. `ReportErrorOptions.tags` remains optional for existing - callers, but an event does not conform to this taxonomy unless its tags - satisfy this record shape. - - New enum values require an entry under Registry Amendments. Dynamic values - such as ids, names, and paths are forbidden in tags; permitted diagnostic - values belong in `context`. + omit it. An assertion event is an `invariant` or `bad_state` check reported + with assertion semantics, including a semantic soft assertion emitted + through `reportError` instead of `assert()`. Assertion events supply + `assert_mode`; non-assertion events omit it. Existing taxonomy-tagged + emitters with other values or with `assert_mode` on non-assertion events do + not conform; reconcile them before acceptance. `ReportErrorOptions.tags` + remains optional for existing callers, but an event does not conform to + this taxonomy unless its tags satisfy this record shape. + + Rule 2's lists are authoritative. After acceptance, add a value there and + append its audit record under Registry Amendments. Dynamic values such as + ids, names, and paths are forbidden in tags; permitted diagnostic values + belong in `context`. Other bounded snake_case tags may accompany these keys, + but monitors must not require them. 3. **Choose levels by impact.** Use `error` for `invariant`, `bad_state`, - `caught_unexpected`, and `outcome:failed`. Use `warning` for `degraded`, - `outcome:recovered`, and `missing_event` unless the user lost work. When - multiple rules match, `error` takes precedence over `warning`. + `caught_unexpected`, and `outcome:failed`. Use `warning` for + `failure_kind:degraded`, `outcome:degraded`, `outcome:recovered`, and + `missing_event` unless the user lost work. When multiple rules match, + `error` takes precedence over `warning`. 4. **Retain default grouping unless a slug must subdivide an issue.** Sentry's default grouping ignores tags, including `error_type`. When separate issues are needed per slug, configure an emitter fingerprint containing both `{{ default }}` and `errorType`. Never use a bare `[errorType]` fingerprint, - and do not set fingerprints globally in `beforeSend`, so each override stays - reviewable at its call site. + and do not add taxonomy fingerprints globally in `beforeSend`, so each + override stays reviewable at its call site. 5. **Use soft assertions by default in the browser.** Hard assertions are only for cases where continuing corrupts document or CRDT state, authentication, or billing. Sampled assertions are only for render and pointer hot paths and never for data-loss invariants. Assertion messages are static strings. 6. **Classify every touched catch.** A catch is expected-handled, degraded-recovered, unexpected, or rethrown. Only degraded-recovered and - unexpected catches report. Never report and rethrow the same error. + unexpected catches report. A degraded-recovered catch reports + `failure_kind:degraded`; an unexpected catch reports + `failure_kind:caught_unexpected`. `outcome` records what happened to the + requested work. Never report and rethrow the same error. 7. **Do not include PII.** Prompts, workflow names or paths, file names, node titles, emails, and tokens are forbidden in tags and context. Limit context - to ids, counts, enums, and booleans. + to ids, counts, enums, and booleans. Before acceptance, update the + `src/AGENTS.md` guidance that permits filenames and URLs in context. 8. **Route operational alerts through Datadog by family.** Following [ADR-TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md), - use one RUM monitor per family rather than per slug: new invariant, hard - assertion, broken user operation, unexpected catch, missing completion, - failure-rate deviation, dynamic anomaly by feature area, and a - recovered-degradation digest. Sentry remains an error exploration and - grouping sink, not the operational alert route. + use one RUM monitor per family rather than per slug: + - new invariant: `failure_kind:invariant` + - hard assertion: `assert_mode:hard` + - broken user operation: `outcome:failed` with `operation` + - unexpected catch: `failure_kind:caught_unexpected` + - missing completion: `failure_kind:missing_event` + - failure-rate deviation: `outcome:failed` divided by the paired success + actions from rule 9, grouped by `feature_area` and `operation` + - feature-area anomaly: conforming events grouped by `feature_area` + - recovered-degradation digest: `failure_kind:degraded` with + `outcome:recovered` or `outcome:degraded` + + Sentry remains an error exploration and grouping sink, not the operational + alert route. + 9. **Add denominators in a later phase.** Paired Datadog RUM success actions should use the same bounded attributes as these error tags so Datadog calculated metrics and monitors can measure failure rates. @@ -108,15 +141,23 @@ below; the decision does not depend on access to an external design document. ## Consequences +### Positive + - Sentry issues become queryable by agents using stable tag conjunctions such as `failure_kind:invariant feature_area:workflow release:X`. -- Event volume rises, so alert families and existing filtering must keep noisy - deployment and chunk-load failures out of these signals. - `feature_area` becomes shared vocabulary for error tags, future metric attributes, and operational follow-up. - Instrumentation changes stay small and reviewable: one feature area or catch cluster per pull request, with no behavior change to recovery paths. +### Negative + +- Event volume rises, so alert families and existing filtering must keep noisy + deployment and chunk-load failures out of these signals. +- Tags remain untyped, so conformance depends on review until emitters enforce + the taxonomy. +- Existing nonconforming emitters must be migrated before the ADR is accepted. + ## Alternatives Considered - A new `reportInvariant()` or `Telemetry` wrapper was rejected because it @@ -134,16 +175,7 @@ below; the decision does not depend on access to an external design document. ## Registry Amendments -Append `YYYY-MM-DD tag=value — reason — PR` lines here. - -- 2026-09-13 `error_type=workflow_insert_aborted_canvas_changed` — workflow - insertion was dropped after the canvas or graph changed while loading — - [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) -- 2026-09-13 `operation=insert` — workflow insertion into an existing graph — - [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) -- 2026-09-13 `outcome=degraded` — operation was dropped safely without - user-visible corruption — - [#17593](https://github.com/Comfy-Org/ComfyUI_frontend/pull/17593) +After acceptance, append `YYYY-MM-DD tag=value — reason — PR` lines here. ## References @@ -153,4 +185,5 @@ Append `YYYY-MM-DD tag=value — reason — PR` lines here. - [Sentry event grouping](https://docs.sentry.io/concepts/data-management/event-grouping/) - [Google SRE: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/) - [TigerBeetle assertion guidance](https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md) +- [ADR-TELEMETRY-DIAGNOSTICS-0019: Recoverable Event Diagnostics](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md) - [ADR-TELEMETRY-ROUTING-0013: Telemetry Routing Across Consumers](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md) diff --git a/src/AGENTS.md b/src/AGENTS.md index e8de3d3dc1b..64eb5c70bd9 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -43,11 +43,9 @@ a migration of affected queries and alerts that accounts for old and new releases reporting different names. - [ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) - proposes agent-consumable tags, assertion modes, catch classification, and - PII rules. It is Proposed, not Accepted, until `assert()` takes an assertion - mode and `ReportErrorOptions` takes a fingerprint, so treat it as direction - rather than a gate. +[ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md) +proposes agent-consumable tags, assertion modes, catch classification, and PII +rules. It is Proposed: treat it as direction rather than a gate. ## Security